Data ecosystems are complex at the best of times when using data on hand within the business, however organisations are increasingly seeking better insights by tapping into third-party data. Data from outside the organisation can add another rich perspective to analytics but using it effectively can be challenging.
COVID-19 has shown us just how relevant external data can be. A swift change in consumer purchasing patterns and behaviours made pre-existing predictive models obsolete overnight, while internal data didn’t help much either in forecasting demand and supply. A wealth of external data however in the cloud offered granular information to help organisations adapt and respond to these changes.
Businesses know they can gain valuable insights by analysing data generated from their operations, but internally generated data can leave gaps which is why companies are looking to third-party data to complete the picture. This data can include almost anything from historical demographic and weather data to satellite imagery or market share information. Companies increasingly operate as part of networks consisting of business partners such as suppliers, resellers, channel partners, regulators, and other stakeholders. These networks are often globally distributed and potentially affected by economic, political, and/or environmental factors. Analysing external data can help companies see risks and opportunities that they would miss, clarifying how factors such as shifting consumer behaviours, weather or competitor initiatives can affect a business.
As most of us know the volume of data being created, shared, and stored is increasing at an exponential pace. According to one study by Cisco, the data stored in data centres will nearly quintuple by 2021 to reach 1.3 zettabytes globally. Along with the volume of data available, the potential value of analysing this data grows bigger by the day.
It’s not surprising, then, that companies on the leading edge of data and analytics are more likely to make use of external data. An MIT Sloan Management Review report found that the companies making the most innovative use of data and analytics were more likely than others to leverage more external data sources, including social, mobile, and publicly available data. While Forrester found that faster-growing companies were more likely to be planning to expand their ability to source external data than companies with lower growth rates.
How To Get Business Value From External Data
External data sources are helping businesses personalise marketing offers, improve HR decisions, gain new revenue streams by launching new products or services, enhance risk visibility and mitigation, and better anticipate shifts in demand for their products and services. There are publicly available data sources like macroeconomic indicators and private business data sources. For instance, a major semiconductor manufacturer used third-party data to build models that could predict the best types of customers to target in marketing campaigns. This external data helped train the models to identify potential targets that fit similar profiles to the company’s most engaged customers. These “lookalike” models helped the organisation optimise marketing spend and reduced a major campaign’s cost-per-engagement.
There are numerous other examples of analytics programs generating value with external data. Monitoring social networking data can help to predict patterns of external job-seeking behaviour and retention risk; potentially being more predictive of an employee’s likelihood of leaving than internal data available. Pharmaceutical firms have been using third party market share and consumer information about where their prescriptions end up for years to better target sales efforts. Australian agricultural firms use geolocation and weather data to predict crop yields, helping farmers optimise their use of fertiliser. Logistics companies use social media data and data from suppliers to predict disruptions to clients’ supply chains, as well as IoT data to monitor the safety of their drivers. Retailers are using economic data and forecasts, data from suppliers, and geolocation data to better predict demand and reduce stockouts. Some companies are using satellite imagery to estimate shopping traffic and predict retail sales; others are using aerial imagery to estimate oil inventories to better underwrite loans to refiners and the list goes on.
The Challenges of Using External Data
Organisations may experience a wide variety of business and technical challenges in gaining insights from external data. Among the business challenges is the size and complexity of the data-provider market, which can make it hard to identify the right data sources and partners. Negotiating acquisition of data can be hard, depending on factors such as whether ongoing access to data is needed for refreshing machine learning models, usage restrictions, whether the vendor wants a share of revenue gained from the data, and liability if the data proves to be inaccurate or tainted. This process can involve lengthy risk and legal reviews of vendor contracts and licensing agreements. The ongoing management of a growing roster of data-sharing relationships and partnerships can be taxing as well. Companies also need to be conscious of privacy concerns and consumer security when they use some types of external data.
The technical challenges include fundamentals such as assessing data quality and accuracy: A variety of studies have demonstrated that third-party data can be riddled with inaccuracies. There can also be inconsistencies between external and internal data to resolve before performing an analysis. Data preprocessing such as cleansing and formatting it for analysis is time-consuming. Some estimates suggest that this can account for most of the effort in data analysis projects. And securely storing and cataloging data in an easily accessible manner can require updating information management processes and capabilities designed to handle only internal data. The longer it takes to work through these challenges, the less time available to react to market trends and external events with agility.
So Where To From Here?
Research suggests that most companies haven’t developed the necessary capabilities to use external data effectively. To be good at using external data means being competent in identifying, evaluating, procuring, and preparing external data in a consistent and timely manner. Companies will need a continuous process for identifying, engaging with, and evaluating new external data sources and partners and, when appropriate, integrating these data sources into analytics processes or product offerings. Establishing a dedicated team for external data sourcing can also help.
Up-front planning starts with an assessment of the existing data environment to determine how it can support ingestion, storage, integration, governance, and use of the data. The assessment covers issues such as how frequently the data come in, the amount of data, how data must be secured, and how external data will be integrated with internal data. This will provide insights about any necessary modifications to the data architecture.
Modifications should be designed to ensure that the data architecture is flexible enough to support the integration of continuous incoming data from a variety of data sources. In other cases, it may require tooling to support large-scale data ingestion, querying, and analysis. Data architecture and underlying systems can be updated over time as needs mature and evolve.
The final process in this step is ensuring an appropriate and consistent level of quality by constantly monitoring the data used. This involves examining data regularly against the established quality framework to identify whether the source data have changed and to understand the drivers of any changes (for example, schema updates, expansion of data products, change in underlying data sources). If the changes are significant, algorithmic models leveraging the data may need to be retrained or even rebuilt.
The pressure on companies to innovate and to improve the efficiency and effectiveness of their operations is unrelenting and for many companies, a well-structured plan for using external data may be a viable solution to provide a competitive edge.
