The New Science of Demand: Digital Transformation and Consumer Engagement

Earlier this year, Target Corporation lost nearly 25% of its $100B market capitalization following a disappointing earnings report. A few weeks later, the stock fell again as the company announced that it would be reducing prices due to rapidly increasing inventories. Walmart, a retailer four times larger than Target, lost 20% of its value over the same period claiming changing consumer behaviors and continued supply chain challenges were responsible. Shockwaves spread across the retail landscape as markets scrambled to process the impact of underlying trends. This may have you wondering: “In a world spinning with constant news of inflation, spiking energy costs and supply-side woes, why would deflationary trends like ramping inventories be hitting some of the world’s largest businesses?”  

If your business is consumer products, many challenges of the pandemic era have become abundantly clear: hiccups at the top of the supply chain due to lockdowns, shortages in shipping containers and port infrastructure, a massive transition from consumption of services to goods and housing and breaking news every week are impacting all these factors as they shift and churn. While understanding how all these dynamic inputs impact your bottom line might seem like an impossible machine, they all boil down to one core concept: forecasting consumer demand.  

Whatever the incarnation, be it sales operations, inventory, or revenue management, it is someone’s job to predict future demand as input to a variety of investment and staffing decisions. It can be done terribly, as a trendline of historic quarterly sales with a seasonal adjustment applied—an approach completely unable to respond to a shifting macro environment. It can also be done incredibly well, with dynamic tools in the hands of multiple stakeholders sitting on real-time data that responds to the slightest change in consumer preference, sentiment or spending power.  

Below, we outline three key elements to building successful demand forecasts that will keep the pulse of consumer engagement no matter how unpredictable the world can be.  

1. Look to First Party Digital Data for an Accurate View of Individual Customer Behaviors Over Time 

The first thing to note: Organizations need to use web and/or app engagement data as the foundation, ideally first-party data blended with media exposure and eCRM for a more holistic view across the customer lifecycle. These data types are most critical and valuable due to their real-time nature and reflection of active shopping behavior. After all, if a consumer is no longer interested in buying a product from you, they won’t be visiting your website to read about it. This is the signal you want. It is also important to have data tracked via a robust web analytics platform (such as Google Analytics or Adobe Analytics) collected via a logged-in state or a first-party cookie. This will enable consistent visibility into the same consumer’s behavior over multiple visits, especially if your products have a longer consideration cycle.  

Additionally, having a good tagging strategy and metadata is critical. Organizations will want data scientists to mine the data to understand exactly what users were engaging with at each stop across digital properties. One of the big mistakes we’ve seen among companies who use digital data for demand prediction today is that they look at all the behaviors on aggregate, which can mask a dip or rise in demand behind outlier behaviors. Organizations’ goal should be to predict the demand for each individual and then aggregate demand at the other end. Otherwise, they risk forecasting inordinate demand for a single consumer or household.  

Also, strive to migrate complex engagement data (“log-level” data) into a flexible big-data environment. This should be done so that data science models and business applications can be easily built on top of it. A good example of this would be the big-data warehousing products within any of the ‘big three’ cloud providers (AWS, GCP and Azure).  

While it might sound like a lot, most mid-to-large-sized organizations already have most of the key elements in place and will simply require a few small pieces to complete the puzzle. Building the infrastructure can take as little as three weeks or as long as three months, depending upon the current maturity and toolkit. But the value is there: Many companies who accurately predicted the Covid-19 demand shock and subsequent demand spike did so by getting real-time signals from individual consumers based on changing digital engagement with their brands and products.  

2. Empower Data Scientists and Engineers to Design and Automate New Demand Models—but Don’t Sleep on Strategy.  

Another lesson every business has learned over the past decade is that all the data in the world is worth nothing if you don’t know how to use it. A small, dedicated task force of data scientists, engineers and at least one strategist is ideal for building this capability.  

The strategist role is critical for developing any sort of data science application, akin to a product manager but with more specialized skills to serve as a subject-matter expert on digital data and sources. This person acts as a steward of the business to ensure data scientists and engineers have the appropriate context in designing their analysis and setting up the infrastructure to support it. “Demand” as a concept isn’t one-size-fits-all. Multiple ideas and approaches need to be evaluated and prioritized over the course of the project. With that in mind, the strategist also acts as a liaison to the stakeholder teams when decisions need to be made regarding proxy measures, model outputs and historical techniques for comparison.  

Data scientists ensure the data is organized to interpret cause and effect, that the model is as accurate as possible, and that the output is responsive to new information entering the ecosystem. If they’re working in a cloud environment, they will have access to data processing tools and ML-as-a-service. The data science team will likely lean on those tools and their native integrations with the data platforms to develop scalable and up-to-date demand models.  

Data engineer(s) should ideally have expertise in ML Ops and some exposure to digital analytics and demand-side platforms, as source data can be somewhat ugly and difficult to work with. Key tasks for this team will be the processing of source data, staging of data for analysis (and eventually reporting) and automating the model outputs. The latter is of critical importance, since getting updated forecasts frequently is the key to understanding shifting trends and reacting before it’s too late.  

Working together, this team can generate not just improved demand forecasts to inform downstream applications such as inventory, but also outputs powering higher-funnel tactics such as dynamic creative optimization and offer management. Keeping the team online as new capabilities launch and managing a roadmap of prioritized future applications is a great way to get continual value out of your digital infrastructure. 

3. You’ll Need User-Friendly Tools if You Want to Drive Adoption of New Techniques  

The only thing better than having all the smartest tools in the industry: actually using them. Getting the outputs of the organization’s data assets into the hands of decision-makers is just as important as developing those assets themselves. Business intelligence (“BI” or Data Visualization) tools and specialists are the keys to disseminating new data that suits each stakeholder’s needs. BI specialists should work with stakeholders to understand their requirements and with the engineers to produce user-friendly outputs from the models upon which they can design visualizations.  

Reframing team roles and advancing technology and tools allows businesses to democratize critical data and serve business units based on their purpose and key decision points. Regional merchandisers may want to see shifts in demand with the ability to drill down on specific geographies. Others may want broader, national demand (or individual product propensity) visualized alongside incentive, inventory or media spend. Targeting a few stakeholders early who are interested in trying new techniques can be important; building internal advocacy and developing case studies early on can speed up the process of getting new tools to market.