Outgrace, home
Technology & AIHow-to guide

How to run a retail demand-forecasting pilot that can show whether it worked

A control group, error measures agreed in advance and a written go or no-go rule. In our experience, many forecasting pilots skip all three, and then cannot say whether they worked.

A forecasting pilot often ends with a shrug. The new tool ran beside the old spreadsheet for a quarter, the planners liked some of its output, and nobody can say which method was more accurate or whether the gap paid for the licence. The fault is seldom the software. It is that no one agreed, before the first forecast ran, what a better forecast would change. Forecasts are hard even for experienced people. According to NBER Working Paper 35231 by Cutler and Klarnet, which compares US health spending with a 2010 CMS actuarial forecast, spending by 2024 was nearly $1 trillion, or 15 percent, below that projection. The field is different, but the lesson carries across: a forecast is a claim, and a claim needs a test.

Before you start

Three things must be true before anyone builds a model. The first is a named decision the forecast will change: a reorder rule, a store allocation, a markdown calendar or a weekly buy. The second is a baseline, meaning the method the business uses now, written down with its recent error record, so the pilot has something concrete to beat. The third is an owner with the authority to stop the test, who ideally did not buy the tool. Without the decision you will end up with a dashboard. Without the baseline you will end up arguing about whether the new numbers look better.

Settle these three questions at the kickoff meeting, and put the answers in writing.

  • Which stores, regions or product groups are in scope, and how many of them sit in the control group.
  • Which data the model may use, and who signs off each data feed before it leaves your systems.
  • How long the test runs, and the date on which the go or no-go decision will be made.

The MIT CTL–Mecalux study, listed by the MIT Center for Transportation and Logistics, finds that 57% of organisations expect AI to decide more than a quarter of warehouse calls without a person by 2028. The point for a retailer is that a forecast feeds decisions both upstream of the shelf and downstream of it. Trace the decision you named to the warehouse, the distribution centre and the store, so you know which operational costs the pilot should measure.

  1. Step 1

    Tie the forecast to one decision

    Pick a single choice the forecast will feed, such as how much stock a distribution centre sends to one cluster of stores over the next four weeks. Write down what the planner will do differently when the forecast is right and when it is wrong. If nothing changes either way, the pilot cannot prove value, however accurate the model is. What usually goes wrong is a scope so broad that nobody can say which result mattered.

  2. Step 2

    Split the estate by lottery

    Choose comparable stores, regions or product groups, and keep a control group on the existing method for the full test. Assign them by random draw rather than by hand, because a manager who picks the easy stores will pick the winners. A randomised comparison is the strongest way to show that a change caused a difference. What usually goes wrong is a control group that quietly receives the new model's numbers through a shared planning tool.

  3. Step 3

    Fix the error measures in writing

    Choose the measures that suit your demand pattern and write them down before the pilot starts. If many lines sell in lumps or with long gaps between sales, be careful with the common measures. A HICSS-53 paper from January 2020 argues that MAPE and RMSE are not suitable for lumpy and intermittent demand, and that they do not sufficiently account for temporal shifts or cost. The paper proposes a metric called Stock-keeping-oriented Prediction Error Costs, which adds cost-related aspects to the er

  4. Step 4

    Write the go or no-go rule

    State the rule in one sentence that names the measure, the comparison and the threshold. A workable pattern is this: the model must beat the baseline on the agreed cost measure in the test stores, across the full period, with no store group performing worse than its control. Set the threshold from what a wrong decision costs your business, not from a vendor's case study. Sign the rule and file it, because a rule written after the results is a story rather than a test.

  5. Step 5

    Run the full cycle

    Fix the length in advance, and cover at least one seasonal peak if your range has one. Do not stop early because the first fortnight looks good, and do not extend the test because the second fortnight looks bad. Log every manual override, since a pilot in which planners often overrule the model is measuring the planners, not the model. What usually goes wrong is an early verdict based on the weeks when little happened.

  6. Step 6

    Apply the rule and record the answer

    On the agreed date, apply the rule as written and nothing else. If the model passes, scale in stages and keep a control group in the first wave, so the next result is still a test. If it fails, write down why, close the pilot and keep the result on file. A clean no-go saves the rollout budget and gives the next pilot a better question to answer.

MIT CTL–Mecalux: firms expecting AI to make over a quarter of warehouse decisions by 2028
57%
Unique randomised trials in education research, 1980–2016 (Queen's Belfast review)
1,017
Spending below a 2010 CMS forecast by 2024 (NBER Working Paper 35231)
15 percent

What usually goes wrong

The most common error is choosing the measure after the results arrive. A team that reviews three metrics and reports the one that flatters the model has run a demonstration, not a pilot. The second error is a control group that is not really a control, such as the stores a regional manager chose because they are easy to run. The third is a short run that ends on the first good fortnight. In our experience all three come from the same habit: deciding to be persuaded before the evidence exists. Fixing the rule, the measures and the control group on paper, before any data moves, removes most of the room for that habit.

Critics also argue that controlled trials are blunt, and that they ignore context and the experience of the people involved. Queen's University Belfast, reviewing randomised trials in education, reports that the review finds evidence to challenge that claim, although its authors also outline key challenges for researchers. The practical answer for a retailer is simple. Log local conditions during the test, such as a store refit or a new competitor opening nearby, so that a surprising result can be explained rather than argued over.

Is a backtest on our own history enough?

A backtest is useful for ruling out weak models, but it does not show how a model performs when planners act on its output in live decisions. In our experience a backtest can flatter a model if the tuning used the same history it is then scored on. Use it as a filter before the live control group, not as the verdict.

Which error measure should we use for slow-moving lines?

Do not rely on the common measures alone. A HICSS-53 paper from January 2020 argues that MAPE and RMSE are unsuitable for lumpy and intermittent demand, and proposes a cost-aware alternative. Check any alternative metric against your own series before you adopt it.

How long should the pilot run?

Long enough to cover the decisions you are testing, including at least one full replenishment cycle and any seasonal peak in your range. Write the length down before you start and keep to it. A pilot of a few weeks shows little if the business has few weeks of variation to learn from.

Does the NIST AI Risk Management Framework settle the design?

No. NIST describes its framework as intended for voluntary use, to help build trustworthiness into how AI systems are designed, used and evaluated. NIST's page notes that version 1.0 is under revision, linked to the White House AI Action Plan, so check the current version before citing it in a board paper.

Who should own the go or no-go decision?

Someone with the authority to stop the test who did not buy the tool and does not report to the person who did. Put the name in the pilot plan. If the owner changes mid-test, the rule and the data should pass to the new owner unchanged.

This week, write three things on one page: the decision the forecast will change, the baseline it must beat, and the go or no-go rule in a single sentence. Send the page to the person who can stop the test, and ask them to sign it before any data goes to a vendor. If they hesitate over the rule, the pilot is not ready to start. Set up the control group the same week, even if the model is not yet built, because the control group is the part of the pilot that cannot be fixed later.

Newsletter

The Briefing

One email a week: the leaders we profiled, the ideas worth stealing, and nothing else.

Weekly, free, one click to leave. Read our privacy policy.