Self-assessment answers from Module 2
One. The three components of an ETS model are Error (additive or multiplicative), Trend (none, additive, multiplicative, or damped), and Seasonal (none, additive, or multiplicative). Error captures the noise structure, Trend captures the direction, Seasonal captures the repeating pattern.
Two. ARIMA outperforms ETS on series with strong autocorrelation structure that ETS does not capture, particularly where past forecast errors are predictive of future values. ETS outperforms ARIMA on short series (under 50 observations), on series where the level, trend, and seasonal decomposition is the dominant signal, and in automated contexts where the simpler model selection is more robust.
Three. Seasonal decomposition is useful as a diagnostic because it separates the trend, seasonal, and residual components. If the seasonal component is unstable, seasonal models will struggle. If the residual is large relative to trend and seasonal, the series is noisy and no model will forecast it accurately. Decomposition tells you the forecastability of the series before you fit any model.
Four. ARIMA requires typically 50 or more observations for reliable fitting. This excludes new products, short-life SKUs, and any series with less than roughly 4 years of monthly data or 1 year of weekly data.
Five. For intermittent demand, none of the three methods (ETS, ARIMA, decomposition) should be the primary forecast method. Use a method designed for intermittent demand, such as Croston's method, SBA (Syntetos-Boylan Approximation), or TSB (Teunter-Syntetos-Babai). These separate the demand-occurrence and demand-size problems.
If you got all five, continue. If you missed any, review Module 2.
What this module covers
This module covers the four metrics every demand planner must know: MAPE, WMAPE, bias, and forecast value added. These are the metrics that tell you whether your forecast is good, whether it is improving, and whether your manual overrides are helping or hurting.
The module is dense because the metrics are interconnected. MAPE is the default but misleads. WMAPE corrects MAPE's biggest flaw. Bias is directional and more actionable than error. FVA tells you whether your planners add value. Read each section carefully, because the relationships between the metrics are what matter in practice.
MAPE: the default that misleads
MAPE, Mean Absolute Percentage Error, is the most widely quoted forecast accuracy metric in supply chain. It is also the most misread. The formula is simple: for each period, compute the absolute error divided by the actual demand, express it as a percentage, then average across all periods. The result is a single percentage representing your average miss rate.
MAPE has three well-documented failure modes that every planner should know.
First, MAPE explodes when actual demand is low. A forecast of 10 units against an actual of 1 produces a 900% error. The same absolute miss against an actual of 100 produces a 90% error. Slow movers, which have low actual demand, dominate MAPE even if they are a small part of revenue.
Second, MAPE is undefined when actual demand is zero. Division by zero. Most tools silently skip these periods, which means MAPE is computed on a biased sample of non-zero periods. If 30% of your SKUs have zero demand in a given month, your MAPE does not reflect 30% of your business.
Third, MAPE weights all errors equally regardless of volume. A 50% error on a SKU selling 2 units per month counts the same as a 50% error on a SKU selling 2,000 units per month. In revenue terms, the second error is 1,000 times more important. MAPE cannot see this.
Despite these flaws, MAPE remains the default because it produces a single percentage that is easy to communicate. The solution is not to abandon MAPE but to read it alongside the metrics below. Never report MAPE alone.
WMAPE: the correction
WMAPE, Weighted Mean Absolute Percentage Error, corrects MAPE's biggest flaw by weighting each error by the actual demand volume. The formula sums the absolute errors and divides by the sum of actual demand, producing a volume-weighted error percentage.
The practical difference is significant. If your MAPE is 30% but your WMAPE is 12%, your errors are concentrated in low-volume SKUs, which is usually tolerable. If your MAPE is 30% and your WMAPE is 28%, your errors are in high-volume SKUs, which is a serious problem that MAPE was hiding.
WMAPE also handles zero-demand periods correctly. A zero-demand period contributes zero to the numerator (absolute error) and zero to the denominator (actual demand), so it does not bias the metric the way it biases MAPE.
The rule is this. If you report MAPE, always report WMAPE alongside it. If the two diverge significantly, the divergence tells you where your errors sit. If you can only report one, report WMAPE.
Bias: the direction of your errors
MAPE and WMAPE tell you the size of your errors but not their direction. Bias measures whether you systematically over-forecast or under-forecast. The formula is the mean of the signed errors (forecast minus actual), not the absolute errors. A positive bias means you over-forecast on average. A negative bias means you under-forecast.
Bias is more actionable than error because it is systematic. If your MAPE is 20% with a bias of zero, your forecast is missing randomly. The fix is better methodology or more data. If your MAPE is 20% with a bias of positive 15%, your forecast is systematically over-predicting. The fix is to investigate why: is it a model issue, a data issue, or an override pattern?
Track bias at three levels. Overall bias tells you whether the entire forecast is shifted. Bias per product family tells you which categories are problematic. Bias per planner tells you whether certain planners consistently override in one direction. The per-planner bias is the most sensitive finding, because it reveals human patterns that are invisible in aggregate.
A persistent bias of more than 5% in either direction warrants investigation. Bias near zero with high MAPE is normal. High bias with low MAPE is suspicious and suggests the forecast is being adjusted to hit a target rather than to predict demand.
Forecast value added: are overrides helping?
FVA, Forecast Value Added, measures whether your manual overrides actually improve the forecast or make it worse. The computation is straightforward but powerful. Compare the accuracy of the statistical baseline (before overrides) to the accuracy of the final published forecast (after overrides). If the final forecast is more accurate than the baseline, the overrides added value. If the final forecast is less accurate, the overrides destroyed value.
FVA is the metric that most planning teams do not want to compute, because the result is often uncomfortable. Research by Michael Gilliland, published in the Foresight journal and summarized in his book "The Business Forecasting Deal" (Wiley, 2010), consistently finds that a significant fraction of manual overrides destroy forecast value. Planners override because they have market intelligence, but the intelligence is often already reflected in the statistical baseline, or the override is based on a hunch that is wrong more often than it is right.
The practical application of FVA is per-planner tracking. Compute the baseline accuracy and the final accuracy for each planner's portfolio. The planners whose overrides improve accuracy should be studied and their methods shared. The planners whose overrides destroy accuracy should be coached or have their override permissions reduced. This sounds harsh, but the alternative is a planning process where overrides feel productive but actually make the forecast worse.
How the four metrics work together
No single metric tells the truth about forecast quality. The four metrics form a system that, read together, give a complete picture.
MAPE tells you the average miss rate, weighted equally across all observations. It is the headline number but the most misleading if read alone.
WMAPE tells you the volume-weighted miss rate. It corrects MAPE's biggest flaw and is the metric to report to finance and operations.
Bias tells you the direction of your errors. It is the most actionable metric because systematic bias can be corrected.
FVA tells you whether your planning process adds value. It is the metric that holds the planning team accountable for the quality of their overrides.
The diagnostic pattern is this. If MAPE is high and WMAPE is low, your errors are in slow movers, which is tolerable. If both are high, you have a systematic accuracy problem. If bias is high regardless of MAPE, you have a directional problem that is more fixable than random error. If FVA is negative, your overrides are destroying value and your planning process needs review.
What a good accuracy report looks like
A monthly accuracy report should contain, at minimum, the following for the most recent forecast cycle:
- MAPE at SKU-location level, 1-month-ahead horizon, stated with the level and horizon
- WMAPE at the same level and horizon
- Bias (signed mean error) at the same level
- FVA (baseline accuracy versus final accuracy) per planner
- A comparison to the previous 3 months, to show trend
- A comparison to a naive forecast (last period actual as this period forecast), to establish whether the model is beating the simplest possible alternative
The last point is important. If your sophisticated statistical model cannot beat a naive random walk, the model is not adding value. This is the test that Gilliland and others advocate as the baseline for any forecasting process. A naive forecast is free. If your forecast is not better than free, you have a problem.
What to do next
This module covered the four accuracy metrics. Module 4 covers the S&OP process that wraps the forecast into a published demand plan. Module 5 covers supply netting and inventory policy. Module 6 covers scenario planning.
Before moving to Module 4, do this exercise. Pull the last 3 months of forecast and actual data for 20 SKUs. Compute MAPE, WMAPE, and bias for each month. If you have the baseline forecast (before overrides), compute FVA. The exercise will take 30 minutes and will tell you more about your forecast quality than any tool report.
Self-assessment
One. What are the three failure modes of MAPE?
Two. Why does WMAPE handle zero-demand periods correctly while MAPE does not?
Three. What does a positive bias of 15% with a MAPE of 20% tell you, and what should you do about it?
Four. What does FVA measure, and why is the result often uncomfortable for planning teams?
Five. What is the naive forecast baseline, and why should your model beat it?
Answers are in Module 4's introduction.