Sales Forecast Error Analysis

Forecasts are worthless if they don’t drive action. This document shows how to turn forecast errors into insights: # 1 - Compare Forecast vs Actual Pattern, Not Just Values Look for trend breaks: promotions, seasonality shifts, competitive actions Insight: shows whether the model or the business behavior changed # 2 - Separate Volume Error from Mix Error Your total forecast may be right but SKU mix is wrong Insight: points to cannibalization, launches, or customer preference shifts # 3 - Slice the MAPE (forecast error) MAPE at total level hides the real problem; slice by SKU, region, channel, and planner Insight: find where the system is breaking, not the average # 4 - Track Bias Consistently MAPE shows how much you miss; bias shows how you think Insight: positive bias = optimism; negative bias = fear of stockouts # 5 - Connect Error Spikes to Events Overlay error trend with business events; launches, stockouts, price changes and map everything Insight: turns disconnected numbers into cause-and-effect stories # 6 - Use FVA (forecast value added) to Check If Adjustments Helped or Hurt Measure whether human overrides improved or worsened accuracy Insight: helps remove emotional adjustments from the process # 7 - Build an Error Heatmap One view showing where the biggest misses are by SKU, month, region Insight: quickly identifies where planning attention is needed # 8 - Weekly Error Deep Dive Pick the top 5 SKUs with the biggest misses; ask: “what changed?” and “who owns the correction?” Insight: makes forecasting a feedback loop, not a ritual Any others to add?

A forecast that’s right half the time isn’t a sales execution problem. It’s a system signal — the model producing the number doesn’t know what it’s producing. At $8M–$30M ARR, the instinct is to tighten stage definitions, add scrutiny to late-stage deals, introduce a second opinion on every forecast call. Those moves produce small improvements. They don’t fix the underlying issue. Because 55% accuracy isn’t noise. It’s signal. Three mechanical causes, in order of prevalence: 1. ICP drift. The accounts converting today aren’t the accounts your conversion rates were calibrated against. Your historical close rates are predicting a buyer profile that’s changed underneath you. 2. Stage definitions that describe activity, not commitment. “Proposal sent” tells you what your team did. It doesn’t tell you what the buyer did. Forecasts built on activity stages will always oscillate. 3. Pipeline carrying its own history. 20–40% of most pipelines at $8M–$30M ARR are deals that should have been disqualified two quarters ago. They’re distorting every ratio the forecast depends on. None of these are sales problems. They’re architecture problems. Which is why adding a second sales review doesn’t fix them — and why tightening your CRM workflow makes the symptom worse by hiding the break deeper in the data. What it means for the board conversation: If you’re reporting forecast accuracy under 70%, the defensible board narrative isn’t “we’re working on sales discipline.” It’s “we’ve identified that our forecast model is calibrated against assumptions that need to be re-validated — here’s the work underway.” The first framing sounds like you don’t control the outcome. The second sounds like you understand the system. One question to bring to your next exec team meeting: When was the last time our conversion rates were recalibrated against the accounts we’re actually closing today — not the ones we were closing 18 months ago? If no one has an answer, the forecast isn’t wrong. The model underneath it is. — Forecast Fridays #01

A sales leader told me "Our forecast is always off by 20-30%. I don't know what's real anymore." I looked at his pipeline. Every deal in "proposal stage" had an 80% close probability. I asked him one question: "Has an executive at the buyer's company authorized solving this problem?" He had no idea. Here's the problem: His CRM stages were measuring seller activity. Not buyer commitment. Discovery meant "we had a discovery call." Not "they acknowledged a costly problem." Demo meant "we showed them the product." Not "multiple stakeholders agreed this needs to be solved." Proposal meant "we sent pricing." Not "an executive authorized budget to fix this." So his forecast was always wrong. Because he was tracking the wrong things. Here's what we did: We rebuilt his qualification framework around buyer stages instead of seller activities. The ADVANCED framework: Acknowledged problem Documented issue Validated by team Authorized by executive Narrowed to external Chosen as vendor Established timeline Deal terms finalized These are buyer commitments. Not seller activities. When we ran his pipeline through this framework, reality hit hard. Most of his "80% deals" were actually at 25%. They had acknowledged a problem but nothing was documented. No executive sponsorship. No validation from multiple stakeholders. 𝗪𝗶𝘁𝗵𝗶𝗻 𝗼𝗻𝗲 𝗾𝘂𝗮𝗿𝘁𝗲𝗿, 𝗵𝗶𝘀 𝗳𝗼𝗿𝗲𝗰𝗮𝘀𝘁 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝘄𝗲𝗻𝘁 𝗳𝗿𝗼𝗺 65% 𝘁𝗼 93%. Not because his team started working harder. Because they started tracking what actually predicts if deals close. BTW: When you can forecast within 3%, you can predict your income. You can plan for your family. You can budget for that house or wedding or kids' school. When your forecast is always off by 20%, you're guessing. Your compensation is unpredictable. Your future is uncertain. This isn't just about making your boss happy. This is about controlling your financial future. Track buyer commitment. Not seller activity. That's how you build forecast accuracy. — Sales Leaders! Your sales team doesn’t need more training. it needs a revenue operating system: https://lnkd.in/ghh8VCaf

Frankly speaking, I didn't see the point in discussing MAPE when I wrote recent posts on error measures. However, I've received several comments and messages asking for clarification. So, here it is. TL;DR: Avoid using MAPE! In fact, anything that has "APE" in it, will cause issues (https://lnkd.in/eH7JQENX + see image). MAPE, or Mean Absolute Percentage Error, is a still-very-popular-in-practice error measure, which is calculated by taking the absolute difference between the actual and forecast, dividing it by the actual value. The rationale is clear: we need to get rid of scale, we want something that measures accuracy well, is easy to calculate and interpret. Unfortunately, MAPE is none of these things, and here is why. 1. It is scale sensitive: if you have sales in thousands then the actual in the denominator will bring the overall measure down and you will have a very low number even if the model is not doing well. Similarly, if you deal with very low values, they will inflate the measure, making it easily hundreds of percents. 2. It is well known that MAPE prefers when you underforecast (https://lnkd.in/eKAaKVBZ). It is not symmetric and misleading. BTW, "symmetric" MAPE is not better and not symmetric either (https://lnkd.in/etfeTHQ4). 3. It is not easy to calculate on intermittent demand. Technically speaking, you get an infinite value, so it is not possible to calculate it in that case. 4. Okay, it is easy to interpret. But the value itself does not tell you anything about performance of your model (see point 1 above). 5. And it is not clear what it is minimised with (remember this post? https://lnkd.in/eSGV2ZMR). So, how can we fix that? The main problem of MAPE is in the denominator. If we change it, we solve problems (1) and (2). Hyndman & Koehler (2006, https://lnkd.in/e2vxfKzD) proposed a solution, taking the Mean Absolute Error (MAE) of forecast and dividing it by the mean absolute differences of the data. The latter step is done purely for scaling reasons, and we end up with something called "MASE" that does not have the issues (1), (2) and (5), but is not easy to interpret. The problem with MASE is that it is minimised by median and as a result not appropriate for intermittent demand (https://lnkd.in/ezX_EVCC). But there is a good alternative based on the Root Mean Squared Error (RMSE), called RMSSE (https://lnkd.in/e7SQznfG) that uses the same logic as MASE: take RMSE and divide it by the in-sample Root Mean Squared differences. It is still hard to interpret, but at least it ticks the other four boxes. If you really need the "interpretation" in your error measure, consider dividing MAE/RMSE by the in-sample mean of the data (https://lnkd.in/enWyQHBs). This might not fix the issue (1) completely, but at least it would solve the other four problems. For more on error measures see my monograph: https://lnkd.in/e_URj36s Read the full post here: https://lnkd.in/eWjhXtqD #datascience #forecasting #machinelearning

In one of my earlier roles in demand forecasting, our BI system used a very simple percentage error formula: (forecast - actuals)/forecast. At the time, I questioned whether dividing by the actuals might provide a more accurate picture of forecast performance, but like many things in practice, it wasn’t a high priority for discussion. Later, at another company, we used MAPE across all products, including those with highly intermittent demand. It was a consistent approach, but no one really questioned whether a different metric might better capture the nuances of different demand patterns. It wasn’t until my time back to university doing my PhD that I encountered the broader landscape of forecast accuracy metrics. That’s when I started asking a bigger question: which metric should be used for which purpose? Forecast accuracy seems simple until you try to measure it consistently across products, teams, or tools. Most people start with MAPE or RMSE because that’s what the software provides. But eventually, the questions come up: – Why does one model look better on RMSE but worse on MAPE? – Why do different teams report accuracy differently? – Why does it feel like the numbers don’t tell the full story? I wrote this article to help unpack those questions: what each accuracy metric emphasizes, when it’s most useful, and what happens when different metrics lead to different conclusions. It includes: – A breakdown of common metrics like RMSE, MAE, MAPE, sMAPE, MASE, and more – Practical examples of when each metric works best — and when it doesn’t – Guidance on how to choose the right metrics based on product portfolios and business goals I'm curious, which forecasting error measures are being used where you work? Are you using more than one?

A few years back, I ran a forecast error report for a global pharma client.   The overall accuracy looked healthy. But inventory write-offs told a different story.   We weren’t losing money on the forecast—we were losing it on the wrong products. So we zoomed in with a sharper lens. We didn’t just look at errors.   We classified SKUs—A, B, C.   Then overlaid forecast bias on each class. And that’s when the picture turned clear. One A-class SKU—high revenue, high velocity—had a persistent under-forecast bias.   Every quarter.   Which meant constant stockouts and lost sales. Meanwhile, several C-class items had over-forecast bias, inflating dead inventory. Same metric (bias), but now targeted at SKU importance.   That’s where real planning intelligence begins. We acted.   Adjusted safety stocks for C SKUs.   Improved forecast models for A SKUs.   And in just one quarter, we slashed working capital by 9% and boosted service levels by 6%. Because in supply planning, accuracy without relevance is just noise.   It’s bias + ABC classification that turns noise into strategy. Supply Planning is not just about what you stock—it's about what you shouldn’t stock. Are you still measuring forecast bias in isolation?

Your MMM forecast said revenue would be $20.5M last quarter. Your business actually drove $18.4M. Was your forecast wrong? It depends on how you measure it. Most teams judge forecasts with metrics designed for a different problem entirely. MAE, MAPE, R-squared, and other metrics like it are designed to judge the accuracy of a model making many predictions at once (e.g. a model predicting which customers will upgrade their subscription). They are not designed to measure your model’s accuracy in making a single (very important) prediction (like what your sales will be this quarter). They can tell you how close your forecast was, but don’t have anything to say about how uncertainty was estimated. That's where CRPS (Continuous Ranked Probability Score) comes in. Where other metrics are like guessing exactly where a dart will land, CRPS is like drawing a probability map on the dartboard – darker where you think landing is likely, lighter where it's not – then scoring well when the dart lands in the dark areas. This has a lot of repercussions for marketers: - A forecast of "$20M ± $12M" is not helpful for planning - A more confident forecast of "$20M ± $0.5M" that misses by $3M is also dangerous - What you need is both accuracy AND appropriate confidence intervals CRPS measures both. It penalizes you for being wrong, but also for being overconfident when you're wrong or underconfident when you're right. In our new deep dive, Chelsea Parlett breaks down exactly how CRPS works and shows you how to start using it to evaluate your probabilistic models. Here’s a link to the article: https://lnkd.in/eA3MFHye

How Forecast Error Analysis Improves Planning Accuracy in Sexual Wellness Forecast error analysis improves planning accuracy in sexual wellness by identifying where assumptions consistently diverge from actual behavior. Data shows that measuring forecast error is more actionable than refining forecasts alone. What the Data Shows 1. Persistent bias matters more than single misses Brands reviewing forecast error over multiple cycles identify systematic over or under forecasting tied to seasonality, promotions, or launch timing. Correcting bias improves future accuracy more than reacting to one off misses. 2. SKU level error reveals operational risk Aggregate forecasts can appear accurate while individual SKUs experience large deviations. SKU level error analysis highlights stockout and overstock risk earlier. 3. Error bands improve cash planning Using forecast ranges rather than point estimates produces more reliable cash flow and inventory decisions. Brands using confidence bands show fewer liquidity surprises. 4. Feedback loops tighten accuracy over time Teams that regularly compare forecasts to outcomes reduce error magnitude across subsequent cycles by adjusting assumptions around repeat timing and demand decay. Why This Matters in Sexual Wellness Demand in sexual wellness is repeat driven and education influenced. Forecasts that ignore behavior patterns produce avoidable inventory and cash strain. V For Vibes benefits from tracking forecast error as a core metric, improving alignment between demand signals, inventory planning, and financial expectations. Forecast error analysis functions as planning correction. In sexual wellness, understanding where forecasts fail improves stability more effectively than increasing forecast complexity.

A groundbreaking study sheds light on the biases inherent in firms' forecasting behavior, revealing how inaccuracies, over-optimism, and over-precision in predictions persist even under varied conditions. Much of Nick Bloom's work has aimed to not only quantitatively embed forecasting and uncertainty into macro models, but also actively measure how firms conduct forecasting and the impact on their behavior. And yet, evidence of rationality in firm forecasts has been mixed and scarce. A new paper that's out takes a major step in that direction. Bloom et al utilize a panel of over 6,000 U.S. firms from 2019 to 2024, collecting detailed sales forecasts through incentivized surveys. Respondents were paid base fees and additional accuracy rewards of up to $400 for predictions within 10% of actual revenue, leveraging transaction data from a financial services firm referred to as “FinTech.” Key Findings on Forecasting Biases 1) Inaccuracy: Only 18% of forecasts were within 10% of actual revenue, with past revenue trends outperforming firms' own predictions by 1.4 percentage points. 2) Over-Optimism: Firms consistently overestimated revenues by 16.3%, regardless of macroeconomic conditions or survey experience. This optimism bias persisted across stable periods, recessions, and recoveries. 3) Predictable Errors: Errors correlated with prior outcomes and forecast errors, indicating systematic patterns in firms' mistakes. 4) Over-Precision: Firms underestimated variance in revenue outcomes, with subjective forecasts showing variances roughly half the true level. The study used randomized control trials to test potential solutions for mitigating these biases: 1) Data Dashboards: Providing firms with recent transaction data significantly reduced inaccuracies, bias, and over-precision. Exposure to dashboards increased forecast accuracy, albeit temporarily, as firms often overestimated their own abilities and underused the dashboards in subsequent waves. 2) Accuracy Rewards: Higher monetary incentives led to a 70% reduction in optimism bias among firms. For every $100 increase in rewards, forecast win rates improved by 0.9 percentage points. This result supports the idea that firms can adjust optimism when properly incentivized. 3) Forecast Training and Contingent Thinking: Surprisingly, neither training exercises nor scenario evaluations improved forecast accuracy or reduced biases, suggesting that these errors stem from more ingrained behavioral tendencies rather than simple skill deficits. Over-optimism, predictable errors, and over-precision are not mere lapses, but reflect entrenched behaviors that require strategic incentives or tools for improvement. The research shows the importance of understanding and addressing forecasting biases, emphasizing that solutions must extend beyond basic training or nudges. #Economics #BusinessForecasting #BehavioralEconomics #DecisionMaking #EconomicPolicy