Zero mean error can coexist with delivery later than predicted
Zero mean error alone cannot tell you how accurately you predicted delivery dates. Treat a delivery four days later than predicted as +4 days and another four days earlier as −4 days, and the differences sum to zero. Neither delivery occurred on its predicted date.
Here, the prediction concerns the number of days from receiving a request to completing delivery. An error is the actual duration minus the predicted duration. A positive value means delivery was later than predicted; a negative value means it was earlier. Averaging these values across projects gives the mean error.
Mean error allows early and late delivery differences to cancel out. To assess how close each prediction was to its actual delivery date, also calculate the size of the difference without distinguishing early from late.
What four predicted and actual durations reveal
The following example uses hypothetical predictions and actual durations for four deliveries. Each prediction is made when the request is received. Actual duration is elapsed time from receipt to completed delivery, expressed in days. One day means 24 hours, including weekends, holidays, and time spent waiting for customer replies. This differs from the sum of the worker's active working hours.
| Project | Days predicted at receipt | Actual days to delivery | Actual minus predicted duration | Difference regardless of early or late delivery |
|---|
| A | 5 days | 9 days | +4 days | 4 days |
| B | 9 days | 5 days | −4 days | 4 days |
| C | 6 days | 8 days | +2 days | 2 days |
| D | 8 days | 6 days | −2 days | 2 days |
A was delivered four days later than predicted, and B four days earlier. C was delivered two days later than predicted, and D two days earlier. The mean error for these four projects is (4 − 4 + 2 − 2) ÷ 4 = 0 days.
Zero mean error therefore tells us that the total days of later-than-predicted delivery equalled the total days of earlier-than-predicted delivery across these four projects. The number delivered on their predicted date was zero out of four.
If a customer scheduled work requiring A's deliverable for the day after its predicted delivery, A's four-day delay would also affect that work's start. Delivering B earlier does not shorten the wait for A's customer. Confusing an average across projects with the difference for one project conceals these consequences.
Use mean absolute error to calculate the size of delivery prediction errors
To average how many days each prediction differed from its actual duration, use the rightmost column. Treat −4 days as four days and −2 days as two days. Adding four, four, two, and two days and dividing by four projects gives three days on average.
The value obtained by removing a difference's sign is its absolute value. The average of those values is mean absolute error (MAE). Both early and late deliveries add to the total, so they cannot cancel each other out. (Formulas for forecast errors and mean absolute error)
The same four projects have a mean error of zero days and an MAE of three days. The former describes whether predictions tended to be too short or too long; the latter describes how many days predictions and actual durations differed on average.
An MAE of three days does not mean every delivery occurred within three days of its predicted date. A and B each differed by four days. Retain each project's difference as well as the average to see how many days its delivery differed from the prediction.
An experiment assessed average bias and individual prediction errors separately
In Study 4 of Buehler and colleagues' 1994 paper, 123 psychology undergraduates predicted a computer assignment's completion date and time. Groups recalled past experience, additionally considered finishing at their usual interval before a deadline, or received neither instruction.
Compared with the group receiving neither instruction, linking past experience to the current prediction reduced the average tendency to predict completion too early. The instruction produced no statistically significant improvement in absolute error. Reduced bias therefore cannot be read as a finding that individual predictions became closer to actual completion times. (Study 4 methods and results)
Distinguish delivery later than predicted from delivery after an agreed deadline
Whether delivery was later than predicted depends on the predicted date. Whether a delivery commitment was met depends on the date agreed with the customer. If these dates differ, the assessments also differ.
Suppose you predict delivery five days after receiving a request and agree with the customer on a deadline seven days after receipt. Actual delivery six days after receipt is one day later than predicted, yet one day before the agreed deadline.
A small MAE alone therefore cannot establish whether delivery commitments were met. To assess missed deadlines, compare each agreed date and time with actual delivery and count both the projects delivered late and the duration of each delay.
Compare estimates and actual durations using consistent definitions
Zero mean error alone does not establish accurate delivery predictions. Compare the duration predicted at receipt with actual elapsed time from receipt to delivery for each project, then calculate mean error and MAE. The appropriate measure depends on whether you are investigating a tendency to underestimate duration or trying to reduce the number of days between predictions and actual deliveries.
After changing your estimation method, continue saving predictions made at receipt and comparing them with actual durations after completion. Rewriting predictions after the outcome is known prevents evaluation of their accuracy before delivery. The principle of evaluating forecast accuracy on new data, separate from the data used to develop a method, is also explained in the forecasting textbook. (Why forecast accuracy is evaluated on new data)
On a service website, stating when the duration begins, what counts as completed delivery, and whether nonworking days are included also explains what a published duration means.
SpreadSite is a no-code website builder that lets you create, publish, and update a website from Google Sheets using ready-made templates.
To create a website explaining your service's request process and delivery schedule in concrete terms, give SpreadSite a try!
Build a website with SpreadSite →
