How to test whether a predictive maintenance alert can be trusted
Precision, recall and lead time are the three numbers that show whether a failure model earns a place in the maintenance plan. This draft is held pending sources.
A predictive maintenance model makes a claim: that a machine will fail soon, and that someone should act before it does. Whether that claim deserves trust depends on three numbers that most vendor brochures leave out. Precision asks how often an alert turned out to be a real failure. Recall asks how many of the real failures the model caught at all. Lead time asks how far ahead of the failure the warning arrived. A model can score well on one and badly on another, and a plant manager who sees only one of them will make the wrong call.
Before you start
Three things must be true before any validation figure means anything. First, the company needs a record of failures: dates, asset identities and failure modes, kept by maintenance rather than reconstructed later. Second, the model must be scored on failures it did not learn from. A model that is tested on its own training data will look better than it is. Third, the business must agree in advance what counts as a useful alert. A warning that arrives two minutes before a bearing seizes is not much use to a team that needs two days to bring a replacement part.
What to ask for
Ask for the precision, recall and lead time, each stated with the failure set it was measured on, the time window and the asset types covered. Ask how many failures were in that set. A recall figure based on four failures says little, and the reader should be told so. In our experience, the most useful single document is a table that lists each failure in the test period, whether the model flagged it, and how many days before the event the first alert fired. Without that table, the summary numbers cannot be checked.
This draft cannot yet give specific thresholds, typical ranges or the names of the reliability standards that govern how such tests should be run. The research sources needed for those claims have not been supplied, and this piece will not supply them from memory. The tests above rest on the plain arithmetic of the three measures and on the judgement described here.
What usually goes wrong
- The model is validated on the same period it was trained on, so its scores flatter it.
- Alerts are counted but not matched to actual failures, so precision is never measured.
- Failures are relabelled after the event, so the record fits the model's predictions.
- Lead time is reported as an average, which hides the cases where the warning came too late to act on.
What to do this week
Ask your maintenance lead for the failure log for the last two years and check whether it is complete. Then ask the vendor or internal data team to score the model against that log, using only failures the model did not see in training, and to return the three numbers with the table behind them. Do not approve a rollout on a summary alone. This article should be reviewed and completed with sourced standards and peer-reviewed studies before it is published.