Articles / AI Learning / Machine Learning & Data
Your model scored 95% on one small test split. Can you trust that number? Not by itself. One split may contain unusually easy or difficult cases. Cross-validation repeats the check on different parts of the same development data, so you see both an average and how much the result moves.
Why one split can mislead
Imagine a quality team with 200 labeled images of sealed packages. It trains a model on 160 images and checks it on the remaining 40. The model gets 38 right, so the reported accuracy is 95%.
That calculation is correct: 38 ÷ 40 = 95%. The problem is the choice of 40. If that set happens to contain clearer images, fewer rare defects or less variation between production lines, the number will look better than the model’s usual performance.
A single check is like inspecting one tray from a shift. It can reveal a problem, but it cannot show whether that tray was typical. The textbook An Introduction to Statistical Learning calls this the validation-set approach and highlights two drawbacks: the estimate changes with the chosen split, and the model learns from only part of the available data.
Rotate which part gets checked
In five-fold cross-validation, the 200 development items are divided into five non-overlapping groups, called folds. The model learns from four folds and is checked on the fifth. This is repeated five times, so every item is held out once. The five results are then averaged.
The model is learning during each round because it is fitted again on a different four-fifths of the data. The checking step is inference: the fitted model makes predictions for the fold it did not use for learning.
Cross-validation does not create new evidence. It reuses the available development data in a disciplined way. A separate untouched test set is still useful for the final check after teams have compared models or settings; otherwise, repeated choices can indirectly tune the work to the evaluation data.
A five-fold package example
Suppose the five held-out folds each contain 40 package images. The model makes 2, 6, 3, 5 and 4 mistakes. Those results correspond to accuracies of 95%, 85%, 92.5%, 87.5% and 90%.
The average is 90%: (95 + 85 + 92.5 + 87.5 + 90) ÷ 5. If the team had used only the first split, it would have reported 95%. The other folds show that the same method can land anywhere from 85% to 95% on these particular groups.
The average is usually more useful than the luckiest fold, but the range matters too. A wide spread is a prompt to inspect whether some folds contain different products, cameras, lines or defect types. It is not proof that the model will score exactly 90% in production.
Use the right kind of split
Random folds assume the items can reasonably be mixed. That assumption often fails in industrial data. Images from the same batch, machine or supplier may be near-duplicates. Time-ordered records may contain future information. In those cases, keep related items together or preserve time order. Cross-validation cannot repair leakage created by the wrong grouping rule.
For a next lesson on that limitation, read Did Your Model See the Future?
The answer
No: one 95% test split is not enough to establish stable performance. Five-fold cross-validation asks the model to succeed across five held-out groups, giving you an average and a spread before the final untouched test.
Three takeaways
- A correct score can still be unrepresentative when it comes from one convenient split.
- Five-fold cross-validation rotates the held-out group and averages five checks.
- The fold-to-fold range can expose sensitivity that a single average hides.
One action
For your next model review, request every fold’s score—not only the average—and record whether the split kept batches, machines, suppliers and time periods appropriately separated.
Limitation
Cross-validation estimates performance from the data you already have. If that data omits a new product, future operating condition or rare failure, repeating the split will not supply the missing evidence.
Sources
- Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani, An Introduction to Statistical Learning: with Applications in R, first edition, Springer, 2013. Chapter 5, “Resampling Methods,” especially §§5.1.1, 5.1.3 and 5.1.4. Official open book accessed September 25, 2026.
- scikit-learn 1.9.1, “Cross-validation: evaluating estimator performance,” including the k-fold procedure, final test-set warning and group/time-aware split guidance. Accessed September 25, 2026.
Leave a Reply