Articles / AI Learning / Machine Learning & Data
If a model fits every training run, should you deploy it?
No. A model can match its training data more closely and still make worse predictions on new batches. The goal is not the smallest training error. It is the lowest reliable error on data the model did not use to learn.
This is the practical meaning of the bias–variance trade-off. It helps explain why increasing model flexibility can first improve performance and then make it less stable.
Three terms are enough
Flexibility is how many different patterns a model can represent. Bias is the error created when the model is too rigid to capture an important pattern. Variance is the model’s sensitivity to which examples happened to be in the training set.
Think of choosing a template to check the shape of a coated strip. A straight ruler misses a real bend. A suitable template follows the stable shape. A highly flexible wire can press into every small notch, including accidental noise that will not repeat on the next strip.
A coating-thickness example
Suppose a quality team has 200 historical coating runs. It wants to predict the final thickness error from process readings. The team compares three model choices and measures mean absolute error (MAE) in micrometres. Lower is better.
The following numbers are fictional teaching data, not a measured factory result. The rigid model has a training MAE of 3.6 µm and a five-fold validation MAE of 4.0 µm. Both errors are fairly high and close together: the model probably misses useful structure.
The balanced model improves to 2.0 µm on training data and 2.5 µm in validation. The highly flexible model reaches just 0.7 µm on training data—but validation error jumps to 5.2 µm.
From balanced to highly flexible, training error falls by 1.3 µm, while validation error rises by 2.7 µm. The last model is better at remembering the 200 runs and worse at predicting held-out runs. Its 5.2 µm validation error is more than twice the balanced model’s 2.5 µm.
Read the gap, not just the best number
When training and validation errors are both high, adding useful flexibility may help. When training error is low but validation error is much higher, further flexibility is a warning, not an achievement. The comparison must use the same target, units, data split and preprocessing for every candidate.
The previous lesson, Can You Trust One Test Split?, explains why rotating the held-out data gives a better view than one lucky split. Here, those validation results become a model-selection tool: choose the level of flexibility that performs well across held-out folds, not the one that wins on training data.
The answer, three takeaways and one action
So, should you deploy the model that fits every training run? No. Prefer the model that keeps error low and stable on unseen, representative data.
- A rigid model can miss repeatable structure: high bias.
- A highly flexible model can follow sample-specific noise: high variance.
- Training error alone cannot choose model complexity; held-out performance must decide.
One action: for the next model comparison, record training error and the same five-fold validation error for a simple, moderate and more flexible candidate. Keep the units and data preparation fixed, then choose from the validation evidence.
A meaningful limitation
The classical trade-off is a useful diagnostic, not a universal law. Modern deep networks can sometimes generalize better as they grow, despite their capacity. Validation also cannot protect against a future process change that is absent from historical data. Use separate drift checks when materials, equipment or operating policy changes.
Sources
- Aston Zhang, Zachary C. Lipton, Mu Li and Alexander J. Smola, Dive into Deep Learning, first edition, Cambridge University Press, 2023, Chapter 4 §4.6, especially §§4.6.3–4.6.4. Accessed September 26, 2026.
- scikit-learn developers, “Underfitting vs. Overfitting,” scikit-learn 1.8 documentation. Accessed September 26, 2026.
Leave a Reply