Do not buy another labeling batch just because validation performance is disappointing. First plot training and validation scores against training-set size. If the validation curve is still rising and the gap is narrowing, more representative examples may help. If both curves have flattened low—or the result changes wildly by product, line or split—spend first on labels, features or evaluation design.
A learning curve does not tell you exactly how many parts to label or guarantee a return. It turns a vague request for “more data” into a testable decision about what evidence is missing.
What a learning curve measures
A learning curve trains the same pipeline on progressively larger subsets and plots a chosen score for the training data and unseen validation data. The shape helps distinguish two common problems.
- Large train–validation gap: the model fits what it saw but does not generalize well. More representative data can reduce variance if the validation score is still improving.
- Both scores low and close: the present features, labels or model assumptions may not express the relationship. More of the same examples is unlikely to fix that bottleneck.
This is a practical extension of the bias–variance trade-off. A learning curve changes the horizontal axis from model flexibility to sample size, so the question becomes whether additional observations are still buying generalization.
Build the curve without changing the question
Keep the model family, preprocessing, target definition, metric and evaluation protocol fixed. Use nested training subsets—such as 250, 500, 1,000, 2,000 and 4,000 parts—and repeat the fit so the chart shows a mean and useful uncertainty, not one fortunate run.
The split must match the operating question. Images of one physical part cannot appear in both train and validation; use non-overlapping groups for parts, lots or assets. Time-dependent predictions need an earlier-to-later split so the model cannot learn from the future. The site’s group-leakage test explains why a new file is not necessarily a new independent example.
Choose a score that reflects the actual decision. The chart uses Macro F1 so each class contributes equally, but another workflow may need recall at a fixed review capacity, cost-weighted error or forecast loss. Also inspect the curve by product, line and operating condition; an aggregate improvement can hide a failing slice.
A synthetic inspection example
Suppose a fictional surface-inspection model has training Macro F1 values of 0.91, 0.90, 0.89, 0.88 and 0.87 as the labeled set grows. Validation rises from 0.64 to 0.71, 0.76, 0.80 and 0.82. The gap shrinks from 0.27 to 0.05, so the early additions clearly help. But doubling from 2,000 to 4,000 adds only 0.02.
The next decision should not be “label another 4,000 at random.” Review where errors remain. If rare finishes, lighting states or suppliers are underrepresented, buy a targeted tranche and redraw both the overall and slice curves. If the residual errors are label disagreements, repair the rules and use the inspection-label agreement check before expanding.
What the curve cannot prove
A curve describes the sampled population and the chosen protocol. It cannot prove that new data will match old data, that annotators will follow the same rule, or that a future product mix will be covered. Extrapolating far beyond the observed sizes is uncertain. A rising curve built on duplicated parts, selectively labeled defects or leaked time periods is still misleading.
Costs also matter. The last score improvement may be statistically real but operationally irrelevant, or it may justify collection if one missed defect is severe. Put the curve beside annotation cost, review capacity and the decision threshold; do not turn it into an isolated model leaderboard.
Three takeaways
- A rising validation curve and narrowing gap support a targeted data-collection test—not an unlimited labeling budget.
- Flat low curves point toward features, labels or model assumptions before sample expansion.
- Group, time and slice discipline matter more than a smooth-looking aggregate plot.
Sources
- Terence Parr and Jeremy Howard, The Mechanics of Machine Learning, book version 0.4, Chapter 2 “How Machine Learning Works”, 2018–2019. Sections 2.1 and 2.4 cover generalization, overfitting, underfitting and validation data.
- scikit-learn 1.9.1, “Validation curves: plotting scores to evaluate models,” Section 3.5.2 Learning curve, accessed October 17, 2026.
- scikit-learn 1.9.1, “Cross-validation: evaluating estimator performance”, including grouped-data validation, accessed October 17, 2026.
- Google for Developers, Machine Learning Crash Course: “Datasets, generalization, and overfitting”, last updated December 3, 2025.
Next action
For one factory model, train the unchanged pipeline on five nested, group-safe sample sizes. Plot the training and validation score with uncertainty, then write one sentence that names the next evidence purchase: representative examples, repaired labels, a missing signal—or none yet.
Leave a Reply