Which Model Error Should You Fix First?

Articles / AI Learning / Practical AI Skills

Your model is wrong on 1 in 10 checks. Should the team collect more data, relabel images, or change the model?

Start by inspecting a bounded set of wrong predictions. Group them by cause, count each group, and estimate the most each proposed fix could recover. This is error analysis: a practical way to choose work from evidence instead of from the loudest opinion.

A score tells you how much, not why

Suppose a vision model checks machined plates. Its accuracy is 90% on a development set—an unchanged set of examples used to guide improvements. The score says 100 of 1,000 cases are wrong. It does not say whether the problem is glare, hidden features, incorrect labels, unfamiliar part variants, or something else.

Think of a returned-parts bin. Counting every return tells you the size of the problem. Sorting the returns by cause tells you where a process change might help. The analogy stops there: model-error categories can overlap, and the data still has to represent real use.

Andrew Ng recommends manually examining misclassified development examples and counting the share associated with each idea in Machine Learning Yearning, Chapters 14 and 15. The aim is not to explain every mistake perfectly. It is to learn which possible project is large enough to matter.

A 100-error worked example

Review the 100 wrong cases and assign one primary cause to each:

• 35 involve glare.
• 25 show a partly hidden feature.
• 15 need label review because the recorded answer may be wrong.
• 15 contain unfamiliar part variants.
• 10 fall into other causes.

The largest group is glare. If a new lighting setup fixed every glare-related mistake and introduced no new errors, it could remove at most 35 of the 100 errors. Across all 1,000 development cases, accuracy would move from 90.0% to at most 93.5%.

That 3.5-percentage-point change is a ceiling, not a forecast. A real lighting change may solve only part of the glare group. It may also alter other images. The calculation is useful because it prevents a team from promising a five-point gain from a problem that accounts for only 3.5 points of current error.

Turn the counts into a work decision

Do not automatically choose the largest category. Compare three questions:

1. How much error is associated with it? The count provides the ceiling.
2. Can the team change it? Better lighting may be easier than collecting rare part variants.
3. Is the reference answer trustworthy? The 15 label-review cases may reveal a data-quality problem rather than a model problem.

Keep the actual and predicted class for every reviewed case. A confusion matrix follows the same discipline: rows record the true class and columns record the predicted class, as the scikit-learn documentation explains. Those counts stop a broad label such as “bad image” from hiding which decisions are failing.

Optional detail: avoid false totals

If one case can receive several tags—for example, both glare and partly hidden—category counts can exceed 100. Do not add those overlapping counts and present the sum as removable errors. Either choose one primary cause for the ceiling calculation, as this example does, or report overlaps separately.

Also keep the final test set out of daily diagnosis. Repeatedly choosing changes from test cases turns that set into another development set and weakens the final check.

The answer

Which error should you fix first? Choose the largest credible, addressable error group—not simply the most visible mistake or the newest model idea. In this example, glare deserves the first small experiment only after the team confirms that the labels are reliable and a lighting change is feasible.

Three takeaways

• Overall accuracy measures the size of error; error analysis reveals its causes.
• A category’s share of all cases is an improvement ceiling, not an expected result.
• Counts matter only when the reviewed development set resembles intended use.

One action

Export the next 100 wrong development examples. Add three columns—primary cause, label confidence, and a possible fix—then count the causes before approving the next model task.

Limitation

A small review can miss rare but costly failures. Use business severity alongside frequency, and repeat the review when products, sensors, or operating conditions change.

Related lesson

Next, learn how to preserve the true-versus-predicted counts in How to Read a Confusion Matrix for AI Quality Inspection.

Sources

• Andrew Ng, Machine Learning Yearning, official DeepLearning.AI draft/sample edition, 2020: Chapter 14, Error analysis: Look at dev set examples to evaluate ideas, Chapter 15, Evaluating multiple ideas in parallel during error analysis, and Chapter 19, Summary of basic error analysis. Accessed September 23, 2026.
• scikit-learn 1.9.1 documentation, Confusion matrix. Accessed September 23, 2026.