Matthews Correlation Coefficient for Rare-Defect Models

For a rare-defect classifier, Matthews correlation coefficient (MCC) is a useful compact check because it uses true positives, true negatives, false positives, and false negatives together. It can expose a model that looks strong on accuracy only because almost every part is good. But MCC is a comparison metric—not an approval rule. Release decisions still need actual miss counts, false-review load, costs, and an accountable owner.

What MCC adds to the confusion matrix

A confusion matrix contains four outcomes. Accuracy collapses them into “correct” and “wrong.” In a factory where only 2% of inspected parts are defective, the large true-negative count can dominate that summary.

MCC instead compares the agreement between predicted and observed classes across all four cells. For binary classification, it ranges from −1 to +1: +1 is perfect agreement, values near 0 indicate little useful association, and negative values indicate systematically reversed decisions. The formula is:

MCC = (TP × TN − FP × FN) / √[(TP + FP)(TP + FN)(TN + FP)(TN + FN)]

You do not need to calculate it by hand. The important point is structural: a score can only rise when the classifier performs coherently on both classes. That makes MCC a useful companion to balanced accuracy, precision, and recall when defects are rare.

A worked inspection example

Consider an illustrative test set of 1,000 parts with 20 true defects. Model A finds 1 defect, misses 19, and creates 1 false alert. Its accuracy is 98.0% because it correctly labels 979 good parts. Its MCC is only 0.153.

Model B finds 15 defects, misses 5, and creates 20 false alerts. Its accuracy is slightly lower at 97.5%, yet its MCC is 0.556. The second model makes far more useful distinctions between the two classes, although its additional review burden is real.

This is synthetic teaching data, not a benchmark. The arithmetic uses the same 1,000-part population and the same positive-class definition for both models. If either changes, a direct score comparison may be misleading.

Do not turn MCC into a false pass mark

There is no universal “good MCC” for manufacturing. A score of 0.56 does not say whether five missed defects are tolerable, whether 20 false alerts fit the review team’s capacity, or whether errors cluster on a critical product family. It also changes with the selected threshold and evaluation population.

Use MCC to compare candidate models only when the label definition, time window, product mix, split method, and decision threshold are comparable. Then disclose the operating consequences separately. If the threshold changes, recompute the full confusion matrix rather than carrying the old MCC forward.

A practical reporting sequence

1. Compare: report MCC for each candidate on the same held-out production population and threshold.
2. Translate: show misses and false alerts per 1,000 inspections, then add review time and consequence estimates.
3. Own: name who approves the threshold and who watches prevalence, segments, and drift after release.

If different error types carry different operational costs, use a documented threshold decision policy. MCC can narrow the comparison, but the policy should decide what the factory does.

Three takeaways

• MCC is valuable when class imbalance makes accuracy look better than the underlying defect detection.
• It summarizes all four confusion-matrix outcomes, but it does not price or authorize those outcomes.
• Always pair the score with per-1,000 counts, a fixed threshold, segment checks, and a named review owner.

Next action

Take one held-out inspection set and produce a one-page comparison containing MCC, the four raw counts, misses and false alerts per 1,000, and the proposed review owner. Then test the operating trade-off with the Alert Trade-off Analyzer.

Sources

• James, Witten, Hastie, Tibshirani, and Taylor, An Introduction to Statistical Learning with Applications in Python (2023), Chapter 4; see the open classification lab for held-out evaluation and confusion-matrix interpretation.
• Google for Developers, Machine Learning Crash Course: accuracy, recall, precision, and related metrics.
• scikit-learn, matthews_corrcoef documentation and model-evaluation user guide.

Related reads

• How AI Makes Right and Wrong Decisions in Manufacturing
• Balanced Accuracy: Evaluating Rare Defects Fairly
• Turning Model Scores into an Operational Decision Policy