A Sensor Ranks Low in Feature Importance. Can You Remove It?

No—not from one feature-importance ranking. Permutation feature importance measures how much a fitted model’s score worsens when one input is shuffled. That is useful evidence of model reliance, but correlated sensors can substitute for one another and make both look unimportant. Validate the model first, use an operationally relevant metric, repeat the shuffle, test related features as a group, and retrain before removing anything.

What permutation importance actually measures

Start with a trained model and a labeled evaluation set the model did not learn from. Record a reference score. Then shuffle one feature column while leaving the outcome and every other column in place. If the score falls, the model was using information in that feature. Repeat the shuffle because one random rearrangement can be lucky or unlucky.

The result belongs to a specific model, dataset and metric. A feature can appear important for F1 but less important for a cost-weighted score. Importance on training data can expose what the model memorized; importance on held-out data is more useful for understanding generalization. Neither version proves the feature causes defects.

Worked example: two temperature sensors share one signal

The following numbers are synthetic teaching data, not factory performance evidence. A defect classifier has held-out F1 of 0.78. Shuffling camera contrast lowers F1 by 0.11 and conveyor speed by 0.06. Two furnace temperature sensors are strongly correlated (ρ = 0.96). Shuffling sensor A alone lowers F1 by only 0.02; shuffling sensor B alone lowers it by 0.03.

It would be wrong to conclude that temperature barely matters. While one sensor is shuffled, the other still carries similar information. When both are shuffled together, F1 falls by 0.15. The group result says the model relies on the shared temperature signal; it does not say both devices are necessary. That requires a separate retraining and redundancy test.

Three ways a clean ranking can mislead

1. The model is weak or the test is contaminated. Explaining a poor model only explains poor behavior. Confirm predictive performance on independent, future-like data, with the same grouping discipline used to prevent inspection-data leakage.
2. The metric answers the wrong operating question. Accuracy may hide rare misses; F1 may ignore review capacity or unequal costs. Choose the score before looking at the ranking, and inspect performance by product, line or shift with slice metrics.
3. Correlation splits or hides reliance. Related temperatures, pressures or derived rolling averages can replace one another. Review correlations and provenance, then shuffle a physically meaningful group or use a conditional method that preserves plausible combinations.

Permutation importance is global and performance-based. It does not show whether increasing a feature raises or lowers one prediction. For case-level contributions, compare it with SHAP explanations—while keeping the same warning that model explanation is not causal proof.

A five-gate review before changing the sensor set

Keep the mean and spread from repeated shuffles, not only a sorted bar chart. Recalculate the result for operating slices that matter. If a low-ranked sensor is a safety backup, diagnostic signal or regulated record, model reliance is not the only removal criterion. Ask engineering, quality and controls owners what would be lost outside the model.

Finally, compare the current model with a retrained candidate that omits the feature or group. Recheck overall and slice performance, calibration, drift monitoring, failure modes and sensor economics. A low permutation score is permission to investigate—not permission to disconnect hardware.

Three takeaways

• Permutation importance measures a particular model’s performance reliance, not physical causality.
• Correlated sensors can make individual importance look small even when their shared signal matters.
• Any removal decision needs group tests, retraining and future-like validation—not a single ranking.

Sources

• Christoph Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 3rd edition, Chapter 23 “Permutation Feature Importance”, accessed October 14, 2026.
• scikit-learn 1.9.1, “Permutation feature importance”, including held-out evaluation, repeated shuffles and metric dependence.
• scikit-learn 1.9.1, “Permutation Importance with Multicollinear or Correlated Features”.
• Aaron Fisher, Cynthia Rudin and Francesca Dominici, “All Models are Wrong, but Many are Useful”, Journal of Machine Learning Research 20(177), published 2019.

Next action

Take one validated model and select a sensor plus its closest correlated substitute. On the same frozen holdout set, repeat individual shuffles 20 times, shuffle the pair together, and record the score mean and spread. If the group drop is much larger, keep the pair together in the engineering review and retrain before proposing removal.