Feature Importance Visualizer

Feature importance is a score that measures how much each input feature contributes to a model's predictions. Calculation methods vary by algorithm: tree-based models (Random Forest, XGBoost) use impurity reduction (how much each split reduces Gini impurity or entropy), while linear models use coefficient magnitudes. Permutation importance — shuffling each feature and measuring performance drop — is model-agnostic and often more reliable because it measures actual predictive contribution rather than how frequently a feature is used for splits.

Permutation importance measures performance loss when a single feature is randomly shuffled, making all other features unchanged. It is fast but can underestimate features that are correlated with others. SHAP (SHapley Additive exPlanations) uses game theory to assign each feature a Shapley value — the average marginal contribution across all possible feature subsets. SHAP is more theoretically principled, handles correlated features better, and provides local (per-sample) explanations, but is computationally expensive for large datasets. For quick global feature ranking, permutation importance is usually sufficient; for regulatory compliance or detailed model auditing, SHAP is preferred.

Multicollinearity (high correlation between features) can severely distort importance scores. When two features carry similar information, tree-based models may arbitrarily split importance between them, making each look less important than it actually is. Linear model coefficients become unstable and hard to interpret. Mitigation strategies: (1) compute Variance Inflation Factor (VIF) and remove features with VIF > 10, (2) use SHAP which handles correlation better than split-based importance, (3) cluster correlated features and select one representative per cluster, (4) use regularization (Lasso/Ridge) to let the model choose among correlated features.

A practical workflow: (1) Train a baseline model on all features and compute importance. (2) Sort features by importance descending and compute cumulative importance. (3) Keep the minimum set of features that explain 80–95% of cumulative importance (this tool highlights that threshold). (4) Retrain with the reduced feature set and compare cross-validation scores — if performance is within 1–2%, the reduction is successful. (5) Watch for data leakage: features with suspiciously high importance (>50%) often indicate target leakage. Recursive Feature Elimination (RFE) automates this iteratively and is more robust for small datasets.

Key pitfalls to avoid: (1) Confusing importance with causation — a feature can be important for prediction but not a cause of the outcome. (2) High-cardinality categorical features (many unique values) are artificially inflated in tree-based importance because they offer more split points. (3) Scale sensitivity — some algorithms are sensitive to feature scale, so always normalize before comparison. (4) Importance is dataset-specific: a feature important on your training set may not generalize. Always validate importance estimates on a held-out set. (5) Ignoring interactions — two features may each seem unimportant individually but interact to be critical together; interaction terms or SHAP interaction values can reveal this.