Beyond Feature Importance: Hierarchical Explanations of Feature Selection

An emerging family of explainable AI methods aims to identify, for each input, a minimal subset of observed feature values sufficient for predicting the output of a black box. These instance-wise feature-selection methods can reveal which features are important for a prediction. In this work, we go one step further and ask: why were these features selected rather than others? We introduce a recursive framework for hierarchical explanations of feature selection. At the first order, a sufficient-subset explainer produces an instance-wise binary feature-selection mask for explaining the original black-box output. Each subsequent order explains the selection mask produced at the previous order. Repeating this process yields an ordered hierarchy in which each level explains why the features at the previous level were selected. We develop a training procedure for learning these hierarchies. On nonlinear mechanisms generated by decision trees, our method recovers the underlying decision-tree structure substantially more accurately than conventional decision-tree baselines, while maintaining high predictive fidelity at every order. Our framework moves beyond identifying what features are selected toward explaining why they are selected.