acceptodds
Under review as a conference paper at ICLR 2027

Do Neural Networks Encode Generalizable Patterns in Specific Feature Directions?

Abstract

This paper focuses on the scientific problem of testing whether DNNs encode generalizable patterns in specific feature directions in a verifiable manner. We build on the recent finding that the output score of a neural network can be decomposed into numerical effects of different interaction patterns, some of which can generalize across DNNs and samples, whereas others cannot. We empirically observe that a DNN typically uses only a few feature directions in an intermediate-layer feature space to encode such generalizable interactions. We refer to these directions as reliable feature directions, as they capture generalizable interaction patterns. We identify such reliable feature directions and empirically examine their properties. The identified reliable feature directions usually play a dominant role in classification and are transferable to different samples.

Primary area:
learning theory
Keywords:
Hypothesis Testing, Mechanism of Overfitting, Interpretability, Representation Structure

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aLgcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether DNNs encode generalizable patterns in specific feature directions. Using AND-OR interactions and a first-order Taylor decomposition, the authors learn a few directions that preserve low-order, more generalizable interactions. They show these directions transfer to unseen samples and largely preserve classification performance.

Critical strengths

The trend that low-order interactions generalize better is shown consistently across CNNs, ViTs, tabular models, and pretrained ImageNet models. This matters for my assessment because it suggests the finding is not specific to a single architecture or dataset. It is the most convincing empirical result in the paper.

Critical weaknesses

1.Patch selection. The paper does not explain how the input patches are selected or justify the choice theoretically. Since generalizability is determined by exact patch-index matching across models, the results may depend heavily on this choice. A sensitivity analysis is needed.

2.Scalability. There is no evidence that 10 patches suffice for high-resolution or complex images. Exact extraction requires 2ⁿ forward passes per sample, which prevents scaling to larger n and limits the generality of the conclusions.

3.Relation to SHAP. The AND interaction is the Harsanyi dividend, from which Shapley values are directly derived. The authors should clarify how their approach differs from SHAP and from Shapley interaction indices, and what concrete advantage it offers.

4.Taylor approximation. It is unclear whether the first-order Taylor approximation of the layer output is accurate. The only reported case study shows substantial error (about 26% of the masking-induced output change). Validation across all models and layers is needed.

  1. Practical utility. It is unclear what the identified directions enable in practice. Unlike CBMs, which allow concept-level intervention, the paper uses projection only for evaluation. It does not show any downstream use, such as improving generalization or model editing.
Questions for authors

See weakness

Reviewer_CBetOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates if DNN encode generalisable patterns in specific directions of their intermediate feature spaces. Building on existing research on AND–OR feature interactions, the authors propose a method to identify reliable feature directions by preserving low-order interactions while suppressing high-order interactions (less generalisable). Experiments across different neural network architectures and datasets suggest that a small number of learned directions can capture most generalisable interactions, transfer to unseen samples, and retain substantial classification utility.

Critical strengths
  1. Interesting perspective on understanding DNN generalization. The paper connects feature-space directions with generalisable feature interactions.
  2. Extensive empirical evaluation. The authors conduct experiments across multiple architectures and datasets, examining interaction generalisability, transferability to unseen samples, and classification utility. These experiments provide empirical support for the proposed approach.
Critical weaknesses
  1. Insufficient justification for feature-direction analysis. Generalisable feature interactions, including the relationship between interaction complexity and generalisation, have been extensively investigated in prior studies. Therefore, the main novelty should arise from investigating these interactions through feature directions. However, the paper does not sufficiently establish what fundamentally new insights this perspective provides compared with directly analysing generalisable interactions using existing methods.
  2. Insufficient evidence supporting the central claim. The proposed method explicitly optimizes feature directions to preserve low-order interactions while suppressing high-order ones. Consequently, the observation that the learned directions predominantly encode low-order interactions is largely expected from the optimization objective. It remains unclear whether the results reveal an intrinsic property of DNN representations or simply demonstrate that the proposed optimisation can construct directions with the desired interaction characteristics.
Questions for authors
  1. Given that generalisable feature interactions have been extensively studied, what fundamentally new insights can be obtained by investigating them through feature directions? What advantages does this perspective offer over existing interaction-based approaches?
  1. How can the authors demonstrate that generalisable interactions are intrinsically concentrated in specific feature directions, rather than this concentration being induced by the proposed optimisation objective?
Reviewer_bLcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work tackles a challenging question of how the learned features of a deep neural network generalize across different models using a feature interaction approach to uncover the low-order interactions. They use the interaction decomposition from [Chen et al. 2024] to find the full model interactions, then they choose a middle layer of the deep neural network and perform the same interaction decomposition. They optimize for the middle layer feature direction which matches these two interaction decompositions on the lower order feature interactions. In their experimental results, they then find that the learned directions have low order feature interactions

Critical strengths

Strengths

This paper tackles an interesting and challenging research question about the generalizability of hidden layer feature directions in deep convolutional neural networks. They use a feature interaction approach which tries to understand which input features are useful both for the full model and for the intermediate layers, providing a more rigid way to investigate the learned feature directions.

Critical weaknesses

Weaknesses

By far the biggest weakness of this work is the writing. It is extremely challenging to understand the goal of the experiments run from the writing alone, and many sections are written as though the reader already understands the experiment which was run before explaining why it was run. I have several questions about the experiments run and conclusions made, and I would be happpy to raise my score if my concerns can be addressed.

  1. Although low order feature interactions are known to be more generalizable than high order feature interactions, is this taken as an assumption during the paper or is this something which is a research question which is empirically verified by the experiments?

  2. When the authors say `generalizable' interaction patterns, does this mean generalizable across models, generalizable across input samples, or something else?

  3. What are the input features? Are they the image patches as shown in Figure 1? I do not see them defined anywhere else.

  4. How are the interaction approximation made by Equation (5) and the Taylor approximation made by Equation (4) justified? I do not see any results which check how accurate these approximations are.

  5. Based on the descriptions of JSJ_S, it seems from Equations (5) and (11) that the definition of these quanitites is circular. Can you clarify further how these coefficients were fit and whether they were fit locally or globally?

  6. How do these results compare to related works like [A] and [B], do they agree or disagree with your conclusions? [A] claims that it is not possible to find global patches which work for all input images, instead requiring local self-attention to decide which patches are important for each input image. [B] claims that using AND-only masking fail for global patches, even when using higher-order interactions. I would like to better understand if these are agreeing with your work or disagreeing with your work.

  7. What are the conclusions in Section 3.3 from Figure 7? I don't understand if the authors believe that Figure 7 is good evidence that the relevant directions of ff have been learned or not.

  8. How do the findings vary across the input feature layers? A priori, it seems that if your methods are not checking whether the Taylor approximation is good, we can simply take the first or second layer, and if your methods are not checking whether the Equation (5) approximation is good, we can simply take the last layer. Is this explored somewhere in the appendix?

[A] "Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups" Weiqiu You, Helen Qu, Marco Gatti, Bhuvnesh Jain, Eric Wong. ICML 2025.

[B] "InstaSHAP: Interpretable Additive Models Explain Shapley Values Instantly" James Enouen, Yan Liu. ICLR 2025.

Questions for authors

(See the discussion in weaknesses.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.