acceptodds
Withdrawn from ICLR 2027

Which neural network layers have stable representations?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many applications that rely on deep neural network (DNN) representations, such as encoding models or transfer learning, require choosing which intermediate layers to use. For such applications, we would like to get similar results for different instances of a model, i.e. stable results. Here, we aim to find out which DNN layers fulfill this criterion. We analyze five instances each of six ImageNet-trained architectures, comparing representations from each layer with their counterparts in other instances. Later network stages tend to be less self-similar, while certain layer types maintain higher stability throughout the network. We identify three criteria that characterize these most stable layers: (i) they are not bypassed by a skip connection, (ii) they are normalized, and (iii) they are placed after the non-linear activation function. Based on these criteria we modify ConvNext to yield an additional stable layer type confirming the validity of our criteria. Furthermore, we predict a large-scale fMRI dataset from each layer's representation, finding that stable layers yield more consistent and improved results for these encoding models. Finally, we tested transfer learning for a fine grained classification task, again observing more stable and better performance for stable layers. To confirm that our observations are not caused by the specific comparison methods, we perform each analysis with two different sets of methods. Based on our results, it seems advisable to focus on stable layers for all analyses that are based on internal representations of DNNs.

Primary area:
applications to neuroscience & cognitive science
Keywords:
representational similarity, vision, stability, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9w6vOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors evaluate intermediate layers of DNNs to find stable representations. They define stability as representations that give similar results across different instances of the same network trained for the same task. They analyze six ImageNet-trained architectures with five instances each. They compare layers across instances with two similarity measures (JSD with BayesCompare, and linear Centered Kernel Alignment) on MS COCO images. From this analysis, they empirically derive three criteria that characterize the most stable layers. Additionally, they modify ConvNeXt Tiny based on these criteria, and a stable layer type emerges, as predicted. Finally, they show on two downstream tasks (fMRI brain encoding and transfer learning) that stable layers give more consistent results across instances and comparable or better performance than non-stable layers.

Critical strengths

The paper tests stability in two very different applications: fMRI brain encoding and transfer learning on CUB-200-2011. In Table 1, stable layers have a higher mean accuracy in most of the cases. However, this strength is limited. The results depend on the stable-layer selection; many differences are small, and there are clear exceptions.

Critical weaknesses
  • The evaluation uses only five instances per architecture: four trained by the authors and one pretrained PyTorch model. The four trained instances differ only by random initialization and “minor modifications in their base training recipes” (lines 106–107). Their accuracies are within about one percentage point. The modified ConvNeXt is tested on only four instances. The current results show stability across small training differences. However, it is unclear whether this stability holds across larger training differences. More instances with clearly different training recipes would make the conclusions stronger.

  • It is unclear how the stable layers are selected. Table 3 orders the layers by the average of JSD and CKA, but the paper gives no rule for which layers are stable. The number of stable layers varies from zero to three. With this average, ResNet50 matches the paper’s labels only with the top one, ViT only with the top two, and EfficientNet only with the top three. In Swin-T and Swin V2, the rule is even more unclear, because Linear1 has the lowest average of all block layers but is marked as non-stable. In the modified ConvNeXt, LayerNorm1 (JSD 0.1426, CKA 0.0849) is marked as stable. However, GeLU1 has almost the same values (JSD 0.1362, CKA 0.0906) in the original ConvNeXt and is marked as non-stable. This is critical because the three criteria, the ConvNeXt test, and the downstream results all depend on these labels.

Questions for authors
  • Do the findings still hold if the training recipes change significantly?

  • Could the authors state the rule they used to classify layers as stable? Why is Linear1 not selected in Swin-T and Swin V2, although it has the lowest average self-distance (JSD and CKA) in both models?

Reviewer_bvTTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of representation stability in deep neural networks. The authors analyze five instances each of six ImageNet-trained vision architectures (ResNet-50, ViT-B/16, Swin Transformer Tiny, Swin Transformer V2 Small, ConvNeXt Tiny, and EfficientNet V2 Small), computing layer-wise self-distances using three similarity metrics (BayesCompare JSD, linear CKA, and Jaccard similarity). They find that later network stages tend to be less self-similar, while certain layer types maintain higher stability throughout the network. Based on these observations, the authors propose three empirical criteria for stable layers: (1) not bypassed by skip connections, (2) followed by normalization, and (3) located after non-linearity. The criteria are validated through architecture modification (modified ConvNeXt) and shown to be useful in brain encoding (fMRI prediction) and transfer learning (CUB-200-2011) tasks.

Critical strengths
  • The analysis spans six diverse architectures, experiments are comprehensive.

  • The ConvNeXt modification experiment provides causal evidence for the proposed criteria, going beyond mere correlation.

  • The connection to brain encoding and transfer learning tasks demonstrates the practical utility of stable layers, not just their theoretical interest.

Critical weaknesses
  • The study is limited to ImageNet-trained vision models.

  • The paper does not adequately address the questions practitioners care about most, such as how to choose a substitute layer.

Questions for authors

The brain-encoding experiment feels only loosely connected to the paper's central claim. Stability, as defined in the paper, is a property of representations across training instances, whereas neural predictivity measures how well a layer resembles brain responses. These are not the same thing, and the paper does not clearly justify why one should imply the other. Why, then, is the brain analysis placed in the main text rather than in the appendix? What does it contribute that the layer-wise similarity analysis does not already provide? If the goal is to show that stable layers are practically useful, a more direct validation would be on robustness to distribution shift, fine-tuning, or pruning, rather than on fMRI fitting. As it stands, the brain section reads more like an add-on that broadens the paper's scope than a test that really falsifies or supports the three empirical criteria.

Reviewer_TvhoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how consistent is image representation in neural networks. To do so they use 1000 images from MS-COCO, after presenting the image to the network they extract the activations at different layers, and use metrics like CKA and BayesCompare to study the inter-image variation. They find that generally decreases with depth, but representations that are not bypassed by skip connections, follow normalization and non lienear activation tend to be more stable representations. Furthermore, they find that these stable layers are generally better at predicting brain activity recorded from FMRI responses. As well, they show in a narrow task that transfer-learning accuracy and consitency on a bird classification task with some exceptions.

Critical strengths

The paper provides a good analysis on layer stability and the usage of bayes on this context is particularly elegant. The paper is clear and provides a simple message that is well developed.

Critical weaknesses

I missed a deeper discussion of the connection between layer normalization, representational stability, and brain-response prediction. Why might more stable representations be particularly effective for predicting brain activity? Does this reflect computations shared with biological visual systems, or general statistical properties that benefit downstream prediction? Although the paper offers intuitions about why certain layers are more stable, it could develop these into testable hypotheses that distinguish these explanations and clarify what the findings reveal about brain mechanisms.

What about other training paradigms? Would these hold under unsupervised/self-supervised pre/training or under multimodal training?

I like the transfer learning experiment, but it sounds a bit limited, can other comparable datasets be included?

The paper cites important work on representational variability, including Mehrer et al. (2020), but its connection to research on representational universality and brain alignment could be developed further. In particular, Chen and Bonner (2024) and Hosseini et al. (2024) link shared representations or inter-model agreement to brain alignment, yet neither is cited.

Questions for authors

Included in the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.