- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates structured depth pruning on vision models via layer-importance scores derived from brain-model alignment scores computed from the Natural Scenes Dataset. The paper proposes 2 metrics, R1D and R2D, which are functions of brain-encoding scores R which is computed by maximally correlating voxel responses and model activations. Using a greedy layer dropping approach, different vision foundation models are pruned according to these metrics, and compared to other layer pruning baselines. With and without additional fine-tuning, these metrics are shown to perform similarly to strong layer pruning baselines.
Critical strengths
The analysis section in this paper is its largest strength because it 1) verifies that the brain data specfically is contributing something useful rather than just a second-derivative metric, and 2) is honest about why random pruning is useful in the layer drop setting. It is an interesting finding that is useful to those working on layer dropping in the vision model context. Additionally, it was also nice to see the results as a function of individual participants on the natural scenes dataset, which shows that even individual participants responses give similar pruning maps.
Critical weaknesses
-
The brain-model alignment score is highly hyperparameter specific, and its motivation is not clear. Why is it the case that the blocks representation should linearly predict the neural code? And why is it the case that is fitted separately per voxel? By carefully fitting as closely as possible this linear map, this seems to artificially inflate , which is then used downstream as an importance metric.
-
The performance difference between the proposed method and baselines, especially after healing, is minimal, with several baselines outperforming the proposed method. It seems like the paper is trying to do two things simultaneously: one is try to determine whether brain model alignment is a functional measurement for layer pruning, and the second is frame this method as outperforming other metrics, which doesn't seem like it should be the point of this more exploratory work. This is because as a functional pruning metric, the proposed method is a bit impractical because it requires brain data on near-distribution image input. Additionally, splitting "best" results across R1D and R2D is not clear; it would be helpful to explain why it is useful to look at both of these metrics in context of performance, and what they tell us about brain-model alignment's role in pruning.
Questions for authors
- How were the 15 ROI regions selected? Is it based on prior work that describes each regions role in vision, or something else?
- Prior work on LLM layer pruning has focused on pruning contiguous blocks of layers. This work suggests that random layer dropping and the proposed method (non-contiguous) are strong due in part to being non-contiguous. Is this perhaps a function of the domain difference or something else? Did prior papers fail to consider the non-contiguous setting?
4 comments
Response to Reviewer yQJu 2/2
3. Performance, framing and practicality.
Significance. We now repeat the full pruning pipeline 5 times; each repetition samples a different calibration set (for R1D/R2D, a different NSD subset for the encoding models), which can change the pruned blocks, and evaluated on a different fifth of the test set. We also include an additional baseline, first-order Taylor importance, a gradient-based criterion that uses labels. Table 2 of the revised manuscript now reports mean ± standard deviation for all models and methods; below, DeiT and Swin after healing, averaged over the five datasets:
| Model | Pruned | EntroDrop | ShortGPT | Angular | P3B | Random | Taylor | R1D | R2D |
|---|---|---|---|---|---|---|---|---|---|
| DeiT-B | 25% | 80.0±0.5 | 82.6±0.4 | 79.9±0.4 | 80.8±0.5 | 81.2±0.6 | 81.7±0.5 | 79.3±0.6 | 81.6±0.5 |
| 50% | 66.5±0.4 | 73.8±0.6 | 55.8±0.4 | 63.3±0.4 | 70.2±1.9 | 71.9±1.0 | 73.3±1.0 | 74.0±0.5 | |
| 75% | 46.8±0.4 | 46.2±0.3 | 45.8±0.4 | 45.4±0.3 | 44.9±1.4 | 47.1±1.4 | 45.0±1.5 | 50.1±0.5 | |
| Swin-B | 25% | 83.1±0.4 | 84.9±0.4 | 83.9±0.4 | 84.7±0.3 | 83.6±0.4 | 82.5±0.7 | 84.6±0.4 | 84.3±0.4 |
| 50% | 78.8±0.3 | 79.4±0.4 | 77.4±0.3 | 79.9±0.4 | 79.7±0.8 | 74.7±1.7 | 78.9±0.3 | 80.9±0.4 | |
| 75% | 57.9±0.2 | 59.0±0.4 | – | 59.0±0.4 | 55.3±3.3 | 57.8±1.7 | 58.5±1.5 | 59.0±1.4 |
Over all 9 models, 5 datasets, 3 pruning ratios and 6 baselines, R2D outperforms the baselines by more than one standard deviation in 77% of comparisons without healing and 58% after healing. We also note that R2D achieves this using only brain responses to natural images, without labels or any target-task data.
Framing. We agree and removed all state-of-the-art claims. The framing focuses now on asking whether brain-model alignment is a useful functional signal for pruning, and our results show that it is.
Near-distribution data. Our metrics do not need brain data on near-distribution images. R1D and R2D are computed once from the public NSD dataset, without labels or target images, and reused for every dataset and pruning ratio. Their advantage holds far from NSD too since without healing, R2D outperforms the baselines in 79% of comparisons on COCO, the source of NSD images, and in 77% on the other 4 datasets.
R1D and R2D. We now report R1D as a first-order ablation. R2D outperforms R1D by more than one standard error in 77% of comparisons without healing and 46% after healing. What matters for pruning in several cases is not how much a block changes brain alignment but whether it marks a transition between processing stages.
4. Contiguous pruning. Prior LLM work considered both settings; ShortGPT [6] and SLEB [7] remove non-contiguous layers, while angular distance [8] removes the best contiguous block. Contiguous removal may be popular because healing then repairs a single interface, but in our vision models it is especially harmful and healing does not compensate. On DINOv3-7B, making the removed blocks adjacent multiplies the output divergence by 10 to 134 times. At 50% pruning, every contiguous run of 20 blocks keeps at most 17.6% accuracy without healing while random sets keep 87.6%, and at 75% contiguous runs recover only to 63% after healing while random sets reach 85 to 88%. We believe this reflects how redundancy is spread across depth in these models rather than the domain, since neighbouring blocks compensate for an isolated removal but not for a whole run. We added this discussion to the revision.
Thank you again for your helpful feedback. We made a considerable effort to address each of your points with new ablations, stability and significance analyses, and a clearer framing of our contribution. We believe these additions strengthen our paper, and we would be grateful if you would reconsider your assessment in light of these revisions.
References:
[1] Schrimpf, Martin, et al. "Brain-score: Which artificial neural network for object recognition is most brain-like?." BioRxiv (2018): 407007.
[2] Yamins, Daniel LK, et al. "Performance-optimized hierarchical models predict neural responses in higher visual cortex." Proceedings of the national academy of sciences 111.23 (2014): 8619-8624.
[3] DiCarlo, James J., Davide Zoccolan, and Nicole C. Rust. "How does the brain solve visual object recognition?." Neuron 73.3 (2012): 415-434.
[4] Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.
[5] Raugel, Joséphine, et al. "Disentangling the factors of convergence between brains and computer vision models." arXiv preprint arXiv:2508.18226 (2025).
[6] Men, Xin, et al. "Shortgpt: Layers in large language models are more redundant than you expect." Findings of the Association for Computational Linguistics: ACL 2025. 2025.
[7] Song, Jiwon, et al. "Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks." arXiv preprint arXiv:2402.09025 (2024).
[8] Gromov, Andrey, et al. "The unreasonable ineffectiveness of the deeper layers." International Conference on Learning Representations. Vol. 2025. 2025.
Thank you for the author rebuttal.
Thanks for the updated clarification on weakness 1, it would be valuable to improve the clarity of this setup in the main text in order to justify the design choices as a product of prior work, and also to properly attribute credit. And thank you for the answer to the ROI selection question and clarifying this with respect to prior work.
I appreciate that the authors attempt a larger overhaul of the paper framing to soften many of the claims regarding state-of-the-art and improve the framing. I do not think this is a level of revision that is doable well within a rebuttal period and also very difficult for individual reviewers to readdreass. To that end, W2 still remains and does not seem well resolvable during the rebuttal period.
Regarding other aspects of W2, like near-distribution, I also do not find this respose to address my concern. Near-distribution or in-distribution does not require that labels are present. The concern instead stems from the fact that the metrics did indeed get computed from NSD (I understand without labels), and then tested only on data very similar to NSD. The concern is that if a different dataset was selected for evalution, R{1,2}D would not generalize well. It is not the true that this needs to be the case, especially in a paper that explores brain data as an importance metric rather than proposes a SOTA Method, but an honest discussion of this limitation seems necessary. I apologize if the original concern was not written clearly.
I appreciate the reframing of R1D as an ablation, but this again, like the above point, is a significant overhal of the presentation and content of the paper.
Finally, re contiguousness. I appreciate these additional numbers regarding contiguousness and randomness. It seems that contiguousness is very unfavored in this setting, perhaps related to model differences in the current work and other papers more related to LLMs. This question was mostly out of curiosity and is well answered. Thank you! This is an interesting hypothesis about depth redundancy.
I want to acknowledge that the authors put in a significant effort during the rebuttal period. I unfortunately still do feel that this amount of revision including in key claims and framing of this paper makes it very difficult to increase the score now that the paper is quite different in spirit.
Thank you for engaging with our rebuttal and for acknowledging the effort we put into it. We are glad that our responses clarified the encoding setup, the ROI selection and the question on contiguous pruning. We would also like to gently clarify that the revisions to the framing and to R1D are much narrower than they may appear, and that all of them are already in the revised manuscript in blue, so they can be checked directly.
Framing. The original submission already framed our work as an investigation of whether brain-model alignment is a useful signal for pruning; the introduction states that "we investigate whether brain-model alignment captures the functional importance of individual blocks in vision models", and Section 2 poses our central question as whether brain-model alignment can guide the identification of the most functionally important blocks. The only state-of-the-art claim in the submission was one phrase in the conclusion, which we removed, and we added two sentences to the introduction to make the existing framing explicit. The abstract, the contributions, the method and all of the original experiments are unchanged, and the new results only add the uncertainty estimates and baselines that the reviewers requested.
R1D as an ablation. Similarly, the submission already presented R2D as our main metric; our contributions state that R2D is competitive with or outperforms the baselines across all pruning strengths and isolate the role of the brain data for R2D, while R1D "also shows considerable performance improvements". We kept every R1D result and only added one sentence to Section 3.3 that describes R1D as a first-order ablation, so this clarifies the existing presentation rather than changing its content.
Encoding setup and credit. Following your suggestion, Section 3.1 of the revised manuscript now motivates the linear, voxel-wise encoding as a product of prior work and credits Schrimpf et al., Yamins et al., DiCarlo et al., Naselaris et al. and Raugel et al., whose setup we follow.
Generalization beyond NSD. Thank you for clarifying this concern. Our transfer datasets already include images that differ substantially from the natural scenes of NSD, such as the textures of DTD and the low-resolution images of CIFAR, and R2D wins more comparisons against the baselines than it loses on every dataset:
| Comparisons where R2D outperforms the baselines | COCO | ImageNet | CIFAR-10 | CIFAR-100 | DTD |
|---|---|---|---|---|---|
| No healing | 79% | 86% | 73% | 79% | 69% |
| Healing | 58% | 73% | 48% | 56% | 55% |
However, the advantage is smaller on the datasets furthest from NSD, and we agree that an honest discussion of this limitation is necessary. We therefore added the following to the conclusion of the revised manuscript: "R1D and R2D are computed once from brain responses to the natural scenes of NSD. Although their advantage holds on ImageNet, CIFAR and DTD, it is smaller on the datasets furthest from NSD, and we did not evaluate domains that differ more strongly from natural images, such as medical or satellite images, which we leave for future work."
We hope these clarifications show that the revised paper keeps the same spirit, method and experiments as the submission, and that our revisions only address the points raised during the review. We are grateful for your engagement throughout the discussion, and we would appreciate it if you could take these clarifications into account in your final assessment.
Response to Reviewer yQJu 1/2
Response to Reviewer yQJu
Thank you for your thoughtful review and for highlighting our analysis section, our discussion of random pruning and our per-participant results. Inspired by your feedback, we tested a nonlinear encoder and automatic region selection, measured the stability of the R scores, added standard deviations to every comparison, and now use R2D as our main metric.
1. Linear, per-voxel encoding. Our encoding approach builds on a substantial body of work on neural representations and brain-deep network alignment (Schrimpf et al. [1], Yamins et al. [2], DiCarlo et al. [3], Naselaris et al. [4], Raugel et al. [5], etc.), which uses the standard definition of a representation as linearly readable information and on the voxel-wise encoding framework reviewed by Naselaris et al. [5]. A linear map tests what a block represents explicitly, while a flexible map could recover brain-relevant information from almost any block and blur the differences our metrics rely on. We fit each voxel separately because voxels differ in noise and selectivity, with each penalty set by cross-validation, not by hand. Fitting closely cannot inflate R, since R is evaluated on held-out images and R1D and R2D only use differences between consecutive blocks, where any shared bias cancels.
To test whether a linear map is sufficient, we also computed the r scores with two nonlinear maps:
Both networks are trained on each participant's own images and tested on the 907 images that all participants saw. We report r on the 15 ROIs (used for R1D and R2D in the paper), the whole brain, well-predicted voxels and automatically selected regions, as the change from ridge fit on the same images (ridge's r in parentheses), averaged over DeiT-S and DINOv3-S; ± is the std over the 8 participants:
The r scores change by at most 0.010, indicating that a linear map is sufficient for modelling the brain-deep-network alignment. Pruning with R2D computed from each encoder's R scores, without healing, gives the following accuracy (%), averaged over nine networks and five datasets (changes from the original):
Neither nonlinear network prunes consistently better than ridge.
2. ROI selection. The 15 ROIs come from Raugel et al. [5], where authors selected a representative set among the regions that vision models encode significantly, spanning the cortex from early visual areas (V1, V2, V3) through ventral temporal (V4, FFC) and temporo-parieto-occipital regions to prefrontal cortex. This coverage makes the curvature of R(k) meaningful; as the network deepens, the best-predicted region moves from early visual to prefrontal cortex, while an evolution of R(k) may reflect a representational transition. Replacing these selected ROIs with all well-predicted regions, about half of the cortex, does not change the results significantly as shown in the table above; it slightly helps R2D and slightly hurts R1D. We clarified this motivation in the revised paper.