acceptodds
Under review as a conference paper at ICLR 2027

Brain Data as a Block-Importance Metric for Depth Pruning

Abstract

Brain-model alignment quantifies how well a deep neural network's internal representations correlate with human brain activity. It remains unclear however whether this alignment actually reflects the functional importance of individual deep network components. In this work, we investigate whether changes in this correlation between brain data and the model's internal representations can be used to infer block importance. Using per-block brain-model alignment scores computed on the Natural Scenes Dataset, we derive two model-agnostic block-importance metrics that capture the first derivative and second derivative of the alignment score, denoted R1D and R2D, respectively. We evaluate R1D and R2D through structured depth pruning by ranking blocks according to these metrics, removing the lowest-scoring blocks, and comparing the resulting performance against conventional pruning baselines. Across multiple models and vision tasks, R2D matches or outperforms several other block-importance metrics for pruning, with the largest margins at moderate compression rates. Notably, after pruning half the blocks of the Swin Small model using our brain-inspired metrics, it retains 61.5% accuracy on ImageNet without any additional finetuning, against 26.9% for the strongest depth pruning baseline. Our work demonstrate that brain-model alignment provides a direct signal that allows for more efficient model design.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-data, compression, pruning, vision, neuroscience, brain

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yQJuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates structured depth pruning on vision models via layer-importance scores derived from brain-model alignment scores computed from the Natural Scenes Dataset. The paper proposes 2 metrics, R1D and R2D, which are functions of brain-encoding scores R which is computed by maximally correlating voxel responses and model activations. Using a greedy layer dropping approach, different vision foundation models are pruned according to these metrics, and compared to other layer pruning baselines. With and without additional fine-tuning, these metrics are shown to perform similarly to strong layer pruning baselines.

Critical strengths

The analysis section in this paper is its largest strength because it 1) verifies that the brain data specfically is contributing something useful rather than just a second-derivative metric, and 2) is honest about why random pruning is useful in the layer drop setting. It is an interesting finding that is useful to those working on layer dropping in the vision model context. Additionally, it was also nice to see the results as a function of individual participants on the natural scenes dataset, which shows that even individual participants responses give similar pruning maps.

Critical weaknesses
  1. The brain-model alignment score is highly hyperparameter specific, and its motivation is not clear. Why is it the case that the blocks representation should linearly predict the neural code? And why is it the case that λ∈Rm\lambda \in \mathbb{R}^m is fitted separately per voxel? By carefully fitting as closely as possible this linear map, this seems to artificially inflate R(k)R(k), which is then used downstream as an importance metric.

  2. The performance difference between the proposed method and baselines, especially after healing, is minimal, with several baselines outperforming the proposed method. It seems like the paper is trying to do two things simultaneously: one is try to determine whether brain model alignment is a functional measurement for layer pruning, and the second is frame this method as outperforming other metrics, which doesn't seem like it should be the point of this more exploratory work. This is because as a functional pruning metric, the proposed method is a bit impractical because it requires brain data on near-distribution image input. Additionally, splitting "best" results across R1D and R2D is not clear; it would be helpful to explain why it is useful to look at both of these metrics in context of performance, and what they tell us about brain-model alignment's role in pruning.

Questions for authors
  1. How were the 15 ROI regions selected? Is it based on prior work that describes each regions role in vision, or something else?
  2. Prior work on LLM layer pruning has focused on pruning contiguous blocks of layers. This work suggests that random layer dropping and the proposed method (non-contiguous) are strong due in part to being non-contiguous. Is this perhaps a function of the domain difference or something else? Did prior papers fail to consider the non-contiguous setting?
4 comments
AuthorsOpenReview ↗

Response to Reviewer yQJu 1/2

Response to Reviewer yQJu

Thank you for your thoughtful review and for highlighting our analysis section, our discussion of random pruning and our per-participant results. Inspired by your feedback, we tested a nonlinear encoder and automatic region selection, measured the stability of the R scores, added standard deviations to every comparison, and now use R2D as our main metric.

1. Linear, per-voxel encoding. Our encoding approach builds on a substantial body of work on neural representations and brain-deep network alignment (Schrimpf et al. [1], Yamins et al. [2], DiCarlo et al. [3], Naselaris et al. [4], Raugel et al. [5], etc.), which uses the standard definition of a representation as linearly readable information and on the voxel-wise encoding framework reviewed by Naselaris et al. [5]. A linear map tests what a block represents explicitly, while a flexible map could recover brain-relevant information from almost any block and blur the differences our metrics rely on. We fit each voxel separately because voxels differ in noise and selectivity, with each penalty set by cross-validation, not by hand. Fitting closely cannot inflate R, since R is evaluated on held-out images and R1D and R2D only use differences between consecutive blocks, where any shared bias cancels.

To test whether a linear map is sufficient, we also computed the r scores with two nonlinear maps:

  • Nonlinear network per participant: an MLP (a 512-unit hidden layer with GELU, then 64 units, then a linear output for every voxel), trained with an MSE loss for each participant.
  • Nonlinear network shared by all participants: the same MLP with its hidden layers shared by all participants and a separate output layer for each.

Both networks are trained on each participant's own images and tested on the 907 images that all participants saw. We report r on the 15 ROIs (used for R1D and R2D in the paper), the whole brain, well-predicted voxels and automatically selected regions, as the change from ridge fit on the same images (ridge's r in parentheses), averaged over DeiT-S and DINOv3-S; ± is the std over the 8 participants:

Encoder15 ROIs (0.184)Whole brain (0.098)Well-predicted voxels (0.144)Automatically selected regions (0.130)
Nonlinear network, per participant-0.005 ± 0.001+0.002 ± 0.0020.000 ± 0.002+0.001 ± 0.002
Nonlinear network, all participants+0.006 ± 0.004+0.009 ± 0.003+0.009 ± 0.004+0.010 ± 0.003

The r scores change by at most 0.010, indicating that a linear map is sufficient for modelling the brain-deep-network alignment. Pruning with R2D computed from each encoder's R scores, without healing, gives the following accuracy (%), averaged over nine networks and five datasets (changes from the original):

R score computation (encoder + region)25% pruned50% pruned75% pruned
Ridge + 15 ROIs (original)62.138.511.3
Nonlinear network per participant + whole brain+0.0-2.1-3.2
Nonlinear network for all participants + 15 ROIs+1.8-4.5-2.1

Neither nonlinear network prunes consistently better than ridge.

2. ROI selection. The 15 ROIs come from Raugel et al. [5], where authors selected a representative set among the regions that vision models encode significantly, spanning the cortex from early visual areas (V1, V2, V3) through ventral temporal (V4, FFC) and temporo-parieto-occipital regions to prefrontal cortex. This coverage makes the curvature of R(k) meaningful; as the network deepens, the best-predicted region moves from early visual to prefrontal cortex, while an evolution of R(k) may reflect a representational transition. Replacing these selected ROIs with all well-predicted regions, about half of the cortex, does not change the results significantly as shown in the table above; it slightly helps R2D and slightly hurts R1D. We clarified this motivation in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer yQJu 2/2

3. Performance, framing and practicality.

Significance. We now repeat the full pruning pipeline 5 times; each repetition samples a different calibration set (for R1D/R2D, a different NSD subset for the encoding models), which can change the pruned blocks, and evaluated on a different fifth of the test set. We also include an additional baseline, first-order Taylor importance, a gradient-based criterion that uses labels. Table 2 of the revised manuscript now reports mean ± standard deviation for all models and methods; below, DeiT and Swin after healing, averaged over the five datasets:

ModelPrunedEntroDropShortGPTAngularP3BRandomTaylorR1DR2D
DeiT-B25%80.0±0.582.6±0.479.9±0.480.8±0.581.2±0.681.7±0.579.3±0.681.6±0.5
50%66.5±0.473.8±0.655.8±0.463.3±0.470.2±1.971.9±1.073.3±1.074.0±0.5
75%46.8±0.446.2±0.345.8±0.445.4±0.344.9±1.447.1±1.445.0±1.550.1±0.5
Swin-B25%83.1±0.484.9±0.483.9±0.484.7±0.383.6±0.482.5±0.784.6±0.484.3±0.4
50%78.8±0.379.4±0.477.4±0.379.9±0.479.7±0.874.7±1.778.9±0.380.9±0.4
75%57.9±0.259.0±0.4–59.0±0.455.3±3.357.8±1.758.5±1.559.0±1.4

Over all 9 models, 5 datasets, 3 pruning ratios and 6 baselines, R2D outperforms the baselines by more than one standard deviation in 77% of comparisons without healing and 58% after healing. We also note that R2D achieves this using only brain responses to natural images, without labels or any target-task data.

Framing. We agree and removed all state-of-the-art claims. The framing focuses now on asking whether brain-model alignment is a useful functional signal for pruning, and our results show that it is.

Near-distribution data. Our metrics do not need brain data on near-distribution images. R1D and R2D are computed once from the public NSD dataset, without labels or target images, and reused for every dataset and pruning ratio. Their advantage holds far from NSD too since without healing, R2D outperforms the baselines in 79% of comparisons on COCO, the source of NSD images, and in 77% on the other 4 datasets.

R1D and R2D. We now report R1D as a first-order ablation. R2D outperforms R1D by more than one standard error in 77% of comparisons without healing and 46% after healing. What matters for pruning in several cases is not how much a block changes brain alignment but whether it marks a transition between processing stages.

4. Contiguous pruning. Prior LLM work considered both settings; ShortGPT [6] and SLEB [7] remove non-contiguous layers, while angular distance [8] removes the best contiguous block. Contiguous removal may be popular because healing then repairs a single interface, but in our vision models it is especially harmful and healing does not compensate. On DINOv3-7B, making the removed blocks adjacent multiplies the output divergence by 10 to 134 times. At 50% pruning, every contiguous run of 20 blocks keeps at most 17.6% accuracy without healing while random sets keep 87.6%, and at 75% contiguous runs recover only to 63% after healing while random sets reach 85 to 88%. We believe this reflects how redundancy is spread across depth in these models rather than the domain, since neighbouring blocks compensate for an isolated removal but not for a whole run. We added this discussion to the revision.

Thank you again for your helpful feedback. We made a considerable effort to address each of your points with new ablations, stability and significance analyses, and a clearer framing of our contribution. We believe these additions strengthen our paper, and we would be grateful if you would reconsider your assessment in light of these revisions.


References:

[1] Schrimpf, Martin, et al. "Brain-score: Which artificial neural network for object recognition is most brain-like?." BioRxiv (2018): 407007.

[2] Yamins, Daniel LK, et al. "Performance-optimized hierarchical models predict neural responses in higher visual cortex." Proceedings of the national academy of sciences 111.23 (2014): 8619-8624.

[3] DiCarlo, James J., Davide Zoccolan, and Nicole C. Rust. "How does the brain solve visual object recognition?." Neuron 73.3 (2012): 415-434.

[4] Naselaris, Thomas, et al. "Encoding and decoding in fMRI." Neuroimage 56.2 (2011): 400-410.

[5] Raugel, Joséphine, et al. "Disentangling the factors of convergence between brains and computer vision models." arXiv preprint arXiv:2508.18226 (2025).

[6] Men, Xin, et al. "Shortgpt: Layers in large language models are more redundant than you expect." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[7] Song, Jiwon, et al. "Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks." arXiv preprint arXiv:2402.09025 (2024).

[8] Gromov, Andrey, et al. "The unreasonable ineffectiveness of the deeper layers." International Conference on Learning Representations. Vol. 2025. 2025.

Thank you for the author rebuttal.

Thanks for the updated clarification on weakness 1, it would be valuable to improve the clarity of this setup in the main text in order to justify the design choices as a product of prior work, and also to properly attribute credit. And thank you for the answer to the ROI selection question and clarifying this with respect to prior work.

I appreciate that the authors attempt a larger overhaul of the paper framing to soften many of the claims regarding state-of-the-art and improve the framing. I do not think this is a level of revision that is doable well within a rebuttal period and also very difficult for individual reviewers to readdreass. To that end, W2 still remains and does not seem well resolvable during the rebuttal period.

Regarding other aspects of W2, like near-distribution, I also do not find this respose to address my concern. Near-distribution or in-distribution does not require that labels are present. The concern instead stems from the fact that the metrics did indeed get computed from NSD (I understand without labels), and then tested only on data very similar to NSD. The concern is that if a different dataset was selected for evalution, R{1,2}D would not generalize well. It is not the true that this needs to be the case, especially in a paper that explores brain data as an importance metric rather than proposes a SOTA Method, but an honest discussion of this limitation seems necessary. I apologize if the original concern was not written clearly.

I appreciate the reframing of R1D as an ablation, but this again, like the above point, is a significant overhal of the presentation and content of the paper.

Finally, re contiguousness. I appreciate these additional numbers regarding contiguousness and randomness. It seems that contiguousness is very unfavored in this setting, perhaps related to model differences in the current work and other papers more related to LLMs. This question was mostly out of curiosity and is well answered. Thank you! This is an interesting hypothesis about depth redundancy.

I want to acknowledge that the authors put in a significant effort during the rebuttal period. I unfortunately still do feel that this amount of revision including in key claims and framing of this paper makes it very difficult to increase the score now that the paper is quite different in spirit.

Thank you for engaging with our rebuttal and for acknowledging the effort we put into it. We are glad that our responses clarified the encoding setup, the ROI selection and the question on contiguous pruning. We would also like to gently clarify that the revisions to the framing and to R1D are much narrower than they may appear, and that all of them are already in the revised manuscript in blue, so they can be checked directly.

Framing. The original submission already framed our work as an investigation of whether brain-model alignment is a useful signal for pruning; the introduction states that "we investigate whether brain-model alignment captures the functional importance of individual blocks in vision models", and Section 2 poses our central question as whether brain-model alignment can guide the identification of the most functionally important blocks. The only state-of-the-art claim in the submission was one phrase in the conclusion, which we removed, and we added two sentences to the introduction to make the existing framing explicit. The abstract, the contributions, the method and all of the original experiments are unchanged, and the new results only add the uncertainty estimates and baselines that the reviewers requested.

R1D as an ablation. Similarly, the submission already presented R2D as our main metric; our contributions state that R2D is competitive with or outperforms the baselines across all pruning strengths and isolate the role of the brain data for R2D, while R1D "also shows considerable performance improvements". We kept every R1D result and only added one sentence to Section 3.3 that describes R1D as a first-order ablation, so this clarifies the existing presentation rather than changing its content.

Encoding setup and credit. Following your suggestion, Section 3.1 of the revised manuscript now motivates the linear, voxel-wise encoding as a product of prior work and credits Schrimpf et al., Yamins et al., DiCarlo et al., Naselaris et al. and Raugel et al., whose setup we follow.

Generalization beyond NSD. Thank you for clarifying this concern. Our transfer datasets already include images that differ substantially from the natural scenes of NSD, such as the textures of DTD and the low-resolution images of CIFAR, and R2D wins more comparisons against the baselines than it loses on every dataset:

Comparisons where R2D outperforms the baselinesCOCOImageNetCIFAR-10CIFAR-100DTD
No healing79%86%73%79%69%
Healing58%73%48%56%55%

However, the advantage is smaller on the datasets furthest from NSD, and we agree that an honest discussion of this limitation is necessary. We therefore added the following to the conclusion of the revised manuscript: "R1D and R2D are computed once from brain responses to the natural scenes of NSD. Although their advantage holds on ImageNet, CIFAR and DTD, it is smaller on the datasets furthest from NSD, and we did not evaluate domains that differ more strongly from natural images, such as medical or satellite images, which we leave for future work."

We hope these clarifications show that the revised paper keeps the same spirit, method and experiments as the submission, and that our revisions only address the points raised during the review. We are grateful for your engagement throughout the discussion, and we would appreciate it if you could take these clarifications into account in your final assessment.

Reviewer_5e7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether changes in the correlation between brain data and a model's internal representations can be used to evaluate block importance. It proposes two brain-inspired block-importance metrics based on the first- and second-order derivatives of brain-model alignment scores. Experiments on different benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper is generally well-written.
  • The experimental results show the effectiveness.
Critical weaknesses
  • Insufficient motivation and explanation for method design.
  • Lack of network efficiency evaluation.
Questions for authors
  • The motivation behind the proposed brain-inspired derivative-based metrics needs further clarification. Although the authors provide empirical evidence of their effectiveness, it remains unclear why the first- and second-order derivatives of brain-model alignment scores can reliably evaluate the importance of individual blocks.
  • COCO is primarily designed for object detection and instance segmentation tasks. However, the authors use COCO for image classification and report Top-1 accuracy. It is unclear how the original multi-label annotations are converted into classification labels.
  • The proposed metrics employ an iterative greedy pruning strategy, whereas most baseline methods use static block selection. Such differences in pruning strategies may affect the fairness of the experimental comparisons. Moreover, the iterative strategy introduces additional computational overhead compared with static selection. Therefore, the authors should provide comparisons under a unified pruning strategy and analyze the computational cost of the proposed method.
  • The paper lacks a comprehensive evaluation of network efficiency.
  • The experimental results in Figures 5 and 6 are difficult to follow. The authors should improve the presentation of these figures to make the numerical comparisons clearer and more intuitive.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5e7x

Thank you for your detailed review and for noting that our paper is well written and that our results show the effectiveness of our method. Inspired by your feedback, we expanded the motivation for R1D and R2D with evidence from neuroscience and pruning, compared all methods under unified pruning strategies and analyzed their cost, added an efficiency evaluation for all 9 models, and described our COCO task in detail.

1. Motivation for the derivative-based metrics. Thank you for this helpful suggestion. We agree that the motivation deserved a more explicit treatment, and we have now expanded Section 3.3 around the following two well-established observations:

Neuroscience perspective: We know from neuroscience literature that the ventral visual stream is organized as a hierarchy, from V1 to V4 to IT, that encodes increasingly complex features. It was also shown by existing works that vision models reproduce this progression, with intermediate layers best predicting V4 and top layers best predicting IT [1, 2]. The alignment curve R(k) therefore traces how a network moves through these processing stages.

Pruning perspective: Early vision works have observed that residual blocks are organized in a similar manner where blocks within a stage refine the same representation, and new representations arise at the transitions between stages [3]. Deleting a single block barely affects accuracy except at these transitions points between representational stages [4], and layers with near-identical representations can be pruned with little loss [5].

Intuition that connect the two perspectives: One can think of R(k) as a route through the visual hierarchy of the brain. From this point of view, R1D measures how far each block moves the representation along this route, while R2D measures whether the block is situated at a transition between two representational regimes. Therefore, removing a block on a straight stretch may have less impact, while removing a block that is situated at a representational transition might have a bigger effect. So R2D finds redundant blocks within a stage from an external, brain-derived signal, and our analysis confirms that the brain data contributes beyond the derivative itself as shown in Section 5.

2. COCO labels. We chose COCO because the NSD images are drawn from it, so it is closest to what the participants saw; that is the reason why we call it “in-distribution in the paper”. Our logic for starting from COCO was that if the brain signal helps pruning at all, COCO gives it the best chance due to being “in-distribution” for the brain signal, and comparing COCO with the other 4 datasets shows how well this signal generalizes “out-of-distribution”. To build the classification task, we first remove all 73k NSD images from COCO, which leaves ~47k training and ~2k test images. We then label each image with the COCO supercategory of its largest object, which gives 12 classes. We added this description to the paper.

3. Unified pruning strategy and cost. First, we would like to note that greedy step adds no meaningful cost for R1D and R2D since it only re-differences a fixed curve, taking less than 7 ms on a CPU with no forward pass, while iterative baselines need one forward pass per removed block. Regardless of this cost difference, we reran every method under both strategies, adding static R1D and R2D and iterative ShortGPT and Angular distance. Greedy and static R2D perform the same after healing, with a median difference of 0.0 points, so our results appear largely robust to the pruning strategy.. R2D also stays ahead of the iterative version of the baselines in the exact same settings showing that the iterative version does not consistently benefit the baselines either; boosting their average accuracy by 1.2% with a standard deviation of 3.5% meaning it worsens their performance in some cases.

4. Network efficiency. We added an efficiency evaluation measuring parameters, FLOPs and throughput on an H100 GPU in bf16 to the paper. The measured speedups are summarized below:

Family25%50%75%
DeiT-T/S/B1.27-1.29x1.72-1.82x2.69-3.11x
Swin-S/B/L1.17-1.69x1.48-2.50x1.98-4.59x
DINOv3-S/L/7B1.30-1.32x1.87-1.97x3.34-3.88x

Since all blocks in DeiT and DINOv3 have the same size, every method gives the same speedup at a given ratio. Swin stages differ in cost, so we report it for each method.

5. Figures 5 and 6. We improved their presentation in the revised paper.

Thank you again for your constructive feedback. We made a considerable effort to address each of your points and believe these additions strengthen our paper. We would be grateful if you would reconsider your assessment in light of these revisions.


References:

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC4060707/

[2] https://arxiv.org/abs/1411.6422

[3] https://arxiv.org/abs/1612.07771

[4] https://arxiv.org/abs/1605.06431

[5] https://arxiv.org/abs/2010.15327

AuthorsOpenReview ↗

Follow-up before the discussion period closes

Dear Reviewer 5e7x,

Thank you again for your detailed feedback. As the discussion period closes soon, we wanted to follow up and ask whether you have any additional questions or remaining concerns regarding our response, in particular regarding the two main weaknesses you raised.

Motivation. We expanded Section 3.3 with intuition derived from prior works in neuroscience and pruning. The ventral visual stream is organized as a hierarchy of processing stages that vision models reproduce, and residual networks are organized in a similar way, refining the same representation within a stage while new representations arise at the transitions between stages. R2D therefore identifies redundant blocks within a stage from an external, brain-derived signal, and our analysis in Section 5 confirms that the brain data contributes beyond the derivative itself.

Network efficiency. We added an efficiency evaluation for all 9 models to the appendix; removing half of the blocks speeds up inference by 1.72 to 1.97 times for DeiT and DINOv3 and by 1.48 to 2.50 times for Swin, and since LoRA adapters are merged after healing, healed models are as fast as unhealed ones.

In addition, we reran every method under both greedy and static selection; greedy and static R2D perform the same after healing, with a median difference of 0.0%, while the greedy step takes less than 7 ms on a CPU without any forward pass. We also described how we build the COCO classification task in the appendix and improved Figures 5 and 6.

We hope these additions address your concerns and that you would consider raising your score. We understand that this is a busy period and greatly appreciate your time, and we remain available to clarify any remaining point before the discussion closes.

Reviewer_ULieOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether brain-model alignment can serve as a block-importance metric for structured depth pruning. The authors fit ridge-regression encoding models that map transformer block representations to fMRI responses from the Natural Scenes Dataset (NSD), obtaining per-block Pearson correlation scores across 15 cortical regions of interest. They introduce two importance metrics based on the absolute finite-difference first (R1D) and second (R2D) derivatives of these alignment scores, averaged across regions, and use them to guide iterative block removal.

The proposed metrics are compared against ShortGPT, angular distance, EntroDrop, P3B, and random pruning across three vision transformer families (DeiT, Swin, and DINOv3), with three model sizes each. Experiments cover COCO and four additional datasets (ImageNet, CIFAR-10, CIFAR-100, and DTD), at 25%, 50%, and 75% depth pruning, both before and after LoRA-based post-pruning adaptation.

The authors report that their brain-derived metrics match or outperform conventional baselines in many settings, particularly at moderate compression rates. They also demonstrate that importance rankings computed using NSD brain responses can transfer to other datasets without using their data to compute the rankings.

Critical strengths
  • Novel and well-motivated research question: The paper explores whether brain-model alignment can provide a useful signal for model compression, extending its traditional role as a descriptive measure of representational similarity. Comparing brain-derived importance metrics against established depth-pruning methods provides an interesting connection between neuroscience and model efficiency.

  • Broad experimental evaluation: The proposed metrics are evaluated across three vision transformer families (DeiT, Swin, and DINOv3), three model sizes per family, five datasets, and three pruning ratios (25%, 50%, and 75%). The evaluation also considers performance both before and after post-pruning adaptation, strengthening the empirical assessment.

  • Transferability across datasets: The brain-derived importance scores are computed using NSD data and transferred to ImageNet, CIFAR-10, CIFAR-100, and DTD without requiring target-dataset examples for ranking.

  • Analysis of competing baselines: The paper acknowledges settings where random pruning is competitive and investigates the relationship between pruning patterns and model performance, rather than focusing exclusively on favorable results.

Critical weaknesses
  • Mixed results and limited statistical evidence: Although the proposed methods perform well in several settings, the improvements are not consistent across architectures and pruning strengths. For instance, random pruning outperforms R2D on DINOv3-Small at 25% and 50% pruning, and slightly outperforms it on DINOv3-7B at 50% and 75% (Table 2). Several reported improvements are within one percentage point, but uncertainty estimates are not provided for the main comparisons. The claim of state-of-the-art performance therefore appears stronger than the evidence supports.

  • Limited comparisons and transfer controls: Most baselines originate from language-model pruning or related activation-based methods. Including additional vision-specific or gradient-based block-importance metrics would strengthen the evaluation. Moreover, the transfer experiments compare fixed brain-derived rankings against conventional rankings recomputed for each target dataset. While this is a favorable setting for the conventional baselines, evaluating their rankings under the same fixed-source transfer protocol would better establish whether the proposed metrics offer a distinctive advantage in transferability.

  • There are several typos in the manuscript. L026 (abstract): “Our work demonstrate” → “demonstrates”, L130 (Sec. 2): “our brain-inspired metrics due indeed outperform” → “do indeed”, L356 (Sec. 4.2): “inexistent decline phases” → “nonexistent” or "inexisting".

Questions for authors

See Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ULie

Thank you for acknowledging the novelty of our research question and our broad experimental evaluation. We appreciate your helpful feedback and respond to your questions below.

  1. Variability across repetitions and central contribution. Thank you for your suggestion to include uncertainty estimates for the results. We repeated the full pipeline 5 times; each repetition samples a different calibration set (for R1D/R2D, a different NSD subset for the encoding models), which can change the pruned blocks, and evaluated on a different fifth of the test set (random pruning: 10 seeds). Following your suggestion, we also include an additional baseline, first-order Taylor importance [1], a gradient-based criterion that uses labels (we expand on it in point 3). Table 2 of the revised manuscript now reports mean ± standard deviation for all models and methods; below, DeiT and Swin after healing, averaged over the five datasets:
ModelPrunedEntroDropShortGPTAngularP3BRandomTaylorR1DR2D
DeiT-B25%80.0±0.582.6±0.479.9±0.480.8±0.581.2±0.681.7±0.579.3±0.681.6±0.5
50%66.5±0.473.8±0.655.8±0.463.3±0.470.2±1.971.9±1.073.3±1.074.0±0.5
75%46.8±0.446.2±0.345.8±0.445.4±0.344.9±1.447.1±1.445.0±1.550.1±0.5
Swin-B25%83.1±0.484.9±0.483.9±0.484.7±0.383.6±0.482.5±0.784.6±0.484.3±0.4
50%78.8±0.379.4±0.477.4±0.379.9±0.479.7±0.874.7±1.778.9±0.380.9±0.4
75%57.9±0.259.0±0.4–59.0±0.455.3±3.357.8±1.758.5±1.559.0±1.4

Over all 9 models, 5 datasets, 3 pruning ratios and 6 baselines, R2D outperforms the baselines by more than one standard deviation in 77% of comparisons without healing and 58% after healing. We also note that R2D achieves this using only brain responses to natural images, without labels or any target-task data. We agree this does not support a state-of-the-art claim and we removed it; rather, it shows that brain-model alignment carries a block-importance signal that activation-based metrics miss in several settings, which we frame as our central contribution.

  1. Random pruning. Random pruning is competitive on DINOv3-7B because removing neighbouring blocks compounds the damage from pruning (at 50% pruning, every contiguous 20-block run keeps ≤17.6% accuracy unhealed vs. 87.6% for random sets), blocks are highly redundant (layer scale grows ~400× with depth), and healing favours sparse pruning. This may reflect how overparameterized a 7B model is for vision tasks and helps explain why random pruning is competitive, rather than indicating a limitation of our metrics.

  2. Vision-specific and gradient-based metrics. Inspired by your feedback, we added first-order Taylor importance [1], a gradient-based criterion proposed for pruning vision models. For each block, it estimates the change in classification loss caused by removing the block from the gradient of the loss with respect to the block's output, using labels and the classification head on the same calibration sets (Taylor column above and in Table 2 of the revised manuscript). Although R1D/R2D use neither labels nor gradients, our brain metrics beats Taylor in 10 of the 12 settings above. We also note that P3B [2], a baseline already included, is a recent ViT-specific block metric reported as state of the art. Our other baselines, though introduced for LLMs initially, have also been applied to ViTs: ShortGPT-style cosine similarity to ViT, DINOv2/v3 and SwinV2 [3], angular distance to the CLIP vision encoder [4], and entropy-guided layer removal to ViTs [5].

  3. Using calibration data for the brain metrics. Since R1D/R2D use only NSD data by design, we also calibrated the baselines on the NSD images. At 50% pruning after healing, the average accuracy drops by 3.7%, 1.2% and 4.8% for EntroDrop, ShortGPT and Taylor, while for Angular it barely moves by +0.4% and P3B seems to benefit from it by +2.0%. So overall, a setting that is fairer and does not favor the baselines leads to the brain metrics outperforming them in 83.3% of the settings. We added this ablation to the manuscript.

  4. Typos. Thank you for pointing them out; we fixed them in the revised manuscript.

Thank you again for your constructive feedback. We made a considerable effort to address each of your points with uncertainty estimates for every comparison, a new gradient-based baseline, a matched-source ablation and an analysis of random pruning. We believe these additions strengthen our paper, and we would be grateful if you would reconsider your assessment in light of these revisions.


References:

[1] https://arxiv.org/abs/1906.10771

[2] https://arxiv.org/abs/2506.23675

[3] https://arxiv.org/abs/2610.15005

[4] https://arxiv.org/abs/2505.02819

[5] https://arxiv.org/abs/2404.05657

AuthorsOpenReview ↗

Follow-up before the discussion period closes

Dear Reviewer ULie,

Thank you again for your helpful feedback and for your supportive review. As the discussion period closes soon, we wanted to follow up and ask whether you have any additional questions or remaining concerns regarding our response.

In response to your feedback, we repeated the full pruning pipeline 5 times with different calibration sets and test splits and now report uncertainty estimates for every model and method; across all 9 models, 5 datasets, 3 pruning ratios and 6 baselines, R2D outperforms the baselines in 77% of comparisons without healing and 58% after healing. We also added first-order Taylor importance as a gradient-based baseline, and although R1D and R2D use neither labels nor gradients, our brain metrics beat Taylor in 10 of the 12 DeiT and Swin settings. Moreover, when we calibrate all baselines on the same NSD images, R1D and R2D outperform them in 83.3% of the settings. Finally, we analyzed why random pruning is competitive on DINOv3-7B, removed all state-of-the-art claims and clarified our central contribution, and we fixed the typos you pointed out.

We included all of these results in the revised manuscript, and we hope they address your questions and that you would consider raising your score. We understand that this is a busy period and greatly appreciate your time, and we remain available to clarify any remaining point.

I thank the authors for the considerable effort put into the rebuttal and the revised manuscript. The additional experiments (uncertainty estimates, the gradient-based baseline, and the matched-source ablation) and the more measured framing of the claims address my main concerns, and the paper is clearly stronger as a result. I also value the originality of the idea: using brain-model alignment as a prescriptive signal for pruning is an interesting direction. Considering the new results and the adjustments to the paper, I am raising my score to 4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up as the discussion period nears its end and summary of updates and new results

We thank all reviewers again for their helpful feedback. As the discussion period nears its end, we wanted to follow up on our responses and summarize the updates we made to the paper. We are glad that the reviewers found our research question novel and our evaluation broad (ULie), the paper well written and our results effective (5e7x), and our analysis section, discussion of random pruning and per-participant results valuable (yQJu). Inspired by the reviewers' feedback, we ran additional experiments and analyses and summarize the main findings below:

Statistical significance and framing. Following Reviewers ULie's and yQJu's feedback, we repeated the full pipeline 5 times, each time with a different calibration set and a different fifth of the test set, and Table 2 now reports uncertainty estimates for all models and methods. Across all 9 models, 5 datasets, 3 pruning ratios and 6 baselines, R2D outperforms the baselines in 77% of comparisons without healing and 58% after healing, using only brain responses to natural images and no labels or target-task data. We now frame our central contribution as showing that brain-model alignment carries a block-importance signal that activation-based metrics miss in several settings.

A new gradient-based baseline and a matched-source comparison. Following Reviewer ULie's suggestion, we added first-order Taylor importance, a gradient-based criterion proposed for pruning vision models that uses labels. Although R1D and R2D use neither labels nor gradients, our brain metrics beats Taylor in 10 of the 12 DeiT and Swin settings. We also calibrated the baselines on the NSD images so that all metrics use the same source data, and in this fairer setting, brain metrics outperform the baselines in 83.3% of the settings.

Motivation for R1D and R2D. Following Reviewer 5e7x's feedback, we expanded Section 3.3 with intuition from the neuroscience and pruning literature. The ventral visual stream is organized as a hierarchy of processing stages that vision models reproduce, and residual networks are organized in a similar way, refining the same representation within a stage while new representations arise at the transitions between stages. R2D therefore identifies redundant blocks within a representational stage from an external, brain-derived signal.

Encoding model and ROI selection. Following Reviewer yQJu's suggestion, we computed the R scores with two nonlinear networks, one per participant and one shared by all participants. The R scores change by at most 0.010 and neither network prunes consistently better than ridge regression, which indicates that a linear map is sufficient. Moreover, replacing our 15 ROIs with all well-predicted regions, about half of the cortex, does not change the results significantly, and we now report R1D as a first-order ablation of R2D.

Pruning strategy, cost and efficiency. Following Reviewer 5e7x's suggestion, we reran every method under both greedy and static selection. Greedy and static R2D perform the same after healing, with a median difference of 0.0%, and the greedy step takes less than 7 ms on a CPU without any forward pass. We also added an efficiency evaluation for all 9 models; removing half of the blocks speeds up inference by 1.72 to 1.97 times for DeiT and DINOv3 and by 1.48 to 2.50 times for Swin.

Random and contiguous pruning. Following Reviewers ULie's and yQJu's feedback, we analyzed why random pruning is competitive on DINOv3-7B. At 50% pruning, every contiguous set of 20 blocks keeps at most 17.6% accuracy without healing, while random sets keep 87.6%, which reflects how redundancy is spread across depth in this overparameterized model.

We included all of these results in the revised paper (additions in blue), with a detailed description of our COCO task and an improved presentation of the figures. Please let us know whether our responses address your questions, and we remain available to clarify any remaining point before the discussion closes. If they do, we would appreciate it if you could revise your assessment in light of these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.