acceptodds
Under review as a conference paper at ICLR 2027

Entropy as a Lens: Revealing Structural Importance in Masked Self-Supervised Vision Transformers

Abstract

Masked self-supervised Vision Transformers (ViT) such as MAE and VideoMAE have demonstrated strong transferability but often exhibit substantial architectural redundancy. In this work, we ask a simple yet underexplored question: Are all Transformer blocks equally important for representation learning? We present a data-free analysis framework that estimates block importance using the entropy of pretrained weight-number, without relying on downstream data or task-specific sensitivity measurements. Through mathematical derivation and empirical analysis, we show that block-wise weight entropy strongly correlates with performance degradation when blocks are removed, revealing a clear hierarchy of importance across Transformer depth. Leveraging this observation, we demonstrate that a large portion of blocks in masked self-supervised ViT can be removed with minimal impact on downstream performance, suggesting significant structural redundancy in pretrained models. Beyond compression, our results provide new insights into how representation capacity is distributed across Transformer blocks, and position entropy as a lightweight diagnostic tool for understanding and analyzing large pretrained vision models.

Primary area:
interpretability and explainable AI
Keywords:
Data-Free Pruning, Efficient Transfer Learning, Information-Theoretic Analysis, Model Compression, Sensitivity Analysis, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_48ASOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper ranks Transformer blocks by the Shannon entropy of their pretrained weight histograms and removes low-scoring blocks without accessing data. The pruned models are then fine-tuned with labeled data. Experiments cover Kinetics-400-pretrained VideoMAE-B on UCF101: removing one or three blocks yields 91.29% or 85.54% WAR, respectively, versus 91.78% for the original model.

Critical strengths
  1. Practical selection setting: Ranking blocks directly from checkpoint weights avoids calibration data and forward/backward passes; whole-block removal preserves ordinary dense computation.
  2. Useful diagnostic evidence: Ablating all 12 blocks reveals substantial differences in downstream impact. Entropy selection also outperforms the reported elementary baselines at several budgets.
Critical weaknesses
  1. Novelty and theoretical support are insufficient: The score is standard empirical histogram entropy. Parameter-based entropy compression predates this work (KSE, CVPR 2019; Tang et al.); although their estimators and pruning units differ, the conceptual advance needs clearer justification. Appendix A.1, Eqs. (7)–(8), asserts rather than derives the connection to sensitivity. Missing binning details also prevent assessing the claimed scale invariance.
  2. The experiments do not establish an advantage beyond depth or generality: From Tables 2/5, Spearman correlation with sensitivity is 0.755 for entropy and 0.748 for depth; entropy and depth correlate at 0.951 across 12 blocks. Deepest-first deletion is therefore an essential missing baseline. Evidence covers only VideoMAE-B→UCF101, without repeated runs or binning ablations; relevant ViT pruning comparisons such as UP-ViTs are also absent, though their data requirements differ. The nonstandard UCF101 split and WAR definition need clarification. Moreover, removing 25% of blocks loses 6.24 percentage points, weakening the “minimal impact” claim.
Questions for authors
  1. Please specify the binning and normalization used in Table 5 and the assumptions justifying Eqs. (7)–(8).
  2. Does entropy outperform deepest-first deletion and random deletion with block 1 protected, across repeated runs and another checkpoint or dataset?
  3. How was the 10,100-video subset constructed from UCF101’s 13,320 videos, with video groups kept disjoint? Why does WAR differ from test accuracy in Table 4, and which split determined the sensitivity ranking?
Reviewer_XVRpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the problem of block-wise pruning in masked video autoencoders with multilayer transformer architectures. The motivation is to reduce the size and capacity of such architectures after training, to adjust them to reduced memory or compute budgets. This is a relevant problem for leveraging video models on limited infrastructures, not at least mobile and edge devices. The basic idea is to find an efficient proxy measure for performance degradation when removing transformer layers from VideoMAE models, which is easier to compute than common measures requiring degradation analysis with repeated forward passes or expensive finetunings / retrainings, and which does not require access to the training data. The proposed solution consists of evaluating a simple approximate entropy measure on the trained weights, which consists of evaluating numbers of nonzero activiations over a predefined binning of a layer. The results are based on a pretrained ViTB model which is evaluated on action classification with a 70/20/10 % split of the UCF101 dataset. The key observation is that the proposed entropy measures yields comparable ranking of layers to an established oracle sensitiy measure, cited from Molchanov et al. 2016 but rather an even longer established standard ablation scheme.

Critical strengths
  • The paper communicates a simple and intuitive idea for data- and gradient free block pruning, which is fast to compute and approximately matches performance degradation and results of a common loss-driven ablation method (oracle) in a relevant (but single) experiment.
  • On the chosen evaluation case, the proposed method is superior to trivial magnitude statistics
  • Limitations of the method are openly reported
Critical weaknesses
  • Only a single experiment is presented, using a single evaluation dataset (UCF101) and pretrained model. This is insufficient for supporting the key claims, leaving unclear wether the entropy measure works on other architectures and models or wether the reported results are only case-specific. The choice of hyperparameters is not adequately justified and evaluated, particularly the choice of the binning K lacks a clear evaluation.
  • The paper builds on a quite limited, single-sided and rather outdated selection of related work. It misses important recent references, foremost recent work in the LLM domain such as ShortGPT (Men et al. 2024) or the work of He et al. (What Matters in Transformers? 2024); but also work on video models, e.g. Glandorf and Rosenhahn 2025 (Pruning by Block Benefit). The main baseline (Molchanov 2016) is 10 years old. Even more, the Leave-one-out sensitivity analysis used as the key baseline is not a methodology proposed in this paper, but a standard ablation method that the paper used in a similar fashion than the present submission. Thus the core experiment is built on a rather trivial setup
Questions for authors

I have no questions to the authors.

Reviewer_mEtCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes weight number entropy to estimate Transformer block importance using only pretrained parameters. Blocks are ranked by the entropy of their weight histograms, and those with lower scores are removed before downstream finetuning. Experiments with VideoMAE Base on UCF101 show competitive pruning performance against the evaluated simple baselines and approximate agreement with sensitivity based rankings.

Critical strengths
  1. The proposed criterion is simple and inexpensive to compute. Selecting blocks directly from pretrained weights avoids the repeated evaluations required by sensitivity based selection, which could be useful when downstream data is unavailable at pruning time.

  2. The individual block removal experiments demonstrate substantial variation in block importance and provide a direct reference for evaluating the proposed ranking.

Critical weaknesses
  1. The central theoretical justification is unsupported and internally inconsistent. Appendix A.1 replaces differentiation with respect to parameters by differentiation with respect to entropy, then approximates this quantity using histogram entropy without establishing the required relationship. Its corollaries state that higher entropy implies lower sensitivity, contradicting the main text, which associates higher entropy with greater importance. The claimed invariance to weight scaling also depends on an unspecified binning procedure. These issues undermine the paper’s main explanation for why the criterion should work.

  2. The empirical evidence is insufficient to establish general utility or isolate the contribution of entropy. All experiments use VideoMAE Base transferred to UCF101, with no reported variability across repeated runs. The comparisons mainly involve simple weight statistics and random selection. Since the reported entropy ranking largely favors retaining earlier blocks, a baseline that simply removes the deepest blocks is particularly necessary. The manuscript also claims robustness to binning choices without providing the corresponding controlled analysis.

Questions for authors
  1. Can the authors provide a valid derivation connecting histogram entropy to block removal sensitivity and resolve the opposite relationships stated in the main text and Appendix A.1?

  2. What exact binning and normalization procedures are used? Please specify the number of bins, interval boundaries, treatment of different parameter tensors, and whether boundaries are shared across blocks. How do the scores and rankings change under weight scaling and alternative binning choices?

  3. Does entropy outperform simply removing the deepest blocks under identical pruning budgets and finetuning settings? Please report repeated runs and rank correlations with the sensitivity reference, including whether the relationship remains after controlling for block depth.

  4. Can the result be validated on additional pretrained models and downstream datasets, including image MAE models and other video recognition benchmarks? This evidence would materially affect whether the observed relationship can support the broader claims about masked self supervised Vision Transformers.

Reviewer_3hMkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether all Transformer blocks in masked self-supervised Vision Transformers contribute equally to downstream transfer, and whether block importance can be estimated without data. The authors propose block-level weight-number entropy to explain the block importance, where the scalar weights of each block are binned into K value intervals, and the fraction of weights per interval forms an empirical distribution, its Shannon entropy is used as the block's importance score. Based on this score, a one-shot pruning algorithm removes the lowest-entropy blocks and then finetunes the pruned model on the downstream task. The pruning decision requires no data, gradients, or forward passes. The paper reports three folded results: (1) removing block 1 drops accuracy from 91.78% to 46.73%, while removing deep blocks only drops less than 1%; (2) entropy ranking of the blocks is compared with the sensitivity ranking and is reported to align closely; (3) the paper's entropy based block pruning can beat the other data-free criteria and come close to sensitivity-based pruning.

Critical strengths

(1) the data-free scoring proposed by the paper is easy and cheap to implement, and is a reasonable thing to try in settings where downstream data is unavailable at pruning time;

(2) the paper clearly shows the different effect of blocks in the single-block removal study. For VideoMAE-B on UCF101, the first block is critical (−45 points), blocks 3-8 matter moderately, and the last blocks are nearly redundant (<1 point).

Critical weaknesses

(1) using entropy to rank the blocks is nearly unnecessary. The trivial ranking "block index" achieves almost the same correlation with sensitivity. In the multi-block experiments, the entropy selection coincides exactly with "remove the last k blocks", and differs by a single block at the other ratios. So I think that "prune the deepest blocks" baseline should be compared, which is also a well known baseline from the layer-pruning literature (e.g., LayerDrop, Fan et al., 2020; Gromov et al., 2024; Men et al., 2024).

(2) the unstated pruning direction of each criterion makes the baseline comparison not convincing. From Tables 5–12, entropy, mean, L1, L2, variance and std remove the lowest-scoring blocks, whereas kurtosis and mutual information (MI) remove the highest-scoring ones. MI's ranking in Table 5 (MI_N) is almost identical to the entropy ranking. If the authors apply it in the same direction as entropy would select exactly the same blocks as entropy at all six pruning ratios. But the paper use MI to prune it in the reverse way, which makes the result looks bad. The paper justifies entropy by its agreement with the oracle sensitivity ranking (Table 2 and Section 4.3, "entropy-based ranking exhibits the strongest alignment with sensitivity ranking"). However, by the same standard, the MI ranking in Table 5 (MI_N) agrees with the oracle at least as well as entropy does (Spearman ρ ≈ 0.78 vs. 0.76, computed from the ranks in Table 5); this also contradicts the claim in Section 4.3. Moreover, MI is never defined in the paper: it is unclear between which quantities it is computed and how it is obtained without data. The authors should define MI, explain how the pruning direction of each criterion was chosen, and report all baselines in both directions.

(3) the unpruned baseline is substantially below standard numbers. The reported unpruned WAR is 91.78%, whereas VideoMAE reports 96.1% for the same backbone and pretraining on UCF101 (Tong et al., 2022, Table 4). The finetuning recipe uses no augmentation and a non-standard train/val/test split (7,070/2,020/1,010). With an under-tuned finetuning protocol, the measured "oracle" sensitivities may not reflect the behavior under standard training, which weakens all downstream conclusions.

(4) I am worried about the paper's reproducibility. In the paper, the number of bins K, the binning range (per-block or global), which parameters are included (attention, MLP, LayerNorm, biases), and the raw entropy values are not reported; Table 5 shows only min, max, and normalized scores. The claim that the rankings are "robust to reasonable choices of binning strategies" is not supported by any experiment. No code is provided, and there is no reproducibility statement.

Questions for authors

(1) In the reported results, the entropy ranking closely follows the block order. Can the authors show that the entropy criterion carries information about block importance that a simple depth-based heuristic does not?

(2) The choices of entropy variant and pruning direction (removing low vs. high scoring blocks) appear to drive the comparison, but the paper does not explain how these choices were made for the proposed criterion or for the baselines. I wonder how they were determined. Were they fixed independently of the oracle results they are evaluated against?

(3) Please provide details for reproducibility. This includes how the entropy and all baseline metrics are computed, the data splits and training setup, and code if possible.

(4) Suggestions: since the pipeline still requires full downstream finetuning, a discussion of when this approach should be preferred over a smaller pretrained model, distillation, or lightweight data-aware criteria would strengthen the practical motivation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.