- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper ranks Transformer blocks by the Shannon entropy of their pretrained weight histograms and removes low-scoring blocks without accessing data. The pruned models are then fine-tuned with labeled data. Experiments cover Kinetics-400-pretrained VideoMAE-B on UCF101: removing one or three blocks yields 91.29% or 85.54% WAR, respectively, versus 91.78% for the original model.
Critical strengths
- Practical selection setting: Ranking blocks directly from checkpoint weights avoids calibration data and forward/backward passes; whole-block removal preserves ordinary dense computation.
- Useful diagnostic evidence: Ablating all 12 blocks reveals substantial differences in downstream impact. Entropy selection also outperforms the reported elementary baselines at several budgets.
Critical weaknesses
- Novelty and theoretical support are insufficient: The score is standard empirical histogram entropy. Parameter-based entropy compression predates this work (KSE, CVPR 2019; Tang et al.); although their estimators and pruning units differ, the conceptual advance needs clearer justification. Appendix A.1, Eqs. (7)–(8), asserts rather than derives the connection to sensitivity. Missing binning details also prevent assessing the claimed scale invariance.
- The experiments do not establish an advantage beyond depth or generality: From Tables 2/5, Spearman correlation with sensitivity is 0.755 for entropy and 0.748 for depth; entropy and depth correlate at 0.951 across 12 blocks. Deepest-first deletion is therefore an essential missing baseline. Evidence covers only VideoMAE-B→UCF101, without repeated runs or binning ablations; relevant ViT pruning comparisons such as UP-ViTs are also absent, though their data requirements differ. The nonstandard UCF101 split and WAR definition need clarification. Moreover, removing 25% of blocks loses 6.24 percentage points, weakening the “minimal impact” claim.
Questions for authors
- Please specify the binning and normalization used in Table 5 and the assumptions justifying Eqs. (7)–(8).
- Does entropy outperform deepest-first deletion and random deletion with block 1 protected, across repeated runs and another checkpoint or dataset?
- How was the 10,100-video subset constructed from UCF101’s 13,320 videos, with video groups kept disjoint? Why does WAR differ from test accuracy in Table 4, and which split determined the sensitivity ranking?