- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DeltA, a token-pruning criterion that multiplies Mamba’s discretization signal by the post-block representation norm. It evaluates the criterion across different vision tasks and backbones.
Critical strengths
The criterion is simple, uses existing backbone signals, and requires no additional learnable selector. Evaluation across multiple tasks and backbones provides initial evidence of its applicability.
Critical weaknesses
-
Incremental contribution with insufficient support for the combination. Table 5 shows that activation-only scoring outperforms DeltA on ViM-Tiny (75.6% vs. 75.3%), while delta-only scoring outperforms it on ViM-Small (80.0% vs. 79.4%). The necessity and claimed robustness of the combined score therefore remain insufficiently established.
-
Modest gains with unclear attribution. Improvements over ToP-ViM are only 0.2/0.6 percentage points. Since DeltA uses feature distillation and the comparison does not establish matched training and pruning settings, these gains cannot be clearly attributed to the new score. Variability across runs is also unreported.
Questions for authors
-
Were all variants in Table 5 evaluated under identical pruning, training, and distillation settings? Since the best individual signal outperforms DeltA on each backbone, what evidence supports the claimed robustness of the combination?
-
Were ToP-ViM and DeltA compared under matched settings, including distillation? Could the authors report variability across runs to establish whether the small gains are stable and attributable to the new score?
-
Why use multiplication rather than a normalized sum or weighted sum? Is there evidence that this particular combination improves token selection?
4 comments
Questions for the author
Q1. Metric component ablation
Yes. Within each backbone, all variants in Table 5 use the same pretrained checkpoint, learning rate, training schedule, pruning locations, retention rates, and distillation settings; only the importance criterion changes. The revised ViM-Tiny experiments use the updated initial learning rate consistently across all three criteria. Our claim of robustness refers to maintaining competitive performance across backbones and training settings without selecting a different importance signal for each architecture. Activation-only scoring performs best on ViM-Tiny, whereas -only scoring performs best on ViM-Small. DeltA remains within 0.2 and 0.6 percentage points of the strongest component, respectively. We have clarified that this supports a unified criterion rather than superiority over the best individual component in every setting. We additionally repeated the component ablation without knowledge distillation, keeping all other settings unchanged.
| Backbone | Training | only | only | DeltA |
|---|---|---|---|---|
| ViM-Tiny | With KD | 75.3 | 75.8 | 75.6 |
| ViM-Tiny | Without KD | 75.2 | 75.5 | 75.5 |
| ViM-Small | With KD | 80.0 | 79.1 | 79.4 |
| ViM-Small | Without KD | 79.5 | 77.3 | 79.1 |
The preferred individual signal still changes with architecture. On ViM-Tiny, DeltA matches activation-only scoring at 75.5%; on ViM-Small, it achieves 79.1%, compared with 79.5% for -only and 77.3% for activation-only scoring. Removing distillation reduces DeltA’s accuracy by only 0.1/0.3 percentage points across the two backbones, whereas activation-only scoring loses 1.8 points on ViM-Small. These results strengthen the evidence that DeltA provides consistently competitive selection across the evaluated settings without requiring architecture-specific choice of the importance signal.
Q2. Fairness of Comparisons, Knowledge Distillation, and Experimental Variability
The difference in the settings between ToP-ViM and DeltA training procedure comes down to a different choice of learning rate and scheduler which was separately chosen for our method and the addition of knowledge distillation in our experiment which was not mentioned in the ToP-ViM paper, while the rest of the setup, such as the FLOPs reduction and the amount of retraining was kept the same. Thank you for pointing this out, since the knowledge distillation in the training procedure could attribute to the part of the gain. As per your comment and another reviewer's suggestion we have added an additional ablation analyzing the change in the ImageNet results depending on whether knowledge distillation was used or not. The starting learning rate for ViM-Tiny architecture was changed from 5e-4 to 1e-4 to suit the architecture better, which has lead to an improvement in both cases where distillation was used, and in the case where it was omitted from the retraining. The value has also been changed in the main results section.
Regarding run-to-run variability, all variants for a given backbone start from the same pretrained checkpoint and use a batch size of 512. This removes variation from backbone initialization, while the large batch reduces noise from minibatch sampling. We therefore expect relatively limited variability compared with training from scratch. Given the computational cost and rebuttal timeframe, we prioritized controlled component and distillation ablations to investigate the sources of the observed performance differences. For a faster comparison in the limited time we have completed the run on ViM-Tiny on the CelebA dataset on three different seeds with the results presented in the following table:
| Run | Avg. accuracy (%) | Precision (%) | Recall (%) | F1 (%) |
|---|---|---|---|---|
| 1 | 91.0 | 76.6 | 65.8 | 69.7 |
| 2 | 91.1 | 77.0 | 65.3 | 69.4 |
| 3 | 91.1 | 77.1 | 65.2 | 69.5 |
| Mean | 91.1 | 76.9 | 65.4 | 69.5 |
| Std. | 0.0 | 0.3 | 0.3 | 0.1 |
The results show limited variation: accuracy ranges from 91.03% to 91.10%, and the sample standard deviations of precision, recall, and F1 are 0.3, 0.3, and 0.1 percentage points, respectively. These additional experiments provide empirical evidence of consistent performance across seeds in the evaluated CelebA setting. As for the ImageNet experiments, we plan to include them in the final version of the paper, should the paper be accepted.
Q3 Motivation for Multiplication over Additive or Weighted Fusion
We chose multiplication to prioritize tokens that exhibit both a strong discretization signal and a large output representation. Unlike an additive score, the product suppresses tokens when either factor is small, reflecting our hypothesis that both quantities should inform retention. It also preserves token rankings under uniform positive rescaling of either component within a pruning layer and requires no mixing coefficient. A normalized or weighted sum is a reasonable alternative, but entails a normalization or weighting choice. These properties motivate the design rather than establish its empirical optimality. Table 5 (6 in the new version of the paper) compares the product with its individual components, but does not directly compare alternative fusion rules. It shows that the product avoids the weaker component’s performance on each tested backbone, although it does not outperform the stronger component in either case.
Summary of Our Response
We thank the reviewer again for the careful evaluation and constructive suggestions. We have clarified the interpretation of the combined importance criterion and the role of its individual components, and added new ablation experiments without knowledge distillation to Section 5.3. We have also quantified the contribution of knowledge distillation in Section 5.2, updated the ViM-Tiny training configuration and corresponding results, and clarified the limitations of comparisons based on published baselines and single-run evaluations. These additional analyses demonstrate that DeltA maintains competitive performance across the evaluated backbone scales and training settings, while providing a more precise understanding of the contribution and limitations of the proposed criterion. We hope these clarifications and results address the reviewer's concerns, and we would appreciate their reassessment of the paper in light of the rebuttal and revised manuscript.
Thank you
Thank you for the transparent clarifications and additional ablations. While I appreciate the honest reporting, my core concerns remain unresolved, and I will maintain my current score.
Specifically, the updated results show that multiplying the two signals still yields no synergistic gain over the best individual signal on either backbone, and no empirical comparison against alternative fusion rules (e.g., normalized addition) is provided to justify the product formulation. Moreover, once distillation is removed, the modest 0.3–0.4% margins over the reported ToP-ViM baselines remain too small to convincingly establish the advantage of the proposed score.
We thank the reviewer for the careful and constructive evaluation of our work. We appreciate their recognition of the simplicity of the proposed pruning criterion, particularly its use of existing backbone signals without requiring additional learnable scoring modules. We also value their positive assessment of the paper's clarity and presentation, as well as their acknowledgment of the method's applicability across different vision architectures and tasks. We thank the reviewer for highlighting aspects of the combined importance criterion and experimental comparisons that require further justification. Below, we address each concern and question individually.
W1. Contribution of Individual Importance Signals and Robustness of Their Combination
We thank the reviewer for the questions. Yes, all variants in Table 5 were evaluated under identical pruning, training, and distillation settings, with only the token-importance criterion changed. The results show that the relative effectiveness of the individual signals depends on the architecture: activation-only scoring performs better on ViM-Tiny, whereas Δ-only scoring performs better on ViM-Small. One possible explanation is that differences in model capacity and representational redundancy affect the usefulness of each signal, although our ablation does not directly test the redundancy level of each model. Following the reviewer’s suggestion, we additionally evaluated both individual components without knowledge distillation, which is shown in the table bellow. These experiments reproduce the same pattern: neither signal consistently outperforms the other across both architecture scales. Thus, the architecture-dependent behavior is not specific to the distillation objective. This makes neither of the components suitable across all versions of the ViM architecture, by combining them we achieve a more stable metric, which outperforms SOTA across all architecture variants.
Our claim of robustness therefore refers to the combination’s cross-backbone consistency, rather than superiority over the strongest individual component in every setting. DeltA provides a unified criterion that maintains competitive accuracy and exceeds the reported results of the compared pruning baselines on both backbones, without selecting a different importance metric for each architecture. By having a unified metric which maintains performance in multiple cases we are avoiding choosing separate pruning metric for each individual variant of vision mamba
W2. Training settings and run-to-run variability
Regarding the reviewer’s concern about the paper's gains and run-to-run variability, we agree that knowledge distillation can influence the final results and have added controlled experiments to assess its contribution. Following another reviewer’s suggestion, we also added DyVM to the comparison, which uses knowledge distillation during fine-tuning. We therefore retain DeltA’s distillation-based results in the main comparison and separately analyze the contribution of distillation through an ablation.
In this ablation, the pretrained checkpoint, learning rate, training schedule, pruning locations, and retention rates are identical; only the distillation objective is removed. DeltA achieves 75.6% and 79.4% Top-1 accuracy with distillation on ViM-Tiny and ViM-Small, respectively, compared with 75.5% and 79.1% without distillation. Distillation therefore contributes 0.1 and 0.3 percentage points, while both results without distillation remain above the reported ToP-ViM accuracies of 75.1% and 78.8%. The updated ViM-Tiny result follows a revision to the initial learning rate, applied consistently across the corresponding ablation variants and documented in the revised manuscript. Consequently, the updated improvements over ToP-ViM are 0.5/0.6 percentage points with distillation and 0.4/0.3 points without it.
To further investigate attribution to the score, we evaluate Δ-only, activation-only, and combined scoring with and without distillation, holding the remaining settings fixed within each backbone. The stronger individual signal changes with architecture, whereas DeltA remains competitive across both scales and training settings. On ViM-Small, removing distillation reduces activation-only accuracy by 1.8 percentage points, compared with 0.3 points for DeltA. These experiments directly examine the scoring components and show that DeltA’s advantage over the reported ToP-ViM results cannot be explained solely by teacher supervision, although differences between the methods’ training protocols remain.