acceptodds
Under review as a conference paper at ICLR 2027

Simple approach to token pruning for vision state-space models

Abstract

Vision Mamba models offer linear sequence complexity but remain costly at token counts of high resolution images. Token pruning can reduce this cost, but methods developed for Vision Transformers do not directly account for the recurrent, order-dependent computation of state-space models. Removing tokens changes subsequent state updates and can disrupt information propagation along the sequence. We introduce DeltA, a parameter-free token-pruning importance metric designed for vision state-space models. DeltA scores each token using the interaction between its input-dependent timescale parameter and the magnitude of its post-block representation. The method uses signals already available in the backbone and requires no learned pruning module. Experiments on image classification, semantic segmentation, and facial-attribute recognition using ViM and PlainMamba architectures show a favorable accuracy–computation trade-off and further indicate that the same criterion generalizes across different vision Mamba architectures and tasks.

Primary area:
optimization
Keywords:
Vision Mamba, State-space models, Token pruning, Model compression, Efficient deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7fW1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DeltA, a token-pruning criterion that multiplies Mamba’s discretization signal by the post-block representation norm. It evaluates the criterion across different vision tasks and backbones.

Critical strengths

The criterion is simple, uses existing backbone signals, and requires no additional learnable selector. Evaluation across multiple tasks and backbones provides initial evidence of its applicability.

Critical weaknesses
  1. Incremental contribution with insufficient support for the combination. Table 5 shows that activation-only scoring outperforms DeltA on ViM-Tiny (75.6% vs. 75.3%), while delta-only scoring outperforms it on ViM-Small (80.0% vs. 79.4%). The necessity and claimed robustness of the combined score therefore remain insufficiently established.

  2. Modest gains with unclear attribution. Improvements over ToP-ViM are only 0.2/0.6 percentage points. Since DeltA uses feature distillation and the comparison does not establish matched training and pruning settings, these gains cannot be clearly attributed to the new score. Variability across runs is also unreported.

Questions for authors
  1. Were all variants in Table 5 evaluated under identical pruning, training, and distillation settings? Since the best individual signal outperforms DeltA on each backbone, what evidence supports the claimed robustness of the combination?

  2. Were ToP-ViM and DeltA compared under matched settings, including distillation? Could the authors report variability across runs to establish whether the small gains are stable and attributable to the new score?

  3. Why use multiplication rather than a normalized sum or weighted sum? Is there evidence that this particular combination improves token selection?

4 comments

We thank the reviewer for the careful and constructive evaluation of our work. We appreciate their recognition of the simplicity of the proposed pruning criterion, particularly its use of existing backbone signals without requiring additional learnable scoring modules. We also value their positive assessment of the paper's clarity and presentation, as well as their acknowledgment of the method's applicability across different vision architectures and tasks. We thank the reviewer for highlighting aspects of the combined importance criterion and experimental comparisons that require further justification. Below, we address each concern and question individually.

W1. Contribution of Individual Importance Signals and Robustness of Their Combination

We thank the reviewer for the questions. Yes, all variants in Table 5 were evaluated under identical pruning, training, and distillation settings, with only the token-importance criterion changed. The results show that the relative effectiveness of the individual signals depends on the architecture: activation-only scoring performs better on ViM-Tiny, whereas Δ\Delta-only scoring performs better on ViM-Small. One possible explanation is that differences in model capacity and representational redundancy affect the usefulness of each signal, although our ablation does not directly test the redundancy level of each model. Following the reviewer’s suggestion, we additionally evaluated both individual components without knowledge distillation, which is shown in the table bellow. These experiments reproduce the same pattern: neither signal consistently outperforms the other across both architecture scales. Thus, the architecture-dependent behavior is not specific to the distillation objective. This makes neither of the components suitable across all versions of the ViM architecture, by combining them we achieve a more stable metric, which outperforms SOTA across all architecture variants.

BackboneTrainingΔt\Delta_t onlyAtA_t onlyDeltA
ViM-TinyWith KD75.375.875.6
ViM-TinyWithout KD75.275.575.5
ViM-SmallWith KD80.079.179.4
ViM-SmallWithout KD79.577.379.1

Our claim of robustness therefore refers to the combination’s cross-backbone consistency, rather than superiority over the strongest individual component in every setting. DeltA provides a unified criterion that maintains competitive accuracy and exceeds the reported results of the compared pruning baselines on both backbones, without selecting a different importance metric for each architecture. By having a unified metric which maintains performance in multiple cases we are avoiding choosing separate pruning metric for each individual variant of vision mamba

W2. Training settings and run-to-run variability

Regarding the reviewer’s concern about the paper's gains and run-to-run variability, we agree that knowledge distillation can influence the final results and have added controlled experiments to assess its contribution. Following another reviewer’s suggestion, we also added DyVM to the comparison, which uses knowledge distillation during fine-tuning. We therefore retain DeltA’s distillation-based results in the main comparison and separately analyze the contribution of distillation through an ablation.

In this ablation, the pretrained checkpoint, learning rate, training schedule, pruning locations, and retention rates are identical; only the distillation objective is removed. DeltA achieves 75.6% and 79.4% Top-1 accuracy with distillation on ViM-Tiny and ViM-Small, respectively, compared with 75.5% and 79.1% without distillation. Distillation therefore contributes 0.1 and 0.3 percentage points, while both results without distillation remain above the reported ToP-ViM accuracies of 75.1% and 78.8%. The updated ViM-Tiny result follows a revision to the initial learning rate, applied consistently across the corresponding ablation variants and documented in the revised manuscript. Consequently, the updated improvements over ToP-ViM are 0.5/0.6 percentage points with distillation and 0.4/0.3 points without it.

To further investigate attribution to the score, we evaluate Δ\Delta-only, activation-only, and combined scoring with and without distillation, holding the remaining settings fixed within each backbone. The stronger individual signal changes with architecture, whereas DeltA remains competitive across both scales and training settings. On ViM-Small, removing distillation reduces activation-only accuracy by 1.8 percentage points, compared with 0.3 points for DeltA. These experiments directly examine the scoring components and show that DeltA’s advantage over the reported ToP-ViM results cannot be explained solely by teacher supervision, although differences between the methods’ training protocols remain.

Questions for the author

Q1. Metric component ablation

Yes. Within each backbone, all variants in Table 5 use the same pretrained checkpoint, learning rate, training schedule, pruning locations, retention rates, and distillation settings; only the importance criterion changes. The revised ViM-Tiny experiments use the updated initial learning rate consistently across all three criteria. Our claim of robustness refers to maintaining competitive performance across backbones and training settings without selecting a different importance signal for each architecture. Activation-only scoring performs best on ViM-Tiny, whereas Δ\Delta-only scoring performs best on ViM-Small. DeltA remains within 0.2 and 0.6 percentage points of the strongest component, respectively. We have clarified that this supports a unified criterion rather than superiority over the best individual component in every setting. We additionally repeated the component ablation without knowledge distillation, keeping all other settings unchanged.

BackboneTrainingΔt\Delta_t onlyAtA_t onlyDeltA
ViM-TinyWith KD75.375.875.6
ViM-TinyWithout KD75.275.575.5
ViM-SmallWith KD80.079.179.4
ViM-SmallWithout KD79.577.379.1

The preferred individual signal still changes with architecture. On ViM-Tiny, DeltA matches activation-only scoring at 75.5%; on ViM-Small, it achieves 79.1%, compared with 79.5% for Δ\Delta-only and 77.3% for activation-only scoring. Removing distillation reduces DeltA’s accuracy by only 0.1/0.3 percentage points across the two backbones, whereas activation-only scoring loses 1.8 points on ViM-Small. These results strengthen the evidence that DeltA provides consistently competitive selection across the evaluated settings without requiring architecture-specific choice of the importance signal.

Q2. Fairness of Comparisons, Knowledge Distillation, and Experimental Variability

The difference in the settings between ToP-ViM and DeltA training procedure comes down to a different choice of learning rate and scheduler which was separately chosen for our method and the addition of knowledge distillation in our experiment which was not mentioned in the ToP-ViM paper, while the rest of the setup, such as the FLOPs reduction and the amount of retraining was kept the same. Thank you for pointing this out, since the knowledge distillation in the training procedure could attribute to the part of the gain. As per your comment and another reviewer's suggestion we have added an additional ablation analyzing the change in the ImageNet results depending on whether knowledge distillation was used or not. The starting learning rate for ViM-Tiny architecture was changed from 5e-4 to 1e-4 to suit the architecture better, which has lead to an improvement in both cases where distillation was used, and in the case where it was omitted from the retraining. The value has also been changed in the main results section.

Regarding run-to-run variability, all variants for a given backbone start from the same pretrained checkpoint and use a batch size of 512. This removes variation from backbone initialization, while the large batch reduces noise from minibatch sampling. We therefore expect relatively limited variability compared with training from scratch. Given the computational cost and rebuttal timeframe, we prioritized controlled component and distillation ablations to investigate the sources of the observed performance differences. For a faster comparison in the limited time we have completed the run on ViM-Tiny on the CelebA dataset on three different seeds with the results presented in the following table:

RunAvg. accuracy (%)Precision (%)Recall (%)F1 (%)
191.076.665.869.7
291.177.065.369.4
391.177.165.269.5
Mean91.176.965.469.5
Std.0.00.30.30.1

The results show limited variation: accuracy ranges from 91.03% to 91.10%, and the sample standard deviations of precision, recall, and F1 are 0.3, 0.3, and 0.1 percentage points, respectively. These additional experiments provide empirical evidence of consistent performance across seeds in the evaluated CelebA setting. As for the ImageNet experiments, we plan to include them in the final version of the paper, should the paper be accepted.

Q3 Motivation for Multiplication over Additive or Weighted Fusion

We chose multiplication to prioritize tokens that exhibit both a strong discretization signal and a large output representation. Unlike an additive score, the product suppresses tokens when either factor is small, reflecting our hypothesis that both quantities should inform retention. It also preserves token rankings under uniform positive rescaling of either component within a pruning layer and requires no mixing coefficient. A normalized or weighted sum is a reasonable alternative, but entails a normalization or weighting choice. These properties motivate the design rather than establish its empirical optimality. Table 5 (6 in the new version of the paper) compares the product with its individual components, but does not directly compare alternative fusion rules. It shows that the product avoids the weaker component’s performance on each tested backbone, although it does not outperform the stronger component in either case.

Summary of Our Response

We thank the reviewer again for the careful evaluation and constructive suggestions. We have clarified the interpretation of the combined importance criterion and the role of its individual components, and added new ablation experiments without knowledge distillation to Section 5.3. We have also quantified the contribution of knowledge distillation in Section 5.2, updated the ViM-Tiny training configuration and corresponding results, and clarified the limitations of comparisons based on published baselines and single-run evaluations. These additional analyses demonstrate that DeltA maintains competitive performance across the evaluated backbone scales and training settings, while providing a more precise understanding of the contribution and limitations of the proposed criterion. We hope these clarifications and results address the reviewer's concerns, and we would appreciate their reassessment of the paper in light of the rebuttal and revised manuscript.

Reviewer_7fW1OpenReview ↗

Thank you

Thank you for the transparent clarifications and additional ablations. While I appreciate the honest reporting, my core concerns remain unresolved, and I will maintain my current score.

Specifically, the updated results show that multiplying the two signals still yields no synergistic gain over the best individual signal on either backbone, and no empirical comparison against alternative fusion rules (e.g., normalized addition) is provided to justify the product formulation. Moreover, once distillation is removed, the modest 0.3–0.4% margins over the reported ToP-ViM baselines remain too small to convincingly establish the advantage of the proposed score.

Reviewer_tQhNOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a simple yet effective way for pruning the token inside vision mamba models. Each token is scored by the product of two quantities: its channel-averaged discretization step Δt\Delta_t (aggregated across scan directions), and the L2 norm of its post-block output. Pruning happens at three stages with geometrically decreasing retention rates. During fine-tuning, a Gumbel-Sigmoid straight-through relaxation is used, together with feature-level distillation from the unpruned model. Experiments on ImageNet-1K, CelebA attribute recognition, and ADE20K segmentation shows positive results.

Critical strengths
  • The method is simple and cheap, seems to be generalizable.

  • The experiment part is comprehensive.

Critical weaknesses
  • I have a concern on its novelty. Δ\Delta as a token-importance signal has been used in DTP[1] which averages Δ\Delta over channels exactly as in Eq. (6). The same is true of MTR[2], which uses Δ\Delta for token reduction on ViM/ImageNet. An output-based importance score is the metric of ToP-ViM[3], which uses the channel-averaged clipped SSM output. ToP-ViM’s own ablation already evaluates the L2-norm variant (78.6% on ViM-S vs. 78.8% for its clipped version).

  • The 1.24× speedup is measured on an RTX 4070 and compared against ToP-ViM’s reported 1.27×, which comes from different hardware (ToP-ViM’s experiments use V100s). Speedup ratios are not transferable across GPUs. Moreover, ToP-ViM reports 1.30× for pruning without alignment at similar FLOPs, so it is unclear why DeltA, which has no alignment overhead, is slower. Batch size, resolution, and kernel details are not reported, and there is no ViM-T throughput.

[1] DTP: Delta-Guided Two Stage Pruning for Mamba-based Multimodal Large Language Models, ICLR 2026 [2] Training-free Token Reduction for Vision Mamba, arXiv 2507.14042 [3] Exploring Token Pruning in Vision State Space Models, NeurIPS 2024

Questions for authors
  • Please elaborate the technical difference between proposed method and DTP, MTR, and ToP-ViM.

  • Please clarify the detail of speed test and flops calculation.

2 comments

We thank the reviewer for the thoughtful evaluation and encouraging feedback. We appreciate their recognition of the simplicity and computational efficiency of our proposed token-pruning approach, as well as its applicability across different architectures and vision tasks. We also value their positive assessment of the comprehensiveness of our experimental evaluation. We thank the reviewer for raising important questions regarding the technical distinctions from related methods and the evaluation of computational efficiency. Below, we address each concern and question individually.

W1/Q1. Technical Differences from DTP, MTR, and ToP-ViM

We thank the reviewer for the opportunity to clarify these differences. DTP uses Δ\Delta-based importance within a two-stage pruning procedure for Mamba-based multimodal LLMs, with pruning locations selected through importance distributions and implicit attention analysis. MTR uses Δ\Delta within a training-free Vision Mamba framework that groups and merges tokens. ToP-ViM instead scores SSM outputs and introduces hidden-state alignment; its L2L_2-norm ablation evaluates output magnitude alone. DeltA introduces the joint criterion It=Δt∣yt∣2\mathcal{I}_t=\Delta_t|\mathbf{y}_t|_2, coupling the discretization response with the magnitude of the representation passed to subsequent layers. This changes the basis for token selection: tokens with similar Δ\Delta values are distinguished by their output magnitudes, and vice versa. Neither Δ\Delta-only scoring nor ToP-ViM’s output-only ablation evaluates this interaction. We use the criterion at fixed pruning stages and fine-tune the backbone without token merging or hidden-state alignment. Our experiments further show that, although the strongest individual signal changes with backbone scale, the joint criterion remains competitive with and without distillation, supporting its use without architecture-specific selection of the scoring signal. Our contribution is therefore a distinct joint importance criterion supported by evaluations across architectures, tasks, and training settings. We have clarified these distinctions in the Introduction and Related Work sections, with changes marked in blue.

W2/Q2. Inference Speedup and FLOPs Calculation

We agree that the original cross-hardware comparison was insufficient. We therefore repeated the measurements on the same RTX 4070 Ti SUPER using batch size 32, 224×224 inputs, stride 16, FP32, and TF32 disabled. We report median throughput over five repetitions, each with 50 warm-up and 100 timed iterations, with GPU synchronization. Using the same implementation and pretrained weights for dense and pruned ViM-S, DeltA achieves 891.2 versus 679.5 images/s (1.31×). Profiling revealed redundant block-selection operations and auxiliary delta-importance calculations at non-pruning layers. Removing these operations improved runtime without changing token selections on the checked inputs. Under the same timing conditions, our ToP-ViM released-code variant achieves 938.6 images/s, remaining faster than DeltA. The release lacks the optimized alignment implementation, so this measurement is explicitly distinguished from the published aligned result. We no longer use the reported V100 ratios as a direct runtime comparison. Absence of alignment overhead alone does not guarantee higher throughput, since selection and kernel implementation costs also matter.

Summary of Our Response

We thank the reviewer again for the encouraging evaluation and constructive suggestions. We have clarified the methodological distinctions between DeltA and existing discretization-based and output-based token reduction approaches, emphasizing the contribution of the combined importance criterion. We have also conducted additional same-hardware throughput measurements, specified the benchmarking protocol, and clarified the FLOPs calculation and limitations of comparisons with published speedup results. We hope these clarifications and additional measurements address the reviewer's concerns, and we would appreciate their reassessment of the paper in light of the rebuttal and revised manuscript.

Thanks for your response, it do resolve my concern. However, I agree with Reviewer 7fW1's concerns. As a result, I would like to downgrade my recommendation to weak rej.

Reviewer_oAxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Token pruning is a standard way to speed up vision transformers, but it does not carry over cleanly to Vision Mamba: the SSM updates its state recurrently along the sequence, so deleting a token changes the states of all positions that follow. The paper builds on recent work showing that the input-dependent discretization parameter Δt\Delta_t is a useful importance signal, and points out that Δt\Delta_t on its own says nothing about how large the representation passed on to later layers actually is. Their score is simply the product It=Δt⋅∣yt∣2I_t = \Delta_t \cdot |y_t|_2, where ∣yt∣2|y_t|_2 is the L2 norm of the Mamba block output, so no extra modules are needed—both terms fall out of the normal forward pass. Pruning is applied in three stages (layers 6, 12, 18) with a geometric retention schedule, and the pruned model is fine-tuned for 30 epochs with a Gumbel-Sigmoid straight-through relaxation plus feature-level knowledge distillation from the dense model. On ImageNet-1K they report 75.3% and 79.4% Top-1 for ViM-Tiny and ViM-Small at 16% and 32% fewer FLOPs, a touch above EViT and ToP-ViM at similar compute (1.24× wall-clock speedup measured on an RTX 4070). The ADE20K results (44.58/46.72 mIoU for PlainMamba-L1/L2) beat both pruning baselines, and CelebA metrics stay within roughly two points of the full models. Their ablation finds the combined score to be the most consistent across scales, although on each architecture one of the two components alone does better.

Critical strengths
  1. The method is about as lightweight as token pruning gets: the score uses only Δt\Delta_t and ∣yt∣2|y_t|_2, which the model computes anyway. There is no selector network, no hidden-state alignment layer, and no extra cost at inference.
  2. The same criterion works across classification, ADE20K segmentation, and CelebA attribute recognition, and across two fairly different scanning schemes (bidirectional ViM versus PlainMamba's 2D scan). The margins over ToP-ViM on ADE20K (44.58 vs. 44.10 mIoU on PlainMamba-L1) are the clearest in the paper.
Critical weaknesses
  1. Limited novelty. The Δt\Delta_t signal itself comes from prior work—DTP (ICLR 2026) uses it for Mamba-based models and TE-VMamba (CVPR 2026) for vision Mamba. What is new here is multiplying Δt\Delta_t by the post-block norm ∣yt∣2|y_t|_2. I could not find any argument for choosing a product over, say, a weighted sum, and the accuracy this buys is small (point 3).
  2. Thin and not fully controlled comparison. Only EViT (2022) and ToP-ViM (2024) appear in the main tables, while a number of recent and directly relevant works—DTP, DyVM, STORM, TE-VMamba, R-MeeTo, Famba-V, Sequential Token Merging, FastVim—are missing, so the "state-of-the-art" claim is hard to take at face value. There are also fairness issues. The retention rates were picked precisely to land on the baselines' FLOPs. The speedup numbers mix hardware: their own RTX 4070 measurement against ToP-ViM's reported figure. And the baseline accuracies are taken from Zhan et al. (2024) rather than re-run under the authors' fine-tuning setup (30 epochs with feature KD). Any one of these differences could plausibly explain the 0.2–0.6 point gaps.
  3. The ablation works against the main claim, and the margins are fragile. Each single component beats the combination on one of the two models (75.6 vs. 75.3 on ViM-T; 80.0 vs. 79.4 on ViM-S), so "complementarity" here really means "the product is never the worst option". Fine-tuning also adds feature KD, and with no without-KD ablation it is hard to tell where the gains come from. The headline improvements over ToP-ViM (+0.2/+0.6) come with no seed variance or significance testing, and a few statements run ahead of the numbers: the CelebA text says precision stays "within approximately two percentage points", but on ViM-Small it falls by 2.3.
Questions for authors
  1. Table 5 is puzzling to me: on each model, one of the single components beats the combination (75.6 vs. 75.3 on ViM-T; 80.0 vs. 79.4 on ViM-S). In which settings does the product actually win? Relatedly, is there a reason to prefer multiplication over an additive or weighted combination?
  2. How much of the gain comes from the feature-level KD rather than the criterion? Numbers without KD for DeltA and both single components would be very helpful.
  3. Would it be possible to re-run ToP-ViM, and ideally DyVM or DTP, under the same fine-tuning protocol, and compare at several pruning ratios on the same hardware?
  4. On PlainMamba-L1 the pruned model (44.58 mIoU) scores above the unpruned one (44.24). Is this just the effect of the extra fine-tuning? If so, the same budget should presumably be given to all methods.
5 comments

We thank the reviewer for the thorough and constructive evaluation of our work. We appreciate their recognition of the simplicity and computational efficiency of the proposed pruning criterion, particularly its use of existing backbone signals without requiring additional learnable modules or introducing inference overhead. We also value their positive assessment of the method's applicability across different vision architectures and tasks, as well as the encouraging results on semantic segmentation. We thank the reviewer for identifying areas where the motivation behind the proposed criterion, experimental comparisons, and interpretation of the ablation results could be further strengthened. Below, we address each concern and question individually. We first address the critical weaknesses one by one after which we answer the questions.

Critical Weaknesses

W1. Novelty and Distinction from Competing Methods

We appreciate the opportunity to clarify the distinction from DTP and TE-VMamba and the motivation for using a product rather than a weighted sum. We do not claim novelty in using Δ\Delta as an importance signal. Our contribution is an output-magnitude-aware criterion that combines this SSM-internal signal with the norm of the representation passed to subsequent layers. This allows the token ranking to reflect both quantities, while requiring no additional learned scoring module. We use multiplication to make each signal modulate the contribution of the other: a large output magnitude is weighted by the corresponding discretization response, while a large discretization response is weighted by the resulting representation magnitude. A small value in either component consequently reduces the token’s score. Multiplication also has a useful property for ranking: multiplying either component by a positive constant shared across tokens rescales all scores equally and leaves their ordering unchanged. A weighted sum, by comparison, requires specifying the relative scaling and weighting of the two signals, which can affect the resulting ranking. The product therefore provides a simple combination without an additional mixing coefficient or normalization rule. These properties motivate our choice; the component ablations assess the combined criterion against its individual signals, but do not establish that multiplication is optimal among all possible fusion strategies. A normalized or weighted sum remains a reasonable alternative for further investigation. The distinction from prior methods also extends beyond the application domain. DTP addresses visual-token reduction in Mamba-based multimodal LLMs through a two-stage procedure whose pruning locations are determined from importance distributions and implicit attention analysis. TE-VMamba combines adaptive pruning, depth-aware merging, and learnable layer-wise thresholds for person re-identification. Our approach uses a fixed staged pruning schedule and the proposed joint criterion, without either layer-selection analysis or token merging. We evaluate this design across classification, semantic segmentation, and attribute recognition, using both ViM and PlainMamba. Thus, our contribution is a distinct scoring criterion and a broadly evaluated pruning design, rather than a claim to originate Δ\Delta-based importance estimation. To address this comment, we revised the introduction to clarify our contribution and its distinction from recent methods. The newly added text is marked in blue.

W2. Controlled comparison

We thank the reviewer for highlighting the need to clarify the comparison scope and experimental protocols. We selected ToP-ViM as a directly relevant published baseline for token pruning in Vision Mamba, together with EViT as a representative transformer-derived approach evaluated on the same backbones. We have now also added DyVM, which combines token pruning with dynamic block selection, to the comparison. The remaining methods address related but distinct settings: DTP targets Mamba-based multimodal LLMs; TE-VMamba combines pruning and merging for person re-identification; STORM evaluates spatially structured token reduction without additional training; R-MeeTo, Famba-V, and Sequential Token Merging investigate token merging or fusion; and FastVim modifies spatial processing within SSM blocks. These differences warrant consideration when interpreting numerical comparisons. Our retention rates were chosen to match the baselines’ computational costs as closely as possible, enabling accuracy comparisons at similar inference budgets. Matching FLOPs is appropriate here because identical retention rates can produce different computational costs depending on pruning locations and additional operations. We also evaluate multiple retention rates to characterize the accuracy-computation trade-off beyond a single operating point. For baseline accuracies, we use the original authors’ reported results under their respective experimental protocols. Reproducing the complete ToP-ViM pipeline is constrained by missing components in the released implementation, including the segmentation implementation and optimized alignment path. Although these reported results do not establish an identical training comparison, our additional controlled ablation directly addresses the potential contribution of distillation: with all other settings held fixed, DeltA without KD achieves 75.5% on ViM-Tiny and 79.1% on ViM-Small, exceeding the reported ToP-ViM results of 75.1% and 78.8%, respectively. The observed advantage therefore cannot be explained solely by distillation. We also note that the newly included DyVM baseline itself uses both output-level and token-level distillation during fine-tuning. Finally, we have replaced the cross-hardware speed comparison with measurements on the same RTX 4070 Ti SUPER, using identical input resolution, batch size, precision, warm-up, and timing procedures. The revised manuscript explicitly distinguishes the evaluated ToP-ViM released-code variant from the authors’ published implementation with optimized hidden-state alignment, so the new measurements compare the available implementations under common hardware conditions.

W3. Ablation studies

We thank the reviewer for identifying the issues in the ablation study of separate importance metric components. We have added experiments and clarified the interpretation of the results. Our intended claim is that DeltA provides a consistently competitive criterion across backbones, rather than outperforming the strongest individual component in every setting. The preferred individual signal changes with architecture: activation-only scoring performs better on ViM-Tiny, whereas Δ\Delta-only scoring performs better on ViM-Small. The additional experiments without KD reproduce this pattern. Notably, on ViM-Small, removing KD reduces activation-only accuracy by 1.8 percentage points, from 79.1% to 77.3%, whereas DeltA decreases by only 0.3 points, from 79.4% to 79.1%. On ViM-Tiny, DeltA without KD matches activation-only scoring at 75.5%. These results support using a unified criterion across the evaluated backbones and training settings, without requiring architecture-specific selection between the individual signals. To isolate the contribution of distillation, we compare DeltA with and without KD while keeping the pretrained checkpoint, learning rate, training schedule, pruning locations, and retention rates identical. Removing KD reduces accuracy from 75.6% to 75.5% on ViM-Tiny and from 79.4% to 79.1% on ViM-Small. DeltA therefore remains above the reported ToP-ViM accuracies of 75.1% and 78.8% even without distillation, showing that the observed advantage cannot be explained solely by teacher supervision, although other differences between the training protocols remain. The updated ViM-Tiny results use the revised initial learning rate, as documented in the revision. Regarding run variability, experiments on each backbone start from the same pretrained checkpoint and use a batch size of 512. This removes variation in backbone initialization, while the large batch reduces stochastic gradient noise relative to smaller batches. We consequently expect less variability than in training from scratch. Given the computational cost and rebuttal timeframe, we prioritized controlled ablations of the scoring components and distillation objective to investigate the sources of the observed performance differences. For a faster comparison in the limited time we have completed the run on ViM-Tiny on the CelebA dataset on three different seeds with the results presented in the following table:

RunAvg. accuracy (%)Precision (%)Recall (%)F1 (%)
191.076.665.869.7
291.177.065.369.4
391.177.165.269.5
Mean91.176.965.469.5
Std.0.00.30.30.1

The results show limited variation: accuracy ranges from 91.03% to 91.10%, and the sample standard deviations of precision, recall, and F1 are 0.3, 0.3, and 0.1 percentage points, respectively. These additional experiments provide empirical evidence of consistent performance across seeds in the evaluated CelebA setting. As for the ImageNet experiments, we plan to include them in the final version of the paper, should the paper be accepted. Finally, we have corrected the CelebA statement to acknowledge the 2.3-percentage-point precision decrease on ViM-Small and thank the reviewer for pointing out this discrepancy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.