- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reference-set dependence in token reduction for ViTs. The authors define token importance as the marginal loss induced by deleting a token from a reference set, distinguishing between , computed on the full token set, and , computed on the candidate set. The paper shows that these two reference sets can yield different rankings, thereby affecting the quality of the retained subset. It further argues that re-evaluating the surviving tokens during reduction is more effective than scoring them only at the initial state.
The experiments primarily cover DeiT-III Small and DINOv2 ViT-S/14, comparing candidate-set construction strategies, loss definitions, and token budgets on the full ImageNet-1K validation set. They also include overlap interventions with a fixed reference-set size, learned-scorer experiments, and controlled comparisons with ToMe. Additional validation is conducted on ImageNet-V2, segmentation, depth estimation, and a larger backbone. Overall, the results support the conclusion that reference choice affects token selection quality. However, the evidence remains insufficient to establish the general limitations of per-token scoring or to identify re-evaluation as the primary source of ToMe’s advantage.
Critical strengths
1. The paper explicitly identifies reference choice as a decision variable in token reduction and provides targeted experiments.
Beyond comparing and the paper varies the overlap between the reference set and the candidate set while holding the reference-set size fixed. This is more convincing than directly comparing two sets of different sizes. The finding has practical implications for understanding existing token importance measures.
2. The main empirical findings are supported by broad validation and reasonably rigorous statistical analysis.
The experiments use all 50,000 ImageNet-1K validation images and cover two backbones, two loss definitions, four candidate-set construction strategies, and three token budgets. Candidate-set scoring is favored in 45 of the 48 configurations, with an aggregate normalized advantage of 0.305. The paper also reports confidence intervals and validation across distributions and tasks. These results make it difficult to attribute reference-set shift to an isolated model or configuration, providing reasonably reliable support for the empirical findings.
Critical weaknesses
1. The current proofs and learning experiments do not support the theoretical claims about scorer expressivity.
Propositions 2 and 3 are important components of the theoretical argument, but the main text primarily provides construction sketches, while the appendix does not sufficiently detail the corresponding loss constructions or proofs of the key inequalities. More importantly, under conditions such as a specified feature map and identical features for certain candidate tokens, Proposition 3 shows that a shared per-token scoring rule of the form cannot guarantee correct selection across all constructed instances. This worst-case result does not directly establish that learned scorers operating on actual ViT features generally cannot recover , nor does it directly constrain a set-attention scorer with access to the entire candidate set.
Nevertheless, the paper also interprets the poor performance of the latter as an empirical manifestation of this proposition. The learning experiments use only approximately 300 images per backbone, which is insufficient to rule out the effects of training data, model capacity, or optimization. Therefore, the paper cannot move from “reference-set dependence” to the stronger conclusion that “improving the scorer cannot solve the problem in principle.”
2. The existing controls are insufficient to attribute ToMe’s advantage primarily to re-evaluation while discounting the contribution of merged content.
In Table 1, ToMe merges the full patch sequence, whereas the other methods prune only within a candidate set of 96 patches. An equal final token count does not imply equal information sources, reduction procedures, or computational budgets. This system-level comparison therefore establishes a performance difference but cannot, by itself, identify re-evaluation as its cause.
The merged-content ablation also fixes contextual keys, matching edges, and reduction rounds. This tests the direct effect of preserving merged content under a fixed matching process, but it simultaneously blocks the potential pathway through which merged content changes subsequent features and matching decisions. The conclusions therefore overlook the importance of preserving merged content in this setting. The conclusions drawn from the audit of EViT attention scoring should also be limited in scope: the original EViT includes token reorganization within layers and fusion of inattentive tokens, rather than simply performing one-shot attention-based selection.
Questions for authors
-
How are the set losses, feature maps, and interaction terms defined explicitly? How can approximate additivity, ranking reversal, and a non-vanishing excess loss be shown to hold simultaneously?
-
What is the scope of Proposition 3 with respect to practical learned scorers? Can contain contextual information, candidate-set identity, or survival status? Does a set-attention scorer with access to the entire candidate set belong to the function class constrained by this proposition? If not, how should its failure be connected to the proposition?
-
Could the negative results for learned scorers be limited by the data scale or training configuration? Please clarify the training objectives, data splits, hyperparameter selection, and convergence behavior, and provide evidence sufficient to assess whether data availability or model capacity is a primary limitation. Please also explain the discrepancy between the ridge correlation coefficients reported in the two sections.
-
When merged content is allowed to affect subsequent features and matching decisions, how does merging perform relative to dropping?
-
Which experimental configurations and aggregation procedure produce the 10.6% Top-1 improvement reported in the abstract? Please report the selection results for teacher-KL and ground-truth-based label-NLL separately, together with the corresponding unnormalized accuracy values.
2 comments
Thank the authors for their detailed response and supplementary experiments. Overall, these additions strengthen the paper.
We sincerely appreciate your time and valuable comments, which help us significantly improve our paper.
Explicit Constructions and Scope of Propositions 2 and 3
Appendix C provides full proofs and exact definitions for Propositions 2 and 3. The revision defines the set losses, feature representations, and token interactions directly. It presents the simultaneous derivation of near-additivity, ranking inversion, and non-vanishing excess loss. Proposition 2 constructs a mirror pair of candidate-conditional edges. Proposition 3 constructs two candidates with identical features that produce opposite ranking orders under the two references.
Proposition 3 applies only to scorers of the form si=g(hi) that use fixed per-token features. It says nothing about scorers that process candidate-set context, survival status, or dynamic features. The set-attention scorer runs as a separate test outside this function class. The text now clarifies this exact boundary.
Scale Analysis and Discrepancy Clarification for Learned Scorers
We evaluated learned scorers on 300, 1,000, 3,000, 10,000, and 30,000 images per model to measure the effect of training volume. Rank correlation with exact d(C) stays near 0.05 across all training sizes on DINOv2. At 30,000 images, the learned scorer scores below full-reference scoring, with normalized differences of −0.016 [−0.036, 0.003] on teacher-KL and −0.319 [−0.355, −0.282] on label-NLL. Appendix A lists the training objectives, splits, seeds, and hyperparameters for every model scale.
The two ridge correlation values reported earlier reflect different aggregation levels. The numbers 0.013 and 0.011 measure the mean per-candidate Spearman correlation. The numbers 0.030 and 0.017 measure the Spearman correlation evaluated over the full 96-token ranking. The revised manuscript defines both metrics and uses the candidate-set-level statistic consistently.
Dynamic Feature Updates and Content Preservation in ToMe
The new experiment allows merged features to update subsequent keys and matching decisions directly. It tests four conditions under one matched layer-wise schedule. When both conditions use dynamic re-matching, merging beats dropping by 0.096 at k=32, 0.067 at k=64, and 0.063 at k=80 on DINOv2. Dynamic re-matching produces larger differences over the frozen baseline, reaching 0.216, 0.156, and 0.150 across the same budgets. The revision reports all four conditions across both backbones.
Clarification of the Top-1 Claim in the Abstract
The +10.6 Top-1 claim in the abstract came from an early proof-of-concept experiment and was included by mistake. We removed this claim from the revised abstract and main text. The revision reports teacher-KL and label-NLL results separately, listing unnormalized Top-1 accuracy for all methods. In Table 1, ToMe outperforms d(C) on teacher-KL Top-1 at k=32, scoring 0.777 against 0.677 on DeiT-III and 0.743 against 0.590 on DINOv2. The new label experiment adds full validation results for label-NLL, reporting unnormalized Top-1 across both loss functions.
Sorry for the late response. We hope our response and the revised manuscript address your concerns. We remain open to any further discussion.