- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1(−1)
Read review
Paper summary
The paper studies the training-free composition of token merging and query-key rank reduction in pretrained vision transformers. It factorizes the bias-augmented query-key product with an activation-weighted truncated SVD. For the resulting logit error, it proves that a fixed correction invariant to arbitrary patch permutations has five parameters, two of which are softmax-invisible row shifts, leaving three effective coefficients per attention head. To reduce the dependence of merge decisions on the current factorization, it computes token similarity from normalized block inputs instead of compressed keys. ImageNet experiments on DeiT and ViT-L identify a compute-dependent crossover where the combination outperforms token reduction alone; throughput measurements on an RTX 3090 partly validate the FLOP-based comparison. Additional ADE20K and CLIP experiments evaluate rank reduction and correction without token merging.
Critical strengths
- Structured logit correction. Proposition 1 characterizes five matrix directions invariant to patch permutations and identifies three that remain observable after row-wise softmax. Table 1 supports the predicted null effect of row-wise shifts and reports the accuracy recovered by the three-coefficient correction without token reduction.
- Conditional evaluation of the combined method. Tables 2&3 report where combining token merging with QK rank reduction improves the accuracy-cost trade-off and where token merging alone performs better. Table 4 isolates the effect of the features used for token matching; Table 3 also reports measured throughput alongside the compute comparisons.
Critical weaknesses
- Incomplete comparison with recent training-free baselines. The paper’s central empirical claim concerns the accuracy-cost benefit of combining token reduction with query-key factorization. However, Table 5 omits recent training-free token-reduction methods such as (Lee & Choi, 2025), despite discussing that work in Section2 Table2 compares the combination against the authors’ block-input token-merging baseline, but does not establish whether the advantage persists against stronger recent single-axis methods
- The abstract highlights gains of “up to 4.34 percentage points” at matched compute and “up to 1.06× throughput” at matched accuracy. The 4.34-point result comes from the DeiT-Tiny, reduction-rate-18, rank-32 setting in Table 2, while many milder-reduction settings show small or negative gains. The 1.06× throughput result comes from separate DeiT-Small development-set experiments in Table 3. Although the paper acknowledges a crossover between regimes, presenting only these maxima in the abstract risks giving an inflated impression of their typical magnitude.
Questions for authors
Please refer to the weaknesses part
3 comments
I thank the reviewer for their response, but my concerns remain. Compared with ATM, a work from 2025, the proposed method achieves lower accuracy (79.76 → 79.46) with a similar computational cost (3.28 GMACs vs. 3.25 GMACs). What, then, is the significance of this improvement?
Thank you for the follow-up. As discussed in Section 4.4 and demonstrated in Table 3, the improvement we claim occurs beyond the crossover, where further token merging becomes costly in accuracy. The ATM comparison at approximately 3.25 GMAC evaluates the mild-compression regime, where adding rank reduction offers little benefit. At approximately 2.06 GFLOPs, composition achieves 77.76% versus 76.66% for token reduction alone using the same matching rule, a 1.10-point accuracy gain at approximately matched compute (Table 14). Our contribution is identifying this regime and enabling effective composition without retraining. These results establish the benefit over the controlled token-reduction baseline.
Thank you for recognizing the structural correction result. We have added the recent baseline you identified and clarified the operating regime and magnitude of the gains.
1. Recent training-free baselines.
Table 6 now includes ATM (Lee & Choi, 2025), evaluated using its official implementation on all 50,000 ImageNet validation images. ATM achieves 79.76% at 3.28 GMAC, and our composition achieves 79.46% at 3.25 GMAC. ATM’s compute is calculated from its observed token schedule using the same counting convention as our method. This provides the requested recent comparison at approximately matched compute and identifies ATM’s accuracy advantage at this mild-compression setting.
Our composition addresses the regime in which dividing compression across both axes becomes beneficial. On the 45,000-image test split, the rate-16/rank-32 configuration achieves 77.76% at 2.064 GFLOPs, compared with 76.66% at 2.059 GFLOPs for token reduction alone using the same block-input matching (Table 14): 1.10 additional accuracy points at approximately matched model compute. This controlled comparison isolates the benefit of composition; it does not establish superiority over ATM at that budget. The revised evaluation places the method alongside the recent baseline while identifying where combining token and rank reduction improves the accuracy–cost trade-off.
2. Magnitude of the highlighted gains.
We revised the abstract and introduction to report 0.6–4.3 percentage points at matched model compute across the evaluated DeiT backbones at reduction rates 16–18, rather than presenting only the maximum. The abstract explicitly states that mild reduction favors merging alone and identifies the stronger-reduction regime in which composition helps.
Table 3 retains the small and negative gains at milder settings. Table 4 separately reports measured throughput on DeiT-Small and labels the development split. Its 1.06× speedup is an additional gain over token merging alone at matched accuracy, rather than a comparison against the dense model. The revised presentation connects each result to the model, compression budget, evaluation split, and baseline that support it.