- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper looks at the downstream perturbation that token merging causes in Vision Transformers, proposes an influence-weighted bound on it, and designs ADAMERGE, which sets the number of merges per layer according to the current features. Direct perturbation measurements, leave-one-out ablations, and experiments on several backbones are used to analyze how the aggregation rule and the depth of merging shape final classification accuracy.
Critical strengths
The theoretical bound and the empirical measurements fit the method design. Section 3 derives an influence-weighted upper bound on the downstream perturbation caused by token merging. Section 5 measures this bound directly and separates the perturbation due to aggregation weights from that due to depth; the result reveals that accuracy is set by where a method merges and by how much it merges. Section 4 then uses these findings to build ADAMERGE, which sets the number of token merges in each layer from influence-weighted similarity.
Critical weaknesses
- The accuracy gains over fairly strong baselines are small, and their stability needs more evidence. At the six FLOPs tiers on ViT-B/16 in Table 1, ADAMERGE improves Top-1 ACC over PiToMe by 0.13, 0.18, 0.20, 0.06, 0.60, and 0.39 pp. The authors point out that the margin over TOME grows with compression, but the gains over PiToMe are overall smaller and do not follow the same trend. Some tiers also match FLOPs only approximately. Because ADAMERGE uses slightly more FLOPs at certain tiers, small accuracy advantages may be confounded by both the method design and differences in the compute budget. Table 1 additionally reports the mean and standard deviation of ADAMERGE over repeated runs but gives no variability for PiToMe, nor the variation of the accuracy difference between the two across runs. The present results therefore do not show whether the smaller gains reproduce reliably, and the advantage over fairly strong baselines still lacks evidence of stability.
- The ablations do not fully separate the effect of total merge count from that of the adaptive per-layer budget. In Table 13, full ADAMERGE beats TOME by 0.33 pp while averaging 12.8 fewer merges (47.2 vs. 60.0). The authors add a 70.7-merge configuration to compare against the 68.8-merge budget-only variant, but matching, aggregation, and budgeting are still confounded.
Questions for authors
- Are the accuracy gains over strong baselines large enough to matter? Across the six tiers in Table 1, ADAMERGE improves over PiToMe by just 0.06–0.60 pp. The margin over TOME does widen with compression, but the gains over PiToMe stay limited and show no such trend. How do the authors argue that the proposed aggregation strategy and adaptive budget constitute a clear and substantive accuracy advance?
- How much of the accuracy gain comes from the adaptive per-layer budget alone? Table 13 shows full ADAMERGE ahead of TOME by 0.33 pp, with 12.8 fewer merges on average (47.2 vs. 60.0). The extra comparison (70.7 vs. 68.8 merges) addresses this in part, yet matching and aggregation still differ. Could the authors hold those rules fixed, match the total merge count, and compare fixed against adaptive per-layer budgets directly? That would isolate what adaptive budgeting contributes on its own.
- What design guidance does the perturbation bound actually provide? When matching and budget are fixed and only the merge weights change, the perturbation is halved, yet Top-1 ACC moves by at most 0.06 pp. A smaller perturbation therefore does not translate directly into accuracy. Which specific choices in ADAMERGE does the bound then support, and what exactly links the bound to the adaptive per-layer budget?
Response to Reviewer yC8J
Line numbers refer to the submitted PDF.
Both weaknesses are answered with the controls you specified. At matched compute there is no margin over PiToMe, and the adaptive budget contributes a depth profile that matters only at strong compression.
Q1. Are the gains over strong baselines large enough to matter?
Over PiToMe, no. With FLOPs matched exactly by interpolation and a paired bootstrap over 50,000 images, AdaMerge and PiToMe are within 0.03 pp at two tiers (CIs include 0) and PiToMe is 0.20 pp higher at the third (CI [−0.36, −0.04]); the submitted tiers were not compute-matched. Isolating the aggregation inside PiToMe gives the same reading: with official PiToMe matching and four merges per layer for both arms (15.205 G), PiToMe 84.930% vs with our aggregation 84.944% (p=0.80). Official ToMe is likewise 0.3–0.7 pp above AdaMerge at matched block FLOPs (GR-4).
The advance is analytical: a label-free per-merge cost, validated against the real block, and a controlled result on what accuracy responds to, now on three operators (GR-1, GR-2). The revised abstract no longer presents a margin over ToMe or PiToMe as a result.
Q2. How much of the gain comes from the adaptive per-layer budget alone?
At mild compression, nothing; at strong compression, +0.2 pp, and it comes from the depth profile, not from adaptivity. Matching and collision-safe aggregation fixed, 50k images:
The fixed profile read off the calibration matches the adaptive gate in both regimes. Table 13's confound (47.2 vs 60.0 merges) is removed and the table is replaced by this one.
To test depth allocation beyond our operator, we kept official ToMe unmodified and changed only its schedule:
The schedule alone moves accuracy by up to 1.17 pp. For ToMe the constant schedule is near-optimal: a label-free KL calibration matches it (n.s.), and our front-loaded profile is worse (−0.37, −1.17 pp). Depth choice is consequential in both operators; the profile that helps is operator-specific.
Q3. What design guidance does the bound provide?
It tells which average minimizes the local perturbation, and the measurements tell that this is not the lever. The proportional average lowers real [CLS] / logit perturbation by 6.9% / 3.5% (2,288 interventions; within-layer Spearman 0.71 / 0.58 between surrogate and real perturbation), yet accuracy moves by ≤0.02 pp in our operator and +0.014 pp in PiToMe. The guidance that survives is where to look: the cost is concentrated by depth, and depth allocation is what accuracy responds to. The link from Eq. 9 to that allocation is a calibrated rule, not a theorem, and §4.4 says so.
Presentation. §3–4 label each statement proved, measured or chosen; the real-block table and the two tables above are added.
If a row is missing from these tables that you would want in order to judge the budget's contribution, please name it and we will add it before the discussion closes.