acceptodds
Under review as a conference paper at ICLR 2027

Later Is Better: Token Reduction for ViTs Under Distribution Shift

Abstract

Training-free token reduction accelerates vision transformers by removing redundant tokens across layers, recovering most of the original accuracy at a fraction of the compute. These methods, however, are designed and evaluated primarily on clean data, and under real-world distribution shift their accuracy gap to the uncompressed model widens with the removal rate. We show that this gap is governed by the reduction *schedule*, the depth profile of removal, usually left fixed as an implementation detail. Concretely, we introduce a one-parameter *late-concentrated* power-law schedule that consistently improves out-of-distribution accuracy over flat at no extra inference cost. On ImageNet-C with DeiT-S, the late schedule closes % of that gap at a % compute reduction (pp), and % of it at a lighter % reduction (pp). The gain cannot be attributed to retaining more tokens or using extra compute: held to flat's compute, the late schedule removes more tokens in total and leaves fewer tokens at the end, yet still wins. Single-layer probes point to a mechanism: earlier reductions perturb features that pass through more remaining layers, front-loading reduction error in depth. The effect is broad, holding across five token-reduction methods (ToMe, EViT, ATS, ATC, PiToMe), nine backbones, all ImageNet-C corruption types, eight further shift suites, and two further modalities, video and vision-language QA. It is also specific to shift, still positive on clean and rising monotonically to 4 that at the highest severity 5. The schedule keeps its gain under six test-time adaptation methods, and needs no per-input or per-domain tuning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
token reduction, vision transformers, distribution shift, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SvodOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the depth allocation of token reduction affects vision-transformer robustness under distribution shift. It proposes a static power-law schedule that concentrates removal in later layers and calibrates its removal budget to use no more compute than a uniform-removal baseline. The authors improves mean accuracy over flat reduction, largely recovering the compression-induced accuracy gap while reducing compute relative to the uncompressed model.

Critical strengths
  1. Changing only the removal schedule produces improvements across multiple reduction operators, backbone families, and distribution shifts, without retraining or input-dependent decisions. The simple transferability of the approach is valuable.
  2. The paper goes beyond reporting aggregate accuracy. The advantage survives excluding the final one or two reduction layers and using global-average-pooling models, addressing the concern that it merely exploits a free final-layer merge. Alternative late-weighted shapes achieve similar gains, and the prediction decomposition shows that the improvement principally preserves predictions that the uncompressed model already gets correct. These experiments support a general depth-allocation effect.
Critical weaknesses
  1. Applying a common flat schedule to every operator is a useful controlled experiment, but it is not equivalent to comparing against each method’s original default. For example, original EViT reduces tokens at selected layers rather than removing a constant number at every layer. The paper should distinguish its training-free adaptation of the operator from the original method. Appendix Q's clean-searched DiffRate schedules outperform the proposed schedule on corrupted data. Thus, the evidence supports a convenient, search-free approximation to effective depth allocation more strongly than a distinctly shift-specific scheduling principle. A native-schedule baseline and a schedule selected exclusively on clean validation data would clarify the practical improvement beyond existing choices.

  2. The empirical decrease in output sensitivity with depth is convincing, but Appendix N does not establish that corruption increases individual downstream Jacobian norms: Table 22 measures the response of the composed network. Likewise, adding Jacobian factors does not generally increase a product’s norm without precise assumptions about contraction and directional alignment. The normalized, token-scaled perturbation probe also needs corresponding scaling factors in its operator-norm bound. Isotropic additive perturbations are useful diagnostics, but they do not directly isolate the propagation of structured merging or pruning errors. The paper should present amplification as a supported hypothesis, or provide additional measurements that separate local reduction error from downstream propagation. This matters because the mechanistic account is a stated contribution.

Questions for authors
  1. How does the late schedule compare with each operator’s native schedule at matched compute, particularly EViT’s selected-layer keep-rate schedule?
  2. Can you state precise assumptions and normalization for Lemma 1, and distinguish measurements of composed sensitivity from evidence about individual layer Jacobians?
  3. Appendix W measures two model widths at one operating point and batch size, yet concludes that every iso-GFLOPs comparison is also iso-latency. Can you restrict this claim to measured settings or provide representative measurements across operators and budgets, including the GPU model and precision?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

[Weakness 1-1] Native schedules and the training-free adaptation

Thank you for the careful reading of the operator comparison; we have now added the native profiles at matched compute (Table L), together with a schedule selected on clean data only. Our main evaluation holds the per-layer budget fixed across operators so that the only variable is the depth profile, and runs every operator training-free, the paper's scope; the revision states this in Section 4.3, so that the training-free adaptation is distinguished from each method as published. A native profile changes where and how much is removed per layer at once, so it is a different experiment.

Table L. Native depth profiles vs flat and late allocations on DeiT-S at matched GFLOPs; ImageNet-C severity 5 mean. Bold rows are the paper's rule.

reducerallocationdetailGFLOPsImageNet-C
ToMenativeconstant rr (= flat)3.2037.99
lateγ=2\gamma{=}2 power law3.1539.16
EViTnativekeep 0.8 at layers 4, 7, 103.2238.29
flatr=8r{=}8 at every layer3.2235.98
lateour variant, cuts moved one layer later (5, 8, 11)3.2238.41
lateγ=2\gamma{=}2 power law3.1738.50
PiToMenativeone ratio per layer (0.95)3.1637.36
flatr=8r{=}8 at every layer3.2037.67
lateγ=2\gamma{=}2 power law3.1537.90

EViT's default is itself later than flat, nothing is removed before layer 4, and it beats flat; moving its cuts earlier at the same compute costs accuracy and moving them later gains, so the direction holds inside EViT's own form, and the γ=2\gamma{=}2 power law edges the default at slightly less compute. PiToMe's default, one ratio per layer, is a geometric and therefore front-loaded count, and it trails both flat and the late schedule. ATS's default is a placement rather than a count. Its sampler keeps an input-dependent number of tokens under a bound that does not bind, so no fixed schedule corresponds to it; where the bound binds, the setting of our main evaluation, the late schedule beats flat at every budget. ATC's default is ToMe's constant rr, the flat arm of that evaluation, which the late schedule beats. In sum, under shift a late arm is the top arm of every group in Table L, and a native default beats flat only to the extent that it is itself late.

[Weakness 1-2] A schedule selected on clean data, and approximation vs principle

A schedule selected on clean data only is late as well. The full γ\gamma grid on clean ImageNet selects a late exponent at every budget, and under shift that exponent lands within 0.070.07pp of the shift-selected one, because the early-layer loss the late direction avoids is already present on clean inputs and shift inflates it without moving it. This is also why approximation and principle are not alternatives. The principle is that late allocation pays more the stronger the shift, as the amplification measured in Section 3 predicts, and the rule, the fixed power law with γ=2\gamma{=}2, is a search-free instance of it; DiffRate's search supports the principle, since its learned schedule is late-concentrated (Table M), on par with the rule on clean data, and ahead under shift only in the setting it was trained for. The revision describes the effect as amplified by shift rather than specific to it, and the practical improvement beyond existing choices is the late direction itself, which the rule delivers without a search.

Table M. Schedules on DiffRate's operator at matched GFLOPs. Tokens removed per layer, their early (1-4) and late (9-12) shares, and ImageNet-C severity-5 mean.

modelscheduletraining-freeGFLOPsremoved per layer, L1 to L12early (1-4)late (9-12)ImageNet-C
DeiT-Sflat✓3.119, 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 836%32%34.33
late γ=2\gamma{=}2 (ours)✓3.120, 1, 3, 5, 7, 11, 14, 19, 24, 30, 36, 425%69%36.30
DiffRate✕3.120, 2, 6, 12, 11, 12, 10, 12, 18, 5, 31, 7210%66%36.82
DeiT-Bflat✓12.978, 8, 8, 7, 7, 7, 7, 7, 7, 7, 7, 736%32%41.17
late γ=2\gamma{=}2 (ours)✓12.990, 1, 2, 4, 6, 9, 13, 17, 21, 26, 31, 374%69%43.45
DiffRate✕13.000, 1, 2, 2, 7, 8, 11, 11, 17, 49, 29, 553%78%43.59
AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Weakness 2-1] Lemma 1 restated, individual vs composed Jacobians, normalization

We appreciate the precise reading, restate Lemma 1 and add the measurements it calls for. Let f=h∘BL∘⋯∘Bℓ+1f = h \circ B_L \circ \dots \circ B_{\ell+1} map the output of block ℓ\ell to the pre-logit class token, with each BkB_k differentiable at the points visited and hh the readout. For a perturbation δ\delta at layer ℓ\ell, f(x+δ)−f(x)=Jℓ(x)δ+o(∥δ∥)f(x+\delta) - f(x) = J_\ell(x)\delta + o(\Vert\delta\Vert) with Jℓ(x)=Jh∏k=Lℓ+1JBkJ_\ell(x) = J_h \prod_{k=L}^{\ell+1} J_{B_k}; for isotropic δ\delta the expected gain is the root-mean-square singular value ∥Jℓ∥F/d\Vert J_\ell\Vert_F/\sqrt{d}, not the operator norm. Submultiplicativity gives ∥Jℓ∥op≤∥Jh∥∏k∥JBk∥op\Vert J_\ell\Vert_{\mathrm{op}} \le \Vert J_h\Vert\prod_k\Vert J_{B_k}\Vert_{\mathrm{op}}, an upper bound only, and the (1+ϵ)L−ℓ(1+\epsilon)^{L-\ell} sentence is a statement about this bound and is removed in the revision. The monotonicity of the gain in ℓ\ell is not guaranteed in general; it requires the per-block gains along the carried direction not to contract on average, which the new probe measures and the revision reports.

Table 22 (23 in the revision) measures the composed map; the new probe measures the individual block gains gk=∥JBkv∥/∥v∥g_k = \Vert J_{B_k} v\Vert/\Vert v\Vert along the direction vv carried from the merge, on clean and corrupted inputs. The per-block gains exceed one for 89-94% of the injection-block pairs, and their geometric mean along every injection path is above one (1.01-1.30) in every condition, so this condition holds in the measured network. Under corruption the change is in the composed gain of early errors, not a uniform rise of block norms. In Table N the composed gain of a layer-1 merge error rises under corruption on average while from layer 7 on it does not move.

On normalization, the isotropic probe perturbs each patch token by α∥xt∥ut\alpha\Vert x_t\Vert u_t, so A(ℓ)A(\ell) is a relative-to-relative gain and its bound is on JℓJ_\ell restricted to the patch tokens, scaled by ∥Xpatch∥/∥z∥\Vert X_{\mathrm{patch}}\Vert/\Vert z\Vert; the revision writes this out.

[Weakness 2-2] Structured merge errors, and the account as a hypothesis

Following the suggestion, we ran an additional probe that injects the actual ToMe merge perturbation at each layer and follows it to the output, and Table N separates the local merge error from its propagated effect. The local error varies little with depth while the propagated distortion falls about fifty-fold, so the front-loading is propagation, not the merge, and the structured gain exceeds the isotropic one in the first layers and falls below it later, so the merge error is most harmful where the late schedule removes the least. The revision presents the amplification account as a hypothesis these measurements support.

Table N. Merge-error probe on DeiT-S, 1024 images: a ToMe merge error injected at layer ℓ\ell and followed to the pre-logit. Gain = propagated distortion / local relative error; isotropic = an equal-norm isotropic perturbation; corr. mean = average over gaussian noise, defocus blur, fog and jpeg at severity 5.

ℓ\elllocal errorpropagatedgain, cleanisotropic gain, cleangain, corr. mean
10.110.1511.360.971.72
70.110.0160.150.370.16
110.080.0030.040.180.04

[Question 1] EViT's native schedule

At matched compute the γ=2\gamma{=}2 rule is ahead of every native profile, including EViT's selected-layer schedule, itself late and the closest to the rule (Table L); please see W1-1.


[Question 2] Lemma 1

Yes; please see W2-1 above.


[Question 3] Latency measured across reducers and budgets

We measured latency across the five reducers, three budgets and both backbones, with GPU model and precision stated (Table O), replacing Appendix W's single-point measurement.

Table O. Batched latency on one TITAN RTX (batch 64, fp32, median of 100 passes after 20 warmup); each cell is ms flat → late (ratio).

backbonereducerrbase=4r_{\mathrm{base}}{=}4rbase=8r_{\mathrm{base}}{=}8rbase=12r_{\mathrm{base}}{=}12
DeiT-SToMe64 → 64 (1.00)58 → 55 (0.96)47 → 48 (1.01)
EViT99 → 99 (1.00)94 → 89 (0.95)80 → 84 (1.06)
PiToMe106 → 107 (1.00)98 → 96 (0.98)90 → 88 (0.98)
ATS131 → 132 (1.01)127 → 129 (1.02)123 → 125 (1.01)
ATC785 → 793 (1.01)704 → 711 (1.01)626 → 646 (1.03)
DeiT-BToMe191 → 191 (1.00)166 → 166 (1.00)141 → 143 (1.01)
EViT256 → 257 (1.00)230 → 230 (1.00)203 → 205 (1.01)
PiToMe271 → 269 (0.99)243 → 240 (0.99)216 → 214 (0.99)
ATS250 → 254 (1.02)239 → 246 (1.03)226 → 235 (1.04)
ATC919 → 921 (1.00)820 → 825 (1.01)720 → 743 (1.03)

At equal GFLOPs the late schedule costs no latency. Across all 30 cells its latency is within −5% to +6% of flat's, with no consistent direction. The two schedules therefore have comparable latency in every measured setting, and the revision says so for these settings.

Reviewer_R3JQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies training-free token reduction for ViTs under distribution shift. It argues that the depth schedule of token removal governs the accuracy loss. It proposes a one-parameter power-law schedule that moves removal to later layers at equal FLOPs. On ImageNet-C with DeiT-S, this improves over the flat schedule by +1.17pp at a 26% compute cut. Gains hold across five reducers, nine backbones, other shift benchmarks, video and VLM.

Critical strengths
  1. The comparison is carefully compute-matched to the flat schedule. Under this constraint, the gain holds across reducers, backbones and all ImageNet-C corruption types. It is near-neutral only on a few video corruptions. It is a free and immediately usable improvement.
  2. The paper rules out several simple explanations. The late schedule actually removes more tokens than flat. The gain survives excluding the last layers and switching to GAP readout.
Critical weaknesses
  1. The claim that the effect is specific to shift is overstated. On clean data, the late schedule also recovers about half of the gap that flat opens (0.28 of ~0.5pp). Under shift the absolute gain is larger, but so is the gap to recover. No clean γ sweep shows that shift changes the optimal schedule.
  2. Prior work already favors deferred or late-heavy removal in some settings. DSM skips early blocks, and DiffRate's searched schedule is also late-heavy. ToMe's own search found late-heavy schedules best at low throughput. Yet the main experiments only compare against flat and a mirrored early schedule. DiffRate's schedule, searched on clean data, even beats the proposed one under shift (36.82 vs 36.30 on DeiT-S). The novelty over these schedules is therefore limited.
  3. The mechanistic explanation adds little. Early perturbations are amplified more simply because more layers follow them. The capacity argument for the interior optimum is assumed, not measured. The account also cannot predict the best γ.
Questions for authors
  1. What is the clean accuracy across the full γ sweep? Does the clean-optimal γ differ from the shift-optimal one?
  2. How does γ=2 compare with ToMe's searched schedule and a DSM-style delayed start at equal FLOPs, on clean and shifted data? For DiffRate, please add the clean comparison.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

[Weakness 1] Whether the effect is specific to shift

Thank you for raising this point. We agree that late allocation also benefits clean data, and clarify that its benefit is amplified by distribution shift rather than specific to it.

Distribution shift amplifies the benefit of late allocation. As Table F shows, the accuracy lost to flat reduction grows from 0.460.46 to 1.411.41pp with severity, and the fraction that late allocation recovers also rises steadily, from 61% to 83%, so the schedule recovers a larger fraction of a larger gap.

Table F. Late γ=2\gamma{=}2 vs flat at rbase=8r_{\mathrm{base}}{=}8 (DeiT-S, ToMe) by ImageNet-C severity (0 = clean), 15-corruption means, pp.

severity0 (clean)12345
accuracy lost to reduction0.460.720.901.091.301.41
recovered by late γ=2\gamma{=}20.280.520.660.831.061.17
fraction recovered61%72%73%76%82%83%

Shift inflates the amplification of early-layer perturbations (Section 3). As Table G summarizes, early A(ℓ)A(\ell) rises from 0.600.60 on clean inputs to 0.740.74-0.920.92 under corruption, while late A(ℓ)A(\ell) barely moves.

Table G. Amplification A(ℓ)A(\ell) of Section 3, mean over each layer range. DeiT-S, 1024 images, severity 5.

layerscleangauss. noisedefocus blurfogjpeg
early (1-4)0.600.920.740.830.77
mid (5-8)0.400.460.380.390.45
late (9-12)0.250.200.200.170.23

We also ran the full clean γ\gamma sweep (Table I). An exponent selected on clean data keeps the shifted gain within 0.070.07pp of the shift-selected one. A similar optimal γ\gamma on clean and shifted data is to be expected and does not conflict with the shift-amplified claim. The late schedule avoids an early-layer loss that is already present on clean data, and shift inflates that loss without moving it, so the benefit of the same schedule grows with shift. The revision states the claim this way.


[Weakness 2] What is new over DSM, ToMe and DiffRate

We appreciate the chance to clarify our novelty over these works. Our contribution is the controlled characterization of the depth axis under distribution shift at matched compute, across five reducers and nine backbones, showing that late allocation helps on clean data and that its benefit is amplified by shift, with a measured account of why and a rule that needs no search. DSM and DiffRate reach the late direction on clean data as a by-product, and ToMe's search notes it only at low throughput; none isolates the depth axis as the only variable or tests it under shift, and we have compared their schedules with ours at matched compute (Tables J and K).

DSM's delay skips only the first two blocks and is flat thereafter. ToMe's search ranks random schedules by clean accuracy per throughput on one backbone and adopts constant and decreasing. At matched compute, DSM and ToMe's increasing option trail our power-law γ=2\gamma{=}2 throughout Table J, and ToMe's decreasing schedule loses to flat. DiffRate learns its schedule with labels, inside its own operator and separately for each architecture and budget; there it edges out the fixed rule by 0.52pp on DeiT-S and 0.14pp on DeiT-B, at a cost the rule does not pay. These comparisons are in Appendix X of the revised manuscript.


[Weakness 3-1] What the mechanism adds beyond the block count

Thank you for pressing on what the mechanism adds; we answer with a new probe. The ordering itself, that an earlier perturbation passes through more blocks, follows from the block count, and Lemma 1 formalizes it. What the account adds, and what the schedule uses, is measured rather than assumed. A block count gives the ordering but not the size of the early-layer damage, nor its source. So we injected the actual ToMe merge error at each layer and followed it to the output (Table H). The error a merge introduces is of similar size at every layer, but by the time it reaches the output it is about fifty times larger when injected at layer 1 than at layer 11, so the damage of early removal comes from propagation, not from a larger merge. This steep profile, not the block count alone, is what late concentration exploits.

Table H. Merge-error probe on DeiT-S (1024 images). Local relative error and propagated output distortion of a ToMe merge error injected at layer ℓ\ell (clean inputs), and their ratio, the composed gain, on clean inputs and averaged over four severity-5 corruptions (gauss. noise, defocus blur, fog, jpeg).

ℓ\elllocal errorpropagatedgain(clean)gain(corr. mean)
10.110.1511.361.72
30.140.0720.510.70
70.110.0160.150.16
110.080.0030.040.04
AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Weakness 3-2] The shift dependence, the capacity hypothesis and the exponent

How much a perturbation at a given layer is amplified by the time it reaches the output also depends on the input, with the number of blocks fixed. In Table H the gain of an early-layer merge error rises under corruption on average while late-layer gains stay put, and the paper's isotropic probe (Table G) shows the same pattern; a block count alone cannot produce this, since the count is the same for every input, so the amplification itself must depend on the input, not on the count alone, and under our account this is what makes the schedule's benefit grow with shift.

On the interior optimum, we agree that the capacity argument is a hypothesis and label it as one in the revision; it is the simplest account consistent with the inverted U in γ\gamma, with its drift toward gentler concentration at larger budgets, and with the shape of DiffRate's searched schedule. The account predicts the ordering and the growth with shift, is consistent with but does not establish the interior optimum, and does not fix the exponent; the revision says so. The rule does not need the exponent predicted; it uses the direction and the existence of an interior optimum, and γ=2\gamma{=}2 sits on the plateau of Table I.


[Question 1] Clean accuracy across the γ\gamma sweep

We ran the full γ\gamma grid on clean ImageNet val (DeiT-S, ToMe, 50k images) and report it in Table I, with the ImageNet-C gain lost by selecting the exponent on clean data.

Table I. Clean ImageNet accuracy across the γ\gamma grid, with flat's accuracy and the other entries as Δ\Delta vs flat in pp (uncompressed 79.86), and the ImageNet-C gain lost by selecting the exponent on clean data.

rbaser_{\mathrm{base}}flatγ=0.5\gamma{=}0.5111.51.5222.52.533clean γ⋆\gamma^\starshift γ⋆\gamma^\starImageNet-C gain, clean γ⋆\gamma^\star −- shift γ⋆\gamma^\star
279.77+0.04+0.07+0.05+0.06+0.07+0.09330.00
479.75+0.00+0.05+0.05+0.08+0.10+0.072.52.50.00
679.57+0.14+0.17+0.24+0.19+0.11+0.191.52-0.07
879.39+0.24+0.28+0.33+0.29+0.31+0.3732-0.01
1079.26+0.25+0.26+0.31+0.35+0.15+0.0721.5-0.03
1279.14+0.08+0.13+0.08-0.14-0.37-0.3411.5-0.02

Selecting the exponent on clean data costs at most 0.070.07pp under shift; the clean-optimal and shift-optimal γ\gamma coincide at the smaller budgets and sit on the same plateau elsewhere, so in this setting the exponent can be chosen on clean data. The clean gain itself is small but real and grows with compression, which is the same sensitivity to early reduction that shift amplifies; the inverted U in γ\gamma and its drift toward gentler concentration at larger budgets are also already present on clean data, so they are properties of the allocation rather than of the shift.


[Question 2] ToMe's schedules, a DSM-style delay, and DiffRate on clean data at matched compute

ToMe does not release the schedules its search found, so we use the schedules it defines, constant (our flat) and decreasing, plus the increasing option its code provides, which realizes the late-heavy pattern its search observed, together with DSM's delayed start, all on ToMe at matched GFLOPs (Table J).

Table J. Schedules on ToMe at matched GFLOPs. ImageNet-C severity 5 mean in the first four columns and clean val top-1 in the last four. DSM merges nothing in the first two blocks.

scheduleImageNet-C, DeiT-S rbase=8r_{\mathrm{base}}{=}8DeiT-S rbase=4r_{\mathrm{base}}{=}4DeiT-B rbase=8r_{\mathrm{base}}{=}8DeiT-B rbase=4r_{\mathrm{base}}{=}4clean, DeiT-S rbase=8r_{\mathrm{base}}{=}8DeiT-S rbase=4r_{\mathrm{base}}{=}4DeiT-B rbase=8r_{\mathrm{base}}{=}8DeiT-B rbase=4r_{\mathrm{base}}{=}4
flat (ToMe default)37.9938.8344.3545.9879.3979.7581.3681.67
ToMe decreasing37.0038.5043.0045.4979.3179.6181.0981.57
ToMe increasing39.1139.3245.5946.4479.6679.7881.4381.83
DSM38.8939.2445.0046.3179.6179.7781.2981.74
late γ=2\gamma{=}2 (ours)39.1639.3745.8946.5079.6879.8281.5981.85

The decreasing schedule, the early direction, loses to flat in every setting on both shifted and clean data. Every schedule that defers removal beats flat under shift and matches or exceeds it on clean data, and γ=2\gamma{=}2 is ahead of every schedule in Table J, under shift and on clean data.

For DiffRate, the clean comparison in its own code at matched GFLOPs (Table K) puts the learned schedule and the fixed γ=2\gamma{=}2 at most 0.110.11pp apart on both backbones, so the closed-form rule is on par with the labeled search on clean data.

Table K. Clean ImageNet val top-1 on DiffRate's operator at matched GFLOPs.

scheduleDeiT-SDeiT-B
flat78.5780.87
DiffRate79.7781.85
late γ=2\gamma{=}2 (ours)79.7481.74
Reviewer_YenAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training-free token reduction for Vision Transformers (ViTs) under distribution shift. Instead of a flat schedule that removes the same number of tokens at every layer, it proposes a power-law schedule, r(ℓ)∝(ℓ/L)γr(\ell) \propto (\ell/L)^\gamma, that removes more tokens in later layers at matched compute (GFLOPs). With DeiT-S on ImageNet-C, it gains +1.17 percentage points over the flat schedule at a 26% compute reduction. The authors attribute this to early reductions being amplified as they pass through more downstream layers.

Critical strengths
  1. Compute is matched fairly in both GFLOPs and measured throughput.
  2. The gain holds when the final layers are excluded or GAP readout is used, and it comes from preserving the correct predictions of the unreduced model.
  3. The method is closed-form and training-free, so it is easy to apply.
Critical weaknesses
  1. It is unclear whether the effect is specific to distribution shift. On clean data, the late schedule already recovers about 60% of the accuracy lost by flat reduction, which is not far from 83% at severity 5. The "~4×" ratio seems to come mainly from the larger loss under corruption. Also, DiffRate [1], searched on clean ImageNet, beats γ=2\gamma=2 on ImageNet-C (Appendix Q). Could the authors explain why the gain should be seen as specific to distribution shift rather than a general reduction in compression error?

  2. Flat is the only schedule baseline. DiffRate [1] appears only in the appendix, with two backbones. Depth-dependent token allocation has also been studied in ToMe [2], DSM [3], ATM [4], and LTMP [5]. A comparison with these schedules at matched GFLOPs would make the contribution clearer.

  3. All backbones are isotropic ViTs. Hierarchical models such as Swin already reduce tokens across stages. Can the method be applied to them? If not, the claims should be limited to isotropic ViTs.

[1] M. Chen et al. DiffRate: Differentiable Compression Rate for Efficient Vision Transformers. ICCV, 2023. [2] D. Bolya et al. Token Merging: Your ViT but Faster. ICLR, 2023. [3] J. H. Heo et al. Training-Free Acceleration of ViTs with Delayed Spatial Merging. ES-FoMo Workshop at ICML, 2024. [4] J. Lee and D.-W. Choi. Lossless Token Merging Even Without Fine-Tuning in Vision Transformers. ECAI, 2025. [5] M. Bonnaerens and J. Dambre. Learned Thresholds Token Merging and Pruning for Vision Transformers. TMLR, 2023.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

[Weakness 1] Whether the gain is specific to shift

Thank you for raising this point. We agree that late allocation also benefits clean data, and clarify that its benefit is amplified by distribution shift rather than specific to it. The additional analyses and experimental evidence below support this clarification.

(i) Distribution shift amplifies the benefit of late allocation. As Table A shows, the accuracy lost to flat reduction grows from 0.460.46 to 1.411.41pp with severity, and the fraction that late allocation recovers also rises steadily, from 61% to 83%. The larger gain under shift is therefore not only a larger loss to recover; late allocation also recovers a larger fraction of that loss.

Table A. Late γ=2\gamma{=}2 vs flat at rbase=8r_{\mathrm{base}}{=}8 (DeiT-S, ToMe) by ImageNet-C severity (0 = clean val), 15-corruption means, pp.

severity0 (clean)12345
accuracy lost to reduction0.460.720.901.091.301.41
recovered by late γ=2\gamma{=}20.280.520.660.831.061.17
fraction recovered61%72%73%76%82%83%

We further support this with the mechanism measured in Section 3. Shift inflates the amplification of early-layer perturbations. As Table B summarizes, early A(ℓ)A(\ell) rises from 0.600.60 on clean inputs to 0.740.74-0.920.92 under corruption, while late A(ℓ)A(\ell) barely moves. This matches Table A, where avoiding early reduction pays more the stronger the shift.

Table B. Amplification A(ℓ)A(\ell) of Section 3, mean over the layers in each range. DeiT-S, 1024 images, corruptions at severity 5.

layerscleangauss. noisedefocus blurfogjpeg
early (1-4)0.600.920.740.830.77
mid (5-8)0.400.460.380.390.45
late (9-12)0.250.200.200.170.23

(ii) DiffRate's result is consistent with this account. DiffRate also learns a late-concentrated schedule on clean ImageNet (Table C), which our account expects, since early-layer amplification already exists on clean data. As shift inflates it, the same late allocation pays more under shift, and a clean-searched schedule carries over to ImageNet-C. In its own setting the search edges out our fixed γ=2\gamma{=}2 by a fraction of a point at severity 5, through a finer split of the same late budget fitted to its operator; moved to another backbone or operator at matched compute, the learned profile trails the rule (Appendix Q).

Table C. Schedules on DiffRate's operator at matched GFLOPs. Tokens removed per layer, their early (1-4) and late (9-12) shares, and ImageNet-C severity-5 mean.

modelscheduletraining-freeGFLOPsremoved per layer, L1 to L12early (1-4)late (9-12)ImageNet-C
DeiT-Sflat✓3.119, 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 836%32%34.33
late γ=2\gamma{=}2 (ours)✓3.120, 1, 3, 5, 7, 11, 14, 19, 24, 30, 36, 425%69%36.30
DiffRate✕3.120, 2, 6, 12, 11, 12, 10, 12, 18, 5, 31, 7210%66%36.82
DeiT-Bflat✓12.978, 8, 8, 7, 7, 7, 7, 7, 7, 7, 7, 736%32%41.17
late γ=2\gamma{=}2 (ours)✓12.990, 1, 2, 4, 6, 9, 13, 17, 21, 26, 31, 374%69%43.45
DiffRate✕13.000, 1, 2, 2, 7, 8, 11, 11, 17, 49, 29, 553%78%43.59

DiffRate can outperform our fixed γ=2\gamma{=}2 schedule in its own optimized setting, but it requires a supervised, setting-specific schedule search. In contrast, our contribution is to show that the benefit of late allocation grows under distribution shift, with a measured mechanism, and to provide a fixed, training-free rule that is competitive without schedule search across diverse reducers and backbones. The revised manuscript reflects this feedback, describing the effect as amplified by shift.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Weakness 2] The other schedules at matched compute

We appreciate the suggestion and ran the additional schedule baselines. We compare the three schedules ToMe provides (its default constant schedule, which is our flat, its decreasing schedule and the increasing option in its code), DSM, whose depth component is a fixed delay, and the allocations of DiffRate, ATM and LTMP, all on ToMe at matched GFLOPs, on DeiT-S and DeiT-B at two budgets (Table D); ATM and LTMP, which allocate dynamically per input, are also compared as published inside their own operators (Table E). Note that our original experiments were scoped to training-free reduction, whereas DiffRate and LTMP learn their schedules with labels and ATM allocates by input-adaptive thresholds.

Table D. Schedules on ToMe at matched GFLOPs, ImageNet-C severity 5 mean. late (9-12) is the share of removal in layers 9-12 on DeiT-S, a range for the input-adaptive ATM and LTMP, which enter as their mean allocation. DSM merges nothing in the first two blocks. Bold identifies our schedule.

schedulelate (9-12)DeiT-S, rbase=8r_{\mathrm{base}}{=}8DeiT-S, rbase=4r_{\mathrm{base}}{=}4DeiT-B, rbase=8r_{\mathrm{base}}{=}8DeiT-B, rbase=4r_{\mathrm{base}}{=}4
flat (ToMe default)33%37.9938.8344.3545.98
ToMe decreasing9%37.0038.5043.0045.49
ToMe increasing58%39.1139.3245.5946.44
DSM40%38.8939.2445.0046.31
DiffRate66%38.9339.3045.9146.55
ATM64-85%39.1539.4245.7746.00
LTMP67-69%39.1939.4145.8246.62
late γ=2\gamma{=}2 (ours)69%39.1639.3745.8946.50

The decreasing schedule, the early direction, loses to flat in every setting, and every schedule that defers removal beats flat in all four settings. The learned or adaptive schedules are also predominantly late-concentrated and achieve accuracy comparable to our fixed rule. In the tested settings, their maximum advantage over our rule is only 0.12pp, despite requiring supervised optimization or input-adaptive allocation.

Table E. Inside each operator, its adaptive allocation as published vs a fixed late γ=2\gamma{=}2 count at matched mean GFLOPs; ImageNet-C severity 5 mean. Bold identifies our schedule.

operatorallocationDeiT-S, rbase=8r_{\mathrm{base}}{=}8DeiT-S, rbase=4r_{\mathrm{base}}{=}4DeiT-B, rbase=8r_{\mathrm{base}}{=}8DeiT-B, rbase=4r_{\mathrm{base}}{=}4
ATMadaptive39.0539.3545.4546.00
late γ=2\gamma{=}2 (ours)39.1739.3645.9546.57
LTMPadaptive39.1839.4045.6246.64
late γ=2\gamma{=}2 (ours)39.6239.4746.6646.71

Even with each method's own operator kept intact, the fixed late schedule beats its dynamic allocation in all eight settings, and it does so with no training or tuning, where LTMP's thresholds are learned with labels and ATM's are hand-tuned. Tables D and E are in Appendix X of the revised manuscript and summarized in Section 4.1; the related work already notes the depth-dependent allocation of these methods.


[Weakness 3] Stating the claims for isotropic ViTs

Thank you for the question on hierarchical backbones; our claims are for isotropic ViTs, and the revised manuscript states this scope explicitly. The paper studies where across depth to remove tokens. In an isotropic ViT this choice is open because any token can be removed at any layer, whereas in a hierarchical model it is largely built into the trained downsampling between stages. Whether late allocation is also a useful principle for the placement of downsampling in a hierarchical backbone is a question of architecture design, which we leave to future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback.

We have run (i) compute-matched comparisons with all the schedules the reviewers raised, from native defaults to learned allocations; (ii) a clean-data sweep of the exponent; (iii) a probe that follows the actual merge error from each layer to the output; and (iv) latency measurements across reducers and budgets.

These results sharpen our contribution: the depth axis of token reduction, where along the network tokens are removed, is characterized in a controlled way under distribution shift at matched compute, across five reducers and nine backbones. Late allocation already helps on clean data, and we provide a comprehensive analysis of how its advantage grows with shift. A measured early-layer amplification supports this direction, and a fixed γ=2\gamma{=}2 power law captures the benefit without labels, search or training. We state the effect as amplified by shift rather than specific to it, present the amplification account as a hypothesis supported by the new measurements, and state the claims for isotropic ViTs.

Complete responses follow under each review, with Tables A-O numbered in one sequence and the new material added to the revised manuscript, where all changes are marked in blue.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewers,

Thank you again for your thoughtful and constructive feedback.

As the discussion period is nearing its end, we wanted to gently remind you of our rebuttal responses in case you haven't had a chance to revisit them.

We would be glad to address any remaining concerns or questions you may have.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.