- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how the depth allocation of token reduction affects vision-transformer robustness under distribution shift. It proposes a static power-law schedule that concentrates removal in later layers and calibrates its removal budget to use no more compute than a uniform-removal baseline. The authors improves mean accuracy over flat reduction, largely recovering the compression-induced accuracy gap while reducing compute relative to the uncompressed model.
Critical strengths
- Changing only the removal schedule produces improvements across multiple reduction operators, backbone families, and distribution shifts, without retraining or input-dependent decisions. The simple transferability of the approach is valuable.
- The paper goes beyond reporting aggregate accuracy. The advantage survives excluding the final one or two reduction layers and using global-average-pooling models, addressing the concern that it merely exploits a free final-layer merge. Alternative late-weighted shapes achieve similar gains, and the prediction decomposition shows that the improvement principally preserves predictions that the uncompressed model already gets correct. These experiments support a general depth-allocation effect.
Critical weaknesses
-
Applying a common flat schedule to every operator is a useful controlled experiment, but it is not equivalent to comparing against each method’s original default. For example, original EViT reduces tokens at selected layers rather than removing a constant number at every layer. The paper should distinguish its training-free adaptation of the operator from the original method. Appendix Q's clean-searched DiffRate schedules outperform the proposed schedule on corrupted data. Thus, the evidence supports a convenient, search-free approximation to effective depth allocation more strongly than a distinctly shift-specific scheduling principle. A native-schedule baseline and a schedule selected exclusively on clean validation data would clarify the practical improvement beyond existing choices.
-
The empirical decrease in output sensitivity with depth is convincing, but Appendix N does not establish that corruption increases individual downstream Jacobian norms: Table 22 measures the response of the composed network. Likewise, adding Jacobian factors does not generally increase a product’s norm without precise assumptions about contraction and directional alignment. The normalized, token-scaled perturbation probe also needs corresponding scaling factors in its operator-norm bound. Isotropic additive perturbations are useful diagnostics, but they do not directly isolate the propagation of structured merging or pruning errors. The paper should present amplification as a supported hypothesis, or provide additional measurements that separate local reduction error from downstream propagation. This matters because the mechanistic account is a stated contribution.
Questions for authors
- How does the late schedule compare with each operator’s native schedule at matched compute, particularly EViT’s selected-layer keep-rate schedule?
- Can you state precise assumptions and normalization for Lemma 1, and distinguish measurements of composed sensitivity from evidence about individual layer Jacobians?
- Appendix W measures two model widths at one operating point and batch size, yet concludes that every iso-GFLOPs comparison is also iso-latency. Can you restrict this claim to measured settings or provide representative measurements across operators and budgets, including the GPU model and precision?
2 comments
Official Comment by Authors (2/2)
[Weakness 2-1] Lemma 1 restated, individual vs composed Jacobians, normalization
We appreciate the precise reading, restate Lemma 1 and add the measurements it calls for. Let map the output of block to the pre-logit class token, with each differentiable at the points visited and the readout. For a perturbation at layer , with ; for isotropic the expected gain is the root-mean-square singular value , not the operator norm. Submultiplicativity gives , an upper bound only, and the sentence is a statement about this bound and is removed in the revision. The monotonicity of the gain in is not guaranteed in general; it requires the per-block gains along the carried direction not to contract on average, which the new probe measures and the revision reports.
Table 22 (23 in the revision) measures the composed map; the new probe measures the individual block gains along the direction carried from the merge, on clean and corrupted inputs. The per-block gains exceed one for 89-94% of the injection-block pairs, and their geometric mean along every injection path is above one (1.01-1.30) in every condition, so this condition holds in the measured network. Under corruption the change is in the composed gain of early errors, not a uniform rise of block norms. In Table N the composed gain of a layer-1 merge error rises under corruption on average while from layer 7 on it does not move.
On normalization, the isotropic probe perturbs each patch token by , so is a relative-to-relative gain and its bound is on restricted to the patch tokens, scaled by ; the revision writes this out.
[Weakness 2-2] Structured merge errors, and the account as a hypothesis
Following the suggestion, we ran an additional probe that injects the actual ToMe merge perturbation at each layer and follows it to the output, and Table N separates the local merge error from its propagated effect. The local error varies little with depth while the propagated distortion falls about fifty-fold, so the front-loading is propagation, not the merge, and the structured gain exceeds the isotropic one in the first layers and falls below it later, so the merge error is most harmful where the late schedule removes the least. The revision presents the amplification account as a hypothesis these measurements support.
Table N. Merge-error probe on DeiT-S, 1024 images: a ToMe merge error injected at layer and followed to the pre-logit. Gain = propagated distortion / local relative error; isotropic = an equal-norm isotropic perturbation; corr. mean = average over gaussian noise, defocus blur, fog and jpeg at severity 5.
| local error | propagated | gain, clean | isotropic gain, clean | gain, corr. mean | |
|---|---|---|---|---|---|
| 1 | 0.11 | 0.151 | 1.36 | 0.97 | 1.72 |
| 7 | 0.11 | 0.016 | 0.15 | 0.37 | 0.16 |
| 11 | 0.08 | 0.003 | 0.04 | 0.18 | 0.04 |
[Question 1] EViT's native schedule
At matched compute the rule is ahead of every native profile, including EViT's selected-layer schedule, itself late and the closest to the rule (Table L); please see W1-1.
[Question 2] Lemma 1
Yes; please see W2-1 above.
[Question 3] Latency measured across reducers and budgets
We measured latency across the five reducers, three budgets and both backbones, with GPU model and precision stated (Table O), replacing Appendix W's single-point measurement.
Table O. Batched latency on one TITAN RTX (batch 64, fp32, median of 100 passes after 20 warmup); each cell is ms flat → late (ratio).
| backbone | reducer | |||
|---|---|---|---|---|
| DeiT-S | ToMe | 64 → 64 (1.00) | 58 → 55 (0.96) | 47 → 48 (1.01) |
| EViT | 99 → 99 (1.00) | 94 → 89 (0.95) | 80 → 84 (1.06) | |
| PiToMe | 106 → 107 (1.00) | 98 → 96 (0.98) | 90 → 88 (0.98) | |
| ATS | 131 → 132 (1.01) | 127 → 129 (1.02) | 123 → 125 (1.01) | |
| ATC | 785 → 793 (1.01) | 704 → 711 (1.01) | 626 → 646 (1.03) | |
| DeiT-B | ToMe | 191 → 191 (1.00) | 166 → 166 (1.00) | 141 → 143 (1.01) |
| EViT | 256 → 257 (1.00) | 230 → 230 (1.00) | 203 → 205 (1.01) | |
| PiToMe | 271 → 269 (0.99) | 243 → 240 (0.99) | 216 → 214 (0.99) | |
| ATS | 250 → 254 (1.02) | 239 → 246 (1.03) | 226 → 235 (1.04) | |
| ATC | 919 → 921 (1.00) | 820 → 825 (1.01) | 720 → 743 (1.03) |
At equal GFLOPs the late schedule costs no latency. Across all 30 cells its latency is within −5% to +6% of flat's, with no consistent direction. The two schedules therefore have comparable latency in every measured setting, and the revision says so for these settings.
Official Comment by Authors (1/2)
[Weakness 1-1] Native schedules and the training-free adaptation
Thank you for the careful reading of the operator comparison; we have now added the native profiles at matched compute (Table L), together with a schedule selected on clean data only. Our main evaluation holds the per-layer budget fixed across operators so that the only variable is the depth profile, and runs every operator training-free, the paper's scope; the revision states this in Section 4.3, so that the training-free adaptation is distinguished from each method as published. A native profile changes where and how much is removed per layer at once, so it is a different experiment.
Table L. Native depth profiles vs flat and late allocations on DeiT-S at matched GFLOPs; ImageNet-C severity 5 mean. Bold rows are the paper's rule.
EViT's default is itself later than flat, nothing is removed before layer 4, and it beats flat; moving its cuts earlier at the same compute costs accuracy and moving them later gains, so the direction holds inside EViT's own form, and the γ=2 power law edges the default at slightly less compute. PiToMe's default, one ratio per layer, is a geometric and therefore front-loaded count, and it trails both flat and the late schedule. ATS's default is a placement rather than a count. Its sampler keeps an input-dependent number of tokens under a bound that does not bind, so no fixed schedule corresponds to it; where the bound binds, the setting of our main evaluation, the late schedule beats flat at every budget. ATC's default is ToMe's constant r, the flat arm of that evaluation, which the late schedule beats. In sum, under shift a late arm is the top arm of every group in Table L, and a native default beats flat only to the extent that it is itself late.
[Weakness 1-2] A schedule selected on clean data, and approximation vs principle
A schedule selected on clean data only is late as well. The full γ grid on clean ImageNet selects a late exponent at every budget, and under shift that exponent lands within 0.07pp of the shift-selected one, because the early-layer loss the late direction avoids is already present on clean inputs and shift inflates it without moving it. This is also why approximation and principle are not alternatives. The principle is that late allocation pays more the stronger the shift, as the amplification measured in Section 3 predicts, and the rule, the fixed power law with γ=2, is a search-free instance of it; DiffRate's search supports the principle, since its learned schedule is late-concentrated (Table M), on par with the rule on clean data, and ahead under shift only in the setting it was trained for. The revision describes the effect as amplified by shift rather than specific to it, and the practical improvement beyond existing choices is the late direction itself, which the rule delivers without a search.
Table M. Schedules on DiffRate's operator at matched GFLOPs. Tokens removed per layer, their early (1-4) and late (9-12) shares, and ImageNet-C severity-5 mean.