acceptodds
Under review as a conference paper at ICLR 2027

DORA: Dynamic Online Reinforcement Agent for Token Pruning in Vision Transformers

Abstract

Vision Transformers (ViTs) incur quadratic self-attention cost in the number of tokens. Most token-reduction methods adapt token identities within a prescribed layer-wise compression schedule, or search a static mask offline, and thus limit online adaptation of when and how much to prune. We propose DORA (Dynamic Online Reinforcement Agent), which learns an input-adaptive pruning policy itself for frozen ViTs. At each eligible block, a hierarchical actor decides whether to prune, how many tokens to remove, and which tokens to remove from each image's evolving representation. Because early deletions change the states observed by later decisions, DORA formulates pruning as a finite-horizon Markov decision process. Complete-prefix shadow evaluations convert final-prediction fidelity into localized per-step credit, while closed-loop accuracy feedback adjusts the fidelity penalty toward a shared accuracy-drop target. A privileged critic and all shadow computations are training-only. Deployment retains the frozen backbone and a lightweight actor that applies hard deletion and packed variable-length FlashAttention, converting token reduction into measured speedups. On ImageNet-1K with DeiT-Base, DORA reduces FLOPs by 38.4% relative to the uncompressed backbone within one percentage point of accuracy loss. Averaged across four ViT-type backbones at matched accuracy, DORA uses 13.2% fewer FLOPs and achieves 32.4% higher throughput than the corresponding per-backbone baseline means. Under zero-shot transfer to ImageNet-A, these gains widen to 20.3% and 45.6%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
token pruning, vision transformers, reinforcement learning, input-adaptive inference, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9U6xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DORA proposes a reinforcement learning framework for token pruning in frozen vision transformers. A hierarchical actor determines whether to prune, how many tokens to remove, and which tokens to select. Complete-prefix shadow evaluations provide localized fidelity rewards, while accuracy feedback adjusts the fidelity penalty during training. Deployment uses hard token deletion and packed variable-length FlashAttention. Experiments on four backbones demonstrate improved accuracy–efficiency trade-offs on ImageNet-1K and ImageNet-A, with additional analyses of token selection, reward design, feedback behavior, and training stability.

Critical strengths
  1. The hierarchical policy provides a coherent approach to jointly learning pruning locations, budgets, and token identities. Complete-prefix shadow evaluations offer a concrete mechanism for assigning local rewards while preserving the overall objective, and the terminal-only reward ablation supports their effectiveness.

  2. The framework achieves useful reductions in computation with measured throughput gains across four frozen backbones. Evaluation includes online decision and compaction costs, unchanged policy transfer to ImageNet-A, and repeated training runs, providing credible evidence of overall effectiveness and stability.

Critical weaknesses
  1. The paper lacks ablations of input-dependent pruning locations and budgets. Figure 3 shows largely fixed intervention locations, while the selector study keeps gates and budgets unchanged. A comparison with a learned shared schedule using the same selector and execution backend is needed to establish the benefit of joint adaptivity.

  2. Despite substantial FLOPs reductions, DORA delivers only marginal additional acceleration over V-Pruner on ImageNet-1K: 0.7% on DeiT-B and 0.2% on ViT-B. These results provide limited support for the practical advantage of its more elaborate online pruning policy on these backbones.

Questions for authors
  1. What evidence shows that input-dependent gates and budgets improve upon a learned shared schedule when the selector and execution backend remain unchanged?

  2. Which operations account for the gap between FLOPs reductions and throughput improvements over V-Pruner on DeiT-B and ViT-B?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9U6x - Part 1

W1 & Q1: Learned Intervention Locations and Input-Adaptive Budgets

Response.

To isolate the contribution of per-image scheduling, we conducted a static shared-schedule control derived from the learned DORA policy (Section 4.4, Table 2; Appendix F.4). We retain the learned intervention blocks, selector parameters, and packed variable-length FlashAttention backend, while replacing per-image gate and budget decisions with fixed per-block deletion budgets. Each budget is computed from the mean deletion count on the corresponding evaluation dataset, counting gate-off decisions as zero and rounding to the nearest feasible budget-grid value. No additional training is performed.

We also conducted gate-placement ablations (Appendix F.5), evaluating cross-backbone placement transfer and uniformly spaced placements under approximately matched GFLOPs while retaining the online budget head and learned selector.

Key results.

DatasetBackboneTop-1 (%)GFLOPsThroughput (images/s)
ImageNet-1KDeiT-B80.5 (−0.3)21.8 (+0.1)4,473 (+13.4%)
ImageNet-1KViT-B83.6 (−0.2)23.7 (+0.1)4,193 (+12.6%)
ImageNet-1KViT-L84.6 (−0.5)74.7 (−0.2)1,485 (+6.5%)
ImageNet-1KDINOv2-B83.4 (−0.3)30.4 (+0.1)2,674 (+8.4%)
ImageNet-ADeiT-B26.2 (−0.5)20.8 (−0.2)4,831 (+14.1%)
ImageNet-AViT-B42.9 (−0.3)23.1 (−0.1)4,348 (+13.5%)
ImageNet-AViT-L53.7 (−1.0)75.3 (+1.0)1,535 (+5.4%)
ImageNet-ADINOv2-B55.4 (−0.6)32.5 (0.0)2,540 (+6.3%)

Parentheses indicate absolute Top-1 changes in percentage points, absolute GFLOPs changes, and relative throughput changes versus dynamic DORA.

Relative to dynamic DORA, static scheduling reduces Top-1 accuracy by 0.2–0.5 percentage points on ImageNet-1K and 0.3–1.0 points on ImageNet-A, while GFLOPs differ by at most 0.2 and 1.0, respectively. Although static scheduling is 5.4%–14.1% faster, dynamic DORA consistently achieves higher accuracy at comparable computational cost, with larger advantages under distribution shift.

We acknowledge that the learned gates are largely fixed across images and that per-image scheduling adaptivity primarily arises from the budget head. Nevertheless, the placement ablations show that, under approximately matched GFLOPs, cross-backbone and uniformly spaced placements reduce Top-1 accuracy by 0.4–1.6 percentage points on ImageNet-1K and 0.5–3.0 points on ImageNet-A.

Together, these results support two distinct contributions: learning backbone-appropriate intervention locations and adapting pruning budgets to individual inputs.

AuthorsOpenReview ↗

Response to Reviewer 9U6x - Part 2

W2 & Q2: FLOPs Reduction Versus Throughput Improvement over V-Pruner

Response.

We agree that the additional speedup over V-Pruner is small on these two backbones and that FLOPs reductions alone do not determine end-to-end runtime. The key difference lies in the inference execution paths. V-Pruner applies a fixed mask once before the first block, whereas DORA makes online gate, budget, and selector decisions at multiple blocks, followed by dynamic token indexing, packing, and compaction.

To identify the associated overhead, we performed low-overhead phase-level profiling on DeiT-B and ViT-B using the main-table configurations, batch size 320, and the FlashAttention backend. The profile separates backbone execution, actor computation, and indexing/control overhead, and includes host-side work on the critical path (See Appendix F.8).

Key results.

At matched accuracy, the ImageNet-1K results in Table 1 show:

BackboneV-Pruner GFLOPsDORA GFLOPsFLOPs ReductionV-Pruner ThroughputDORA ThroughputThroughput Gain
DeiT-B28.321.723.3%3,9173,944+0.7%
ViT-B29.923.621.1%3,7193,725+0.2%

Throughput is reported in images/s. The phase-level profile, reported in milliseconds per batch of 320 images, is as follows:

OperationDeiT-B ΔLatencyViT-B ΔLatency
Backbone stage−15.34 ms−13.51 ms
Actor stage+5.63 ms+5.73 ms
Indexing and control stage+7.43 ms+7.55 ms
Total latency−2.28 ms-0.23 ms

Relative to V-Pruner, DORA reduces backbone-stage latency by 15.34 ms/batch on DeiT-B and 13.51 ms/batch on ViT-B. However, actor computation and additional indexing/control operations introduce approximately 13.06 and 13.28 ms/batch of additional overhead, respectively, offsetting most of these savings.

DORA's online path requires repeated controller execution, token gathering and compaction, dynamic sequence-length metadata handling, additional kernel launches, and CPU/GPU synchronization. In the profiled implementations, DORA launches approximately 1,203 kernels per batch, compared with 128 for V-Pruner. These operations incur execution and memory-management costs that are not captured by FLOPs.

These results explain why DORA's substantial reduction in backbone computation produces only modest throughput improvements over V-Pruner. The profiling results are diagnostic measurements rather than a direct decomposition of the main-table timings.

We hope these controls and profiling results clarify the distinct contributions of learned intervention locations and input-adaptive budgeting, as well as the systems overhead that limits the conversion of FLOPs reductions into measured throughput gains.

Thank you for the detailed rebuttal and additional experiments. The static-schedule control in Table 2 addresses my main concern about input-adaptive pruning budgets. While the static variant achieves higher throughput, DORA consistently improves accuracy retention at comparable FLOPs, by 0.2–0.5 percentage points on ImageNet-1K and 0.3–1.0 points on ImageNet-A. Since the static budgets are derived from DORA's decisions without separate optimization, this is not a fully independently optimized static-policy baseline, but the consistent results across all eight settings are useful evidence for the value of per-image adaptation. The gate-placement ablations in Appendix F.5 further clarify the contribution of learned intervention locations.

The phase-level profiling in Appendix F.8 explains the limited throughput gains over V-Pruner on DeiT-B and ViT-B: actor and indexing/control overhead of approximately 13 ms per batch offsets most of the 13.5–15.3 ms saved in backbone execution, with roughly 1,203 kernel launches for DORA versus 128 for V-Pruner. The static variant's higher throughput at slightly lower accuracy further illustrates the runtime cost of per-image scheduling, although it is not an accuracy-matched comparison with V-Pruner.

Overall, the additional experiments adequately address my original concerns. The modest acceleration over the strongest baseline on some backbones remains a practical limitation, but it is now better characterized. I will maintain my original score.

Reviewer_q2AqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DORA addresses efficient inference for frozen Vision Transformers by learning when, how much and which tokens to prune for each input. A hierarchical gate–budget–selector policy is trained through reinforcement learning. Complete-prefix shadow evaluations provide incremental prediction-fidelity rewards, while an accuracy-feedback controller adjusts the fidelity penalty. Deployment retains the frozen backbone and lightweight actor, using physical token deletion with packed variable-length FlashAttention to convert compression into measured throughput gains. Experiments cover four backbones on ImageNet-1K and zero-shot transfer to ImageNet-A. The paper reports a 38.4% FLOPs reduction on DeiT-B within a one-percentage-point accuracy loss, supported by selector, reward and feedback ablations.

Critical strengths
  1. The reward decomposition is sound.
  2. Deployment overhead is included, and experiments cover four backbones and distribution shift.
  3. ImageNet-A results show useful gains over strong matched baselines.
Critical weaknesses
  1. Direct comparisons with LTMP, ATS and frozen-backbone RL4EViT are missing.
  2. Fixed-gate/fixed-budget controls do not establish the distinctive benefit of dynamic scheduling.
  3. Throughput gains over the fastest ImageNet baseline are below 1% on two backbones, without timing uncertainty.
  4. Training cost and small-batch performance remain unclear.
Questions for authors
  1. How much improvement remains with fixed gates and budgets but the same selector and packed implementation?
  2. How does DORA compare with direct adaptive baselines under the same protocol?
  3. Can the authors provide complete training costs, timing variability and small-batch latency?
3 comments
AuthorsOpenReview ↗

Response to Reviewer q2Aq - Part 1

W1 & Q2: Direct comparisons with adaptive token-compression baselines

Response.

In response to your suggestion to include ATS, LTMP, and frozen-backbone RL4EViT, we have added ATS and LTMP to our evaluation and updated Table 1 and Figures 1 and 5 accordingly. For ATS, we follow the original attention-score-based token-sampling procedure. For LTMP, we use the authors' released implementation and train the merging and pruning thresholds for one epoch on the ImageNet-1K training set. Our local reproduction of RL4EViT now trains under our frozen-backbone evaluation protocol. However, training and evaluation across all four backbones have not been completed yet. We therefore do not report partial results in the current revision. We will include the complete accuracy-matched comparison once all runs are finished.

ATS uses FlashAttention by default, switching to dense attention only in blocks that require explicit attention probabilities. LTMP retains dense attention because its compression operations rely on attention information. Accordingly, their reported results are obtained using padded dense-attention execution in blocks where attention probabilities are required, and computations on padded positions are included in the GFLOPs accounting.

Key results.

BackboneMethodImageNet-1K Top-1 (%)ImageNet-1K GFLOPsImageNet-1K Throughput (images/s)ImageNet-A Top-1 (%)ImageNet-A GFLOPsImageNet-A Throughput (images/s)
DeiT-BATS80.824.91,83826.821.22,406
DeiT-BLTMP80.828.51,93326.729.61,846
DeiT-BDORA80.821.73,94426.721.04,233
ViT-BATS83.728.41,95743.223.42,032
ViT-BLTMP83.928.91,89143.328.51,911
ViT-BDORA83.823.63,72543.223.23,831
ViT-LATS85.0103.777554.778.3757
ViT-LLTMP85.196.767854.696.7679
ViT-LDORA85.174.91,39454.774.31,456
DINOv2-BATS83.643.11,06956.033.01,203
DINOv2-BLTMP83.639.090755.935.11,019
DINOv2-BDORA83.730.32,46656.032.52,389

Relative to these two newly added baselines, DORA reduces GFLOPs by 12.9%–29.7% on ImageNet-1K and by 0.9%–29.1% on ImageNet-A, while improving throughput by 79.9%–171.9% and 75.9%–134.4%, respectively. These results demonstrate that, under the common evaluation protocol, DORA achieves a better end-to-end accuracy–efficiency trade-off than the newly added adaptive token-compression baselines.

W2 & Q1: Fixed-gate/fixed-budget control with the same selector and backend

Response.

We conducted the requested static-schedule control (Section 4.4, Table 2; Appendix F.4).

We retain the learned intervention blocks, selector parameters, and packed variable-length FlashAttention backend, while replacing per-image gate and budget decisions with fixed per-block deletion budgets. Each budget is computed from the mean deletion count on the corresponding evaluation dataset, counting gate-off decisions as zero and rounding to the nearest feasible budget-grid value. No additional training is performed.

Key results.

DatasetBackboneTop-1 (%)GFLOPsThroughput (img/s)
ImageNet-1KDeiT-B80.5 (−0.3)21.8 (+0.1)4,473 (+13.4%)
ImageNet-1KViT-B83.6 (−0.2)23.7 (+0.1)4,193 (+12.6%)
ImageNet-1KViT-L84.6 (−0.5)74.7 (−0.2)1,485 (+6.5%)
ImageNet-1KDINOv2-B83.4 (−0.3)30.4 (+0.1)2,674 (+8.4%)
ImageNet-ADeiT-B26.2 (−0.5)20.8 (−0.2)4,831 (+14.1%)
ImageNet-AViT-B42.9 (−0.3)23.1 (−0.1)4,348 (+13.5%)
ImageNet-AViT-L53.7 (−1.0)75.3 (+1.0)1,535 (+5.4%)
ImageNet-ADINOv2-B55.4 (−0.6)32.5 (0.0)2,540 (+6.3%)

Static scheduling reduces Top-1 accuracy on every backbone, by 0.2--0.5 percentage points on ImageNet-1K and 0.3--1.0 points on ImageNet-A. GFLOPs remain within 0.2 and 1.0 of dynamic DORA on the two datasets, respectively. Static scheduling also increases throughput by 5.4%--14.1% across the two datasets. The results show that per-image adaptation improves accuracy retention at comparable computational cost, with larger benefits under distribution shift. Although static scheduling is faster at the evaluated operating points, the consistent gains in accuracy retention highlight the value of per-image adaptation for accuracy-constrained inference.

AuthorsOpenReview ↗

Response to Reviewer q2Aq - Part 2

W3 & Q3—Timing variability: Uncertainty of small throughput gains

Response.

To quantify timing uncertainty, we analyze five repeated throughput measurements obtained under the protocol in Appendix E.5 for DORA and the highest-throughput accuracy-matched baseline on each backbone and dataset. In addition to the median throughputs in Table 1, we report the mean, sample standard deviation, coefficient of variation (CV), and Student’s t-based 95% confidence interval for mean throughput (Appendix E.6, Tables 6–7).

Across all reported comparisons, the CV ranges from 0.03% to 0.97%, indicating small repeat-to-repeat variation under the fixed measurement setup. The marginal 95% confidence intervals do not overlap except for ViT-B on ImageNet-1K.

Because the throughput gap on ViT-B is particularly small, we further conducted 20 measurement pairs between DORA and V-Pruner on ViT-B/ImageNet-1K. All measurements use the same RTX 4090 GPU, batch size 320, fixed pool of 3,200 preprocessed images, inference backend, and numerical settings. Each method is executed in a fresh process, with 200 warm-up forwards followed by 200 timed forwards. Execution order is counterbalanced across pairs to reduce temporal and GPU-state effects (Table 8).

Key results.

Table 8: Paired throughput comparison on ViT-B/ImageNet-1K.

ComparisonPairsDORA mean (images/s)V-Pruner mean (images/s)Paired gain (%)95% CI for paired gain (%)
ViT-B/ImageNet-1K203,7133,705+0.22 ± 0.29[+0.08, +0.35]

Paired gain is reported as mean ± sample standard deviation over 20 measurement pairs, with the standard deviation expressed in percentage points.

In the reported 20-pair experiment, DORA achieves a mean throughput gain of 0.22%, with a 95% confidence interval of [0.08%, 0.35%]. This indicates a small positive difference under the controlled paired measurement setup. Nevertheless, the practical throughput advantage on ViT-B remains marginal, and the paired means do not replace the median throughputs in Table 1.

W4 & Q3—Training and deployment costs: Training cost and small-batch latency

Response.

We have added the complete end-to-end training costs in Appendix E.2. All DORA models are trained on a single NVIDIA RTX 4090 GPU. The reported times include rollout collection, complete-prefix shadow evaluations, PPO actor and critic updates, closed-loop accuracy-feedback evaluations, and all associated data-transfer and synchronization overhead. Shadow evaluations and the privileged critic are used only during training; deployment retains only the frozen backbone and deterministic actor.

We have also added a small-batch evaluation in Appendix F.7. We benchmark all four backbones at batch sizes B∈1,8,16,32,64,128,256B\in\\{1,8,16,32,64,128,256\\}, reporting both throughput and batch latency for the original non-graph execution path and CUDA graph replay. Both paths use the same learned policy and model parameters without retraining and preserve Top-1 accuracy across all tested batch sizes.

Key results.

Training cost.

BackboneEnd-to-end training time (hours)
DeiT-B16
ViT-B14
ViT-L21
DINOv2-B18

Small-batch inference. Representative results at B=1B=1 and B=256B=256 are shown below; the complete batch-size sweep is provided in Appendix F.7, Table 15.

BackboneBatchOriginal throughput (images/s)Graph Replay throughput (images/s)Original batch latency (ms)Graph Replay batch latency (ms)
DeiT-B131.64488.1431.612.05
DeiT-B2563,567.922,672.7371.7595.78
ViT-B131.84485.1331.412.06
ViT-B2563,331.112,646.0176.8596.75
ViT-L118.30222.4454.644.50
ViT-L2561,320.38864.42193.88296.15
DINOv2-B138.11279.8626.243.57
DINOv2-B2562,260.111,633.72113.27156.70

CUDA graph replay is preferable for small batches because it reduces CPU-side kernel-launch overhead. At larger batch sizes, however, the additional GPU work from fixed-capacity execution outweighs these savings, making the original non-graph execution path more efficient. For B>1B>1, latency denotes the total processing time of a batch rather than per-image latency.

We thank the reviewer for these suggestions. The additional experiments clarify the performance of DORA relative to adaptive baselines, the accuracy benefit of input-adaptive scheduling, and the training and deployment costs of the proposed method.

AuthorsOpenReview ↗

Follow-up to W1 & Q2: Completed RL4EViT comparison

Follow-up to W1 & Q2: Completed RL4EViT comparison

Response.

We would like to update our earlier response: the RL4EViT experiments that were still in progress have now been completed for all four backbones on both ImageNet-1K and ImageNet-A. The revised manuscript now includes complete comparisons with ATS, LTMP, and RL4EViT (Table 1 and Figures 1 and 5).

For RL4EViT, we start from the authors’ released implementation and locally train its MAPPO-based policy to make binary keep-or-prune decisions for individual tokens, while keeping the backbone frozen. We retain its three predefined pruning stages at one-quarter, one-half, and three-quarters of the network depth—after blocks 3, 6, and 9 for the 12-block backbones, and after blocks 6, 12, and 18 for ViT-L. Training uses the original attention masking. For final accuracy and throughput evaluation, we apply the learned decisions through token gathering and sequence compaction, physically removing pruned tokens so that subsequent Transformer blocks process only the remaining tokens. The adapted implementation uses packed variable-length FlashAttention without batch-wide padding. Full-set re-evaluation of all selected operating points shows a maximum absolute change in mean Top-1 accuracy of only 0.016 percentage points between masked and compacted execution. Further details are provided in Appendix E.3.

Key results.

BackboneMethodImageNet-1K (Top-1 / GFLOPs / images/s)ImageNet-A (Top-1 / GFLOPs / images/s)
DeiT-BATS80.8 / 24.9 / 1,83826.8 / 21.2 / 2,406
LTMP80.8 / 28.5 / 1,93326.7 / 29.6 / 1,846
RL4EViT80.8 / 28.5 / 3,00526.7 / 25.8 / 3,323
DORA80.8 / 21.7 / 3,94426.7 / 21.0 / 4,233
ViT-BATS83.7 / 28.4 / 1,95743.2 / 23.4 / 2,032
LTMP83.9 / 28.9 / 1,89143.3 / 28.5 / 1,911
RL4EViT83.8 / 33.7 / 2,58543.2 / 34.8 / 2,511
DORA83.8 / 23.6 / 3,72543.2 / 23.2 / 3,831
ViT-LATS85.0 / 103.7 / 77554.7 / 78.3 / 757
LTMP85.1 / 96.7 / 67854.6 / 96.7 / 679
RL4EViT85.1 / 102.7 / 97754.7 / 120.3 / 848
DORA85.1 / 74.9 / 1,39454.7 / 74.3 / 1,456
DINOv2-BATS83.6 / 43.1 / 1,06956.0 / 33.0 / 1,203
LTMP83.6 / 39.0 / 90755.9 / 35.1 / 1,019
RL4EViT83.7 / 39.9 / 1,98456.0 / 38.1 / 1,966
DORA83.7 / 30.3 / 2,46656.0 / 32.5 / 2,389

At every corresponding accuracy-matched operating point, DORA achieves both lower GFLOPs and higher throughput than ATS, LTMP, and RL4EViT. Averaged over the four backbones, DORA uses 16.7% fewer GFLOPs and achieves 43.2% higher throughput than the unweighted means of all available accuracy-matched baselines in Table 1 on ImageNet-1K; on ImageNet-A, the corresponding improvements are 19.5% and 54.1%. Notably, even compared with RL4EViT—which is likewise trained using reinforcement learning and employs the same packed variable-length FlashAttention backend—DORA consistently achieves both lower GFLOPs and higher throughput across all four backbones and both datasets, demonstrating a clear advantage.

Reviewer_NEZ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats token pruning in a frozen ViT as a finite-horizon MDP with one decision per block. A lightweight hierarchical actor reads the attention keys and decides whether to prune (a gate on the CLS key), how many tokens to remove (a categorical budget head over a grid) and which tokens to remove (a Plackett–Luce selector conditioned on k_l/N_l). Tokens are physically deleted after the attention residual. The per-step reward is r_l=25 Δ C_l-a_u (D_l-D_(l-1) ), where Δ C_l is the token-layer compression credited to block l and D_l is a τ^2-scaled KL divergence to the unpruned logits, obtained by a "complete-prefix shadow evaluation" that replays the actions up to block l and runs the remaining blocks without pruning. The coefficient a_u is updated in closed loop toward a 1-pp Top-1 drop measured on a held-out shard. Training uses PPO with a privileged critic. At inference only the actor remains, and the surviving variable-length sequences run with packed variable-length FlashAttention. On ImageNet-1K with four backbones (DeiT-B, ViT-B, ViT-L, DINOv2-B), DORA is compared with seven frozen-backbone baselines at accuracy matched within 0.1 pp, and is reported to use 13.2% fewer FLOPs and reach 32.4% higher throughput than the baseline means, widening to 20.3% and 45.6% on ImageNet-A.

Critical strengths

1.The efficiency claims are measured end to end, and DORA is the fastest method at matched accuracy on every backbone. Throughput includes the actor, token selection, packing and compaction (Section 3.5, Appendix E.5), GFLOPs include the actor (Appendix E.4), and baseline operating points are matched to DORA's Top-1 within 0.1 pp (Table 1, Appendix E.2). Under this protocol DORA has the highest ImageNet-1K throughput on all four backbones, and the policies transfer to ImageNet-A without re-selection while the baselines are allowed to choose their closest ImageNet-A point, which favors the baselines. 2.The closed-loop fidelity coefficient is shown to be necessary and to control the operating point. With a fixed a=30, DeiT-B has no checkpoint that satisfies the development constraint and ends at 78.65% Top-1 with a 3.14-pp drop (Table 2), whereas the closed loop drives a to 186.2 on DeiT-B and 32.4 on DINOv2-B from the same initialization (Section 4.4). The target sweep in Figure 5 produces three distinct operating points by changing only d^⋆, and four DeiT-B seeds give 80.82±0.07% Top-1 and 21.76±0.18 GFLOPs (Table 6). This makes the method usable across backbones without per-backbone tuning, which is a practical contribution in its own right.

Critical weaknesses

1.The claim that motivates the RL formulation, that the compression budget itself should be decided per image, is never isolated. The introduction states "We argue that the compression budget itself should be input-adaptive, decided online from each image's evolving representations" (Section 1), and the MDP, the shadow evaluations and the critic exist to learn that schedule. Section 4.3 shows that the learned placement is close to static: the gate converges to four blocks per backbone (DeiT-B {1ⓜ,2ⓜ,7ⓜ,11}, ViT-B {1ⓜ,4ⓜ,9ⓜ,11}, ViT-L {3ⓜ,4ⓜ,16ⓜ,23}, DINOv2-B {1ⓜ,2ⓜ,3ⓜ,6}), and the only conditional block reported is block 3 of DINOv2-B, active on 66.5% of validation images. Figure 3(b) also shows blocks whose budget collapses to a single value. Per-image adaptivity therefore lives mainly in the budget head at a few blocks. The only component study (Section 4.3, Table 5) goes in the other direction: it keeps DORA's per-image gate and budget and replaces the selector. The missing control keeps the selector and replaces the schedule: at the same blocks, give every image the block's mean validation budget (rounded to the grid) and report Top-1, GFLOPs and throughput. This requires no retraining, only an evaluation pass per backbone. Table 1 shows why it matters: V-Pruner, whose mask is fixed across images and applied before the first block (Appendix E.3), reaches 3,917 images/s on DeiT-B against DORA's 3,944 at the same 80.8% Top-1. If a static schedule with the learned selector lands close to DORA, the gains come from hard deletion and a learned token scorer rather than from input-adaptive budgets, and the paper's framing would need to change accordingly. 2.The headline gains are relative to the unweighted mean of the baselines, which includes methods run with dense attention, and the margin over the strongest baseline is much smaller. The 13.2% and 32.4% in the abstract are computed against the "Average of baselines" rows of Table 1. Appendix E.5 states that ToMe, MCTF and DiffRate use dense attention and Zero-TPrune uses it in some blocks, and these are the slowest entries (MCTF: 2,122, 1,900, 755 and 701 images/s across the four backbones). Against the fastest accuracy-matched baseline on each backbone, DORA's ImageNet-1K throughput is higher by 0.7% on DeiT-B (V-Pruner, 3,917), 0.2% on ViT-B (V-Pruner, 3,719), 8.2% on ViT-L (Representation Shift, 1,288) and 7.2% on DINOv2-B (V-Pruner, 2,301), computed from Table 1. On GFLOPs, DORA is not the lowest on ViT-B or ViT-L, which Section 4.2 acknowledges. I do not consider keeping dense attention for proportional-attention methods unfair in itself, but it means that the statement in Section 4.2 that "lower FLOPs do not necessarily imply higher throughput", illustrated with MCTF (21.9 GFLOPs, 2,122 images/s) against DORA (21.7 GFLOPs, 3,944 images/s), mostly reflects the attention kernel rather than the pruning policy. Stating the gain against the strongest baseline in the abstract, and reporting DORA's throughput when its variable-length sequences are padded and run through the same dense path, would separate the contribution of the kernel from that of the policy. Both are measurements on the existing checkpoints.

Questions for authors
  1. At the blocks DORA selects, what Top-1, GFLOPs and throughput does a static schedule reach when every image receives the block's mean validation budget and tokens are chosen by DORA's learned selector?
  2. What is DORA's throughput on each backbone when its sequences are padded and processed with the dense attention path used for ToMe, MCTF and DiffRate, rather than with packed variable-length FlashAttention?
  3. Apart from block 3 of DINOv2-B, is the gate decision at every block identical across all validation images? If so, how much of the per-image GFLOPs range reported in Section 4.3 (for example 26.4–55.4% on DeiT-B) comes from the budget head alone?
2 comments
AuthorsOpenReview ↗

Response to Reviewer NEZ8

W1 & Q1: Contribution of Input-Adaptive Scheduling

Response.

As requested, we conducted the static-schedule control. We retain the learned intervention blocks, selector parameters, and packed variable-length FlashAttention backend, while replacing per-image gate and budget decisions with fixed per-block deletion budgets. Each budget is computed from the mean deletion count on the corresponding evaluation dataset, counting gate-off decisions as zero and rounding to the nearest feasible budget-grid value. No additional training is performed. The new control is presented in Section 4.4 and Table 2, with the complete static-schedule configurations provided in Appendix F.4.

Key results.

DatasetBackboneTop-1 (%)GFLOPsThroughput (img/s)
ImageNet-1KDeiT-B80.5 (−0.3)21.8 (+0.1)4,473 (+13.4%)
ImageNet-1KViT-B83.6 (−0.2)23.7 (+0.1)4,193 (+12.6%)
ImageNet-1KViT-L84.6 (−0.5)74.7 (−0.2)1,485 (+6.5%)
ImageNet-1KDINOv2-B83.4 (−0.3)30.4 (+0.1)2,674 (+8.4%)
ImageNet-ADeiT-B26.2 (−0.5)20.8 (−0.2)4,831 (+14.1%)
ImageNet-AViT-B42.9 (−0.3)23.1 (−0.1)4,348 (+13.5%)
ImageNet-AViT-L53.7 (−1.0)75.3 (+1.0)1,535 (+5.4%)
ImageNet-ADINOv2-B55.4 (−0.6)32.5 (0.0)2,540 (+6.3%)

Static scheduling reduces Top-1 accuracy on every backbone, by 0.2--0.5 percentage points on ImageNet-1K and 0.3--1.0 points on ImageNet-A. GFLOPs remain within 0.2 and 1.0 of dynamic DORA on the two datasets, respectively. Static scheduling also increases throughput by 5.4%--14.1% across the two datasets. The results show that per-image adaptation improves accuracy retention at comparable computational cost, with larger benefits under distribution shift. Although static scheduling is faster at the evaluated operating points, the consistent gains in accuracy retention highlight the value of per-image adaptation for accuracy-constrained inference.

W2 & Q2: Strongest-Baseline Comparison and Execution Backend

Response.

The abstract and Section 4.2 now report DORA's average throughput gains over the highest-throughput accuracy-matched baseline on each backbone. We also conducted the requested dense-attention control while holding all per-image pruning decisions fixed. The surviving sequences in each batch are padded and masked to the maximum valid length and processed using the dense-attention path employed in our baseline implementations of ATS, ToMe, LTMP, MCTF, and DiffRate (ATS and LTMP are newly added adaptive token-compression baselines in the revised paper). The complete results are provided in Appendix F.6, Table 14.

Key results.

BackboneTop-1 (%)GFLOPsThroughput (img/s)
DeiT-B80.824.9 (+3.2)2,296 (−41.8%)
ViT-B83.825.8 (+2.2)2,267 (−39.1%)
ViT-L85.180.5 (+5.6)872 (−37.4%)
DINOv2-B83.733.6 (+3.3)1,547 (−37.3%)

Computations on padded positions increase GFLOPs by 2.2--5.6. Throughput decreases to 2,296, 2,267, 872, and 1,547 images/s on DeiT-B, ViT-B, ViT-L, and DINOv2-B, respectively, corresponding to reductions of 37.3%--41.8% relative to packed variable-length DORA. This gap reflects both the additional computation on padded positions and the mismatch between dense batch execution and the variable-length sequences produced by adaptive pruning.

These results clarify the complementary roles of the pruning policy and the execution backend: the policy reduces input-dependent token computation, while a compatible variable-length backend converts these savings into realized wall-clock speedups. Packed variable-length FlashAttention therefore provides an important systems foundation for deploying adaptive token compression effectively.

Q3: Gate Decisions and Sources of Per-Image Adaptivity

Yes. Apart from DINOv2-B block 3, which activates on 66.5% of validation images, every listed intervention gate is always active. Therefore, the per-image GFLOPs ranges on DeiT-B, ViT-B, and ViT-L are entirely budget-head-driven.

  • DeiT-B: 26.4%–55.4%
  • ViT-B: 21.7%–37.5%
  • ViT-L: 26.0%–46.2%

On DINOv2-B, block 3 is conditionally activated on 66.5% of validation images. Its 22.9%–40.1% GFLOPs-reduction range therefore reflects both input-adaptive budgets and conditional gate activation. These observations suggest that gate learning primarily captures backbone-specific rather than image-specific adaptivity.

Appendix F.5 further evaluates cross-backbone placement transfer and uniformly spaced placements under approximately matched GFLOPs. Relative to the learned intervention locations of each backbone, these alternatives reduce Top-1 accuracy by 0.4--1.6 percentage points on ImageNet-1K and by 0.5--3.0 percentage points on ImageNet-A.

We thank the reviewer for these suggestions. The additional controls have helped us distinguish the accuracy-retention benefit of input-adaptive budgeting from the systems contribution of packed variable-length execution, and to present DORA's efficiency gains more precisely.

AuthorsOpenReview ↗

Response to Reviewer NEZ8

We would like to follow up on our previous response and thank you again for your constructive suggestions. To facilitate your review, the corresponding additions and revisions are summarized below:

  • Input-adaptive scheduling: the requested static-schedule control is provided in Section 4.4, Table 2, and Appendix F.4.
  • Strongest-baseline comparison and execution backend: Section 4.2 now reports gains over the highest-throughput accuracy-matched baselines, and the dense-attention control is provided in Appendix F.6, Table 14.
  • Gate decisions and per-image adaptivity: we have clarified the gate behavior in Section 4.3 and added further intervention-placement analyses in Appendix F.5.

We hope these additions address your main concerns. We would be grateful to hear whether you feel these concerns have been addressed, and we would be happy to clarify any remaining questions. Thank you again for your time and constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.