- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DORA proposes a reinforcement learning framework for token pruning in frozen vision transformers. A hierarchical actor determines whether to prune, how many tokens to remove, and which tokens to select. Complete-prefix shadow evaluations provide localized fidelity rewards, while accuracy feedback adjusts the fidelity penalty during training. Deployment uses hard token deletion and packed variable-length FlashAttention. Experiments on four backbones demonstrate improved accuracy–efficiency trade-offs on ImageNet-1K and ImageNet-A, with additional analyses of token selection, reward design, feedback behavior, and training stability.
Critical strengths
-
The hierarchical policy provides a coherent approach to jointly learning pruning locations, budgets, and token identities. Complete-prefix shadow evaluations offer a concrete mechanism for assigning local rewards while preserving the overall objective, and the terminal-only reward ablation supports their effectiveness.
-
The framework achieves useful reductions in computation with measured throughput gains across four frozen backbones. Evaluation includes online decision and compaction costs, unchanged policy transfer to ImageNet-A, and repeated training runs, providing credible evidence of overall effectiveness and stability.
Critical weaknesses
-
The paper lacks ablations of input-dependent pruning locations and budgets. Figure 3 shows largely fixed intervention locations, while the selector study keeps gates and budgets unchanged. A comparison with a learned shared schedule using the same selector and execution backend is needed to establish the benefit of joint adaptivity.
-
Despite substantial FLOPs reductions, DORA delivers only marginal additional acceleration over V-Pruner on ImageNet-1K: 0.7% on DeiT-B and 0.2% on ViT-B. These results provide limited support for the practical advantage of its more elaborate online pruning policy on these backbones.
Questions for authors
-
What evidence shows that input-dependent gates and budgets improve upon a learned shared schedule when the selector and execution backend remain unchanged?
-
Which operations account for the gap between FLOPs reductions and throughput improvements over V-Pruner on DeiT-B and ViT-B?
5 comments
Response to Reviewer 9U6x - Part 2
W2 & Q2: FLOPs Reduction Versus Throughput Improvement over V-Pruner
Response.
We agree that the additional speedup over V-Pruner is small on these two backbones and that FLOPs reductions alone do not determine end-to-end runtime. The key difference lies in the inference execution paths. V-Pruner applies a fixed mask once before the first block, whereas DORA makes online gate, budget, and selector decisions at multiple blocks, followed by dynamic token indexing, packing, and compaction.
To identify the associated overhead, we performed low-overhead phase-level profiling on DeiT-B and ViT-B using the main-table configurations, batch size 320, and the FlashAttention backend. The profile separates backbone execution, actor computation, and indexing/control overhead, and includes host-side work on the critical path (See Appendix F.8).
Key results.
At matched accuracy, the ImageNet-1K results in Table 1 show:
| Backbone | V-Pruner GFLOPs | DORA GFLOPs | FLOPs Reduction | V-Pruner Throughput | DORA Throughput | Throughput Gain |
|---|---|---|---|---|---|---|
| DeiT-B | 28.3 | 21.7 | 23.3% | 3,917 | 3,944 | +0.7% |
| ViT-B | 29.9 | 23.6 | 21.1% | 3,719 | 3,725 | +0.2% |
Throughput is reported in images/s. The phase-level profile, reported in milliseconds per batch of 320 images, is as follows:
| Operation | DeiT-B ΔLatency | ViT-B ΔLatency |
|---|---|---|
| Backbone stage | −15.34 ms | −13.51 ms |
| Actor stage | +5.63 ms | +5.73 ms |
| Indexing and control stage | +7.43 ms | +7.55 ms |
| Total latency | −2.28 ms | -0.23 ms |
Relative to V-Pruner, DORA reduces backbone-stage latency by 15.34 ms/batch on DeiT-B and 13.51 ms/batch on ViT-B. However, actor computation and additional indexing/control operations introduce approximately 13.06 and 13.28 ms/batch of additional overhead, respectively, offsetting most of these savings.
DORA's online path requires repeated controller execution, token gathering and compaction, dynamic sequence-length metadata handling, additional kernel launches, and CPU/GPU synchronization. In the profiled implementations, DORA launches approximately 1,203 kernels per batch, compared with 128 for V-Pruner. These operations incur execution and memory-management costs that are not captured by FLOPs.
These results explain why DORA's substantial reduction in backbone computation produces only modest throughput improvements over V-Pruner. The profiling results are diagnostic measurements rather than a direct decomposition of the main-table timings.
We hope these controls and profiling results clarify the distinct contributions of learned intervention locations and input-adaptive budgeting, as well as the systems overhead that limits the conversion of FLOPs reductions into measured throughput gains.
Thank you for the detailed rebuttal and additional experiments. The static-schedule control in Table 2 addresses my main concern about input-adaptive pruning budgets. While the static variant achieves higher throughput, DORA consistently improves accuracy retention at comparable FLOPs, by 0.2–0.5 percentage points on ImageNet-1K and 0.3–1.0 points on ImageNet-A. Since the static budgets are derived from DORA's decisions without separate optimization, this is not a fully independently optimized static-policy baseline, but the consistent results across all eight settings are useful evidence for the value of per-image adaptation. The gate-placement ablations in Appendix F.5 further clarify the contribution of learned intervention locations.
The phase-level profiling in Appendix F.8 explains the limited throughput gains over V-Pruner on DeiT-B and ViT-B: actor and indexing/control overhead of approximately 13 ms per batch offsets most of the 13.5–15.3 ms saved in backbone execution, with roughly 1,203 kernel launches for DORA versus 128 for V-Pruner. The static variant's higher throughput at slightly lower accuracy further illustrates the runtime cost of per-image scheduling, although it is not an accuracy-matched comparison with V-Pruner.
Overall, the additional experiments adequately address my original concerns. The modest acceleration over the strongest baseline on some backbones remains a practical limitation, but it is now better characterized. I will maintain my original score.
Response to Reviewer 9U6x - Part 1
W1 & Q1: Learned Intervention Locations and Input-Adaptive Budgets
Response.
To isolate the contribution of per-image scheduling, we conducted a static shared-schedule control derived from the learned DORA policy (Section 4.4, Table 2; Appendix F.4). We retain the learned intervention blocks, selector parameters, and packed variable-length FlashAttention backend, while replacing per-image gate and budget decisions with fixed per-block deletion budgets. Each budget is computed from the mean deletion count on the corresponding evaluation dataset, counting gate-off decisions as zero and rounding to the nearest feasible budget-grid value. No additional training is performed.
We also conducted gate-placement ablations (Appendix F.5), evaluating cross-backbone placement transfer and uniformly spaced placements under approximately matched GFLOPs while retaining the online budget head and learned selector.
Key results.
Parentheses indicate absolute Top-1 changes in percentage points, absolute GFLOPs changes, and relative throughput changes versus dynamic DORA.
Relative to dynamic DORA, static scheduling reduces Top-1 accuracy by 0.2–0.5 percentage points on ImageNet-1K and 0.3–1.0 points on ImageNet-A, while GFLOPs differ by at most 0.2 and 1.0, respectively. Although static scheduling is 5.4%–14.1% faster, dynamic DORA consistently achieves higher accuracy at comparable computational cost, with larger advantages under distribution shift.
We acknowledge that the learned gates are largely fixed across images and that per-image scheduling adaptivity primarily arises from the budget head. Nevertheless, the placement ablations show that, under approximately matched GFLOPs, cross-backbone and uniformly spaced placements reduce Top-1 accuracy by 0.4–1.6 percentage points on ImageNet-1K and 0.5–3.0 points on ImageNet-A.
Together, these results support two distinct contributions: learning backbone-appropriate intervention locations and adapting pruning budgets to individual inputs.