acceptodds
Under review as a conference paper at ICLR 2027

Not Every Action Needs Every Token: Dynamic Visual Token Allocation for Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models repeatedly execute the vision-language backbone at every control step, where visual tokens constitute a dominant portion of the inference cost. Existing visual-token pruning methods accelerate inference by applying a fixed retention ratio throughout deployment. However, we observe that sensitivity to sustained token reduction varies across tasks and control steps within a rollout. A uniform ratio can therefore miss opportunities to reduce computation while preserving task performance. To address this issue, we propose **D**ecision-importance-driven **V**isual-token **A**llocation (**DVA**), a dynamic token allocation framework that determines the retention ratio online based on the current observation and instruction. The key idea is to introduce **D**ecision **I**mportance (**DI**), a per-step supervision signal derived from counterfactual suffix rollbacks, which quantifies the downstream degradation caused by sustained token reduction starting at a given control step. A lightweight importance predictor learns to estimate DI from features already available in VLA models, and a monotone calibrator converts the predicted importance into the deployed retention ratio. DVA serves as a plug-in module for existing token pruners, introducing only about 1% per-step latency overhead while keeping both the pruning strategy and the policy parameters unchanged. Experiments show improved performance-efficiency trade-offs over fixed-ratio pruning. On OpenVLA-OFT, retaining 37% of visual tokens yields a success rate comparable to the full-token baseline at a 1.20× speedup, while retaining 22% maintains a 96.2% success rate at a 1.28× speedup. At matched average retention targets, DVA outperforms fixed-ratio pruning across three action heads, three pruning methods, and two benchmarks, demonstrating its effectiveness and generalization ability. Code and models will be released publicly.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Visual Token Pruning, Dynamic Token Allocation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KGbTOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DVA, which predicts how important the current control step is and allocates the visual-token budget accordingly. The supervision comes from counterfactual suffix rollbacks, measuring how much sustained pruning from a given step hurts the rest of the rollout. A predictor and monotone calibrator are added on top of existing pruners, and experiments show consistent but generally bit performance gains over static pruning at matched average retention.

Critical strengths
  • This paper is well-organized with relatively clear figures and tables.
  • The main assumption seems reasonable: The right token budget varies across control phases, so a fixed ratio can waste computation or starve difficult steps.
  • The devised DVA is modular and keeps the original pruner and policy weights unchanged.
  • I like the idea of the approach to generating counterfactual labels.
Critical weaknesses
  • Sometimes an important frame which seems hard to human intuitions actually require less tokens than we expected. This process is actually can be learned, since I found some related works which aim to guide the VLA/VLM model learns how to adaptively select tokens [1-3]. Authors are expected to follow and compare some highly related works beyond the tradition baselines including FastV, SparseVLM and other classic baselines. The author shouldn't avoid comparing with more powerful VLA pruning or lightweight VLA methods.
  • From the experimental perspective, though as mentioned above in the strengths, the decision-importance labels rely on simulator suffix rollbacks and are expensive or unavailable on a real robot. The main training signal is therefore tied to resettable simulation.
  • The FLOPs comparison is estimated and excludes some components, including parts of action-head computation and the DVA predictor. A full hardware-level accounting would make the efficiency claim more convincing. Authors can give the wall-clock time report and this is acceptable for me.

[1] VLA-ADP: ICLR'26

[2] LightVLA, ICRA'26

[3] SpecPrune-VLA, ICML'26

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KGbT (Part 1/2): Weaknesses 1–2 (VLA baselines and real-robot adaptation)

To Weakness 1 (VLA-Specific Pruning Baselines):

We appreciate your recommendations of relevant work and suggestions for broader baseline comparisons. We have added baseline results to Section 4.2 and Table 2, and expanded the discussion of pruning objectives in Appendix A.3 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The original Table 2 includes VLA-ADP at Ratio=0.50 and 0.30, reporting success, measured latency, and speedup alongside the general VLM pruners [1]. Your suggestion helped us extend this VLA-specific comparison and explain more clearly which pruning objective DVA addresses.

LightVLA learns to reduce visual computation within a model call while preserving action prediction, but that objective does not directly measure the effect of sustained pruning on the remaining task [2]. SpecPrune-VLA exploits spatial and temporal redundancy and adjusts pruning according to action granularity [3]. However, states with similar action speeds can require different visual information, so action granularity does not directly measure whether continued pruning will cause later failure or extra completion steps.

DVA learns from those execution consequences. As Section 3.1 explains, full suffixes under different ratios are compared from the same state, and failures and additional completion steps supervise the budget module. The module is learned separately from the frozen policy, and Table 3 and Figure 4 evaluate its use across action heads. Our revised characterization therefore includes learned adaptive pruning and distinguishes DVA by its execution-based supervision. The Abstract and Introduction state this distinction.

To compare the resulting success–latency trade-off with VLA-specific alternatives, we extended the VLA-ADP sweep to Ratio=0.05 and evaluated SpecPrune-VLA on OpenVLA-OFT. Both follow the standard LIBERO success-rate protocol and the Table 2 timing procedure, with speedups referenced to the corresponding full-token run.

MethodVLA-ADP (Ratio=0.05)SpecPrune-VLADVA
Success (%)78.295.097.0
Speedup1.16×1.01×1.20×

At these measured operating points, DVA achieves both higher success and greater speedup, showing a practical benefit in comparisons with VLA-specific dynamic pruning.

To Weakness 2 (Real-Robot Labeling and Adaptation):

Thank you for your feedback on presenting the application of DVA to real robots. We have expanded the labeling and adaptation protocol in Section 4.2 and Appendix D.7 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Comparable restored states are needed to construct DI labels, because the candidate retention ratios must be tested from the same starting conditions. Simulation supports this comparison directly, while physical resets cost more. We therefore collect the main supervision in simulation and adapt the resulting predictor with a small set of real-robot labels. Once trained, the predictor and calibrator assign budgets from current observations without online rollback.

The real-robot experiment follows this separation by fine-tuning the simulator-pretrained predictor on four labeled states per task, as described in Section 4.1. To obtain each label, we return the robot to a predefined pose, reset the objects, and replay the full-token prefix. Each candidate ratio then remains fixed through the suffix, and its success and additional completion steps relative to the full-token reference contribute to the state’s DI label. Thus, the reported deployment follows simulator pretraining and a small real-robot adaptation stage. The Conclusion in Section 5 also states this adaptation protocol.

AuthorsOpenReview ↗

Response to Reviewer KGbT (Part 2/2): Weakness 3 (Full-model latency and hardware costs)

To Weakness 3 (Measured Latency and Predictor Overhead):

We appreciate your constructive advice on reporting practical inference costs. We have added StarVLA runtime measurements to Section 4.2, Table 7, Appendix D.2, and Table D.2 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The wall-clock measurements in Table 2 already include the costs omitted from the FLOPs estimate. They time visual encoding, budget prediction and selection, the decoder, and complete action generation. In particular, the parenthesized DVA value is the predictor and calibrator latency already included in the total, as stated in the caption. The approximately 37% retention result is therefore a speedup with budget-control overhead included:

ArchitectureFull tokens (ms)DVA (ms)DVA speedup
OpenVLA-OFT107.088.8 (1.12)1.20×

The 1.12 ms contribution is part of the reported 88.8 ms total. To test whether this practical benefit extends to other action heads, we measured the VLM, complete action head, and full call on StarVLA-π and GR00T. Each comparison uses identical inference settings on an exclusive A800 and the Table 2 sampling procedure, with the corresponding full-token run as reference.

ArchitectureStarVLA-πStarVLA-πStarVLA-πStarVLA-GR00TStarVLA-GR00TStarVLA-GR00T
ComponentVLMAction headFull model callVLMAction headFull model call
Full tokens (ms)160.8181.5348.9161.621.1189.3
DVA (ms)135.6149.0291.2143.318.9168.7
Speedup1.19×1.22×1.20×1.13×1.12×1.12×

These calls include the corresponding predictor, calibrator, and token-selection operations, while the VLM component includes visual encoding and the language decoder. Shorter visual conditioning reduces computation in both the VLM and the action head’s key/value projections and cross-attention, so the measured full-model gains include savings from both components after budget-control costs. Appendix D.5 specifies the timing setup.

References

[1] Xiaohuan Pei et al. Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulation. arXiv:2509.22093, 2025.

[2] Titong Jiang et al. The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning. arXiv:2509.12594v2, 2025.

[3] Hanzhen Wang et al. SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning. ICML, 2026.

If you have more questions or suggestions, please let us know, and we would be happy to have further discussions.

Reviewer_KGbTOpenReview ↗

Reply

I appreciate authors' response. Though I think the experimental improvement is rational and not that promising for all evaluated benchmarks, the extra results and LIBERO-plus experiments enrich the experimental analysis to a certain extent though I still think more related pruning methods for VLA should be incorporated. I raise my rating from 2 to 3 with the low confidence.

Best wishes,

Reviewer KGbT

AuthorsOpenReview ↗

Response to Comments by Reviewer KGbT

To the Follow-up Comment (Additional VLA Pruning Comparisons)

Thank you for reassessing our work, raising your rating, and suggesting further VLA pruning comparisons. We have added SAFE-Pruner, VLA-ACL, and VLA-Pruner results to Section 4.2 and Table 2, and expanded the related-work discussion in Appendix A.3 of the updated submission, with changes highlighted in purple\color{purple}{\text{purple}}.

Building on the existing EfficientVLA [1], VLA-Cache [2], VLA-ADP [3], and SpecPrune-VLA [4] comparisons, the updated table includes SAFE-Pruner [5], VLA-ACL [6], and VLA-Pruner [7]. This gives seven VLA acceleration methods alongside DVA at three retention ratios. VLA-ADP uses Ratio=0.05, whose speedup is close to DVA's, to support comparison at similar inference cost.

These methods cover a range of acceleration approaches: VLA-ACL learns token selection under a fixed budget, VLA-ADP and SpecPrune-VLA adjust pruning using action information, and VLA-Pruner and SAFE-Pruner use historical information for training-free pruning. VLA-Cache and EfficientVLA additionally introduce caching across frames and model-structure compression, respectively. The comparisons therefore cover static and dynamic budgets, learned selection, temporal information, and computation reuse.

MethodModel-call latency (ms)SpeedupSuccess (%)
EfficientVLA [1]70.41.52×88.9
VLA-Cache [2]103.91.03×96.8
VLA-ADP (Ratio=0.05) [3]92.41.16×78.2
SpecPrune-VLA [4]106.41.01×95.0
SAFE-Pruner [5]88.21.21×96.4
VLA-ACL (K=128) [6]91.11.17×96.5
VLA-Pruner (Ratio=0.10) [7]89.51.20×93.8
DVA (r̄=0.37)88.8 (1.12)1.20×97.0
DVA (r̄=0.28)85.4 (1.08)1.25×96.5
DVA (r̄=0.22)83.4 (1.09)1.28×96.2

At operating points near DVA's 1.20× speedup, SAFE-Pruner, VLA-ACL, and VLA-Pruner achieve 96.4%, 96.5%, and 93.8% success, respectively, compared with 97.0% for DVA. At tighter budgets, DVA maintains 96.5% and 96.2% success at 1.25× and 1.28× speedups. Thus, the expanded comparison with recent VLA-specific pruners continues to show higher success for DVA at the evaluated operating points with similar runtime cost. Parenthesized values report the budget-control overhead already included in total latency.

If you have more questions or suggestions, please let us know, and we would be happy to have further discussions.

References

[1] Yantai Yang et al. EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models. arXiv:2506.10100, 2025.

[2] Siyu Xu et al. VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching. NeurIPS, 2025.

[3] Xiaohuan Pei et al. Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulation. arXiv:2509.22093, 2025.

[4] Hanzhen Wang et al. SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning. ICML, 2026.

[5] Shilin Ma et al. SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation. ECCV, 2026.

[6] Owen Du et al. VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models. arXiv:2610.08133, 2026.

[7] Ziyan Liu et al. Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inference. arXiv:2511.16449, 2025.

Reviewer_m8BFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DVA, a framework for dynamic visual-token budget allocation in vision-language-action (VLA) models. It derives Decision Importance (DI) labels from counterfactual suffix rollouts, accounting for task failures and additional control steps caused by sustained pruning. At deployment, a lightweight predictor estimates DI from existing VLA features, and a monotone calibrator maps the estimate to a per-step retention ratio. DVA leaves the underlying policy weights and the pruner's token-scoring rule unchanged. Experiments on OpenVLA-OFT, multiple StarVLA action heads, LIBERO, LIBERO-PLUS, and two real-robot tasks report improved performance–compute trade-offs over fixed-ratio pruning.

Critical strengths
  1. Clear motivation and modular design. Using the effects of sustained pruning on task success and completion length provides a concrete formulation of compute allocation for closed-loop control. Separating budget prediction from token scoring allows DVA to work with existing pruners while preserving the underlying policy weights. This modular design is well motivated and potentially useful in practice.

  2. Broad evaluation with measured runtime gains. The evaluation spans three StarVLA action heads, three pruners, and two benchmarks, with positive average gains under matched mean retention ratios. The OpenVLA-OFT timing results include predictor and calibrator overhead, supporting a practical speedup on that platform. Ablations on pruning depth, smoothing, pooling, and cross-head transfer, together with disclosure of offline labeling costs, improve experimental transparency.

Critical weaknesses
  1. Limited evidence isolating the key design choices. Table 3 matches the mean retention ratio between DVA and the static baseline at the suite level. Since each suite contains multiple tasks, this comparison does not isolate the benefit of adjusting budgets within a task from that of assigning different average budgets across tasks. The specific contribution of per-step allocation therefore remains insufficiently established. Appendices B.3 and B.4 primarily compare label distributions and rank correlations across intervention horizons, rather than the closed-loop success and latency achieved by controllers trained with those labels. Different label distributions do not establish that full-suffix supervision produces a better budget controller, a limitation acknowledged in Appendix B.3. Given the offline cost of counterfactual suffix labeling, evidence of its advantage over simpler supervision would strengthen the central contribution.

  2. Incomplete runtime evaluation across action heads. The main measured performance–latency comparison is limited to OpenVLA-OFT. For StarVLA-π and StarVLA-GR00T, Table 3 reports success rates and mean retention ratios, Table D.4 provides estimated FLOPs, and Table D.2 reports module timings only for the full-token baseline. These action heads have substantially different latency profiles from OFT, so reductions in visual-token counts or estimated FLOPs do not determine their end-to-end speedups. The FLOPs accounting in Appendix D.6 also excludes parts of the complete control pipeline. Measurements of dynamic and static pruning on the same hardware at comparable budgets are needed to establish the practical efficiency gains across action architectures.

Questions for authors
  1. Attribution of the gains. Could you provide controls that distinguish task-level budget allocation from within-task adaptation, such as a static baseline matched to DVA's mean retention ratio for each task? Could you also compare full-suffix supervision with single-step or finite-horizon supervision in terms of downstream closed-loop performance, with other components, samples, and budgets kept comparable?

  2. Runtime gains across action heads. Could you report end-to-end latency, speedup, or success–latency curves for dynamic and static pruning on StarVLA-π and StarVLA-GR00T, including predictor, calibrator, and associated processing overhead? These results would directly inform my assessment of the method's practical applicability.

  3. Positioning relative to learned adaptive pruning. The Abstract and Introduction broadly characterize prior methods as using fixed retention ratios or handcrafted dynamic rules. However, LightVLA (Jiang et al., 2025) already uses differentiable selection to produce input-dependent retained-token sets. Please clarify how supervision from closed-loop consequences differs from existing learned adaptive pruning and whether a fair comparison is feasible. If joint policy fine-tuning versus frozen-policy deployment limits direct comparison, please explain this distinction and narrow the characterization of prior work accordingly.

Minor Comments

  • Please include an image of the real-robot experimental setup and representative rollout visualizations to help readers understand the tasks and failure cases.
  • In Fig. 3, please clarify whether the black patches in the bottom visualization indicate pruned visual tokens and how they relate to the retention ratio.

The paper addresses a worthwhile problem with a clear framework and a broad evaluation. However, the independent benefits of per-step allocation and full-suffix supervision remain insufficiently established, and runtime gains across action heads are unclear. I currently lean toward weak rejection, close to the acceptance boundary. I would consider raising my recommendation if the authors provide convincing controlled comparisons clarifying the source of the gains and end-to-end timing results across action heads.

7 comments
AuthorsOpenReview ↗

Response to Reviewer m8BF (Part 1/4): Weaknesses 1–2 (Controlled comparisons and cross-head runtime)

To Weakness 1 (Task-Static Controls and Supervision Horizon):

We appreciate your suggestions for isolating the contributions of our design choices. We have added task-static and supervision-horizon results to Section 4.3, Appendices B.4 and E.1, and Tables B.5 and E.1 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

To isolate allocation within a task, we fixed task-static retention to each task’s mean DVA budget on all three StarVLA action heads. The policy, TQA scoring, and pruning layer remain unchanged. Unlike the suite-level matching in Table 3, this comparison also controls differences in mean budgets between tasks, so it tests whether adapting retention within each task improves success.

ArchitectureStarVLA-OFTStarVLA-πStarVLA-GR00T
Task-static success (%)94.196.597.4
DVA success (%)94.797.698.6

DVA achieves higher success on all three heads after this task-level matching, providing further evidence for allocating tokens from the current observation within a task.

The horizon comparison tests a different choice: whether supervision should cover the entire remaining task. Full suffixes hold each candidate ratio until success or timeout, as Figure 3 shows. An H=8 window covers the whole suffix only when execution terminates within eight decisions. Otherwise, subsequent full-token observations can affect the final outcome, so the label measures only a partially pruned execution.

To test this distinction in closed-loop control, we trained an H=8 variant on OpenVLA-OFT. Labeling prunes for eight policy decisions and then restores full tokens until termination, with the label determined by the final outcome. Deployment still predicts a budget at each observation. We fixed the tolerance and adjusted offsets from token usage alone to match each suite’s mean retention to DVA, then evaluated both controllers under the standard LIBERO protocol.

Supervision horizonSuccess (%)Actual mean retention (%)
H=896.737.1
Full suffix97.037.3

At similar retention, full-suffix supervision achieves 0.3 percentage points higher success. The comparison thus adds closed-loop evidence for the choice whose labels consistently cover the entire remaining task.

To Weakness 2 (Dynamic and Static Runtime at Matched Budgets):

Thank you for your advice on strengthening the runtime comparisons. We have added dynamic and static runtime comparisons to Section 4.2, Table 7, Appendix D.2, and Tables D.2–D.3 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

We timed full-token and DVA inference on an exclusive A800 under identical settings within each architecture. Following the Table 2 sampling procedure over complete rollouts, we measured the VLM, complete action head, and full model call. The VLM includes visual encoding and the language decoder, and the full call includes the corresponding predictor, calibrator, and token selection.

ArchitectureStarVLA-πStarVLA-πStarVLA-πStarVLA-GR00TStarVLA-GR00TStarVLA-GR00T
ComponentVLMAction headFull model callVLMAction headFull model call
Full tokens (ms)160.8181.5348.9161.621.1189.3
DVA (ms)135.6149.0291.2143.318.9168.7
Speedup1.19×1.22×1.20×1.13×1.12×1.12×

Shortening the visual context reduces both subsequent VLM computation and the action head’s conditional key/value projections and cross-attention. Although action-token counts, action-side feed-forward computation, and denoising iterations remain fixed, these conditional savings yield 1.20× and 1.12× full-model speedups after budget-control overhead.

For the requested comparison with static pruning, we timed static TQA at the same precision and with the same procedure. Its fixed ratio in each suite matches DVA’s realized mean retention to within 0.3 percentage points, and both speedups use the same full-token reference.

ArchitectureStatic TQA latency (ms)DVA latency (ms)Static TQA speedupDVA speedup
StarVLA-π288.8291.21.21×1.20×
StarVLA-GR00T168.5168.71.12×1.12×

DVA and static TQA differ by less than 1% in full-model latency at these matched budgets. Observation-dependent allocation therefore preserves the measured speedup over full tokens while costing close to fixed-budget pruning. Appendix D.5 specifies the hardware and timing setup.

AuthorsOpenReview ↗

Response to Reviewer m8BF (Part 2/4): Question 1 (Budget-allocation controls and supervision horizon)

To Question 1 (Budget Allocation and Supervision Horizon):

Thank you for recommending these controlled comparisons. We have added allocation and supervision-horizon results to Section 4.3, Table 8, Appendices B.4 and E.1, and Table B.5 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

We used three OpenVLA-OFT controls to separate the allocation effects. Global-static fixes one ratio across all four suites, while task-static uses each task’s mean DVA budget. Progress-only allows variation within a task but determines it from a five-bin curve using elapsed steps divided by the task duration estimated from training trajectories. Because the curve is fitted to counterfactual labels, it still requires trajectory collection and labeling.

All methods share the policy weights, TQA scoring, and pruning layer, with comparable mean retention. Against these controls, DVA predicts DI from each current observation, then applies calibration and EMA. The selected evaluations therefore test whether using that observation improves task success over fixed budgets or progress-only allocation.

Mean retentionDVA success (%)Global-static success (%)Task-static success (%)Progress-only success (%)
37%97.096.696.796.9
28%96.595.796.396.0
22%96.295.295.195.4

The success rates are close at approximately 37% retention, but at approximately 22%, DVA improves over global-static, task-static, and progress-only by 1.0, 1.1, and 0.8 percentage points. Using the current observation thus better preserves success at the lower budget.

For supervision, Table B.3 shows that 49.6% of H=1 labels fall below DI=0.02, compared with 4.8% for full suffixes. The latter provide more differentiated labels and consistently cover the entire remaining task. In an H=8 window, by contrast, an execution lasting longer than eight decisions can use subsequent full-token observations to recover, while full-suffix labeling keeps those corrections under the restricted budget until success or timeout.

To test whether this distinction affects the deployed controller, we trained an H=8 variant on OpenVLA-OFT using the final task outcome after eight pruned decisions followed by full-token execution. Both controllers allocate budgets from each current observation at deployment. We fixed the H=8 tolerance and adjusted offsets using token usage alone to match DVA’s average retention, then followed the standard LIBERO evaluation protocol.

Supervision horizonSuccess (%)Actual mean retention (%)
H=896.737.1
Full suffix97.037.3

The full-suffix controller achieves 0.3 percentage points higher success at similar retention, adding closed-loop evidence to the label comparison. It also maintains the same supervision interpretation across states with different remaining task lengths.

AuthorsOpenReview ↗

Response to Reviewer m8BF (Part 3/4): Question 2 (Complete runtime and budget-control overhead)

To Question 2 (Full-Model Latency and Budget-Control Overhead):

We appreciate your suggestions for making the efficiency evaluation more complete. We have added full-model latency and budget-control costs to Section 4.2, Table 7, Appendix D.2, and Tables D.2–D.3 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The analytical FLOPs model in Appendix D.6 does not include the vision encoder, action-head feed-forward layers, or the DVA predictor. We therefore timed complete model calls, including those operations as well as the language decoder, calibration, token selection, and the rest of action generation. Measurements use an exclusive A800, identical inference settings within each architecture, and the Table 2 sampling procedure across all four LIBERO suites. The resulting speedup thus includes the budget module’s execution cost.

ArchitectureStarVLA-πStarVLA-πStarVLA-πStarVLA-GR00TStarVLA-GR00TStarVLA-GR00T
ComponentVLMAction headFull model callVLMAction headFull model call
Full tokens (ms)160.8181.5348.9161.621.1189.3
DVA (ms)135.6149.0291.2143.318.9168.7
Speedup1.19×1.22×1.20×1.13×1.12×1.12×

Both components benefit from the shorter visual context: it reduces subsequent VLM computation and the action head’s conditional key/value projections and cross-attention. Because action-token counts, action-side feed-forward computation, and denoising iterations stay fixed, the comparison measures the savings from compressing the visual condition with budget control included.

The StarVLA-π and GR00T comparisons include their corresponding predictors and calibrators. As reported in Appendix D.2, predictor latency is below 1 ms and is already included in both the VLM and full-model measurements:

ArchitectureMean predictor-call latency (ms)
StarVLA-π0.91
StarVLA-GR00T0.98

To assess the complete cost of dynamic allocation, including calibration and token selection, we also compare against static TQA at matched mean retention and identical inference and timing settings:

ArchitectureStatic TQA latency (ms)DVA latency (ms)Static TQA speedupDVA speedup
StarVLA-π288.8291.21.21×1.20×
StarVLA-GR00T168.5168.71.12×1.12×

With all operations included, DVA differs from static TQA by less than 1% in full-model latency and retains a speedup over full tokens on both heads. Thus, per-step allocation runs at a cost close to fixed-budget pruning. Appendix D.5 specifies the timing setup.

Reviewer_nb1TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dynamic pruning method for VLA models, motivated by the observation that different tasks and control steps tolerate different levels of visual-token reduction. The method uses counterfactual suffix rollbacks across a range of retention ratios to compute “Decision Importance” (DI) combining task success or failure with additional execution steps, which measures sensitivity to sustained pruning from the current step. This signal is then converted into per-step retention ratios. Experiments demonstrate that the method achieves higher task success rates and inference efficiency.

Critical strengths
  1. The paper addresses a practical problem in VLA inference. Using task success or failure together with additional execution steps to guide token allocation is an interesting approach, and the lightweight predictor makes the offline signal usable during deployment.

  2. Transfer experiments are particularly valuable given the method’s reliance on task-specific rollout supervision. The paper explores reuse and adaptation across different pruners, action heads, and task environments including the real physical world.

Critical weaknesses
  1. There is a gap between the observations and the actual method. The observations mainly motivate dynamic pruning and the need to consider execution cost, but do not sufficiently justify the choice of counterfactual suffix rollback or its advantage over other dynamic pruning signals.

  2. The mapping from supervision to the deployed budget needs further validation. The appendix shows that single-step pruning produces a weaker sensitivity signal (therefore it is not feasible in this methodology), but this does not justify why fixed-ratio suffix supervision is a reliable basis for per-step dynamic allocation. Also, the authors acknowledge that quality curves can be non-monotonic. Why is it appropriate to apply a running maximum before estimating the minimum retention ratio? How much do this processing and the order-preserving assumption between DI and retention ratio affect the final results?

Questions for authors

Compared with methods that dynamically adjust pruning strength without counterfactual rollouts, such as VLA-IAP and SpecPrune-VLA, what is the methodological advantage of learning per-step budgets from counterfactual suffix rollouts? Comparisons with a broader range of dynamic pruning baselines would help clarify this advantage.

6 comments
AuthorsOpenReview ↗

Response to Reviewer nb1T (Part 1/3): Weakness 1 (Why use counterfactual suffix supervision?)

To Weakness 1 (Counterfactual Suffix Supervision):

Thank you for your constructive feedback on the motivation of our method. We have expanded the discussion of closed-loop supervision in Section 3.1 and Appendices A.3 and B.3 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

DVA’s budget must preserve the remaining manipulation under sustained pruning. VLA-IAP and SpecPrune-VLA use signals relevant to the current interaction or action to reduce redundant visual computation, but those signals do not directly measure how much pruning the remaining task can tolerate [1, 2]. This distinction matters in closed-loop control: an action changes the next observation, so information loss can accumulate through later interactions and cause failure or additional completion steps. Information useful for the current action alone therefore does not determine the budget needed for the remaining task.

We use counterfactual suffixes to measure those downstream consequences directly. Starting from the same state, each candidate ratio is held until success or timeout and compared with a full-token reference through failures and additional completion steps. DI summarizes this effect, allowing the deployed predictor to estimate cumulative pruning sensitivity from the current observation and instruction.

Holding the ratio through the suffix is important because a single pruned step can be followed by full-token observations that compensate for the temporary information loss. With a full-suffix intervention, later corrections also operate under a restricted budget, making the label reflect sustained compression. Appendix B.3 and Table B.3 show the resulting difference:

Supervision horizonLabels with DI < 0.02Labels with DI > 0.5
H=149.6%3.5%
Full suffix4.8%18.1%

Nearly half of the single-step labels are close to zero, making sustained-pruning sensitivity difficult to distinguish across those states. Full suffixes provide more differentiated labels because they also constrain the subsequent correction process, aligning the supervision with continued pruning at deployment.

AuthorsOpenReview ↗

Response to Reviewer nb1T (Part 2/3): Weakness 2 (Suffix labels, the running maximum, and calibration)

To Weakness 2.1 (Fixed-Ratio Labels and Dynamic Allocation):

Your comments have helped us explain the link between supervision and deployment more clearly. We have clarified this connection in Section 3.1 and the Figure 3 caption in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Fixed-ratio suffixes measure a state’s sensitivity by comparing its remaining-task outcomes across candidate ratios. DI summarizes the resulting quality curve, and calibration uses the minimum ratio meeting the quality requirement to convert predicted sensitivity into a budget. The fixed intervention thus defines what the predictor should estimate at that state.

After an action chunk, both the observation and remaining task change, so DVA predicts DI again and assigns a new budget from the new state. A fixed ratio therefore defines the offline measurement of sensitivity, while repeated predictions implement the online allocation rule. Calibration and EMA are applied as the budget is recomputed at each control step.

To Weakness 2.2 (Running Maximum and Threshold Interpolation):

Thank you for your suggestions for clarifying the calibration procedure. We have expanded its explanation and the threshold-interpolation analysis in Section 3.2 and Appendix C.4 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The critical ratio is the first tested retention whose quality reaches the threshold. A running maximum preserves that crossing even if a later candidate has lower quality: before the first crossing, all observed values and their cumulative maximum remain below the threshold, and at the crossing both reach it. Let rkr_k denote the evaluated ratios in ascending order, including the full-token anchor r=1r=1. For threshold q=1−τq=1-\tau, this gives

=\min\left\lbrace k:\phi\left(r_k\right)\geq q\right\rbrace.$$ The running maximum is the smallest non-decreasing envelope above the observed curve, and the identity shows why using it leaves the minimum qualifying discrete candidate unchanged. The identity is stated in **Equation (C.9)**. The finite candidate grid provides a first qualifying tested ratio, but calibration benefits from a continuous target. We therefore interpolate between that candidate and its predecessor. The running-maximum envelope can raise the predecessor’s quality and move the interpolated crossing within this interval, while preserving which interval contains the crossing. The interpolated target is defined in **Equation (C.10)**. The envelope consequently affects extraction of the critical-ratio target. DI itself is computed from the original quality curve’s AUC, so the sensitivity label still retains the observed degradation across pruning strengths. This separates the continuous budget estimate used in calibration from the underlying supervision signal. > **To Weakness 2.3 (Order-Preserving Calibration):** We appreciate your guidance on explaining the assumptions behind calibration. We have expanded the discussion of order-preserving calibration in **Appendix C.4** in the updated submission, with changes highlighted in $\color{blue}{\text{blue}}$. The isotonic calibrator encodes the trend that greater pruning sensitivity calls for more visual information. The quality curves motivate this direction: if one state has no higher quality than another at every retention ratio, its DI is no smaller, and reaching the same quality threshold requires at least as much retention. Since actual curves can cross, calibration fits this overall trend while allowing individual targets to deviate from the fitted relationship. As **Section 3.2** and **Equation (8)** describe, the predictor supplies a relative sensitivity score and the calibrator fits that score to the measured minimum feasible retention. Isotonic regression pools adjacent samples that reverse the prescribed direction and minimizes weighted squared error under a non-decreasing constraint. The data therefore determine the mapping’s shape, while the constraint prevents local reversals from assigning less information to higher predicted sensitivity. The running maximum preserves the first qualifying tested ratio within one state’s quality curve, whereas isotonic regression constrains the score-to-budget fit across states. The former leaves the DI label unchanged because DI uses the original curve’s AUC. The latter encodes the direction of the allocation rule while fitting its shape to measured critical ratios.
AuthorsOpenReview ↗

Response to Reviewer nb1T (Part 3/3): Question 1 (Comparison with dynamic pruning methods)

To Question 1 (Comparison with Dynamic Pruning Methods):

Thank you for your recommendations for broader comparisons with related work. We have added SpecPrune-VLA results to Section 4.2 and Table 2, and expanded the discussion of pruning objectives in Appendix A.3 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

VLA-IAP and SpecPrune-VLA relate observation or action signals to pruning strength through a specified rule [1, 2]. DVA learns that relationship from the execution consequences of sustained pruning: different ratios are tested from the same state, and failures and additional completion steps measure the effect on the remaining task. The budget objective therefore directly reflects downstream control performance, as explained in Section 3.1.

Because these executions take place offline, deployment only requires the learned budget module while preserving policy weights and the pruner’s scoring rule. Table 3 evaluates this integration across action heads and pruners. To assess its practical benefit against a dynamic pruning method, we also compared SpecPrune-VLA with DVA on OpenVLA-OFT. VLA-IAP’s official repository has not yet released implementation code [1].

MethodSuccess (%)Model-call latency (ms)Speedup
SpecPrune-VLA95.0106.41.01×
DVA97.088.81.20×

With the Table 2 timing procedure and the full-token reference from the same run, DVA achieves higher success at lower model-call latency. The comparison thus connects the execution-supervised budget to a practical performance and efficiency benefit.

References

[1] Jintao Cheng et al. Training-Free Interaction-Aligned Visual Token Pruning for Efficient Embodied Manipulation. arXiv:2603.22991v2, 2026.

[2] Hanzhen Wang et al. SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning. ICML, 2026.

If you have more questions or suggestions, please let us know, and we would be happy to have further discussions.

Reviewer_ij5eOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

To address the inefficiency of a uniform visual-token retention ratio in VLA models, this paper points out that the number of tokens to keep should follow the measured downstream cost of pruning. It therefore proposes Decision Importance (DI), a label built by counterfactual suffix rollback: restore the simulator, hold a candidate retention ratio to the end of the episode, and score the remaining rollout against a full-token reference. The resulting framework, DVA, adds two plug-in modules that turn DI into a budget online: a predictor that regresses DI from features the VLA already computes, and a per-head, per-pruner isotonic calibrator that maps the predicted score to a retention ratio.

The study covers LIBERO and LIBERO-PLUS, using OpenVLA-OFT plus three action heads over one Qwen3-VL backbone, crossed with FastV, SparseVLM and TQA. On OpenVLA-OFT, 37% mean retention is reported to give 97.0% success at a 1.20× speedup, against 96.8% at 1.00×; 22% gives 96.2% at 1.28×. At matched mean retention, DVA is reported to improve all 18 head × pruner × benchmark settings by 0.3–2.1 points. There is a Franka-arm study, layer and EMA ablations, a horizon comparison, and a cost audit.

Critical strengths

S1. The paper measures what "sustained" pruning means rather than assuming it — and the cheap alternative turns out to measure something else. Appendix B.3–B.4 holds the restored states, the ratio grid and the budget fixed and varies one thing: how long the ratio is held, one control step or to termination. The two label sets barely agree (rank correlation 0.10 over 4,882 paired steps), and the single-step version makes pruning look harmless — about half its labels sit at the bottom of the scale, against 5% for the suffix version. That is the paper's best argument that the budget has to be judged over the remaining rollout, and it is reusable: the horizon is a design choice with a large effect, not a detail.

S2. The operating point is a real result on the practical frontier, and the comparison is built so that the number can be trusted. On OpenVLA-OFT, 37% mean retention gives 97.0% success at a 1.20× end-to-end speedup, for 1.1 ms of added latency per step; 22% still gives 96.2% at 1.28×. The alternatives bracket the trade-off rather than beat it: FastV is faster (1.35×) but pays 2.4 points against the full-token baseline, and VLA-Cache holds that baseline (96.8%) but is only 3% faster. What makes the "no cost" half believable is the design: one harness for the three re-implemented pruners, static arms matched to DVA's mean retention, the residual budget mismatch bounded at 0.41 pp of full-token cost (Table D.4), FastV placed at a layer that costs it accuracy (D.3), and the offline cost disclosed in evaluation-equivalent units (Table B.5). Three action heads and a Franka table broaden the base.

Critical weaknesses

W1. The paper's account of repeated runs contradicts itself, and two of its claims do not survive its own numbers.

The clearest instance is the repeat statement. Appendix D.5 argues "One seed suffices because the evaluation is deterministic by construction"; the footnote under Table D.4 reports that the StarVLA-OFT results with FastV and TQA "are averages over three repeated runs". If the evaluation is deterministic, three runs are identical and the footnote describes nothing; if it is not, the remaining settings need the same treatment, and the +1.8 cell it covers reports no spread.

The first claim is from the contribution list: DVA "even surpasses the full-token baseline while retaining only 37% of visual tokens". In Table 2 that is 97.0% against 96.8% over 2,000 episodes — four episodes, and no further reporting can make it significant, since with a net of four the exact paired test cannot fall below p ≈ 0.13. Eighteen positive signs across the settings are real evidence of direction, and the paper is entitled to that; they are not evidence of magnitude. The median gain is 0.8 points (16 episodes on LIBERO, 80 on LIBERO-PLUS), 15 of the 18 settings remain below their own full-token baseline, and no interval, test or per-episode outcome is reported for any of them. The paired counts exist in the logs, and Q2 asks for them.

The other claim is the real-robot one, which is bounded by the paper's own limitation: §4.2 says the Franka results "confirm that our simulation findings transfer to the physical world", while §5 says the labels "require resettable environments and are costly to approximate on real robots", and §4.1 fine-tunes on "4 labeled steps per task" without saying where they came from. Four labels cannot show that the label construction transfers.

W2. The gains are attributed to per-step allocation, but nothing separates per-step from per-task allocation, and the authors' own ablations point to the latter.

The abstract claims a ratio "determine[d] online based on the current observation and instruction". Yet §4.3 reads its own smoothing ablation as showing that "the predictor primarily captures slow, phase-level variation rather than high-frequency fluctuations": at α = 0.25 the deployed ratio's dispersion drops to 69–75% of the unsmoothed one, and success moves inside a 94.8–97.3 band with no trend across α. The predictor explains R² = 0.416 of per-step DI variance (Table C.6), and Figure 2 shows the measured quantity itself is a smooth function of rollout progress.

If the sensitivity is low-frequency, a ratio held constant within a task would capture much of it — and it is free at deployment, needing one ratio sweep instead of 187,064 + 66,550 rollbacks (Table B.5). The static arm in Table 3 is constant only within a suite, so it cannot separate the per-task from the per-step component, and no per-task or progress-conditioned control is run. Those two evaluations, plus an oracle arm that swaps the predictor for the measured DI, would show whether the 22M predictor earns its cost.

The direction is consistent and the OpenVLA-OFT operating point is useful; the claim at issue is the one the abstract makes for the mechanism.

Additional concerns.

  • TQA supplies the labels and is also the headline pruner. The predictor is trained only on TQA-derived DI (§4.1), and Table 2 has no static TQA row, so on the main backbone the value of allocation cannot be separated from the value of TQA scoring. The StarVLA table does supply that comparison (92.9 → 94.7); the headline table does not.
  • The long-horizon reading does not survive the other benchmark. §4.2 writes "This gain includes a 5.8-point improvement on the Long suite (89.0% to 94.8%), highlighting the benefit of adapting the budget on extended manipulation tasks." The same Long column on LIBERO-PLUS moves the other way in six of nine settings (−1.5 to +1.1).
  • For π and GR00T the efficiency is decoder FLOPs, not latency. Table D.2 puts the VLM at 13.7% of StarVLA-π's 375.6 ms and 25.1% of GR00T's 190.7, so the reported decoder savings correspond to roughly 1.05× and 1.10× end-to-end at those operating points.
  • Labelled steps and deployed steps are different populations. 11.2% of selected StarVLA steps and 14.6% of OpenVLA-OFT steps are dropped because the full-token reference fails (B.5): they are neither trained nor calibrated on, yet they are scored at deployment.
  • "Plug-in" holds at inference only. Every new head–pruner pair needs its own calibration batch (118 episodes, 3.4–5.0 evaluation equivalents; B.5).
Questions for authors

My rating turns on W1, W2 and Q1–Q2. Q3–Q7 are narrower.

  1. Where do the calibration data come from? τ* is chosen as the largest tolerance within a success-rate-drop budget on a "fixed held-out set" (Eq. 9), and for Table 3 it fixes the mean retention the static arms are matched to. Is that set the 118-episode (86–93 on LIBERO-PLUS) calibration batch of B.5, or the evaluation episodes? And B.5 says the OpenVLA-OFT calibrator "is fitted directly on the predictor training set" while the StarVLA calibrators use a separate batch: is the 37% row of Table 2, and with it the abstract's headline, produced by an in-sample calibration?

  2. Paired counts. For all eighteen Table 3 settings — or at least the OpenVLA-OFT 37% row and the four largest gains — please report the discordant counts (static-only correct / DVA-only correct) so a McNemar test can be computed, together with the three-run standard deviation for the repeated cells. With shared initial states and deterministic decoding this costs no additional evaluation.

  3. VLA-ADP at matched speedup. It is reported only at Ratio = 0.50 and 0.30 (1.09× speedup at 96.3%, 1.11× at 94.4%), and its speedup barely moves between them. Can you add an operating point tuned to 1.20×, so that the comparison with DVA's 97.0 is at equal cost rather than at each method's default?

  4. The entropy trigger as a free budget rule. B.2 already uses ViT patch-attention entropy to decide which control steps matter. Allocating the budget directly from that statistic needs no rollbacks, no labels and no predictor. Has it been tried, and how does it compare at matched mean retention?

  5. Real-robot labels. §4.1 fine-tunes the predictor "on 4 labeled steps per task" for the robot experiments. Where do those four labels come from, given that DI requires holding a ratio to termination from a restored state? Please describe the reset protocol, and whether the ten trials per task vary the initial state.

  6. Figure 2. Every curve falls to the bottom of the axis as rollout progress approaches 1, where little of the rollout remains. How much of the within-rollout variation survives if the final part of each rollout is excluded, and is it larger than the variation across the 50 initial states of the same task?

  7. Ablation precision. Tables 5 and 6 use the first 10 initial states of every task — 400 episodes — with a single run. With no interval, how much of the spread across α and τ (94.8 to 97.3) and across L (92.5 to 98.0) is meaningful, and does "L = 1 within 1.8 points of the best" survive a paired comparison?

12 comments
AuthorsOpenReview ↗

Response to Reviewer ij5e (Part 1/7): Weakness 1 (Evaluation protocol and the scope of the claims)

To Weakness 1.1 (Evaluation Determinism and FLOPs Stability):

Thank you for your careful reading and suggestions for improving the presentation. We have clarified the evaluation protocol in Appendix D.5 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The evaluation setup uses LIBERO’s stored initial states and StarVLA-OFT’s regression action head, without random action sampling. Within this setup, we repeated the FastV and TQA configurations three times to check the stability of the FLOPs recorded during execution.

Under the accounting in Appendix D.6, each inference call’s FLOPs depend on its token count, and those costs are aggregated over the rollout. Identical token and query sequences therefore give identical FLOPs. The recorded ranges were 0.0038 percentage points for FastV and 0.0439 for TQA, indicating that the computational budgets remained stable across the repeated runs.

To Weakness 1.2 (Success-Rate Claims and Paired Evidence):

We appreciate your advice on stating the conclusions more precisely. We have revised the performance claim in the Introduction and Section 4.2, and added paired results to Appendix E.2 and Table E.3 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The phrase “even surpasses” placed too much emphasis on the small difference between 97.0% and 96.8% success in Table 2. The main finding is that DVA retains approximately 37% of visual tokens while maintaining comparable task performance at a 1.20× speedup. The revised wording therefore describes comparable success at lower inference cost, consistent with the Abstract.

For fixed-budget pruning, the paired analysis counts episodes in which only DVA or only the static baseline succeeds under the same initial condition. DVA-only successes are more numerous in every setting, providing evidence for the direction of the allocation gains.

To Weakness 1.3 (Real-Robot Adaptation Protocol):

Thank you for your suggestions for making the experimental protocol clearer. We have expanded the real-robot adaptation protocol in Section 4.2 and Appendix D.7 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

The real-robot results evaluate deployment after fine-tuning the simulator-pretrained budget predictor on four labeled states per task, as described in Section 4.1. Each state’s label requires comparing several candidate retention ratios. We therefore reset the robot and objects, replay the full-token prefix to that state, and hold each candidate ratio until success or timeout. The resulting failures and additional completion steps relative to the full-token reference are combined into DI, using the same quality-curve definition as in simulation.

The finding is deployment on the two physical tasks after this limited real-robot adaptation. The Conclusion in Section 5 also states this adaptation protocol. Evaluation uses a fixed initial arrangement for each task.

AuthorsOpenReview ↗

Response to Reviewer ij5e (Part 2/7): Weakness 2 (Within-task allocation, progress controls, and oracle DI)

To Weakness 2 (Within-Task Allocation, Progress Controls, and Oracle DI):

We appreciate your constructive suggestions for strengthening the controlled comparisons. We have added budget-control results to Section 4.3, Table 8, and Appendix E.1, and budget-variation analyses to Appendix E.3 and Table E.4 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

EMA reduces step-to-step prediction noise while allowing the budget to differ across stages of the same task, because DVA still predicts DI from each current observation before calibration and smoothing. To quantify the variation that remains, we computed retention standard deviation within each episode and averaged these values across episodes. At approximately 37% mean retention, the result is 9.79 percentage points, showing that the smoothed budget remains dynamic within tasks. The revised Section 4.3 explains the smoothing choice more precisely: α=0.5 preserves over 80% of the unsmoothed retention dispersion while maintaining similar success. The analysis also covers trajectory prefixes with the rollout tail removed.

Following your suggestion, we used task-static and progress-only controls to distinguish two possible sources of the gains. Task-static fixes each task’s ratio to its average DVA budget, thereby controlling for differences in average allocation between tasks. Progress-only allows the budget to change within a task, but determines it from progress alone. It divides normalized progress into five equal-width bins, averages the minimum required retention from DVA’s counterfactual training labels within each bin, and linearly interpolates a shared curve. At deployment, elapsed steps are divided by the task duration estimated from training trajectories, and a fixed offset per suite matches its mean budget to DVA.

With the frozen policy, TQA scoring, and pruning layer held fixed, comparing DVA with progress-only tests whether using the current observation to allocate budgets improves task success over allocation based on rollout progress. Because the progress curve is learned from counterfactual labels, this control still requires trajectory collection and labeling. The control columns below report the selected supplementary evaluations.

Mean retentionDVA success (%)Task-static success (%)Progress-only success (%)
37%97.096.796.9
28%96.596.396.0
22%96.295.195.4

The success rates are close when more tokens are retained, whereas DVA achieves higher success at the lower budgets. At approximately 22% retention, its gains are 1.1 percentage points over task-static and 0.8 over progress-only. Observation-dependent allocation therefore preserves success more effectively at this lower budget.

The closed-loop controls also connect the predictor’s offline fit to its deployment role: the R² of 0.416 in Table C.6 measures explained variance in DI labels, while task success evaluates the budget obtained after calibration and EMA. An oracle comparison would require new counterfactual suffixes at every state reached by the controller, because each existing DI label belongs to the state from which it was collected and cannot be reused on another trajectory by time index.

AuthorsOpenReview ↗

Response to Reviewer ij5e (Part 3/7): Additional Concerns 1–3 (TQA controls, Long-suite results, and runtime)

To Additional Concern 1 (Static TQA on the Main Backbone):

Thank you for suggesting a more informative baseline comparison. We have added the global-static comparison to Section 4.3, Table 8, and Appendix E.1 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

To separate budget allocation from TQA scoring on OpenVLA-OFT, we added a global-static TQA control with identical policy weights, scores, and pruning layers. Its single retention ratio is shared across all four suites and matches DVA’s overall mean retention. The comparison therefore changes how the budget is allocated while keeping token selection fixed. The static column reports the selected supplementary evaluations.

Mean retentionDVA success (%)Global-static TQA success (%)
37%97.096.6
28%96.595.7
22%96.295.2

Against the full-token baseline’s 96.8% success in Table 2, the difference is small at the higher retention ratio and grows to 1.0 percentage point at approximately 22%. Since the scoring rule is unchanged, the higher success at that budget reflects a benefit from dynamic allocation.

The same distinction applies beyond TQA: Table 3 holds the pruner fixed within each matched-retention comparison across three StarVLA action heads, FastV, TQA, and SparseVLM, on LIBERO and LIBERO-PLUS. DVA improves average success in all 18 settings, showing that the observed allocation benefit also extends to the other evaluated scoring rules and architectures.

To Additional Concern 2 (Interpretation of Long-Suite Results):

Your feedback has helped us make the interpretation of the Long-suite results more precise. We have revised the interpretation of Long-suite results in Section 4.2 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

DVA allocates tokens according to the pruning sensitivity of the states encountered during execution, as estimated by its predictor and calibrator. Task duration alone therefore does not determine the gain. The increase from 89.0% to 94.8% on LIBERO-Long describes the benefit observed in that configuration, rather than a general relationship between task length and improvement.

The Long-suite results on LIBERO-PLUS illustrate why the gain should be interpreted for each configuration: six of nine settings decrease, with changes ranging from −1.5 to +1.1 percentage points. As Section 4.2 notes, individual suite scores do not improve uniformly. We have therefore removed the generalization about extended manipulation tasks from that section while retaining the specific LIBERO-Long result. The overall comparison remains the mean success across all suites in Table 3, which accounts for both increases and decreases in individual suites.

To Additional Concern 3 (StarVLA Runtime and Component Costs):

Thank you for your recommendations for a more complete runtime evaluation. We have added StarVLA latency results to Section 4.2, Table 7, Appendix D.2, and Table D.2 in the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Visual-context compression reduces computation in both the language decoder and the π and GR00T action heads, because these heads consume the compressed context. Fewer conditioning tokens reduce their key/value projections and cross-attention, while action-token counts, action-side feed-forward computation, and denoising iterations remain unchanged. To measure the combined effect of these savings and budget-control overhead, we timed the VLM, complete action head, and full model call on an exclusive A800 under identical inference settings within each architecture. The VLM measurement includes visual encoding and the language decoder.

ArchitectureStarVLA-πStarVLA-πStarVLA-πStarVLA-GR00TStarVLA-GR00TStarVLA-GR00T
ComponentVLMAction headFull model callVLMAction headFull model call
Full tokens (ms)160.8181.5348.9161.621.1189.3
DVA (ms)135.6149.0291.2143.318.9168.7
Speedup1.19×1.22×1.20×1.13×1.12×1.12×

Both components become faster, yielding full-model speedups of 1.20× on π and 1.12× on GR00T after including prediction, calibration, and token selection. Thus, the measured gain includes conditional computation saved in the action head as well as in the VLM. Appendix D.5 specifies the timing setup.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chairs,

Thank you for your consideration and the opportunity to discuss our work with the reviewers. Following our responses, ij5e raised the rating from 3 to 4 and recommended acceptance, and KGbT raised it from 2 to 3, while nb1T and m8BF remain at 2. m8BF said they would consider raising the rating if the concerns were addressed. We have replied to both and await their final feedback. Below, we address all four reviews and follow-up questions.

Concern 1. Supervision for the current token budget (ij5e, nb1T, m8BF)

Response: Sensitivity to sustained pruning measures how maintaining a given visual token pruning ratio throughout the remaining rollout affects task success and completion steps. We explained why it can guide the current budget (Section 3.1) and the rationale for using a running maximum in calibration (Appendix C.4).

Evidence: Unlike the other dynamic budget signals we evaluated, including entropy, DVA’s signal reflects pruning’s effects on future task completion. In these comparisons, DVA achieves higher success than methods using the alternative signals (Tables 2 and E.2). Within DVA, we compared supervision lengths: full-suffix supervision gives slightly higher success than eight-step supervision at similar retention (Table B.5). These results support sensitivity to sustained pruning as a useful proxy for the current budget.

Concern 2. Contribution of within-task allocation (ij5e, m8BF)

Response: Budget variation persists after excluding the final part of each rollout (Table E.5). To test its benefit, we compared DVA with fixed global, fixed per-task, and progress-only budgets at matched mean retention, keeping the policy, token scoring, and pruning layer fixed (Table 8).

Evidence: DVA, global-static, task-static, and progress-only achieve similar success at 37% retention. At 22%, DVA achieves 96.2%, versus 95.1% for task-static and 95.4% for progress-only. Thus, adapting budgets to the current observation can improve success under tighter budgets.

Concern 3. Recent VLA baselines and comparable inference cost (ij5e, nb1T, m8BF, KGbT)

Response: We expanded the evaluation to seven VLA acceleration methods, covering fixed and dynamic budgets, learned pruning, caching, and model compression, including comparisons at similar inference cost (Table 2).

Evidence: On OpenVLA-OFT, DVA achieves 97.0% success at a 1.20× speedup, the highest success rate among the VLA acceleration methods evaluated at similar inference cost.

Concern 4. Actual inference speed across action heads (ij5e, m8BF, KGbT)

Response: We timed complete model calls, including predictor and calibrator latency, and separately timed the backbone and action head. For FLOPs, we clarified the accounting and the use of repeats to check stability (Appendices D.5–D.6).

Evidence: DVA accelerates model calls by 1.20× on StarVLA-π and 1.12× on GR00T, with reductions in both backbone and action-head latency (Table 7 and Table D.2). DVA thus provides net speedups for both action heads.

Concern 5. Statistical evidence and ablation reliability (ij5e)

Response: We reported paired counts and exact McNemar tests for all 18 Table 3 settings and 9 Table 8 comparisons (Tables E.3–E.4). To check the ablation choices, we repeated layer and smoothing comparisons on random and evenly spaced subsets (Tables E.6–E.7).

Evidence: All paired comparisons show higher success for DVA. Across subsets, the default layer’s success is close to that of the best tested layer, while default smoothing gives similar success to no smoothing. This consistency supports both default choices.

Concern 6. Calibration data, overlap, and reference failures (ij5e)

Response: We documented label-collection costs, calibration data, and reuse across tolerances (Appendix B.5). We checked task coverage after reference-failure filtering and gains beyond the calibration and tolerance-selection subsets.

Evidence: All 40 tasks retain valid labels in both the OpenVLA-OFT and StarVLA-OFT collections, preserving task coverage. Gains also extend beyond the calibration and tolerance-selection subsets, averaging 1.3 points on LIBERO and 0.6 points on LIBERO-PLUS (Table B.6).

Concern 7. Real-robot deployment (ij5e, m8BF, KGbT)

Response: We clarified the real-robot protocol.

Evidence: The simulation-trained predictor is adapted with a small real-robot label set collected through resets and action replay, then selects budgets from observations (Appendix D.7). Figure D.1 shows representative camera sequences. DVA therefore supports real-robot deployment after limited adaptation without online rollbacks.

In summary, we believe our responses have addressed the reviewers’ concerns. We are looking for your feedback and suggestions. Thank you again for your time.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.