acceptodds
Under review as a conference paper at ICLR 2027

How Deep Should a VLA Think When Thinking Costs Time? Budget-Constrained RL for Early Exit

Abstract

Vision-language-action (VLA) models are becoming capable generalist robot policies, but they are large, and their inference is slow on robotic hardware. Methods that make VLAs efficient are evaluated with the simulator paused while the model computes, as if the world waited. On a robot it does not: computation is consumed during control, and the value of an extra layer depends on how much control time it costs. We show in a controlled intervention that the same inference latency lowers task success by 3 to 40 points depending on how it reaches the robot, and we therefore ask how deep a VLA should think when thinking costs time. We cast adaptive depth as decision-making under a budget on the fraction of time spent inferring, and learn, with budget-constrained reinforcement learning, a lightweight policy that decides at each intermediate exit of a frozen VLA whether to act or to keep computing. Trained once per task suite in a simulator that keeps running while the model thinks, it runs \SPEEDFULL faster per inference than the full model on LIBERO-Plus, trading success for time as the budget sets, is on par with early-exit rules calibrated separately on every task at the same compute without any per-task calibration, and is ahead of an AVA-VLA-style exit gate on every task where their compute matches.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Action Models, Reinforcement Learning, Early Exit, Efficient AI, Efficient VLA, Constrainted RL

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1523OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between paused-simulation benchmarks and real-time robotic deployment, where VLA inference latency consumes physical control steps and degrades closed-loop task execution. To mitigate this latency penalty, the authors formulate adaptive model depth as a budget-constrained reinforcement learning problem, training a lightweight MLP scheduler via PPO to decide at intermediate exit rungs of a frozen backbone whether to execute the current action or compute deeper layers. The scheduler is initialized with an action-difference threshold heuristic (DeeR-style) and evaluated on LIBERO-Plus under an unpaused simulator that charges compute in whole control steps. The authors report that the method achieves inference speedup over the full 32-layer backbone, matches task-specific heuristic exit baselines without per-task tuning, and demonstrates that real-time execution constraints can make intermediate-depth models outperform full-depth computation.

Critical strengths
  1. highlights artificial advantage of paused simulators. Contains a reasonable formulation of compute cost (burned control steps), exposing how physical staleness degrades closed loop execution
  2. provides empirical evidence to validate the counterintuitive premise that shallower execution can outperform full-depth models in unpaused environments
Critical weaknesses
  1. the main contributed method fails to justify its complexity over simple heuristic baselines. All the performance results are within error margins or favor the heuristic
  2. Method relies on a lot of adhoc design choices, like handcrafted network weights and magic constants. Paper writing and experiments seem hastily done or incomplete, such as 5 action chunks used in Table 1 while method used 8 chunks, evaluating DeeR on a single task, omitting AVA-VLA baseline, uncleaned experiment artifact labels. No real robot validation despite criticizing simulation benchmarks.
Questions for authors
  1. Given that the RL policy was initialized to mimic DeeR, what concrete empirical benefit does the constrained RL framework provide over simply sweeping the heuristic threshold t?
  2. Have you evaluated this method on a physical robot platform, and does the discrete step-burning model accurately capture real-world communication jitter and asynchronous motor buffer execution?
  3. In Table 4, statically truncating the backbone at Layer 24 achieves the highest overall success (0.786), finishes fastest in robot time (7.11 s), and runs 1.33× faster than full depth. Under what deployment regimes would a practitioner justify accepting the lower success rate of the dynamic scheduler (0.714) over simply pruning to Layer 24?
  4. Why does Table 1 evaluate latency using 5-action chunks, whereas the proposed method and backbone fix the chunk size to 8 actions?
  5. Why does Table 2 compare against DeeR-style thresholding on only 1 task, while evaluating full depth across 6 tasks?
  6. Why is the AVA-VLA baseline listed in the caption of Table 4 completely absent from the actual table data?
  7. What is the ablation performance when initializing the exit head randomly versus applying the proposed manual transformation? How sensitive is convergence to the chosen constants?
Reviewer_t6Z8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

It talks about VLA early exit when inference latency consumes control time. A lightweight exit policy on a frozen OpenVLA-OFT is initialized from an action-consistency rule and optimized using budget-constrained PPO. Experiments on LIBERO-Plus examine the trade-off between task success and inference time.

Critical strengths
  1. The problem is practically relevant. Evaluating efficiency with the simulator paused can conceal the control-time cost of inference.

  2. The controlled comparison of latency execution modes is informative. The paper also candidly reports limited RL gains and substantial degradation on fresh tasks.

Critical weaknesses
  1. The evaluation does not adequately test the central motivation. The LIBERO-Plus experiments lack dynamic tasks requiring timely responses, while the predominant pause mode mainly charges latency against episode deadlines.

  2. Real-world validation is missing. Given the paper’s deployment-centered claims, real-world closed-loop validation is essential evidence rather than merely future work. Real-world tasks with explicit response windows, such as conveyor-belt picking or moving-target manipulation, are needed to establish improvements in closed-loop success and task throughput, including comparisons with asynchronous execution.

  3. Limited contribution and unclear necessity of RL. RL only adjusts exit decisions on a frozen VLA and mainly shifts an existing consistency threshold without significantly improving success. A simple threshold-search baseline using the same development data and validation budget is essential to justify the additional machinery.

  4. Unclear generality beyond OpenVLA-OFT. The method relies on distilled intermediate action exits and a predefined latency table. Its applicability to diffusion/flow-based VLAs remains untested, particularly the cost of decoding intermediate actions and the reliability of consistency-based stopping under stochastic sampling.

  5. Insufficient budget and trade-off analysis. Most experiments use one budget, and budget violations remain possible. On fresh tasks, success drops from 53.1% to 24.5%, without a demonstrated advantage over the matched-budget constant mixture. Multiple-budget success–latency curves and violation statistics are needed.

  6. Weak writing and presentation. Figure 1 emphasizes joint scheduling absent from the main evaluation, and inconsistent typography reduces readability. Overall, the work reads as a narrow exploratory study with an insufficiently established technical contribution.

Questions for authors
  1. What existing evidence demonstrates benefits in dynamic, latency-sensitive control beyond deadline consumption? Additional results on simulation benchmark with dynamic tasks are required.

  2. Real-world experiments are required in Weaknesses #2.

  3. Does RL outperform simple threshold search with the same development data and validation budget? See Weakness #3.

  4. What modifications would diffusion/flow-based backbones require, and would intermediate action decoding preserve the computational savings? See Weakness #4.

  5. Multiple-budget success–latency curves and violation statistics are needed. See Weakness #5.

  6. Repolish the writings and chart format. Ensure consistency of font within different tables.

Reviewer_QJ8gOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between VLA inference depth and control latency. It first examines how inference delays affect task success under different execution schemes, then proposes budget-constrained reinforcement learning for early exit. Intermediate action exits are trained through distillation on a frozen OpenVLA-OFT backbone. A lightweight MLP, initialized to implement an action-consistency rule and optimized with PPO, decides whether to exit or continue computing. Experiments on LIBERO-Plus evaluate task success, computational cost, and generalization to held-out tasks.

Critical strengths

The paper addresses an interesting and practically relevant question by connecting inference cost to closed-loop control performance. While prior work explores VLA early exit, explicitly accounting for the control time consumed by computation provides a useful perspective beyond isolated latency or FLOPs measurements.

Critical weaknesses
  • Limited experimental coverage and deployment evidence. Evaluation is restricted to LIBERO-Plus perturbations based on LIBERO-Spatial, without results on Object, Goal, Long, or a systematic evaluation across the standard LIBERO suites. This limits conclusions about generalization across manipulation tasks and longer horizons. Moreover, there are no real-robot experiments, and latency is charged through a model that discretizes measured times into control steps. The evaluation therefore does not establish the actual deployment overhead of intermediate readouts, gating, and dynamic execution, or their effect on real-world closed-loop performance. The transferability of the simulation-trained exit policy also remains unverified.

  • Cross-architecture applicability and the connection to the motivation remain unclear. The method requires usable action predictions at intermediate language-model layers and is validated only on OpenVLA-OFT. How this design extends to architectures with different action-generation mechanisms, such as π₀.₅, is not explained. Furthermore, the latency motivation uses π₀.₅, whereas the proposed method is evaluated on OpenVLA-OFT. This leaves an incomplete connection between the diagnosed latency problem and the demonstrated benefits under the same model and control setting.

Questions for authors
  • Q1. What is the full PPO training cost, and how would the policy transfer to real tasks? How would simulation-to-real differences affect exit decisions, budget compliance, and task success?

  • Q2. Does PPO provide enough benefit to justify its additional cost? A supervised gate could be trained using labels derived from discrepancies between intermediate and final-exit actions. Please compare against this baseline using the same exits, input features, and matched compute, reporting both success and training cost. Given that Section 6.3 finds PPO mainly relaxes the consistency threshold, what benefit does it offer over supervised gating or a global threshold search?

  • Q3. When does the policy choose shallow versus deep computation? Table 10 analyzes exit frequencies and action consistency, but the relationship to task state remains unclear. Representative trajectory visualizations and aggregate statistics linking exit depth to manipulation phases, prediction errors, and outcomes would help establish whether the policy learns meaningful state-dependent behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.