acceptodds
Withdrawn from ICLR 2027

WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies generate robot controls autoregressively, making closed-loop latency dominated by repeated target model forward passes. Speculative decoding reduces this cost by verifying blocks of draft action tokens in parallel, and recent VLA methods relax token-level acceptance because small differences in action token space often map to similar continuous controls. However, a fixed token-distance tolerance does not account for the scene-dependent consequences of action deviations: deviations that are harmless in free space can cause collisions or grasp failures near contact. We propose WA-SpecDec, a world-aware speculative decoding framework that injects world-model-derived physical scene information during the VLA prefill stage. The resulting world-aware prefill states are shared between draft generation and target verification, while the relaxed acceptance rule remains unchanged. Evaluated across four VLA policies and three relaxed acceptance rules on LIBERO, SIMPLER-Env, and four real-robot tasks, WA-SpecDec improves the trade-off between inference speed and task success. Across five equally weighted policy–benchmark settings, it increases task success by an average of percentage points over the speculative baselines and achieves an average inference speedup of over autoregressive decoding of the base policies. For the two policies evaluated on LIBERO, it reduces the near-contact failure (NCF) metric by on average relative to the speculative baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, speculative decoding, world models, robot learning, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_krd2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a visual bias module trained with action prediction and auxiliary next-frame latent prediction. This bias enriches the shared prefill states of the target and draft models while leaving the relaxed acceptance rule unchanged. Experiments report improved success–latency trade-offs across several VLA policies, simulation benchmarks, and physical-robot tasks.

Critical strengths
  • The evaluation is relatively broad, covering four policies, three relaxed acceptance rules, and four real-robot tasks. Consistent improvements across these settings make the empirical benefit more credible than a result obtained with one policy or verifier.

  • The matched continued-training control and world-aware autoregressive evaluation are useful. They address an important alternative explanation: that the gains simply come from additional policy training rather than improved speculative decoding.

Critical weaknesses
  • The decoding-specific contribution remains insufficiently established. Most of the improvement comes from adding WAB, while next-frame supervision increases speedup only from 1.31× to 1.34× and narrows the AR-to-speculative gap from 3.5 to 2.5 points. The results support useful visual adaptation, but provide weaker evidence for the proposed physical-world explanation. Repeated descriptions such as “physical scene direction” are more assertive than the supporting analysis.

  • NCF is a coarse proxy: a failed episode is counted whenever it approaches an object or receptacle at any point. The additional episode counts and threshold sensitivity are helpful, but do not show whether accepted token deviations caused collisions or grasp failures. A generally stronger policy could also improve this metric.

  • The method design is heavy and seems hard to follow. It remains unclear if the method can generalize to more broadly backbones or scenes.

Questions for authors
  • Does the reduction in the AR-to-speculative success gap hold across the other policies under matched world-aware AR controls?

  • Can you separate actual collision, grasp, and placement failures from episodes that merely enter a near-contact region?

1 comment

Thank you again for your time. We hope that our clarifications and additional experiments have addressed your concerns. We would greatly appreciate any further feedback and would be happy to address any remaining questions during the discussion period.

Reviewer_VosqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper claimes that existing relaxed speculative decoding methods in VLA are scene-agnostic, however the safety of action deviations token should be scene-dependent.The same action deviation may cause more failures near contact. The paper proposes a method called WA-SpecDec, a novel world-aware speculatvie decoding framework that introduces world-model-derived infromaion into the draft and target model. The proposed method is evaluated on LIBERO, SIMPLER-Env and four real-robot tasks to demonstrate the benefits in success rate and inference time. Moreover, the authors also presented the near-contact failures on LIBERO to support their arguments.

Critical strengths
  1. The scene-dependence problem of relaxed speculative decoding in VLA studied in this paper is very meaningful, and the paper proposes a plausible method to introduce next-frame-prediction information into taget and draft model to mitigate the problem.
  2. The design of World-Aware Bias shows a lightweight way to enhance the scene-understanding in VLA which is proved can finally improving success rate in ablation studies.
  3. Comprehensive experiments demonstrate the proposed WA-SecDec obtain success rate and inference speed gains consistently on different benchmark.
  4. The paper is easy to follow.
Critical weaknesses
  1. Experimental results are not enough to support the arguments of the paper. Lack of rigorous experiments to prove the gain of success rate credit to the World-Aware state that helps to reject unsafe action near contact rather than a better draft–target agreement.
  2. NCF, the most importance metric in this paper, is only demonstrated on LIBERO. Moreover, the proposed NCF score cannot correctly reflect the benefits of the method. NCF is defined as " NCF is the total number of near-contact failures across these suites divided by the total number of failed episodes across the same suites, expressed as a percentage." in B.5, rather than the conditional failure probability given near-contact. I dont think the proposed NCF can directly reflect the improvement in the near-contact safety, and NCF could also be improved by reducing the proportion of near-contact action.
  3. The design of World-Aware Bias Module is not persuasive. It looks like the post-training stage of v-jepa 2 without action condition. It is not clear why a precise future state could be predicted without any instruction or action from a single input current frame, and why it can be helpful to the speculative decoding.
Questions for authors
  1. Why a precise future state could be predicted without any instruction or action from a single input current frame
  2. How does the method compare with a lightweight scene-adaptive acceptance threshold?
  3. How does this method specifically perform in rejecting unsafe behaviors near contact, and are there any quantitative results?
1 comment

Thank you again for your time. We hope that our clarifications and additional experiments have addressed your concerns. We would greatly appreciate any further feedback and would be happy to address any remaining questions during the discussion period.

Reviewer_m9FcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WA-SpecDec, a framework for improving the success–efficiency trade-off of relaxed speculative decoding in vision-language-action models. A frozen VAE encodes the current observation, and a trainable World-Aware Bias module produces spatially aligned residual features that are added to the VLA’s visual embeddings before prefill. The module and target-policy LoRA adapters are trained using action prediction and an auxiliary next-frame latent prediction objective. A draft model is subsequently distilled from the adapted target. During inference, drafting and verification share the enriched prefill representation, while the selected acceptance rule remains unchanged. The paper reports an average success improvement of 5.3 percentage points over Standard SpecDec, a 1.94× inference speedup over base-policy autoregressive decoding, and a 19.8% relative reduction in the NCF metric on LIBERO.

Critical strengths
  • Broad empirical evaluation, consistent reported benefits
  • Existing ablation provides evidence beyond additional training alone (In Figure 5, matched continued training without WAB produces little improvement, whereas the complete method reduces the AR-to-speculative success gap from approximately 6.3 percentage points for the matched-training control to 2.5 points. This supports the possibility that the method improves performance preservation under relaxed decoding and is an important consideration in favor of acceptance.)
Critical weaknesses
  • The central physical-awareness explanation is insufficiently distinguished from ordinary visual representation enhancement, and NCF does not establish the proposed mechanism. The next-frame objective in Equation 6 predicts future visual latents from the current observation without explicit action conditioning. This can encourage useful predictive features, but it does not establish that the representation captures the consequences of action deviations near contact. Furthermore, NCF measures the fraction of failed episodes that entered a near-contact region at any point. It does not measure failure probability conditional on contact, or identify failures caused by accepted action deviations. An episode can approach an object correctly and fail later for an unrelated reason. The additional episode counts and distance-threshold analysis are helpful, but do not resolve this attribution problem. This weakens the evidence for the paper’s central contact-sensitive mechanism.

  • The broader evaluation does not sufficiently isolate decoding-specific gains, and the uncertainty of the improvements remains unclear. WA-SpecDec changes both the target policy and its distilled draft, whereas Standard SpecDec uses the base target. The paper discloses this clearly, and the OpenVLA/LIBERO decomposition is useful. However, equivalent autoregressive controls for the adapted targets are missing from the other four settings. Consequently, it remains unclear how much of their success improvement comes from policy adaptation and whether their decoding-induced performance gaps also shrink.

Questions for authors
  • What are the autoregressive success rates and inference latencies of the adapted world-aware targets in the other four settings? Does WA-SpecDec consistently reduce the success gap relative to each adapted target’s own AR performance?
  • Do you have a capacity- and training-budget-matched visual residual-branch control? What evidence distinguishes the benefit of predictive physical information from additional visual capacity or spatial detail?
  • Which components of KERV and HeiSD are retained in the evaluations? In particular, are KERV’s Kalman compensation and HeiSD’s hybrid retrieval and verify-skip mechanisms included, or are only their acceptance rules transferred into the common implementation?
1 comment

Thank you again for your time. We hope that our clarifications and additional experiments have addressed your concerns. We would greatly appreciate any further feedback and would be happy to address any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.