acceptodds
Under review as a conference paper at ICLR 2027

dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models have established a powerful paradigm for generalist robotic manipulation by grounding control into the semantic reasoning of large Vision-Language Models (VLMs). Prevailing architectures typically model actions continuously via diffusion or flow processes, or discretely through either autoregressive generation or parallel decoding. Recently, Discrete Diffusion VLAs (dVLAs) have emerged as a distinct alternative, unifying vision, language, and action into a single discrete token space via masked generative modeling. While this paradigm successfully combines iterative action refinement with unified representations, its training has thus far been restricted to Supervised Fine-Tuning (SFT), leaving the potential of Reinforcement Learning (RL) for further policy refinement largely unexplored. A fundamental challenge in RL for dVLAs is that the marginal probability of the final action remains intractable. To address this challenge, we propose dVLA-RL, which shifts policy optimization from the intractable marginal likelihood of the final action to a tractable trajectory-level objective defined over the sampled denoising path. By factorizing this objective across denoising transitions, dVLA-RL naturally accommodates variable denoising horizons. Leveraging this flexibility, we introduce a unified step scheduling approach for complex multi-task learning, tailoring denoising steps to specific task complexities to maximize both success rates and computational efficiency. Extensive evaluations show that dVLA-RL achieves 99.7% success on LIBERO and establishes strong VLA-based results on RoboTwin 2.0, improving over the SFT baseline by 30.6 percentage points while remaining competitive with strong World-Action Model baselines. On real-world tasks, dVLA-RL improves the aggregate score by 25.8 percentage points over the MM-ACT baseline, with performance comparable to the representative continuous-action VLA .

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Discrete Diffusion, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_t6QqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents dVLA-RL, a framework for reinforcement learning (RL) of native discrete diffusion Vision-Language-Action (dVLA) models. The key challenge is that computing the marginal likelihood of the final action is intractable due to multiple possible denoising paths. To solve this problem, the authors propose a tractable trajectory-level likelihood surrogate that accounts for token-generation decisions along the sampled denoising path. This formulation naturally accommodates different denoising horizons and enables a task-adaptive hybrid strategy that balances policy performance and computational efficiency.

The experimental results are impressive. dVLA-RL achieves 99.7% success on LIBERO and improves RoboTwin 2.0 performance from 61.4% to 92.0% over the SFT baseline. The gains also transfer to real-world tasks after supervised adaptation, improving the aggregate score from 55.4% to 81.3%. The ablation studies and efficiency analyses further support the proposed design. Overall, the paper introduces an elegant and well-motivated solution to a challenging optimization problem, making a meaningful contribution to RL for dVLAs.

Critical strengths
  1. The paper tackles a novel and underexplored problem: RL for dVLAs. The proposed formulation provides a well-motivated solution to the intractable action likelihood problem.
  2. The trajectory-level likelihood formulation is simple and intuitive, and naturally supports different denoising horizons within a unified framework.
  3. The improvements over the SFT baseline are substantial and consistent across LIBERO, RoboTwin 2.0, and real-world ALOHA experiments.
  4. The paper is well written and easy to follow, with clear figures, pseudocode, and detailed implementation descriptions.
Critical weaknesses
  1. The simulation results report the best online success rates during training, with RoboTwin checkpoints selected based on the multi-task average. It would be helpful to clarify whether these results were independently evaluated and how stable the selected checkpoints are.
  2. The MM-ACT baseline uses a different SFT setup from the original MM-ACT, while the WAM and π0.5 comparisons involve different training or pretraining settings. Although some of these differences are discussed, further clarification would help interpret the comparisons.
  3. The hybrid denoising steps are selected based on initial SFT performance, but the exact criterion for choosing different step counts is not fully specified.
Questions for authors
  1. Were the reported peak success rates evaluated independently of the online RL rollouts? If available, could the authors share results from final or nearby checkpoints?
  2. Could the authors provide some further discussion of the differences between MM-ACT and MM-ACT*? It would also be helpful to elaborate on the comparisons with WAM and π0.5, considering their different training setups.
  3. How exactly are hybrid denoising steps chosen for each task? If available, could the authors share the SFT results used to make these choices?

Overall, I have a positive impression of this work, and I consider the above concerns relatively minor. I would be happy to raise my score if the authors can address these points through further clarifications and, where available, additional supporting results.

3 comments

W1&Q1 - The independent evaluation and stability of the selected checkpoints need to be clarified.

We thank the reviewer for raising this important point. In the original main tables, the RL results were reported as the best online success rates observed during training. For RoboTwin, each optimization step uses 64 scenarios per task, resulting in 512 rollout trajectories across the eight tasks. In contrast, the SFT baseline was evaluated separately using independent evaluation scenarios.

To provide a direct comparison, we evaluated the saved dVLA-RL checkpoint at RL step 670 using the same evaluation protocol as the SFT baseline. Checkpoints were saved every 10 training steps, and the step-670 checkpoint was selected near the best online rollout performance at step 672.

RoboTwin unified evaluation

TaskMM-ACT* SFTdVLA-RLImprovement
Beat Block Hammer (BBH)85.0%96.0%+11.0
Place Phone Stand (PPS)62.0%96.0%+34.0
Pick Dual Bottles (PDB)71.0%94.0%+23.0
Lift Pot (LP)46.0%88.0%+42.0
Move Can Pot (MCP)40.0%90.0%+50.0
Place A2B Left (PAL)57.0%76.0%+19.0
Place Empty Cup (PEC)81.0%96.0%+15.0
Handover Mic (HM)49.0%90.0%+41.0
Average61.38%90.75%+29.38

LIBERO unified evaluation

The checkpoint column denotes the RL training step of the evaluated dVLA-RL checkpoint. For each suite, the checkpoint was selected near the best online rollout performance.

SuitedVLA-RL checkpointMM-ACT* SFTdVLA-RLTest improvement
LIBERO-Spatial16091.2%98.6%+7.4
LIBERO-Object8082.6%99.4%+16.8
LIBERO-Goal20090.0%98.4%+8.4
LIBERO-Long22088.7%98.6%+9.9
Average—88.1%98.75%+10.65

The independently evaluated results remain strong: dVLA-RL reaches 90.75% on RoboTwin and 98.75% on the LIBERO test split, with a 29.38-point improvement over the MM-ACT* SFT baseline on RoboTwin. These results show that the gains are not solely caused by reporting the peak online rollout success rate.

W2&Q2 - Differences in training settings across baselines need to be clarified for fair comparison.

Thank you for these comments. We address W2 and Q2 together by clarifying the MM-ACT* configuration and the training setups used for the WAM and π0.5 comparisons.

MM-ACT versus MM-ACT*. MM-ACT* retains the MM-ACT architecture and uses action-only SFT, joint multi-task training, and an action chunk length of 16. The original LIBERO configuration uses separately trained models for individual suites and a chunk length of 8. Both use 500 demonstrations per suite. Our reported MM-ACT* results correspond to the checkpoints used to initialize RL: epoch 1 for Object and epoch 4 for the other suites. Object improves rapidly during SFT, reaching 82.6% at epoch 1 and 98.2% at epoch 4. Using its epoch-4 result increases the four-suite SFT average from 88.1% to 92.0%. We will include this additional SFT reference.

WAM comparisons. The WAM baselines follow their respective pretraining and fine-tuning pipelines. For example, LingBot-VA uses 27,500 RoboTwin demonstrations across 50 tasks. Our setup uses 1,000 demonstrations per task across eight tasks for SFT, followed by online RL.

Comparison with π0.5. π0.5 undergoes large-scale robot pretraining, while our MM-ACT-based policies start from general multimodal pretraining. For real-world adaptation, all three methods use the same demonstration mixtures: 300 demonstrations for the short-horizon tasks and 450 when including Organize Table, with batch size 128 and action chunk length 16. dVLA-RL is initialized from its RoboTwin RL checkpoint, while MM-ACT* and π0.5 use their respective RoboTwin SFT checkpoints. Other SFT hyperparameters follow each model’s training configuration.

W3&Q3 - The criterion for selecting hybrid denoising steps needs to be clarified.

We select the denoising horizon for each task using an initial SFT sweep over 1, 2, and 4 steps, with the sweep results measured on the initial rollout batch, together with the performance-efficiency trade-off observed during RL. We choose the smallest horizon that either preserves performance comparable to longer decoding or provides a clear optimization benefit, while avoiding unnecessary denoising transitions. The selected assignment is fixed before evaluation.

Task1-Step (PD)2-Step4-StepSelected Steps
Beat Block Hammer (BBH)84%83%81%1
Place Phone Stand (PPS)59%39%53%1
Pick Dual Bottles (PDB)3%20%38%4
Lift Pot (LP)31%61%70%2
Move Can Pot (MCP)22%33%36%2
Place A2B Left (PAL)36%33%41%2
Place Empty Cup (PEC)84%64%81%1
Handover Mic (HM)19%30%44%4
Average42.3%45.4%55.5%—

Accordingly, tasks that are already strong with parallel decoding use one step; tasks that benefit from moderate refinement use two steps; and tasks that require substantial iterative refinement use four steps. For Place A2B Left, the initial SFT success rates of the 1-step and 2-step settings are comparable. However, in our earlier RL training comparison between PD and 2-step denoising, the 2-step denoising improved more rapidly and achieved better optimization behavior, as shown in Appendix Figure 7. This indicates that additional denoising refinement benefits RL optimization even when the initial SFT performance is similar. We therefore select 2 steps for this task.

Appendix Figures 6 and 7 further validate these choices by comparing the 1-step, 2-step, and Hybrid training curves under the same RL protocol. In particular, the curves show that Hybrid preserves the benefit of longer denoising on Pick Dual Bottles and Handover Mic, while avoiding unnecessary computation on tasks such as Beat Block Hammer, Place Phone Stand, and Place Empty Cup.

Reviewer_t6QqOpenReview ↗

response to author

Thanks to the authors for their response, additional experiments, and revised paper. I believe my concerns have been adequately addressed.

Reviewer_1T55OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper studies online RL for discrete-diffusion VLA models. Building on MM-ACT, the method records the sampled action-denoising trajectory and token-commitment schedule during rollout. During PPO updates, it holds this schedule fixed and aggregates the log-probabilities of newly committed tokens across denoising steps to construct a trajectory-level surrogate. The formulation supports both single-step and multi-step generation. A Hybrid strategy assigns different denoising steps to each task based on initial SFT performance. Experiments cover simulation and real robot adaptation.

Critical strengths
  1. The proposed RL method improves performance over the SFT baseline, suggesting that online RL can further improve discrete-diffusion VLA policies beyond their supervised initialization.

  2. The proposed surrogate objective reuses the existing MM-ACT backbone and standard PPO/GAE machinery while supporting different denoising horizons.

  3. Evaluating the RL method on the real world robot provides valuable experimental validation.

Critical weaknesses
  1. The current real-world evaluation follows the pipeline of simulation RL, followed by SFT on real-world demonstrations and then real-world robot testing. This does not directly establish the effectiveness of the proposed RL method on physical robots. I suggest applying RL directly on the model after real-world SFT. The current comparison does not isolate the source of the real-world performance gains.

  2. I find the motivation insufficiently developed. The limitations of SFT make sense, but they are shared motivations for most work on policy RL. The paper should more clearly motivate RL for discrete-diffusion VLAs and explain the specific opportunities offered by combining RL with this action-generation paradigm. If the main contribution is to adapt PPO to this paradigm by changing the construction of the policy likelihood ratio, the work reads more like a practically useful technical report. My suggestion is to strengthen the motivation and its presentation, especially given the growing body of work adapting PPO and GRPO to different VLA action-generation paradigms.

  3. Please clarify the substantive differences between the proposed PPO formulation and the following related methods, beyond differences in the VLA action-generation paradigm. SimpleVLA-RL already applies token-probability-based online RL to an OpenVLA-OFT variant with parallel action decoding, the original MM-ACT discusses multi-step action re-masking, and TraceRL [1] optimizes token probability ratios along inference traces using a PPO-style objective. In particular, what distinguishes the proposed method in its likelihood-ratio construction, advantage estimation, and treatment of the denoising schedule?

  4. Comparisons with additional VLA RL baselines under matched training and evaluation conditions would also help establish the contribution.

[1] Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

Questions for authors
  1. The MM-ACT baseline in this paper achieves 88.1% on LIBERO, compared with 95.0% for the Vanilla configuration in the original MM-ACT paper. Was less data used for SFT, or are there other reasons for this gap?

  2. Figures 3 and 5 report standard deviations, but it is unclear whether they are computed across independent training seeds, rollout batches, or points within a smoothing window. These measure different sources of variation and are not interchangeable.

  3. Appendix B states that LIBERO-Object uses the epoch 1 checkpoint, while the other suites use the epoch 4 checkpoint. However, Appendix C says that the same checkpoint is used, with only the number of denoising steps during rollouts changed. Please clarify this inconsistency.

5 comments

W1 – Additional Gains from Offline RL after Real-World SFT need further validation.

Thank you for the suggestion. We add a real-world experiment on Organize Table that applies offline RL directly after real-world SFT, following the iterative offline RL framework of RL-100 [1]. Starting from the Sim RL + Real SFT checkpoint, we use an IQL critic trained on all collected trajectories (150 demos + 400 rollouts) to estimate advantages for our PPO-style objective over discrete-diffusion denoising trajectories.

MethodPick and Place Carrot (%)Lift Pot (%)Place Empty Cup (%)Avg. (%)
Real SFT80504056.7
Sim RL + Real SFT95857585.0
Sim RL + Real SFT + Real Offline RL1001009598.3

The comparison shows improvements at both RL stages. Simulation RL improves mean stage success after real-world SFT from 56.7% to 85.0%, a gain of 28.3 percentage points. Further offline RL on real-world data increases it to 98.3%, providing an additional 13.3-point gain over the same policy after real-world SFT. All three stages improve, with the largest additional gains on Lift Pot (85% → 100%) and Place Empty Cup (75% → 95%). These results support the effectiveness of applying our discrete-diffusion RL objective to further improve a policy after supervised adaptation to the real robot.

Following Table 3, we report individual stage success rates for Pick and Place Carrot, Lift Pot, and Place Empty Cup. Avg. is their arithmetic mean. Each policy is evaluated for 20 trials under identical conditions.

[1] K. Lei et al., “Performant robotic manipulation with real-world reinforcement learning.” Science Robotics, 2026.

W2 - The motivation and benefits of applying RL to discrete-diffusion VLAs need to be clarified.

Thank you for your question. Firstly, we want to clarify the importance of the discrete diffusion paradigm for VLA: (1) It unifies multiple modalities in a shared discrete token space. (2) It provides a stronger modeling capability for complex action trajectories compared with autoregressive generation and parallel decoding. (3) It provides exact token-level log-probabilities for likelihood-based RL, compared with flow-based methods. However, existing methods mostly focus on imitation learning which limits the potential of discrete diffusion VLAs. We therefore investigate whether RL can further improve the performance of native dVLAs by complementing imitation learning with task-level reward feedback.

RL is particularly challenging in this setting because a dVLA generates an action chunk through multiple masked-denoising transitions. The executed action can be reached through many intermediate denoising paths, so its exact likelihood requires intractable path marginalization. This prevents standard PPO or GRPO objectives from being directly applied as in autoregressive or parallel action-token policies. We also experimented with a random-masking RL objective inspired by diffusion language-model RL, but found that it led to unstable optimization in the dVLA setting, as shown by the corresponding training curve in Appendix F.

dVLA-RL addresses this dVLA-specific challenge by recording the sampled denoising trajectory and token-commitment schedule during rollout, fixing the schedule during PPO updates, and constructing a tractable likelihood surrogate over the executed path. The same formulation supports different denoising horizons and enables the Hybrid strategy, which uses task-dependent denoising steps to balance action refinement, RL sample efficiency, and computation.

In our experiments, dVLA-RL also outperforms alternative VLA-RL baselines such as SimpleVLA-RL and π-RL; we provide further details and clarify the comparison settings in our response to Weakness 4. This provides supporting evidence for the potential of applying RL to the native discrete-diffusion VLA paradigm.

We have revised the paragraph introducing dVLAs in the Introduction (Section 1, p. 2) to make our research motivation explicit: building on the unified multimodal modeling and action-generation capabilities of native dVLAs, we investigate whether task-level reward feedback can further improve their closed-loop control performance beyond supervised imitation.

W3 - The key differences between our PPO formulation and related methods need to be clarified.

We thank the reviewer for requesting a clearer comparison with SimpleVLA-RL, MM-ACT, and TraceRL.

MethodLikelihood ratioAdvantage estimation
SimpleVLA-RLToken-level ratioGroup-relative, trajectory-shared
TraceRLToken-level ratioSequence-level standardized; optional token-wise GAE
dVLA-RLChunk-level ratioOuter-MDP GAE, chunk-shared

SimpleVLA-RL uses parallel action-token decoding, so each action token has a directly computable probability and can be optimized with a token-level GRPO ratio. Its group-relative advantage is computed from the normalized total outcome reward and shared across the action tokens.

TraceRL computes token-level ratios along the sampled inference trace. Its default formulation uses a sequence-level standardized advantage, while its optional diffusion value model provides token-wise GAE advantages.

In contrast, dVLA-RL involves two nested processes: the inner denoising process generates an action chunk, and the outer robotic environment transitions after the chunk is executed. We therefore aggregate token probabilities along the recorded denoising path into one chunk-level trajectory ratio and use outer-environment GAE to obtain one action-chunk-level advantage.

Regarding the denoising schedule, dVLA-RL follows the multi-step re-masking mechanism used by MM-ACT. However, MM-ACT is trained entirely with SFT and does not define an online RL objective. dVLA-RL records the schedule during rollout and holds it fixed during each PPO update.

We also tested a TraceRL-style formulation that assigns the final outcome reward uniformly to the entire action trajectory and all denoising tokens. It performed poorly in our robotic setting, as shown in Appendix F.

W4 – Additional VLA-RL Comparisons are required.

Thank you for your suggestion. We provide futhur training setting clarifications and additional experiments below.

Single-task training. In our RoboTwin RL setting, single-task training is easier than joint multi-task training because the policy only needs to optimize one task. On Beat Block Hammer, dVLA-RL achieves a success rate of 99.6%, compared with 87.5% reported by SimpleVLA-RL and 80.9% obtained by π-RL in our addtional experiment.

Joint multi-task training. SimpleVLA-RL achieves 68.7% average success rate on eight RoboTwin 2.0 tasks using a separately trained policy for each task, while dVLA-RL train one policy jointly on eight tasks, achieving an average success rate of 92.0%, which demostrates the effectness of our method, and the high potential of discrete diffusion VLAs. We additionally tested π-RL under joint multi-task training, but did not observe consistent gains across repeated experiments in our tested configurations.

Additional comparison details and training curves are provided in Appendix G.

Q1&Q3 - SFT baseline and checkpoint selection

Thank you for raising these points. We use 500 SFT demonstrations per LIBERO suite, matching the data scale of the original MM-ACT.

The 88.1% result reports the performance of the SFT checkpoints actually used to initialize RL. Since LIBERO-Object improves rapidly during SFT, we started RL from its epoch-1 checkpoint (82.6%), while using epoch-4 checkpoints for the other suites. The Object checkpoint at epoch 4 achieves 98.2%; substituting this result while keeping the other suites unchanged gives a four-suite SFT average of 92.0%. Thus, checkpoint selection accounts for 3.9 percentage points of the difference from the original result.

Our setup also uses joint multi-task SFT and an action chunk length of 16, whereas the original Vanilla configuration trains separate models for individual suites with a chunk length of 8.

Regarding Q3, Appendix B describes the actual checkpoint-selection protocol. The statement “the same checkpoint” in Appendix C is inaccurate. We have corrected it to specify epoch 1 for Object and epoch 4 for Spatial, Goal, and Long, consistent with Appendix B.

Q2 - Standard deviation computation.

We thank the reviewer for pointing out this ambiguity. The shaded regions in Figures 3 and 5 are not computed across independent training seeds or rollout batches. In the plotting procedure, records with the same training step are first averaged to obtain one success-rate value per step. We then compute an exponential moving average with α=0.2\alpha=0.2 and an exponentially weighted standard deviation over the historical sequence of per-step mean success rates. The shaded region is given by the EMA plus or minus this exponentially weighted temporal dispersion, clipped to the valid range [0,1][0,1].

Thus, the shaded regions should be interpreted as temporal variation in the training curves rather than uncertainty across independent runs or rollout batches. We agree that labeling them as ±1\pm 1 STD was ambiguous, and we will revise the captions to explicitly describe them as EMA ±\pm exponentially weighted temporal dispersion and avoid interpreting them as seed-level variability.

Reviewer_MTv1OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces dVLA-RL, which constructs a PPO surrogate objective by aggregating the log-probabilities of newly decoded tokens along sampled denoising trajectories and assigns task-specific denoising horizons. The method is evaluated on LIBERO, RoboTwin, and real-world adaptation tasks.

Critical strengths
  1. The paper addresses an important problem with an intuitive and effective method, providing a simple, practical approach to online reinforcement learning for discrete diffusion VLAs.
  2. The paper is clearly written and easy to follow, with broad experimental coverage spanning simulation, real-world adaptation, and key ablations.
Critical weaknesses

According to Section 5.1 and Appendix B.4/D, the main tables report the highest online success rates observed during RL training, where the SFT baseline undergoes separate evaluation. This protocol mismatch may limit the reliability of the performance comparison.

Questions for authors

Please clarify the evaluation procedures used for RL and the SFT baseline in the main tables. If the protocols differ, please re-evaluate both under a unified test protocol and report the results.

2 comments

Evaluation protocol. We thank the reviewer for identifying the difference between the online RL and SFT evaluation procedures. In the original main tables, the RL results were reported as the best online success rates observed during training. For RoboTwin, each optimization step uses 64 scenarios per task, resulting in 512 rollout trajectories across the eight tasks. In contrast, the SFT baseline was evaluated separately using independent evaluation scenarios.

To provide a direct comparison, we evaluated the saved dVLA-RL checkpoint at RL step 670 using the same evaluation protocol as the SFT baseline. Checkpoints were saved every 10 training steps, and the step-670 checkpoint was selected near the best online rollout performance (step-672).

RoboTwin unified evaluation

TaskMM-ACT* SFTdVLA-RLImprovement
Beat Block Hammer (BBH)85.0%96.0%+11.0
Place Phone Stand (PPS)62.0%96.0%+34.0
Pick Dual Bottles (PDB)71.0%94.0%+23.0
Lift Pot (LP)46.0%88.0%+42.0
Move Can Pot (MCP)40.0%90.0%+50.0
Place A2B Left (PAL)57.0%76.0%+19.0
Place Empty Cup (PEC)81.0%96.0%+15.0
Handover Mic (HM)49.0%90.0%+41.0
Average61.38%90.75%+29.38

LIBERO unified evaluation

The numbers in the checkpoint column denote the RL training step of the evaluated dVLA-RL checkpoint. The checkpoints were also selected near the best online rollout performance.

SuitedVLA-RL checkpointMM-ACT* SFTdVLA-RLTest improvement
LIBERO-Spatial16091.2%98.6%+7.4
LIBERO-Object8082.6%99.4%+16.8
LIBERO-Goal20090.0%98.4%+8.4
LIBERO-Long22088.7%98.6%+9.9
Average—88.1%98.75%+10.65

The independently evaluated results remain strong: dVLA-RL reaches 90.75% on RoboTwin and 98.75% on the LIBERO test split, with a 29.38-point improvement over the MM-ACT* SFT baseline on RoboTwin. These results show that the gains are not solely caused by reporting the peak online rollout success rate.

Thank you for the detailed rebuttal. The unified evaluation results address my main concern. I encourage the authors to incorporate these results and clarify the checkpoint selection and evaluation procedures in the revised manuscript. I will maintain my original rating (Clear Accept).

Reviewer_yZLLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes dVLA-RL, which records discrete action-denoising trajectories and combines newly generated token probabilities into a PPO objective while freezing the recorded decoding schedule during updates. It also assigns task-specific denoising horizons. Experiments cover LIBERO, RoboTwin, and real-robot adaptation, with supervised fine-tuning rather than online RL on physical robots.

Critical strengths
  1. The method addresses the structure of discrete diffusion policies. It combines token decisions along denoising paths with action-chunk-level advantages. Controlled ablations show more stable training than using only the final denoising step.

  2. Task-specific denoising provides practical efficiency gains. Hybrid assigns different denoising horizons across tasks, reducing action-chunk generation latency from 728.92 ms to 387.24 ms compared with fixed four-step decoding, a reduction of approximately 46.9%.

  3. Real-robot adaptation provides additional evidence. After fine-tuning on the same real-world demonstrations, RL initialization improves average success on the three short-horizon tasks from 55% to 80%, supporting the transfer value of simulation-based RL.

Critical weaknesses
  1. Evaluation protocols differ: RL reports peak training success, while SFT uses separate evaluation. RoboTwin RL scores also come from training scenarios, weakening the claimed gains (§5.1).

  2. Selection bias is not analyzed: Top-K selection depends on sampled tokens, but the surrogate scores only retained tokens. Its implications for PPO updates remain unaddressed (§4.2, Algorithm 1).

Questions for authors
  1. Do the main gains persist when SFT and RL checkpoints are evaluated on the same held-out scenarios not used for RL training?

  2. When Top-K positions depend on sampled tokens, how does Equation 5 account for selection bias? Can you derive its relationship to the actual sampling distribution?

2 comments

W1&Q1 - SFT and RL checkpoints need to be evaluated on the same held-out scenarios

Thank you for raising this important point. In the original main tables, the RL results were reported as the best online success rates observed during training. For RoboTwin, each optimization step uses 64 scenarios per task, resulting in 512 rollout trajectories across the eight tasks. In contrast, the SFT baseline was evaluated separately using independent evaluation scenarios.

To provide a direct comparison, we evaluated the saved dVLA-RL checkpoint at RL step 670 using the same evaluation protocol as the SFT baseline. Checkpoints were saved every 10 training steps, and the step-670 checkpoint was selected near the best online rollout performance at step 672.

RoboTwin unified evaluation

TaskMM-ACT* SFTdVLA-RLImprovement
Beat Block Hammer (BBH)85.0%96.0%+11.0
Place Phone Stand (PPS)62.0%96.0%+34.0
Pick Dual Bottles (PDB)71.0%94.0%+23.0
Lift Pot (LP)46.0%88.0%+42.0
Move Can Pot (MCP)40.0%90.0%+50.0
Place A2B Left (PAL)57.0%76.0%+19.0
Place Empty Cup (PEC)81.0%96.0%+15.0
Handover Mic (HM)49.0%90.0%+41.0
Average61.38%90.75%+29.38

LIBERO unified evaluation

The checkpoint column denotes the RL training step of the evaluated dVLA-RL checkpoint. For each suite, the checkpoint was selected near the best online rollout performance.

SuitedVLA-RL checkpointMM-ACT* SFTdVLA-RLTest improvement
LIBERO-Spatial16091.2%98.6%+7.4
LIBERO-Object8082.6%99.4%+16.8
LIBERO-Goal20090.0%98.4%+8.4
LIBERO-Long22088.7%98.6%+9.9
Average—88.1%98.75%+10.65

The independently evaluated results remain strong: dVLA-RL reaches 90.75% on RoboTwin and 98.75% on the LIBERO test split, with a 29.38-point improvement over the MM-ACT* SFT baseline on RoboTwin. These results show that the gains are not solely caused by reporting the peak online rollout success rate in the training scenarios.

W2&Q2 — Selection bias and scheduler optimization

Thank you for raising this inspiring question! We provide the derivation below.

At environment step tt, let sts_t denote the observed state and let xK,…,x0x_K,\ldots,x_0 denote the action-token states over KK denoising steps, where xKx_K is fully masked and x0x_0 is the final action sequence. Following Algorithm 1, let HkH_k denote the masked positions at denoising step kk. Let Mk⊆HkM_k\subseteq H_k denote the positions newly committed in the recorded transition from xkx_k to xk−1x_{k-1}, and define Uk=Hk∖MkU_k=H_k\setminus M_k as the positions that remain masked in xk−1x_{k-1}.

Under policy parameters θ\theta, a provisional token x~i\tilde{x}^i is sampled at each position i∈Hki\in H_k from the corresponding categorical distribution pθ(⋅∣xk,st)p_\theta(\cdot\mid x_k,s_t). These samples are conditionally independent across positions.

Write x~Hk=(x~i)i∈Hk\tilde{x}^{H_k}=(\tilde{x}^i)_{i\in H_k} for the complete vector of provisional tokens.

Write x~Uk=(x~j)j∈Uk\tilde{x}^{U_k}=(\tilde{x}^j)_{j\in U_k} for its uncommitted subvector.

Let Sθ(x~Hk;xk,st)\mathcal{S}_\theta(\tilde{x}^{H_k};x_k,s_t) denote the set of positions selected by the deterministic Top-K rule at step kk, using the sampled-token probabilities as confidence scores.

We use 1{⋅}\mathbf{1}\{\cdot\} for the indicator function, which equals one when its condition holds and zero otherwise.

When evaluating the recorded transition, we fix x~i=xk−1i\tilde{x}^i = x_{k-1}^i for i∈Mki \in M_k. The summation ∑x~Uk\sum_{\tilde{x}^{U_k}} below ranges over all possible token assignments to the uncommitted positions, while these committed-token values remain fixed. We define the selection-correction factor Cθ,kC_{\theta,k} as

Cθ,k=∑x~Uk[∏j∈Ukpθ(x~j∣xk,st)]1[Sθ(x~Hk;xk,st)=Mk].C_{\theta,k}=\sum_{\tilde{x}^{U_k}}\left[\prod_{j\in U_k}p_\theta(\tilde{x}^j\mid x_k,s_t)\right]\mathbf{1}\left[\mathcal{S}_\theta(\tilde{x}^{H_k};x_k,s_t)=M_k\right].

Thus, Cθ,kC_{\theta,k} is the probability, over the remaining provisional tokens, that the selector returns exactly MkM_k, given the specified candidate values at the committed positions.

Marginalizing the discarded proposals yields the actual denoising transition probability:

Pθ(xk−1∣xk,st)=∑x~Uk[∏i∈Hkpθ(x~i∣xk,st)]1[Sθ(x~Hk;xk,st)=Mk].P_\theta(x_{k-1}\mid x_k,s_t)=\sum_{\tilde{x}^{U_k}}\left[\prod_{i\in H_k}p_\theta(\tilde{x}^i\mid x_k,s_t)\right]\mathbf{1}\left[\mathcal{S}_\theta(\tilde{x}^{H_k};x_k,s_t)=M_k\right].

Equivalently,

Pθ(xk−1∣xk,st)=[∏i∈Mkpθ(xk−1i∣xk,st)]Cθ,k.P_\theta(x_{k-1}\mid x_k,s_t)=\left[\prod_{i\in M_k}p_\theta(x_{k-1}^i\mid x_k,s_t)\right]C_{\theta,k}.

Let θˉ=θold\bar{\theta}=\theta_{\mathrm{old}} denote the behavior parameters, and let ρtpath(θ)\rho_t^{\mathrm{path}}(\theta) denote the likelihood ratio of the same recorded denoising path under θ\theta and θˉ\bar{\theta}. Its definition and relationship to the committed-token ratio ρ^t(θ)\widehat{\rho}_t(\theta) in Equation 5 are

ρtpath(θ)=∏k=1KPθ(xk−1∣xk,st)∏k=1KPθˉ(xk−1∣xk,st).\rho_t^{\mathrm{path}}(\theta)=\frac{\prod_{k=1}^{K}P_\theta(x_{k-1}\mid x_k,s_t)}{\prod_{k=1}^{K}P_{\bar{\theta}}(x_{k-1}\mid x_k,s_t)}.

Using the factorization above,

ρtpath(θ)=ρ^t(θ)∏k=1KCθ,kCθˉ,k.\rho_t^{\mathrm{path}}(\theta)=\hat{\rho}_t(\theta)\prod_{k=1}^{K}\frac{C_{\theta,k}}{C_{\bar{\theta},k}}.

where Cθˉ,kC_{\bar{\theta},k} is the selection-correction factor evaluated under the behavior parameters.

Equation 5 retains the committed-token factor and omits this selection-correction ratio, defining our fixed-schedule surrogate. Incorporating the selection-correction ratio would allow PPO to explicitly account for the probability of the recorded commitment decisions. Specifically, the correction introduces additional scheduler-related gradient contributions and modifies the importance-sampling ratios used in the PPO clipping objective.

Our current objective also provides an implicit mechanism for optimizing confidence-based scheduling. For positive-advantage trajectories, it encourages higher likelihood for the committed tokens at each recorded denoising state. For negative-advantage trajectories, it encourages lower likelihood for those tokens, discouraging decoding decisions associated with worse-than-expected outcomes. These probability updates affect both candidate generation and the confidence scores used in subsequent rollouts, allowing the scheduler to adapt implicitly to both positive and negative advantage signals as the token policy is updated. This coupling offers a mechanism for reward-driven scheduler adaptation. The substantial improvements reported on LIBERO and RoboTwin support the practical effectiveness of the current surrogate.

We will clarify this distinction in Section 4.2 and explore the effects of explicitly incorporating the selection correction in future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Closing Summary for Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank reviewers 1T55, MTv1, t6Qq, and yZLL for their thoughtful and constructive feedback. We are encouraged by the broad consensus on the core strengths of our work: all four reviewers rated the presentation as clear and easy to follow (MTv1, t6Qq, yZLL, 1T55). Reviewers recognized that dVLA-RL tackles an important, underexplored problem (MTv1, t6Qq) with a practical trajectory-level PPO formulation tailored to discrete-diffusion VLAs (MTv1, t6Qq, yZLL, 1T55) that naturally supports varying denoising horizons (t6Qq, 1T55). Reviewers further highlighted the practical efficiency gains of task-adaptive hybrid denoising (yZLL, t6Qq) and the substantial empirical improvements across simulation benchmarks and real-world robot adaptation, supported by comprehensive ablation studies (yZLL, MTv1, 1T55, t6Qq).

Building on this positive foundation, we are pleased that the follow-up comments confirm that our rebuttal has substantially addressed the reviewers' main concerns: MTv1 confirmed that the unified evaluation resolves the main concern and maintained Clear Accept; t6Qq confirmed that their concerns have been adequately addressed; and 1T55 acknowledged that the real-world offline RL results substantially address the first concern and that the clarifications resolve related questions, raising their rating from Weak Reject (2) to Weak Accept (3). We also provided the requested follow-up clarification on baseline comparison settings.

We completed the requested experiments and revised the manuscript accordingly.

  • Matched held-out evaluation and baseline fairness (MTv1, t6Qq, yZLL, 1T55). We addressed the online-RL versus SFT protocol mismatch and clarified checkpoint selection, SFT initialization, rollout budgets, evaluation procedures, and baseline settings. Under the same independent held-out protocol, RoboTwin performance improves from 61.38% to 90.75% across all eight tasks. LIBERO performance improves from 88.1% to 98.75%.

  • Real-world offline RL after SFT (1T55). We added an Organize Table experiment applying offline RL directly after real-world SFT. Average success increases from 56.7% with Real SFT, to 85.0% with Sim RL + Real SFT, and to 98.3% after additional real-world offline RL.

  • Additional VLA-RL comparisons (1T55). dVLA-RL reaches 99.6% on Beat Block Hammer, compared with 87.5% for SimpleVLA-RL and 80.9% for π-RL. In the harder joint eight-task setting, one dVLA-RL policy achieves 92.0%, compared with 68.7% from eight separately trained SimpleVLA-RL policies. We additionally tested π-RL under joint multi-task training, but did not observe consistent gains in our tested configurations. Training curves and comparison settings are provided in Appendix G.

  • Method, hybrid denoising, and Top-K clarification (1T55, t6Qq, yZLL). We clarified the motivation for RL in discrete-diffusion VLAs and the distinction between our chunk-level ratio with outer-MDP GAE and token-level alternatives. In our robotic experiments, the TraceRL-style token-level variant performed poorly, as reported in Appendix F. For the hybrid denoising concern, we specify that each task’s horizon is selected based on an initial SFT sweep over 1, 2, and 4 steps, together with the observed RL performance–efficiency trade-offs. We further derive the selection-correction factor omitted by the fixed-schedule surrogate and discuss its implications.

All new results consistently support the effectiveness of dVLA-RL. We thank the reviewers again for their constructive feedback. Please refer to our individual responses and the revised manuscript for full details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.