- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents dVLA-RL, a framework for reinforcement learning (RL) of native discrete diffusion Vision-Language-Action (dVLA) models. The key challenge is that computing the marginal likelihood of the final action is intractable due to multiple possible denoising paths. To solve this problem, the authors propose a tractable trajectory-level likelihood surrogate that accounts for token-generation decisions along the sampled denoising path. This formulation naturally accommodates different denoising horizons and enables a task-adaptive hybrid strategy that balances policy performance and computational efficiency.
The experimental results are impressive. dVLA-RL achieves 99.7% success on LIBERO and improves RoboTwin 2.0 performance from 61.4% to 92.0% over the SFT baseline. The gains also transfer to real-world tasks after supervised adaptation, improving the aggregate score from 55.4% to 81.3%. The ablation studies and efficiency analyses further support the proposed design. Overall, the paper introduces an elegant and well-motivated solution to a challenging optimization problem, making a meaningful contribution to RL for dVLAs.
Critical strengths
- The paper tackles a novel and underexplored problem: RL for dVLAs. The proposed formulation provides a well-motivated solution to the intractable action likelihood problem.
- The trajectory-level likelihood formulation is simple and intuitive, and naturally supports different denoising horizons within a unified framework.
- The improvements over the SFT baseline are substantial and consistent across LIBERO, RoboTwin 2.0, and real-world ALOHA experiments.
- The paper is well written and easy to follow, with clear figures, pseudocode, and detailed implementation descriptions.
Critical weaknesses
- The simulation results report the best online success rates during training, with RoboTwin checkpoints selected based on the multi-task average. It would be helpful to clarify whether these results were independently evaluated and how stable the selected checkpoints are.
- The MM-ACT baseline uses a different SFT setup from the original MM-ACT, while the WAM and π0.5 comparisons involve different training or pretraining settings. Although some of these differences are discussed, further clarification would help interpret the comparisons.
- The hybrid denoising steps are selected based on initial SFT performance, but the exact criterion for choosing different step counts is not fully specified.
Questions for authors
- Were the reported peak success rates evaluated independently of the online RL rollouts? If available, could the authors share results from final or nearby checkpoints?
- Could the authors provide some further discussion of the differences between MM-ACT and MM-ACT*? It would also be helpful to elaborate on the comparisons with WAM and π0.5, considering their different training setups.
- How exactly are hybrid denoising steps chosen for each task? If available, could the authors share the SFT results used to make these choices?
Overall, I have a positive impression of this work, and I consider the above concerns relatively minor. I would be happy to raise my score if the authors can address these points through further clarifications and, where available, additional supporting results.
3 comments
W3&Q3 - The criterion for selecting hybrid denoising steps needs to be clarified.
We select the denoising horizon for each task using an initial SFT sweep over 1, 2, and 4 steps, with the sweep results measured on the initial rollout batch, together with the performance-efficiency trade-off observed during RL. We choose the smallest horizon that either preserves performance comparable to longer decoding or provides a clear optimization benefit, while avoiding unnecessary denoising transitions. The selected assignment is fixed before evaluation.
| Task | 1-Step (PD) | 2-Step | 4-Step | Selected Steps |
|---|---|---|---|---|
| Beat Block Hammer (BBH) | 84% | 83% | 81% | 1 |
| Place Phone Stand (PPS) | 59% | 39% | 53% | 1 |
| Pick Dual Bottles (PDB) | 3% | 20% | 38% | 4 |
| Lift Pot (LP) | 31% | 61% | 70% | 2 |
| Move Can Pot (MCP) | 22% | 33% | 36% | 2 |
| Place A2B Left (PAL) | 36% | 33% | 41% | 2 |
| Place Empty Cup (PEC) | 84% | 64% | 81% | 1 |
| Handover Mic (HM) | 19% | 30% | 44% | 4 |
| Average | 42.3% | 45.4% | 55.5% | — |
Accordingly, tasks that are already strong with parallel decoding use one step; tasks that benefit from moderate refinement use two steps; and tasks that require substantial iterative refinement use four steps. For Place A2B Left, the initial SFT success rates of the 1-step and 2-step settings are comparable. However, in our earlier RL training comparison between PD and 2-step denoising, the 2-step denoising improved more rapidly and achieved better optimization behavior, as shown in Appendix Figure 7. This indicates that additional denoising refinement benefits RL optimization even when the initial SFT performance is similar. We therefore select 2 steps for this task.
Appendix Figures 6 and 7 further validate these choices by comparing the 1-step, 2-step, and Hybrid training curves under the same RL protocol. In particular, the curves show that Hybrid preserves the benefit of longer denoising on Pick Dual Bottles and Handover Mic, while avoiding unnecessary computation on tasks such as Beat Block Hammer, Place Phone Stand, and Place Empty Cup.
response to author
Thanks to the authors for their response, additional experiments, and revised paper. I believe my concerns have been adequately addressed.
W1&Q1 - The independent evaluation and stability of the selected checkpoints need to be clarified.
We thank the reviewer for raising this important point. In the original main tables, the RL results were reported as the best online success rates observed during training. For RoboTwin, each optimization step uses 64 scenarios per task, resulting in 512 rollout trajectories across the eight tasks. In contrast, the SFT baseline was evaluated separately using independent evaluation scenarios.
To provide a direct comparison, we evaluated the saved dVLA-RL checkpoint at RL step 670 using the same evaluation protocol as the SFT baseline. Checkpoints were saved every 10 training steps, and the step-670 checkpoint was selected near the best online rollout performance at step 672.
RoboTwin unified evaluation
LIBERO unified evaluation
The checkpoint column denotes the RL training step of the evaluated dVLA-RL checkpoint. For each suite, the checkpoint was selected near the best online rollout performance.
The independently evaluated results remain strong: dVLA-RL reaches 90.75% on RoboTwin and 98.75% on the LIBERO test split, with a 29.38-point improvement over the MM-ACT* SFT baseline on RoboTwin. These results show that the gains are not solely caused by reporting the peak online rollout success rate.
W2&Q2 - Differences in training settings across baselines need to be clarified for fair comparison.
Thank you for these comments. We address W2 and Q2 together by clarifying the MM-ACT* configuration and the training setups used for the WAM and π0.5 comparisons.
MM-ACT versus MM-ACT*. MM-ACT* retains the MM-ACT architecture and uses action-only SFT, joint multi-task training, and an action chunk length of 16. The original LIBERO configuration uses separately trained models for individual suites and a chunk length of 8. Both use 500 demonstrations per suite. Our reported MM-ACT* results correspond to the checkpoints used to initialize RL: epoch 1 for Object and epoch 4 for the other suites. Object improves rapidly during SFT, reaching 82.6% at epoch 1 and 98.2% at epoch 4. Using its epoch-4 result increases the four-suite SFT average from 88.1% to 92.0%. We will include this additional SFT reference.
WAM comparisons. The WAM baselines follow their respective pretraining and fine-tuning pipelines. For example, LingBot-VA uses 27,500 RoboTwin demonstrations across 50 tasks. Our setup uses 1,000 demonstrations per task across eight tasks for SFT, followed by online RL.
Comparison with π0.5. π0.5 undergoes large-scale robot pretraining, while our MM-ACT-based policies start from general multimodal pretraining. For real-world adaptation, all three methods use the same demonstration mixtures: 300 demonstrations for the short-horizon tasks and 450 when including Organize Table, with batch size 128 and action chunk length 16. dVLA-RL is initialized from its RoboTwin RL checkpoint, while MM-ACT* and π0.5 use their respective RoboTwin SFT checkpoints. Other SFT hyperparameters follow each model’s training configuration.