- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ROAD-VLA, an advantage-guided self-distillation method for online VLA adaptation. It constructs a teacher by adjusting the original policy predictions based on rollout advantages and adds a distillation loss to PPO. The method is evaluated on OpenVLA, OpenVLA-OFT, and π0.5, showing improvements over PPO across several simulated distribution shifts.
Critical strengths
The method is simple and applicable to different VLA action heads, including discrete tokens, continuous actions, and flow matching. The experiments show fairly consistent improvements over PPO across several simulation settings.
Critical weaknesses
-
The method does not fully address its stated motivation. The paper argues that scalar advantages provide coarse supervision for structured action predictions. However, ROAD-VLA still uses the same scalar advantage for all action dimensions and timesteps within a chunk. While the proposed loss provides explicit training targets, it does not introduce finer-grained credit assignment. It is unclear how this fundamentally addresses the limitation of PPO.
-
Limited algorithm novelty. The core method essentially adds an advantage-weighted distillation loss to PPO, which is closely related to existing advantage-weighted regression methods. More direct comparisons with AWR/AWAC-style objectives or simpler auxiliary regression losses are needed to establish the benefit of the proposed teacher construction.
-
Limited experimental validation. All experiments are conducted in simulation, mostly on pick-and-place tasks, without real-world evaluation. Some improvements on LIBERO-Pro are also quite small. Moreover, the additional training cost (1.41–1.61× over PPO) makes the claimed efficiency gains less convincing.
-
The theoretical results offer limited justification. The policy-improvement bound relies on assumptions about teacher calibration and distillation stability, but the proposed teacher construction does not guarantee these conditions. The analysis therefore provides limited evidence for why the method should consistently improve policy performance.
Questions for authors
-
Since all action dimensions and timesteps share the same scalar advantage, how does the proposed distillation provide more informative supervision than PPO?
-
How does ROAD-VLA compare with PPO augmented by a standard advantage-weighted regression loss? What is the specific benefit of constructing the proximal teacher?
-
Does ROAD-VLA still improve training efficiency when comparing wall-clock time rather than optimization steps?
5 comments
Response to Reviewer LCrk (2/3)
W3 + Q3
Evaluation scope. We thank the reviewer for this comment. We agree that the tasks are manipulation tasks in simulation. However, we would like to note that the difficulty of our evaluation comes from the distribution shifts, which are a known weakness of current VLA models, and not from the manipulation skill itself. We use two different benchmarks. ManiSkill has seven OOD environments (unseen table textures, dynamic texture and noise, distractor objects and receptacles, initial pose, repositioning). LIBERO is trained on the LIBERO-Goal suite (10 tasks) and tested on three LIBERO-Pro perturbations (object attributes, target swap, object position). All OOD shifts are not used for training. ROAD-VLA improves over PPO in every setting (Section 5.2 and Table 1, average success, %).
| Policy and action head | Benchmark | ID: PPO → ROAD-VLA | OOD: PPO → ROAD-VLA |
|---|---|---|---|
| OpenVLA, discrete tokens | ManiSkill | 76.86 → 82.36 | 68.00 → 73.59 |
| OpenVLA-OFT, discretized | ManiSkill | 87.16 → 89.58 | 69.10 → 73.61 |
| OpenVLA-OFT, L1 regression | LIBERO | 96.88 → 98.44 | 26.39 → 29.17 |
| , flow matching | LIBERO | 83.33 → 91.15 | 29.34 → 33.86 |
We agree that evaluation on physical robots is an important next step (Appendix H).
Gains on LIBERO-Pro. These perturbations are very challenging. Under target swap and object position shifts, PPO reaches only about 20% with , so there is limited room for any fine-tuning method. ROAD-VLA is still better than PPO on all three perturbations, with a lower standard deviation (Table 12, 3 seeds).
| LIBERO-Pro perturbation | PPO | ROAD-VLA | Gain |
|---|---|---|---|
| VR-ObjectAttr | 46.35 ± 6.32 | 57.29 ± 5.49 | +10.94 |
| CR-TargetSwap | 21.35 ± 6.26 | 21.88 ± 1.57 | +0.53 |
| ER-ObjectPos | 20.31 ± 6.81 | 22.40 ± 0.90 | +2.09 |
| OOD average | 29.34 ± 5.89 | 33.86 ± 2.09 | +4.52 |
Wall-clock time. Yes, ROAD-VLA still outperforms PPO given the same training time. The extra cost comes only from the distillation loss in the update step, and both methods use the same number of environment interactions. We have added this comparison in Appendix J (Figures 13 and 14). On all three environments, ROAD-VLA reaches its best evaluation before PPO finishes training (Table 15, ID / OOD success, %). Given the limited time during the rebuttal, these runs use one seed, and we chose to cover three environments instead of more seeds on one. We will add more seeds in the final version.
| Environment | PPO | ROAD-VLA |
|---|---|---|
| CR-MultiObject | 62.50 / 50.00 | 65.63 / 57.81 |
| ER-InitPose | 81.25 / 71.88 | 87.50 / 78.13 |
| VR-UnseenTable | 84.38 / 81.25 | 92.19 / 89.06 |
Response to Reviewer LCrk (3/3)
W4
We thank the reviewer for this comment. We agree that the teacher construction does not guarantee the assumptions, and this is why we state Theorem 1 as a conditional bound (Section 4.3, Appendix A.5). We would like to note that this is common for policy-improvement bounds in RL, which also hold under conditions such as exact advantages [1]. We use the theorem to identify the quantities that decide whether the policy improves, and we designed each part of ROAD-VLA to control one of them, as summarized below.
| Term in the bound (Eq. 13) | How it is controlled in ROAD-VLA |
|---|---|
| Teacher calibration (A1, Eq. 11) | the signed, clipped weight and the agreement gate (Eqs. 4 and 5), which uses the reference critic only when it agrees in sign with the online critic |
| Student–teacher mismatch (A2, Eq. 12) | the distillation loss (Eq. 10), which directly minimizes the divergence between the student and the teacher |
| Change in visited states () | the PPO clipping and the bounded teacher step ( on the logit) |
We can also say more about (A1) in the token case. For a single categorical prediction, the expected true advantage of the teacher in Eq. (7) is , where and . This value is positive when has the same sign as the true advantage of the sampled token. In other words, the teacher moves in an improving direction as long as the sign of the advantage estimate is correct, and the signed weight and the agreement gate are there to protect this sign. We have added this remark to Appendix A.5.
The experiments are consistent with this analysis. Removing the gate lowers the success rate at the end of training on VR-UnseenTable (Figure 7). Without the reference critic, ROAD () still improves over PPO on all three OOD environments (Table 2). In the main results, ROAD-VLA improves over PPO in all 14 comparisons with OpenVLA (Section 5.2, 3 seeds) and in all four settings listed in our response to W3 + Q3.
[1] Schulman et al. Trust Region Policy Optimization. ICML 2015.
We thank the reviewer again. We hope these address the reviewer's concerns and would be grateful if they are taken into account in the final assessment. We are happy to discuss further.
Response to Reviewer LCrk (1/3)
We thank the reviewer for the careful reading and constructive comments. We apologize for the late response. The requested comparisons required additional training runs, and we preferred to report them once they were complete.
Shared scalar advantage. We agree that ROAD-VLA uses one scalar advantage for the whole chunk and does not introduce finer-grained credit assignment (Appendix A.1). The limitation of PPO we address is how this scalar is used. PPO uses it to weight the likelihood ratio of the whole action, which gives a direction but no target for each prediction. ROAD-VLA uses it to build a fixed target for every prediction.
For example, take an action with a positive advantage and ηωt=ln2. By Eq. (7), the teacher doubles the probability of the sampled token at every position and renormalizes:
The same advantage therefore gives each token its own target, which PPO does not provide. Similarly, the teacher is a full target action chunk for regression (Eq. 8) and a full target velocity field for flow matching (Eq. 9), so every action dimension and timestep has its own target value.
Empirical evidence. We compare methods that use the same advantage with and without the teacher loss (Table 15, one seed, mean ID / OOD success over three environments, %).
With the same scalar advantage, adding the teacher loss improves success on all three environments, for both ID and OOD. The gain therefore comes from how the advantage is used in ROAD-VLA. We have revised the introduction to state this more precisely.
Difference from AWR. We would like to clarify that the auxiliary loss in ROAD-VLA is a distillation loss toward a teacher, and the advantage is used to build this teacher. In AWR and AWAC, the advantage is a positive weight on the likelihood of the sampled action. The benefit of the proximal teacher is that every token receives a bounded target, and a negative advantage moves this target away from the sampled action.
Comparison with PPO + AWR. AWR and AWAC share the same policy loss, so we use PPO + AWR as the baseline. Both methods keep the PPO update and use the same gated advantage and the same loss weight. Table 2 reports OOD success (%, one seed).
PPO + AWR gives the same mean as PPO, while ROAD-VLA improves on all three environments.
Comparison with a simpler auxiliary loss. If negative advantages are clipped to zero, our loss becomes a positively weighted likelihood term plus a KL term to the rollout policy (Eq. 19, Appendix A.2). We compare this simpler loss with ROAD-VLA in Table 13 (CR-MultiObject, 3 seeds).
The signed teacher performs better in both cases, so the gain comes from the teacher construction and not from adding an advantage-weighted term to PPO.