acceptodds
Under review as a conference paper at ICLR 2027

ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models

Abstract

Online reinforcement learning can adapt vision-language-action (VLA) models to new deployment conditions, but scalar advantages provide coarse supervision for their high-dimensional, structured action predictions. We propose ROAD-VLA, an advantage-guided self-distillation framework that turns rollout feedback into a structured teacher directly in the policy's native action space. After each on-policy rollout, ROAD-VLA takes a proximal mirror-descent step from the rollout prediction, moving toward better-than-expected executed behavior and away from worse-than-expected behavior. To improve reliability under distribution shift, a frozen reference critic is incorporated only when its advantage estimate agrees in sign with the online critic. This yields a unified construction for discrete action tokens, continuous action chunks, and flow-matching velocity predictions, without requiring an external expert. We derive a conditional policy-improvement bound under explicit calibration and stability assumptions, providing a common theoretical interpretation across all three action representations. Across ManiSkill environments spanning visual, compositional, and execution shifts, ROAD-VLA improves OpenVLA-7B and OpenVLA-OFT over PPO and outperforms language-guided and external-teacher distillation baselines. We further demonstrate the same mechanism with the flow-matching pi_0.5 policy on LIBERO. Across these settings, ROAD-VLA enables faster online adaptation while maintaining higher policy entropy and lower late-stage variance than PPO.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Online Adaptation, On-Policy Self-Distillation, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LCrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROAD-VLA, an advantage-guided self-distillation method for online VLA adaptation. It constructs a teacher by adjusting the original policy predictions based on rollout advantages and adds a distillation loss to PPO. The method is evaluated on OpenVLA, OpenVLA-OFT, and π0.5, showing improvements over PPO across several simulated distribution shifts.

Critical strengths

The method is simple and applicable to different VLA action heads, including discrete tokens, continuous actions, and flow matching. The experiments show fairly consistent improvements over PPO across several simulation settings.

Critical weaknesses
  1. The method does not fully address its stated motivation. The paper argues that scalar advantages provide coarse supervision for structured action predictions. However, ROAD-VLA still uses the same scalar advantage for all action dimensions and timesteps within a chunk. While the proposed loss provides explicit training targets, it does not introduce finer-grained credit assignment. It is unclear how this fundamentally addresses the limitation of PPO.

  2. Limited algorithm novelty. The core method essentially adds an advantage-weighted distillation loss to PPO, which is closely related to existing advantage-weighted regression methods. More direct comparisons with AWR/AWAC-style objectives or simpler auxiliary regression losses are needed to establish the benefit of the proposed teacher construction.

  3. Limited experimental validation. All experiments are conducted in simulation, mostly on pick-and-place tasks, without real-world evaluation. Some improvements on LIBERO-Pro are also quite small. Moreover, the additional training cost (1.41–1.61× over PPO) makes the claimed efficiency gains less convincing.

  4. The theoretical results offer limited justification. The policy-improvement bound relies on assumptions about teacher calibration and distillation stability, but the proposed teacher construction does not guarantee these conditions. The analysis therefore provides limited evidence for why the method should consistently improve policy performance.

Questions for authors
  1. Since all action dimensions and timesteps share the same scalar advantage, how does the proposed distillation provide more informative supervision than PPO?

  2. How does ROAD-VLA compare with PPO augmented by a standard advantage-weighted regression loss? What is the specific benefit of constructing the proximal teacher?

  3. Does ROAD-VLA still improve training efficiency when comparing wall-clock time rather than optimization steps?

5 comments
AuthorsOpenReview ↗

Response to Reviewer LCrk (1/3)

We thank the reviewer for the careful reading and constructive comments. We apologize for the late response. The requested comparisons required additional training runs, and we preferred to report them once they were complete.

W1 + Q1

Shared scalar advantage. We agree that ROAD-VLA uses one scalar advantage for the whole chunk and does not introduce finer-grained credit assignment (Appendix A.1). The limitation of PPO we address is how this scalar is used. PPO uses it to weight the likelihood ratio of the whole action, which gives a direction but no target for each prediction. ROAD-VLA uses it to build a fixed target for every prediction.

PPOROAD-VLA
Signal per tokena gradient direction scaled by the advantagea full target distribution over the vocabulary
Constraintclipping on the likelihood ratio of the whole action (7 or 56 tokens)each token has its own bounded target (∣ηωt∣≤1\lvert\eta\omega_t\rvert\le 1 on the logit)
Negative advantagelowers the log-likelihood of the sampled action, with no targetthe teacher moves away from the sampled token and remains a valid distribution

For example, take an action with a positive advantage and ηωt=ln⁡2\eta\omega_t=\ln 2. By Eq. (7), the teacher doubles the probability of the sampled token at every position and renormalizes:

Token positionPolicy probability of the sampled tokenTeacher target
First20%2×202×20+80=33.3%\frac{2\times 20}{2\times 20+80}=33.3\%
Second80%2×802×80+20=88.9%\frac{2\times 80}{2\times 80+20}=88.9\%

The same advantage therefore gives each token its own target, which PPO does not provide. Similarly, the teacher is a full target action chunk for regression (Eq. 8) and a full target velocity field for flow matching (Eq. 9), so every action dimension and timestep has its own target value.

Empirical evidence. We compare methods that use the same advantage with and without the teacher loss (Table 15, one seed, mean ID / OOD success over three environments, %).

AdvantageWithout teacher lossWith teacher loss
Online onlyPPO: 76.04 / 67.71ROAD (α=0\alpha=0): 78.65 / 70.31
MixedPPO + mixed advantage: 77.08 / 70.31ROAD-VLA: 81.77 / 75.00

With the same scalar advantage, adding the teacher loss improves success on all three environments, for both ID and OOD. The gain therefore comes from how the advantage is used in ROAD-VLA. We have revised the introduction to state this more precisely.

W2 + Q2

Difference from AWR. We would like to clarify that the auxiliary loss in ROAD-VLA is a distillation loss toward a teacher, and the advantage is used to build this teacher. In AWR and AWAC, the advantage is a positive weight on the likelihood of the sampled action. The benefit of the proximal teacher is that every token receives a bounded target, and a negative advantage moves this target away from the sampled action.

AWR / AWACROAD-VLA
How the advantage is usedas a positive weight on the likelihood of the sampled actionto build the teacher (Eq. 7): above the rollout probability if the advantage is positive, below it if negative
Actions with negative advantagestill reinforced, with a smaller weightthe teacher moves away from them
Action headsneeds an action likelihoodany head that outputs a prediction: tokens, action chunks, velocity fields

Comparison with PPO + AWR. AWR and AWAC share the same policy loss, so we use PPO + AWR as the baseline. Both methods keep the PPO update and use the same gated advantage and the same loss weight. Table 2 reports OOD success (%, one seed).

EnvironmentPPOPPO + AWRROAD-VLA
CR-MultiObject50.0048.4457.81
ER-InitPose71.8873.4478.13
VR-UnseenTable81.2581.2589.06
Mean67.7167.7175.00

PPO + AWR gives the same mean as PPO, while ROAD-VLA improves on all three environments.

Comparison with a simpler auxiliary loss. If negative advantages are clipped to zero, our loss becomes a positively weighted likelihood term plus a KL term to the rollout policy (Eq. 19, Appendix A.2). We compare this simpler loss with ROAD-VLA in Table 13 (CR-MultiObject, 3 seeds).

Teacher weightsIDOOD
Positive-only65.55 ± 8.1257.02 ± 7.12
Signed (ROAD-VLA)70.31 ± 6.6360.94 ± 3.12

The signed teacher performs better in both cases, so the gain comes from the teacher construction and not from adding an advantage-weighted term to PPO.

AuthorsOpenReview ↗

Response to Reviewer LCrk (2/3)

W3 + Q3

Evaluation scope. We thank the reviewer for this comment. We agree that the tasks are manipulation tasks in simulation. However, we would like to note that the difficulty of our evaluation comes from the distribution shifts, which are a known weakness of current VLA models, and not from the manipulation skill itself. We use two different benchmarks. ManiSkill has seven OOD environments (unseen table textures, dynamic texture and noise, distractor objects and receptacles, initial pose, repositioning). LIBERO is trained on the LIBERO-Goal suite (10 tasks) and tested on three LIBERO-Pro perturbations (object attributes, target swap, object position). All OOD shifts are not used for training. ROAD-VLA improves over PPO in every setting (Section 5.2 and Table 1, average success, %).

Policy and action headBenchmarkID: PPO → ROAD-VLAOOD: PPO → ROAD-VLA
OpenVLA, discrete tokensManiSkill76.86 → 82.3668.00 → 73.59
OpenVLA-OFT, discretizedManiSkill87.16 → 89.5869.10 → 73.61
OpenVLA-OFT, L1 regressionLIBERO96.88 → 98.4426.39 → 29.17
π0.5\pi_{0.5}, flow matchingLIBERO83.33 → 91.1529.34 → 33.86

We agree that evaluation on physical robots is an important next step (Appendix H).

Gains on LIBERO-Pro. These perturbations are very challenging. Under target swap and object position shifts, PPO reaches only about 20% with π0.5\pi_{0.5}, so there is limited room for any fine-tuning method. ROAD-VLA is still better than PPO on all three perturbations, with a lower standard deviation (Table 12, 3 seeds).

LIBERO-Pro perturbationPPOROAD-VLAGain
VR-ObjectAttr46.35 ± 6.3257.29 ± 5.49+10.94
CR-TargetSwap21.35 ± 6.2621.88 ± 1.57+0.53
ER-ObjectPos20.31 ± 6.8122.40 ± 0.90+2.09
OOD average29.34 ± 5.8933.86 ± 2.09+4.52

Wall-clock time. Yes, ROAD-VLA still outperforms PPO given the same training time. The extra cost comes only from the distillation loss in the update step, and both methods use the same number of environment interactions. We have added this comparison in Appendix J (Figures 13 and 14). On all three environments, ROAD-VLA reaches its best evaluation before PPO finishes training (Table 15, ID / OOD success, %). Given the limited time during the rebuttal, these runs use one seed, and we chose to cover three environments instead of more seeds on one. We will add more seeds in the final version.

EnvironmentPPOROAD-VLA
CR-MultiObject62.50 / 50.0065.63 / 57.81
ER-InitPose81.25 / 71.8887.50 / 78.13
VR-UnseenTable84.38 / 81.2592.19 / 89.06
AuthorsOpenReview ↗

Response to Reviewer LCrk (3/3)

W4

We thank the reviewer for this comment. We agree that the teacher construction does not guarantee the assumptions, and this is why we state Theorem 1 as a conditional bound (Section 4.3, Appendix A.5). We would like to note that this is common for policy-improvement bounds in RL, which also hold under conditions such as exact advantages [1]. We use the theorem to identify the quantities that decide whether the policy improves, and we designed each part of ROAD-VLA to control one of them, as summarized below.

Term in the bound (Eq. 13)How it is controlled in ROAD-VLA
Teacher calibration (A1, Eq. 11)the signed, clipped weight and the agreement gate (Eqs. 4 and 5), which uses the reference critic only when it agrees in sign with the online critic
Student–teacher mismatch (A2, Eq. 12)the distillation loss (Eq. 10), which directly minimizes the divergence between the student and the teacher
Change in visited states (Δocc\Delta_{\mathrm{occ}})the PPO clipping and the bounded teacher step (∣ηωt∣≤1\lvert\eta\omega_t\rvert\le 1 on the logit)

We can also say more about (A1) in the token case. For a single categorical prediction, the expected true advantage of the teacher in Eq. (7) is wtA(a^)/Zw_t A(\hat a)/Z, where wt=pold(a^)(eηωt−1)w_t=p_{\mathrm{old}}(\hat a)(e^{\eta\omega_t}-1) and Z=1+wtZ=1+w_t. This value is positive when ωt\omega_t has the same sign as the true advantage of the sampled token. In other words, the teacher moves in an improving direction as long as the sign of the advantage estimate is correct, and the signed weight and the agreement gate are there to protect this sign. We have added this remark to Appendix A.5.

The experiments are consistent with this analysis. Removing the gate lowers the success rate at the end of training on VR-UnseenTable (Figure 7). Without the reference critic, ROAD (α=0\alpha=0) still improves over PPO on all three OOD environments (Table 2). In the main results, ROAD-VLA improves over PPO in all 14 comparisons with OpenVLA (Section 5.2, 3 seeds) and in all four settings listed in our response to W3 + Q3.

[1] Schulman et al. Trust Region Policy Optimization. ICML 2015.

We thank the reviewer again. We hope these address the reviewer's concerns and would be grateful if they are taken into account in the final assessment. We are happy to discuss further.

Reviewer_5uP8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROAD-VLA, which augments PPO with advantage-guided self-distillation. Specifically, it first constructs a fixed teacher by moving the rollout prediction toward positively advantaged executed actions and away from negatively advantaged ones. A frozen reference critic contributes when its advantage agrees in sign with the online critic. The framework is formulated for either discrete, continuous-regression, and flow-matching head. Experiments on ManiSkill and LIBERO shows the proposed method is effective.

Critical strengths
  1. Interesting idea: using the fixed teacher on the rollout prediction provides an interesting way to incorporate negative-advantage samples without directly optimizing an unbounded negatively weighted regression loss. This idea can be apply to RL for VLA in general.

  2. Strong performance: The main experiments show consistent improvements over PPO on OpenVLA across seven perturbation environments, increasing average OOD success from 68.00% to 73.59%. In addition, these results include variability over three seeds.

Critical weaknesses
  1. The contribution beyond advantage-weighted learning and proximal regularization remains unclear. The teacher still applies one scalar advantage to the entire action chunk and has no component-level credit assignment. This may improve optimization, but the motivating claim of richer structured supervision is not grounded or well explained

  2. The source of the empirical gains is not adequately ablated. ROAD-VLA introduces both the auxiliary teacher loss and guidance from an additional pretrained reference critic. The reported ablations do not establish how much improvement comes from each component. Specifically, PPO with the same agreement-gated mixed advantage and ROAD-VLA without reference guidance are important missing controls.

Questions for authors
  1. What benefits does teacher matching provide over a matched advantage-weighted objective with an old-policy regularizer?

  2. How does PPO with the same mixed advantage and ROAD-VLA without the reference critic perform?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5uP8 (1/2)

We thank the reviewer for the thoughtful comments. We apologize for the late response. The requested comparisons required additional training runs, and we preferred to report them once they were complete.

W1 + Q1

Difference from advantage-weighted regression. We would like to clarify that ROAD-VLA is an on-policy self-distillation method. AWR and AWAC use the advantage as a positive weight on the likelihood of the sampled action. In ROAD-VLA, we use the advantage to shift the policy's own rollout prediction into a teacher, and the policy is then distilled toward this teacher. This is related to on-policy self-distillation, a recent and popular fine-tuning technique in LLMs [1, 2]. Our contribution is the teacher, which we construct from the advantage in closed form for discrete tokens, action chunks, and flow matching. We summarize the differences in the table below.

Advantage-weighted regressionROAD-VLA
How the advantage is usedas a positive weight on the likelihood of the sampled actionto build the teacher: it shifts the policy's prediction toward the sampled action if the advantage is positive, and away from it if negative
Actions with negative advantagestill reinforced, with a smaller weightthe teacher moves away from them
Action headsneeds an action likelihoodany head that outputs a prediction: tokens, action chunks, velocity fields

Role of Eq. (6). We think Eq. (6) may have been read as a proximal regularizer on the RL objective, and we apologize for the unclear presentation. Eq. (6) is solved over the target yy and does not involve the policy parameters θ\theta. It is only used to define the teacher. The term DΦ(y,yold)D_\Phi(y, y^{\mathrm{old}}) keeps the teacher close to the rollout prediction, and it is not a penalty on the policy update. We have added a note after Eq. (6) to make this clear.

Comparison with a matched objective. For discrete tokens, the distillation loss of the student can be written as a weighted likelihood term plus a KL term to the rollout policy. We have added this equation to Appendix A.2 (shown here for τKD=1\tau_{\mathrm{KD}}=1; the same form holds with a temperature):

KL(q⋆∥pθ)=1Z[KL(pold∥pθ)−wt log⁡pθ(a^)]+const,wt=pold(a^) (eηωt−1),Z=1+wt.\mathrm{KL}(q^\star\|p_\theta)=\frac{1}{Z}\Big[\mathrm{KL}(p_{\mathrm{old}}\|p_\theta)-w_t\,\log p_\theta(\hat a)\Big]+\text{const},\qquad w_t=p_{\mathrm{old}}(\hat a)\,(e^{\eta\omega_t}-1),\quad Z=1+w_t.

Here wtw_t has the same sign as the advantage. When the advantage is negative, wt<0w_t<0 and the loss lowers the probability of the sampled action, which AWR cannot do because its weight exp⁡(A/β)\exp(A/\beta) is always positive. If we clip negative advantages to zero, we get wt≥0w_t\ge 0, and the loss becomes a positively weighted likelihood term plus a KL term to the rollout policy. This corresponds to the matched objective in the reviewer's question, and we refer to it as the positive-only teacher.

We compare ROAD-VLA with PPO + AWR using the same gated advantage (Table 2, OOD success, %) and with the positive-only teacher (Table 13, CR-MultiObject, 3 seeds).

EnvironmentPPOPPO + AWRROAD-VLA
CR-MultiObject50.0048.4457.81
ER-InitPose71.8873.4478.13
VR-UnseenTable81.2581.2589.06
Teacher weightsIDOOD
Positive-only65.55 ± 8.1257.02 ± 7.12
Signed (ROAD-VLA)70.31 ± 6.6360.94 ± 3.12

PPO + AWR does not improve over PPO on average, while ROAD-VLA improves on all three environments. The signed teacher also performs better than the positive-only teacher.

Structured supervision. We agree that the advantage is one scalar shared by the whole action, and we do not claim that it tells which component is good or bad (Appendix A.1). What we mean by dense, structured supervision is the target that each output receives. For example, with discrete tokens (7 per action in OpenVLA, 56 in OpenVLA-OFT), PPO constrains the likelihood ratio of the whole action and does not give a target to any single token. In ROAD-VLA, every token has its own teacher distribution, which stays close to the rollout prediction (∣ηωt∣≤1|\eta\omega_t|\le 1 on the logit). Similarly, the teacher is a full target action chunk for regression (Eq. 8) and a full target velocity field for flow matching in π0.5\pi_{0.5} (Eq. 9).

Overall, these results show that the gain comes from the teacher construction, which is the main difference between ROAD-VLA and AWR with proximal regularization.

[1] Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

[2] Shenfeld et al. Self-Distillation Enables Continual Learning. ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer 5uP8 (2/2)

W2 + Q2

Component ablation. We thank the reviewer for suggesting these two controls. We have run both of them. PPO + mixed advantage uses the same mixed, gated, and clipped advantage as ROAD-VLA but without the teacher loss. ROAD (α=0\alpha=0) keeps the teacher loss and disables the reference critic. The results are reported in Section 5.4.1 (Table 2) and Appendix I (Table 15), and we summarize them below (success, %).

SplitEnvironmentPPOPPO + mixed advantageROAD (α=0\alpha=0)ROAD-VLA
IDCR-MultiObject62.5064.0665.6365.63
IDER-InitPose81.2582.8184.3887.50
IDVR-UnseenTable84.3884.3885.9492.19
IDMean76.0477.0878.6581.77
OODCR-MultiObject50.0050.0051.5657.81
OODER-InitPose71.8875.0075.0078.13
OODVR-UnseenTable81.2585.9484.3889.06
OODMean67.7170.3170.3175.00

Both components contribute to the gain. Without the reference critic, ROAD (α=0\alpha=0) is better than PPO on all three environments, for both ID and OOD. With the reference critic, ROAD-VLA is better than PPO + mixed advantage on all three environments, again for both ID and OOD. When the same mixed advantage is given to PPO without the teacher loss, only part of the gain is recovered (OOD mean 70.31 compared with 75.00 for ROAD-VLA).

These ablations use one seed (64 evaluation episodes per split, best evaluation of each run). Given the limited compute during the rebuttal period, we chose to cover three environments instead of running more seeds on one environment, and the results are consistent across the three. We will add more seeds in the final version.

We thank the reviewer again. We hope these address the reviewer's concerns and would be grateful if they are taken into account in the final assessment. We are happy to discuss further.

Dear Reviewer 5uP8,

Thank you again for your detailed and helpful comments. To fully address your concerns, we ran the additional experiments across several environments and seeds. This took some time, so our responses were posted close to the end of the discussion period. They are now available, together with the revised paper, and we would be happy to answer any remaining questions. If you find our responses satisfactory, we hope this may be reflected in your final assessment.

Best regards, The Authors

Reviewer_xnChOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ROAD-VLA augments PPO with a self-distillation target constructed from the rollout policy, the executed action, and a signed advantage. The distillation weight mixes online and frozen-reference advantage estimates when their signs agree after scale matching. The paper develops targets for different action representations and evaluates adaptation under visual, compositional, and execution shifts.

Critical strengths
  1. The OpenVLA results show consistent gains across the seven ManiSkill conditions: average ID success improves from 76.86% to 82.36%, and OOD success from 68.00% to 73.59%. Reporting both splits and several types of shift gives a more informative picture than a single aggregate adaptation score.

  2. The teacher construction is explicit, and the ablations examine sign gating, reference-critic quality, and positive versus signed feedback. The appendix also makes clear that the method does not infer separate credit for individual actions within a chunk, and reports the added training cost.

Critical weaknesses
  1. The appendix gives conflicting descriptions of the experiments. Appendix F.1.1 presents OpenVLA-OFT as a continuous regression experiment, but E.1.2 describes 56 categorical tokens and a token-logit KL teacher for that model. The PPO-OFT baseline is said to use 128 environments, batch size 640, and an upper clip ratio of 0.28, while Table 3 lists 16 environments, batch size 160, and clipping at 0.2 as shared settings. These differences leave both the claimed regression-head validation and the fairness of the OFT comparison unclear.

  2. The PPO comparison changes both the distillation objective and access to a frozen reference critic. Tables 3 and 4 list reference mixing under ROAD-VLA-only settings. Figure 7 varies the mixture and gate, but does not include distillation with the online critic alone. The gains could therefore reflect better guidance from the reference critic as well as the proposed teacher construction. Training also takes 40.8 versus 25.4 hours for OpenVLA on ER-Repositioning (Table 11), so earlier progress per iteration does not by itself establish a wall-clock advantage.

Questions for authors
  1. Which action head and distillation loss produced the OpenVLA-OFT results, and which settings were actually shared with PPO? Please reconcile E.1.2, F.1.1, and Table 3. Also clarify whether the π0.5 results use two training seeds, as stated in E.2, or three, as stated in Figure 5 and Table 9.

  2. On one representative condition, can the authors compare ROAD-VLA with the reference critic disabled and PPO supplied with the same mixed, gated advantage used to construct ROAD-VLA’s teacher? These controls would separate the two changes. Given the measured overhead, how does progress compare against elapsed training time?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xnCh (1/2)

We thank the reviewer for the detailed comments, especially the careful check of the appendix. We apologize for the late response. The requested controls required additional training runs, and we preferred to report them once they were complete.

W1 + Q1

Action head and loss. The OpenVLA-OFT results in Figure 5a and Table 11 were obtained with the discretized action head of the RLinf implementation (Appendix F.1.1). The teacher is the token-logit teacher in Eq. (7), and the distillation loss is the forward KL from teacher to student. We used this head because RLinf provides its ManiSkill OpenVLA-OFT checkpoints for it, which allows our PPO baseline and initialization to follow the public pipeline.

Settings shared with PPO. PPO-OFT and ROAD-VLA-OFT were trained with the same settings, listed below. The only difference is the distillation loss in ROAD-VLA-OFT. We have corrected Appendix E.1.2 and Table 5 (Table 3 in the original submission) so that they report these values consistently. We have also updated Table 11 to report raw, unsmoothed results, ensuring consistency with the reporting format used in the other tables.

ItemPPO-OFT and ROAD-VLA-OFT (ManiSkill)
Initializationsame RLinf ManiSkill checkpoint
Action headRLinf discretized head, 8 actions = 56 tokens per call
Parallel environments16
Global batch size160
PPO clip0.28 for OpenVLA-OFT (0.2 is the OpenVLA-7B value)
Distillation loss (ROAD-VLA only)forward KL to the token-logit teacher, Eq. (7)

Regression head. To evaluate the regression teacher as well, we added an experiment with the official OpenVLA-OFT L1 regression head on LIBERO (Appendix F.2, Table 8). The student is trained to match the continuous teacher in Eq. (8) with a squared ℓ2\ell_2 loss. PPO and ROAD-VLA start from the same public checkpoint and follow the π0.5\pi_{0.5} setup (32 environments, batch size 128, clip ratio 0.2, 200 iterations). The reference critic is the value head of a PPO policy trained under the same setting. Success rates (%) over 3 seeds are:

SplitPerturbationPPOROAD-VLA
IDLIBERO-Goal96.88 ± 2.7198.44 ± 1.56
OODVR-ObjectAttr70.83 ± 3.6076.04 ± 2.36
OODCR-TargetSwap8.33 ± 0.909.90 ± 0.90
OODER-ObjectPos0.00 ± 0.001.56 ± 0.00

ROAD-VLA obtains higher success than PPO on all four splits. Together with the existing results, ROAD-VLA has now been evaluated on discrete-token, continuous-regression, and flow-matching action heads.

π0.5\pi_{0.5} seeds. The π0.5\pi_{0.5} results use three training seeds (1, 2, and 3), as stated in Figure 5b and Table 12. We have corrected Appendix E.2 accordingly.

AuthorsOpenReview ↗

Response to Reviewer xnCh (2/2)

W2 + Q2

Component ablation: teacher loss and reference critic. We ran the two suggested controls on three conditions, one from each shift category (Table 2; Appendix I, Table 15). Success rates (%) are taken at the best evaluation of each run, using the same rule for all methods.

SplitMethodTeacher lossReference criticCR-MultiObjectER-InitPoseVR-UnseenTableMean
IDPPO✗✗62.5081.2584.3876.04
IDPPO + gated mixed advantage✗✓64.0682.8184.3877.08
IDROAD (α=0\alpha=0, reference disabled)✓✗65.6384.3885.9478.65
IDROAD-VLA✓✓65.6387.5092.1981.77
OODPPO✗✗50.0071.8881.2567.71
OODPPO + gated mixed advantage✗✓50.0075.0085.9470.31
OODROAD (α=0\alpha=0, reference disabled)✓✗51.5675.0084.3870.31
OODROAD-VLA✓✓57.8178.1389.0675.00

Teacher matching improves over PPO in both settings, on all three conditions and for both ID and OOD. Without the reference critic, ROAD (α=0\alpha=0) is better than PPO. With the reference critic, ROAD-VLA is better than PPO given the same mixed, gated, and clipped advantage. This is consistent with the design: the reference advantage is used only when it agrees in sign with the online advantage, so it changes how large a teacher update is, but never its direction (Section 4.2.2).

These controls use one seed per method. With the compute available during the rebuttal, we chose to cover three shift categories instead of repeating seeds on one. The ranking is the same in all three, and it agrees with our three-seed main results (Tables 9 and 10). We will add multi-seed results in the camera-ready version.

Elapsed training time. Both methods use the same number of environment interactions per iteration; the 1.411.41 to 1.61×1.61\times overhead (Table 14) comes only from the distillation update. In Appendix J (Figures 13 and 14) we compare them by training time, up to the point where PPO finishes its 400 iterations. ROAD-VLA reaches its best evaluation within this time on all three tasks. Best success rate (%):

TaskID: PPOID: ROAD-VLAOOD: PPOOOD: ROAD-VLA
CR-MultiObject62.5065.6350.0057.81
ER-InitPose81.2587.5071.8878.13
VR-UnseenTable84.3892.1981.2589.06

Given the same training time, ROAD-VLA outperforms PPO on all three tasks, for both ID and OOD.

We thank the reviewer again. We hope these address the reviewer's concerns and would be grateful if they are taken into account in the final assessment. We are happy to discuss further.

Dear Reviewer xnCh,

Thank you again for your detailed and helpful comments. To fully address your concerns, we ran the additional experiments across several environments and seeds. This took some time, so our responses were posted close to the end of the discussion period. They are now available, together with the revised paper, and we would be happy to answer any remaining questions. If you find our responses satisfactory, we hope this may be reflected in your final assessment.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Rebuttal

We thank the reviewers, ACs, and PCs for their time and constructive feedback. We apologize for the late response. Several of the requested comparisons required new training runs, and we wanted to verify all numbers before posting. All main results, including the new OpenVLA-OFT experiment with the L1 regression head, are averaged over 3 seeds. Only the component ablation (Table A) uses one seed per run. For this ablation, we ran each comparison on three different environments, so that the conclusions do not rely on a single task. The results are consistent across all three, and we will add more seeds in the final version.

We are glad that the reviewers found the gains over PPO consistent across the seven ManiSkill environments (xnCh, 5uP8), the teacher construction explicit and well ablated (xnCh), the fixed teacher an interesting way to use negative-advantage samples (5uP8), and the method simple and applicable to discrete, continuous, and flow-matching action heads (LCrk).

Clarification on the method (5uP8, LCrk). We would like to clarify that ROAD-VLA is an on-policy self-distillation method, which is different from adding an advantage-weighted regression loss to PPO. We use the advantage to build a teacher from the policy's own prediction, and the policy is then distilled toward this teacher. When the advantage is negative, the teacher moves away from the sampled action, which a positively weighted likelihood loss cannot do. Eq. (6) is only used to define this teacher and is not a regularizer on the policy update.

Summary of concerns and answers. We have run all the comparisons requested by the reviewers and revised the paper accordingly.

ConcernOur answerIn the paper
Source of the gain (xnCh, 5uP8)The teacher loss improves over PPO both with the reference critic (OOD mean 70.31 → 75.00) and without it (67.71 → 70.31), on all three environments.Section 5.4.1, Table 2, Appendix I
Difference from AWR (5uP8, LCrk)PPO + AWR gives the same OOD mean as PPO (67.71), and a positive-only teacher is worse than our signed teacher (OOD 57.02 vs 60.94, 3 seeds).Note after Eq. (6), Eq. (19) in Appendix A.2, Table 13
One advantage per chunk (5uP8, LCrk)We do not claim finer credit assignment. PPO only constrains the likelihood ratio of the whole action, while ROAD-VLA gives every prediction its own bounded target.Introduction, Appendix A.1
Inconsistent settings (xnCh)The ManiSkill OpenVLA-OFT results use the discretized head, and both methods share 16 environments, batch 160, and PPO clip 0.28. π0.5\pi_{0.5} uses 3 seeds.Appendix E and F, Table 5
Action heads and scope (xnCh, LCrk)With the official L1 regression head, ROAD-VLA improves ID from 96.88 to 98.44 and OOD from 26.39 to 29.17 (Table B).Appendix F.2, Tables 1 and 8
Training time (xnCh, LCrk)ROAD-VLA takes 1.41 to 1.61 times longer per run, but it reaches its best evaluation before PPO finishes training, so it still outperforms PPO given the same time.Appendix J, Table 14
Theory (LCrk)Theorem 1 is a conditional bound. For a categorical prediction, we show that the teacher moves in an improving direction when the sign of the advantage estimate is correct, which motivates the agreement gate.Appendix A.5

Table A. Component ablation (OOD success, %; one seed, 64 episodes, best evaluation). ID results are in Table 15, and ID and OOD learning curves are in Figures 11 and 12 (Appendix I).

MethodCR-MultiObjectER-InitPoseVR-UnseenTableMean
PPO50.0071.8881.2567.71
PPO + AWR48.4473.4481.2567.71
PPO + mixed advantage50.0075.0085.9470.31
ROAD (α=0\alpha=0, no reference)51.5675.0084.3870.31
ROAD-VLA57.8178.1389.0675.00

Table B. Results across action heads (average success, %). Per-environment results with standard deviations are in Appendix F, and learning curves are in Figures 3 and 6.

Policy and action headBenchmarkID: PPO → ROAD-VLAOOD: PPO → ROAD-VLA
OpenVLA, discrete tokensManiSkill76.86 → 82.3668.00 → 73.59
OpenVLA-OFT, discretizedManiSkill87.16 → 89.5869.10 → 73.61
OpenVLA-OFT, L1 regressionLIBERO96.88 → 98.4426.39 → 29.17
π0.5\pi_{0.5}, flow matchingLIBERO83.33 → 91.1529.34 → 33.86

With OpenVLA, ROAD-VLA improves in all 14 comparisons. All OOD results are measured on visual, compositional, and execution shifts not used for training.

We thank the reviewers again for their comments, which have made the paper much stronger. We believe that the new results and revisions fully address the concerns raised, and we would be grateful if they are taken into account in the final assessment. We are happy to discuss further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.