acceptodds
Under review as a conference paper at ICLR 2027

Multi-student On-Policy Self-Distillation with Alternating Reinforcement Learning for Robust Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models have emerged as a paradigm for general-purpose robotic control by translating visual observations and natural language instructions into executable actions. Recent reinforcement learning (RL) post-training further improves their task performance through online environment interaction and task-completion rewards. However, VLA policies remain sensitive to prompt variation, exhibiting substantially different success rates even across semantically equivalent instructions. For the same task, high-performing prompts can reliably elicit successful behavior, whereas low-performing prompts often provide few successful trajectories for direct RL. To exploit this performance asymmetry, we propose Multi-student On-policy Self-distillation with Alternating Reinforcement learning (MOSAR). MOSAR uses a high-performing prompt as a teacher for multiple semantically equivalent student prompts and distills teacher actions at student-visited observations, enabling cross-prompt behavior transfer within a shared VLA policy. For multi-task training, MOSAR adaptively weights distillation according to task-specific teacher–student performance gaps and alternates RL and distillation with separate optimizer states, allowing policy improvement and cross-prompt knowledge transfer to reinforce each other throughout training. From extensive evaluations on RoboLab, REALM, and a real Franka robot, MOSAR consistently improves success on unseen prompts over RL- and OPSD-based baselines while maintaining strong performance on targeted and unseen tasks. Additional evaluations under visual distractors further demonstrate robustness beyond the prompt variations explicitly addressed during training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, Self-Distilled Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P48cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how we can improve VLA sensitivity to semantically-equivalent prompts. The authors propose MOSAR - a method that uses the best-performing prompt as a teacher for the other student prompts. This is done via on-policy self-distillation using the teacher action outputs. However, since distillation alone cannot exceed the performance of the teacher, the authors propose to alternate the distillation steps with PPO updates. Experiments use \pi_0.5 on RoboLab, REALM, and a real Franka robot, improving success on held-out prompts and unseen tasks against baselines.

Critical strengths
  1. Studies an important problem of prompt sensitivity in VLAs. The main idea behind the paper (using best-performing prompt as a teacher model) is quite intuitive and nice.

  2. The practical implementation of the idea is based on the sensible observation regarding gradient discrepancies between the RL and BC objectives.

  3. The presentation is very clear and easy-to-follow

Critical weaknesses
  1. I happened to notice that some of the result numbers aren't matching. For instance, in the abstract and the introduction, you claim "MOSAR achieves 53.2% success on unseen prompts in RoboLab"; however, Table 1 reports 56.8%. Perhaps, I am missing something?

  2. If I read App. A6 correctly, the unseen prompt results for pi0.5 and OPSD-M come from a different evaluation. Would it be possible to re-evaluate these two baselines under exactly the same protocol as the rest?

  3. All results seem to come from a single training run and without error bars. It would be quite important to report multiple seeds for establishing the superiority of the method.

  4. I believe it would also be informative to add a couple of baselines. For instance, PPO from teacher prompt alone and PPO from all 11 prompts.

  5. Why isn't PPO / GRPO reported in Tables 3-4?

  6. As far as I understand, MOSAR has access to the stronger specific instruction (RoboLab specific instruction), extra distillation steps, and an adaptive step size. An important question for me is how much of the gain remains if the baselines get the teacher prompt and a matched compute budget?

Minor:

  • In Sec. 4.3: "We cast training as improving return (ii) ...". Not sure if this is a typo, but what does (ii) refer to?
Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal for reviewer P48c [1/2]

We thank the reviewer for the positive assessment and constructive suggestions, and for recognizing the importance of addressing prompt sensitivity in VLAs and the intuitive appeal of using the best-performing prompt as a teacher. We also appreciate the reviewer’s recognition of the sensible observation regarding gradient discrepancies between the RL and BC objectives underlying our implementation, and of our clear and easy-to-follow presentation. We address the numerical consistency, evaluation protocol, additional baselines, and clarification requests below.

[W1] I happened to notice that some of the result numbers aren't matching. For instance, in the abstract and the introduction, you claim "MOSAR achieves 53.2% success on unseen prompts in RoboLab"; however, Table 1 reports 56.8%. Perhaps, I am missing something?

We have aligned the reported RoboLab success rates throughout the manuscript with Table 1 to ensure consistency. The corresponding updates appear in the Introduction (L84) and Table A13 (formerly Table A12, L1512–L1525 in the revision).

[W2] If I read App. A6 correctly, the unseen prompt results for π0.5\pi_{0.5} and OPSD-M come from a different evaluation. Would it be possible to re-evaluate these two baselines under exactly the same protocol as the rest?

We clarify that both π0.5\pi_{0.5} and OPSD-M were evaluated on the same 100 unseen prompts and under the same protocol as all other methods, including MOSAR. This evaluation subset was randomly selected from an initial pool of 360 unseen prompts to make evaluation across a broader range of baselines computationally feasible. We have revised Appendix A.6 (L784–L792) to distinguish the initial 360-prompt pool from the final 100-prompt evaluation subset and clarify that all methods used the same protocol.

[W3] All results seem to come from a single training run and without error bars. It would be quite important to report multiple seeds for establishing the superiority of the method

Our RoboLab prompt evaluations already use four evaluation seeds, with multiple rollouts across different prompts for each task. To make evaluation uncertainty explicit, we report success rates with Wilson 95% confidence-interval half-widths in Tables R1–R2. Tables R1 and R2 add these intervals to the trained- and unseen-prompt results of Table 1 in the main paper, respectively.

From the tables, MOSAR achieves the highest overall success rate in all two tables, with confidence-interval widths comparable to those of the baselines. For the reported MOSAR–baseline comparisons, McNemar’s tests yield p<2×10−4p<2\times10^{-4} on trained prompts and p<10−5p<10^{-5} on unseen prompts, providing statistical evidence for the observed improvements under the evaluation protocol.

We added these results in the revision as a new App. C.4 (L1528–L1537), with Table A14 (L1538–L1558) and Table A15 (L1566–L1576).

Table R1. RoboLab success rates (%) on trained prompts from Table 1, with Wilson 95% confidence-interval half-widths, from four evaluation seeds.

MethodSauceMustardRaisinBoxBowlStackingOverall
MOSAR75.0±9.354.4±10.746.2±10.736.2±10.353.0±5.4
MOSAR w/o PPO73.4±9.655.7±10.741.0±10.78.8±6.344.7±5.4
OPSD-S59.5±10.656.4±10.741.8±10.616.2±8.043.5±5.4
OPSD-M53.2±10.748.7±10.738.0±10.513.8±7.638.4±5.3
PPO63.8±10.348.1±10.841.2±10.518.8±8.543.0±5.4
GRPO50.8±10.738.5±10.632.9±10.13.8±4.631.5±5.1
π0.555.1±10.849.4±10.834.6±10.312.5±7.337.9±5.3

Table R2. RoboLab success rates (%) on unseen prompts from Table 1, with Wilson 95% confidence-interval half-widths from four evaluation seeds.

MethodSauceMustardRaisinBoxBowlStackingOverall
MOSAR78.0±8.062.0±9.354.1±9.733.0±9.156.8±4.8
MOSAR w/o PPO73.7±8.650.5±9.730.6±9.06.0±4.840.2±4.8
OPSD-S58.0±9.550.0±9.643.9±9.613.0±6.641.2±4.8
OPSD-M38.8±9.547.0±9.636.0±9.216.0±7.234.4±4.6
PPO49.5±9.740.4±9.540.8±9.64.0±4.133.7±4.6
GRPO49.0±9.647.5±9.729.6±8.96.0±4.833.0±4.6
π0.542.9±9.641.0±9.534.0±9.111.0±6.232.2±4.6
AuthorsOpenReview ↗

Rebuttal for reviewer P48c [2/2]

[W4,6] I believe it would also be informative to add a couple of baselines. For instance, PPO from teacher prompt alone and PPO from all 11 prompts. As far as I understand, MOSAR has access to the stronger specific instruction (RoboLab specific instruction), extra distillation steps, and an adaptive step size. An important question for me is how much of the gain remains if the baselines get the teacher prompt and a matched compute budget?

As suggested, we evaluated teacher-only PPO and PPO with MOSAR’s teacher/student split using 1, 3, or 10 student prompts per iteration. MOSAR outperforms all these controls, as well as the Table 1 PPO and GRPO baselines, on both trained and unseen prompts (Table R3). The added PPO controls match MOSAR’s interaction budget and PPO optimization settings. In the split controls, half of the environments use the teacher prompt, while the remainder use student prompts sampled from the ten-prompt pool.

Teacher-only PPO achieves 30.6% on trained prompts and 33.2% on unseen prompts. The strongest PPO control uses three students per iteration and achieves 47.2% / 44.8%, compared with MOSAR’s 53.0% / 56.8%. MOSAR therefore retains gains of 5.8 and 12.0 percentage points, respectively, supporting the additional benefit of multi-student distillation. MOSAR additionally performs teacher-target inference and a distillation backward pass on the collected rollouts, adding computation without requiring additional environment interactions.

We added these comparisons and a discussion of interaction budgets and distillation costs to Appendix C.5–C.6 and Tables A16–A17 (L1560–L1644).

Table R3. RoboLab success rates (%) on trained and unseen prompts for PPO controls with teacher-prompt access and the Table 1 baselines.

MethodTrainedUnseen
PPO, fixed best teacher prompt30.633.2
PPO, teacher/student split (1 student)35.330.1
PPO, teacher/student split (3 students)47.244.8
PPO, teacher/student split (10 students)40.938.2
PPO, 10 training prompts (Table 1)43.033.7
GRPO, 10 training prompts (Table 1)31.533.0
MOSAR53.056.8

[W5] Why isn't PPO / GRPO reported in Tables 3-4?

We added PPO and GRPO to both evaluations under the same protocols in Table R4, R5. On goal swap, they achieve 36.5% and 34.4%, compared with 37.5% for π0.5\pi_{0.5} and 54.2% for MOSAR. Under visual distractors, they improve over π0.5\pi_{0.5} but remain below OPSD-S and MOSAR. These results highlight the benefits of MOSAR’s combination of RL and multi-student distillation for following changed instructions and maintaining robustness to visual distractors.

In the revision, we added PPO and GRPO rows to Table 3 (L385–L386) and Table 4 (L398–L399), together with one sentence summarizing them in Sec. 5.4 (L376–L377).

Table R4. Goal-swap success rates (%) under default instructions, reproduced from Table 3.

MethodBBQ → saladRight → left binLarger raisin → smaller butterLeft-on-right → right-on-leftOverall
π0.537.541.750.020.837.5
PPO41.741.758.34.236.5
GRPO37.537.554.28.334.4
OPSD-S16.745.866.733.340.6
OPSD-M12.54.241.720.819.8
MOSAR37.550.066.762.554.2

Table R5. Success rates (%) under visual distractors, reproduced from Table 4.

MethodSauceMustardRaisinBoxBowlStackingOverall
π0.558.337.54.20.025.0
PPO66.762.54.24.234.4
GRPO62.541.70.020.831.3
OPSD-S62.566.716.716.740.6
OPSD-M62.533.312.58.329.2
MOSAR75.066.720.841.751.1

[W7] In Sec. 4.3: "We cast training as improving return (ii) ...". Not sure if this is a typo, but what does (ii) refer to?

(i)–(iii) are the three goals from the introduction — (i) close the gap among phrasings, (ii) raise the ceiling of the strongest phrasing, (iii) preserve other tasks. We restate them at the start of Sec. 4 and rewrite the sentence as “we cast training as improving return, i.e. raising the ceiling of the strongest phrasing (goal (ii)), while …”. In the revision, we clarified in Sec. 4.3 (L274–L275) that “(ii)” refers to goal (ii) in the Introduction, namely raising the ceiling of the strongest phrasing.

AuthorsOpenReview ↗

A gentle note on our response

Dear Reviewer,

Thank you again for your thoughtful review and helpful comments on our paper.

We just wanted to share a brief note on our response. Following your comments, we corrected the inconsistent success rate, clarified that all baselines were evaluated under the same protocol, reported 95% confidence intervals with McNemar's tests (Tables R1–R2), added PPO controls with teacher-prompt access and a matched interaction budget (Table R3), and added PPO and GRPO to the goal-swap and visual-distractor evaluations (Tables R4–R5).

We understand this is a very busy time, and we sincerely appreciate the time and effort you have already devoted to reviewing our paper. If time permits before the discussion period closes, we would be grateful if you could read through our response. Any further thoughts or feedback would be very welcome, and we would be happy to clarify anything.

Best regards, The Authors

Reviewer_DoAjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the sensitivity of VLA models to different wordings of the same instruction. MOSAR uses a high-performing prompt as the teacher to guide multiple student prompts at their visited states. It combines PPO with self-distillation through alternating updates and adjusts the distillation weights based on teacher-student performance gaps. Experiments on RoboLab, REALM, and a real Franka robot show improved performance on unseen prompts, with additional evaluations on unseen tasks and visual distractors.

Critical strengths

1.The paper studies an important and practical weakness of VLA models: semantically equivalent instructions can lead to very different task success rates. The motivation is clear, and combining PPO for task improvement with on-policy self-distillation for cross-prompt transfer is well aligned with the problem.
2.The multi-student design is also sensible. A shared VLA policy is trained under multiple semantically equivalent student prompts, while a stronger teacher prompt provides supervision at the states visited by those students. The evaluation is reasonably broad, covering RoboLab, REALM, unseen prompts, unseen tasks, and real-robot experiments.

Critical weaknesses
  1. The paper tracks teacher/student success rates and reduces the distillation weight when the teacher degrades, but it does not show how these quantities evolve throughout training. It is therefore unclear whether the fixed teacher remains consistently stronger for most of training. More importantly, teacher quality is judged by overall rollout success, while distillation is applied at student-visited states; the paper does not directly verify that the teacher actions are actually better at those states.
  2. The current ablations show that PPO, weighting, and the optimization design are useful, but they do not directly verify the proposed mechanism of improving the teacher with PPO and then transferring that improvement through refreshed multi-student distillation. Key comparisons are missing, including best-teacher-only PPO, fresh vs. frozen teacher targets, and different numbers of student prompts.
Questions for authors
  1. Could the authors report how rkTr_k^T, rkSr_k^S, bkb_k, and wkw_k evolve throughout training? In particular, how often does the teacher prompt remain stronger than the student prompts, and how frequently is the deterioration gate activated?
  2. What is the performance of PPO trained only with the fixed best teacher prompt, without OPSD, when evaluated on student and unseen prompts? This would help clarify the additional contribution of OPSD.
  3. Could the authors compare refreshed teacher targets with frozen or stale teacher targets, and report results for different numbers of students (M=1,2,3,…M=1,2,3,\ldots)? These comparisons would help isolate the contributions of teacher refreshing and the multi-student design.
3 comments
AuthorsOpenReview ↗

Rebuttal for reviewer DoAj [1/2]

We thank the reviewer for the constructive assessment and for highlighting the important and practical weakness of VLA models that semantically equivalent instructions can lead to very different task success rates. We particularly appreciate the reviewer’s recognition that our motivation is clear and that combining PPO for task improvement with on-policy self-distillation for cross-prompt transfer is well aligned with this problem. We also appreciate the positive assessment of our multi-student design and evaluation. We address the questions below.

[W1, Q1] The paper tracks teacher/student success rates and reduces the distillation weight when the teacher degrades, but it does not show how these quantities evolve throughout training. It is therefore unclear whether the fixed teacher remains consistently stronger for most of training. Could the authors report how rkTr_k^T, rkSr_k^S, bkb_k and ωk\omega_k evolve throughout training? More importantly, teacher quality is judged by overall rollout success, while distillation is applied at student-visited states; the paper does not directly verify that the teacher actions are actually better at those states. How often does the teacher prompt remain stronger than the student prompts, and how frequently is the deterioration gate activated?

Following the reviewer's suggestions, we conducted analyses in Tables R1–R2 and Figure A1 (in the revision), showing that 1) the teacher is substantially stronger early in training and then distillation decreases as students catch up, 2) and teacher-conditioned continuations from student-visited states are actually better than the students' rollouts

Specifically, we report the training dynamics of rkTr_k^T, rkSr_k^S, Δk\Delta_k, bkb_k, and the normalized distillation weight wk=λk/5.5w_k=\lambda_k/5.5. As summarized in Table R1, the average teacher–student success gap decreases from 0.130 in the first quarter to −0.003 in the final quarter, while the normalized distillation weight decreases from 0.356 to 0.015. Figure A1 shows these trajectories, including the factor bkb_k and the number of tasks receiving distillation. Distillation is applied to most tasks early in training and to fewer tasks as students catch up. The deterioration gate is active during the early stages of training and remains active in approximately 75–80% of iterations overall (wk>0w_k>0).

Table R1. Training dynamics averaged over four tasks and iterations within each quarter. Success rates are EMA-smoothed with α=0.5\alpha=0.5.

ProgressrTr^TrSr^SΔ=rT−rS\Delta=r^T-r^Sbbww
0–25%0.3140.184+0.1300.740.356
25–50%0.2890.238+0.0510.380.079
50–75%0.3480.309+0.0390.550.038
75–100%0.3610.364−0.0030.440.015

We also evaluate teacher-conditioned continuations from states reached under student prompts. As shown in Table R2, switching to the teacher prompt at 25% of rollout progress improves all four tasks, raising average success from 36.7% to 43.4%. Switching at 50% raises average success to 39.9%, with task-dependent effects. These results support the benefit of teacher-conditioned continuations from student-visited states.

We added these analyses to Appendix C.9 (L1721–L1741), with training dynamics in Table A20 (L1742–L1749) and Figure A1 (L1751–L1771), and the prompt-switch test in Table A21 (L1773–L1779).

Table R2. Task success rates (%) for teacher-conditioned continuations from student-visited states.

ConditionAvg.SauceMustardRaisinBoxBowlStacking
Student only36.741.847.538.818.8
Student → Teacher at 25%43.445.653.255.020.0
Student → Teacher at 50%39.958.252.536.712.5
AuthorsOpenReview ↗

Rebuttal for reviewer DoAj [2/2]

[W2, Q2, Q3] The current ablations show that PPO, weighting, and the optimization design are useful, .... Key comparisons are missing, including best-teacher-only PPO, fresh vs. frozen teacher targets, and different numbers of student prompts. What is the performance of PPO trained only with the fixed best teacher prompt, without OPSD, when evaluated on student and unseen prompts? Moreover, could the authors compare refreshed teacher targets with frozen or stale teacher targets, and report results for different numbers of students?

To address the reviewer’s comments, we report three additional comparisons: 1) PPO with the fixed best teacher prompt, 2) refreshed versus frozen teacher targets, and 3) different numbers of students. Each comparison is now a separate subsection of App. C in the revision (C.6, C.7 and C.8, respectively).

1) PPO with the fixed best teacher prompt. PPO trained exclusively with the fixed best teacher prompt substantially underperforms MOSAR on both trained student and unseen prompts, supporting the need for explicit cross-prompt distillation. We trained this baseline with all 128 environments using the teacher prompt, keeping the other PPO baseline settings unchanged. It achieves 30.6% on trained prompts (37.9% for π0.5\pi_{0.5} and 53.0% for MOSAR). On unseen prompts, it reaches 33.2%, compared with 32.2% and 56.8%, respectively. (See Appendix C.6 of the revision)

Table R3. RoboLab success rates (%) after PPO training exclusively under the fixed best teacher prompt.

MethodSauceMustardRaisinBoxBowlStackingAvg.
Trained student prompts
π0.555.149.434.612.537.9
PPO, teacher prompt only53.843.611.413.830.6
MOSAR75.054.446.236.253.0
Unseen prompts
π0.542.941.034.011.032.2
PPO, teacher prompt only60.644.917.310.033.2
MOSAR78.062.054.133.056.8

2) Refreshed versus frozen teacher targets. Refreshing teacher targets increases success from 50.6% to 53.0% on trained prompts and from 45.3% to 56.8% on unseen prompts, with the larger gain on unseen prompts supporting its contribution to generalization (Table R4). Together with the component baselines, these results support the complementary contributions of PPO, multi-student distillation, and refreshed teacher targets, with their full combination in MOSAR achieving the strongest performance. (See Appendix C.7 of the revision)

Table R4. RoboLab success rates (%) for MOSAR, its frozen-target variant, and individual components.

Methodw/ OPSDw/ PPOTeacherTrainedUnseen
π0.5 (no training)———37.932.2
MOSAR✓✓refreshed53.056.8
MOSAR w/ frozen teacher✓✓frozen50.645.3
Multi-student OPSD only✓✗frozen45.142.9
PPO only✗✓—47.244.8

3) Different numbers of students. Under a fixed student rollout budget, performance is highest among the tested configurations with seven prompts when all pool prompts are included each iteration (Table R5), and with three prompts per iteration when sampling from a fixed pool of ten (Table R6), consistent with a trade-off between prompt coverage and supervision per prompt.

Both studies use multi-student OPSD alone with a frozen teacher. In Table R5, we vary the student prompt pool size and include every prompt in each iteration, dividing the rollout budget among them. Performance improves as the pool grows from one to seven prompts but declines at ten, suggesting that broader coverage can become less beneficial when too few rollouts are allocated to each prompt.

Table R6 fixes the pool at ten prompts, as in MOSAR, and varies the number sampled per iteration. Sampling three prompts achieves the strongest performance on both trained and unseen prompts, supporting our choice of three students per iteration to balance coverage and sampling depth. (See Appendix C.8 of the revision)

Table R5. RoboLab success rates (%) for different student counts under a frozen teacher and fixed rollout budget.

Setting# of student in prompt poolTrained (%)Unseen (%)
π0.5π_{0.5}—37.932.2
Multi-student OPSD139.939.2
Multi-student OPSD442.740.3
Multi-student OPSD742.841.1
Multi-student OPSD1040.238.5

Table R6. RoboLab success rates (%) for different student-prompt sampling number with 10 student prompt pool.

MethodSampled prompts/iterationTrained (%)Unseen (%)
π0.5π_{0.5}—37.932.2
Multi-student OPSD140.340.9
Multi-student OPSD3 (our setup)45.142.9
Multi-student OPSD1040.238.5
AuthorsOpenReview ↗

A gentle note on our response

Dear Reviewer,

Thank you again for your thoughtful review and helpful comments on our paper.

We just wanted to share a brief note on our response. Following your comments, we reported the training dynamics of teacher/student success rates and distillation weights (Table R1, Figure A1), verified that teacher-conditioned continuations improve success from student-visited states (Table R2), and added comparisons with PPO under the fixed best teacher prompt (Table R3), refreshed versus frozen teacher targets (Table R4), and different numbers of student prompts (Tables R5–R6).

We understand this is a very busy time, and we sincerely appreciate the time and effort you have already devoted to reviewing our paper. If time permits before the discussion period closes, we would be grateful if you could read through our response. Any further thoughts or feedback would be very welcome, and we would be happy to clarify anything.

Best regards, The Authors

Reviewer_ECo1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the sensitivity of VLA models to semantically equivalent instructions and proposes MOSAR. The method uses the same model conditioned on a stronger prompt to provide action supervision for trajectories generated under multiple student prompts. It combines task-level adaptive weighting with alternating RL and distillation updates using separate optimizer states. The authors evaluate task performance and generalization on RoboLab, REALM, and a real Franka robot.

Critical strengths
  1. The problem has practical significance. Performance variation across different instruction formulations is an important challenge for the reliable deployment of VLA models.
  2. The method is clearly motivated. It uses performance differences across prompts within the same model to provide supervision, while RL further improves the policy, resulting in a coherent overall design.
  3. The evaluation covers multiple dimensions. Experiments include two simulation benchmarks, a real robot, and settings involving unseen prompts, unseen tasks, and visual distractors. The reported performance gains are promising.
Critical weaknesses
  1. The relationship between the main results and the CV analysis is unclear. The introduction and Table A12 report a RoboLab unseen-prompt success rate of 53.2% for MOSAR, whereas Tables 1 and 7 and the conclusion report 56.8%; the per-task results in Table 1 also support the latter value. This discrepancy makes it unclear how the main performance results relate to the analysis of the coefficient of variation (CV) across prompts. Please clarify whether these values correspond to different checkpoints, evaluation sets, or aggregation methods, and specify the data used for the main results and the CV analysis.
  2. The PPO baseline uses a different prompt sampling configuration, limiting attribution of the gains to distillation. MOSAR allocates half of its environments to the teacher prompt, whereas PPO samples from the other ten training prompts without using the same teacher/student prompt allocation. The comparison therefore changes both the training objective and the rollout prompt configuration. The current results cannot rule out the possibility that some gains arise from a more favorable prompt allocation and the resulting reward signals. Could the authors include a PPO baseline that uses exactly the same allocation of environments to teacher and student prompts as MOSAR, matches its RL optimization settings, and performs only RL updates? This comparison would more clearly establish the incremental benefit of adding distillation.
  3. The independent benefit of the multi-student design over OPD with random prompt sampling has not been sufficiently established. The substantive distinction between the multi-student method and standard OPD that randomly samples from the same prompt pool remains unclear. Standard OPD can also visit states induced by different student prompts by sampling prompts across batches or independently for each trajectory. The current coverage analysis does not account for sample allocation under a fixed rollout budget, nor does it explain why using multiple prompts within a batch should outperform these alternatives. Moreover, increasing the number of student prompts per iteration reduces the number of trajectories allocated to each prompt, potentially introducing a trade-off between broader coverage and sufficient sampling for each prompt. I recommend comparing these sampling schemes under the same teacher, prompt pool, interaction budget, and optimization settings to establish whether the gains are attributable to the multi-student design itself.
Questions for authors

Please address the questions raised in the weaknesses above.

5 comments
AuthorsOpenReview ↗

Rebuttal for reviewer ECo1 [1/2]

We thank the reviewer for the constructive feedback and for recognizing the practical significance of addressing VLA prompt sensitivity, the clear motivation and coherent overall design of our method, and the breadth of our evaluation across simulation benchmarks, a real robot, and diverse generalization settings. We also appreciate the reviewer’s assessment that the reported performance gains are promising. We address each concern below.

[W1] The relationship between the main results and the CV analysis is unclear. The introduction and Table A12 report a RoboLab unseen-prompt success rate of 53.2% for MOSAR, whereas Tables 1 and 7 and the conclusion report 56.8%; the per-task results in Table 1 also support the latter value. This discrepancy makes it unclear how the main performance results relate to the analysis of the coefficient of variation (CV) across prompts. Please clarify whether these values correspond to different checkpoints, evaluation sets, or aggregation methods, and specify the data used for the main results and the CV analysis.

The discrepancy was a typographical error; we have corrected inconsistent values throughout the manuscript to match Table 1 and recomputed the prompt-level CV using Eq. A5, confirming that MOSAR retains the lowest CV on both splits. These corrections appear in the Introduction (L84) and Table A12 (Table A13 in the revision, L1512–L1525).

[W2] The PPO baseline uses a different prompt sampling configuration, limiting attribution of the gains to distillation. MOSAR allocates half of its environments to the teacher prompt, whereas PPO samples from the other ten training prompts without using the same teacher/student prompt allocation. The comparison therefore changes both the training objective and the rollout prompt configuration. The current results cannot rule out the possibility that some gains arise from a more favorable prompt allocation and the resulting reward signals. Could the authors include a PPO baseline that uses exactly the same allocation of environments to teacher and student prompts as MOSAR, matches its RL optimization settings, and performs only RL updates? This comparison would more clearly establish the incremental benefit of adding distillation.

We trained an RL-only PPO baseline with exactly the same teacher/student environment allocation and RL optimization settings as MOSAR, and it still underperforms MOSAR on both trained and unseen prompts, supporting the incremental benefit of distillation, as shown in Table R1. For each task, half of the environments use the teacher prompt and half use student prompts from the same 10-prompt pool, with identical numbers of environments and training configurations. We also varied the number of student prompts selected per iteration across 1, 3, and 10 to examine its effect on PPO performance.

As shown in Table R1, selecting one student prompt per iteration yields no improvement over π0.5π_{0.5} on either trained or unseen prompts. Three students per iteration, the configuration used in MOSAR, gives the strongest PPO results, while increasing the number to ten yields smaller gains. Nevertheless, even the strongest PPO variant clearly underperforms MOSAR on both trained and unseen prompts. Thus, prompt allocation alone does not fully explain MOSAR’s gains, supporting the benefit of our multi-student OPSD.

These experiments are reported in the new Appendix C.5 (L1562–L1597) and Table A16 (L1599–L1619), which also includes the PPO baseline from Table 1 for direct comparison.

Table R1. Success rates (%) on trained and unseen prompts in RoboLab for PPO controls with MOSAR’s prompt allocation and different numbers of student prompts per iteration (1, 3, or 10).

MethodTrained(%)Unseen(%)
π0.5π_{0.5}37.932.2
PPO, MOSAR's split, 1 student per iteration35.330.1
PPO, MOSAR's split, 3 students per iteration (MOSAR's setting)47.244.8
PPO, MOSAR's split, 10 students per iteration40.938.2
MOSAR53.056.8
AuthorsOpenReview ↗

Rebuttal for reviewer ECo1 [2/2]

[W3] The independent benefit of the multi-student design over OPD with random prompt sampling has not been sufficiently established. The substantive distinction between the multi-student method and standard OPD that randomly samples from the same prompt pool remains unclear. Standard OPD can also visit states induced by different student prompts by sampling prompts across batches or independently for each trajectory. The current coverage analysis does not account for sample allocation under a fixed rollout budget, nor does it explain why using multiple prompts within a batch should outperform these alternatives. Moreover, increasing the number of student prompts per iteration reduces the number of trajectories allocated to each prompt, potentially introducing a trade-off between broader coverage and sufficient sampling for each prompt. I recommend comparing these sampling schemes under the same teacher, prompt pool, interaction budget, and optimization settings to establish whether the gains are attributable to the multi-student design itself.

We address this concern by clarifying (1) our consideration of single-student OPSD baseline (OPSD-S, in the manuscript) as standard OPD for VLAs and (2) the random prompt sampling alternative suggested by the reviewer and the coverage–sampling trade-off that motivates MOSAR’s choice of three students per iteration.

We considered our single-student baseline (OPSD-S), using one fixed student prompt per task such as the vague instruction provided by RoboLab, as the standard OPD. This setup follows the task-specific, single-prompt training used in on-policy RL studies for VLAs, including π\pi-StepNFT (Wang et al., 2026b) and π\pi-RL (Chen et al., 2025). Therefore, we denote the alternative suggested by the reviewer as OPSD-R, which randomly selects one student prompt per iteration, whereas MOSAR jointly trains on multiple student prompts within each iteration. To evaluate this alternative and the trade-off between broader prompt coverage and sufficient sampling per prompt, we compare OPSD-R and a ten-student variant against our three-student setup under the same frozen teacher, prompt pool, and interaction budget, with RL disabled.

As shown in Table R2, three students per iteration (our setup) achieves the highest success rates (45.1% trained / 42.9% unseen), outperforming both OPSD-R (40.3% / 40.9%) and ten students per iteration (40.2% / 38.5%). OPSD-R also performs slightly below OPSD-S, suggesting that broader prompt exposure alone does not ensure better transfer. OPSD-R may reduce repeated supervision by changing prompt-conditioned state distributions across iterations, while using all ten prompts may leave too few trajectories per prompt under a fixed rollout budget. These results are consistent with a trade-off between prompt coverage and sampling depth and support our choice of three students per iteration.

We added this comparison to an App. C.8 (L1670–L1696) and Table A19 (L1698–L1715) in the revision.

Table R2. RoboLab success rates (%) for different student-prompt sampling schemes. OPSD-R and multi-student OPSD under the same frozen teacher, 10-prompt pool, interaction budget.

MethodStudent-prompt samplingPrompts/iterationTrained (%)Unseen (%)
π0.5π_{0.5}——37.932.2
OPSD-S (Table 1)One fixed prompt (RoboLab vague instruction)143.541.2
OPSD-ROne random prompt each iteration140.340.9
Multi-student OPSD (our setup)Random subset each iteration345.142.9
Multi-student OPSDAll training prompts each iteration1040.238.5
Reference: MOSAR (OPSD+RL)Random subset each iteration353.056.8
AuthorsOpenReview ↗

A gentle note on our response

Dear Reviewer,

Thank you again for your thoughtful review and helpful comments on our paper.

We just wanted to share a brief note on our response. Following your comments, we corrected the inconsistent unseen-prompt results, added an RL-only PPO baseline with exactly the same teacher/student allocation and optimization settings as MOSAR (Table R1), and compared multi-student OPSD with random single-prompt sampling under the same teacher, prompt pool, and interaction budget (Table R2).

We understand this is a very busy time, and we sincerely appreciate the time and effort you have already devoted to reviewing our paper. If time permits before the discussion period closes, we would be grateful if you could read through our response. Any further thoughts or feedback would be very welcome, and we would be happy to clarify anything.

Best regards, The Authors

Comments on the paper

2 comments

Dear Reviewers,

Please read the authors’ response and let us know whether it addresses your concerns. If any concerns remain, please briefly indicate which ones.

Thank you for your time and for your thoughtful reviews.

Best,

AC

AuthorsOpenReview ↗

General Response

We thank the Area Chair and all reviewers for their efforts and constructive feedback. Before the end of the author–reviewer discussion, we briefly summarize the strengths noted by the reviewers, their main concerns, and how we addressed them, with pointers to the discussion threads and revised manuscript.

Strengths noted by the reviewers

  1. An important and practical problem: semantically equivalent instructions can lead to substantially different success rates in VLA models [ECo1, DoAj, P48c].
  2. A clearly motivated and coherent approach that combines policy improvement through RL with cross-prompt transfer through on-policy self-distillation [ECo1, DoAj]; an intuitive use of a stronger prompt as the teacher [P48c].
  3. Broad evaluation across two simulation benchmarks and a real Franka robot, including unseen prompts, unseen tasks, and visual distractors, with promising performance gains [ECo1, DoAj].
  4. Clear presentation and a sensible optimization design motivated by discrepancies between the RL and behavior-cloning gradients [P48c].

Concerns and our responses

  1. Numerical consistency and evaluation protocol [ECo1 W1; P48c W1–W2]. We corrected the RoboLab unseen-prompt success rate throughout the manuscript, still confirming that MOSAR retains the lowest CV on both splits. We also clarified that all methods used the same 100 unseen prompts, sampled from an initial pool of 360, under the same evaluation protocol [Appendix A.6, Table A13].
  2. Attribution of gains: teacher-prompt access, prompt allocation, and training budget [ECo1 W2; DoAj Q2; P48c W4/W6]. We added teacher-only PPO and PPO controls matching MOSAR’s teacher/student allocation, interaction budget, and PPO settings. The strongest control achieves 47.2% / 44.8% on trained/unseen prompts, versus MOSAR’s 53.0% / 56.8%, supporting an additional benefit from distillation beyond prompt allocation. These controls match environment interactions; MOSAR additionally incurs teacher-inference and distillation computation [Appendix C.5–C.6, Tables A16–A17].
  3. Benefit of multiple students and the coverage–sampling trade-off [ECo1 W3; DoAj Q3]. Under the same frozen teacher, ten-prompt pool, and interaction budget, three students per iteration achieve 45.1% / 42.9% on trained/unseen prompts, outperforming random single-student sampling (40.3% / 40.9%) and ten students (40.2% / 38.5%). Together with prompt-pool-size experiments, these results support balancing prompt coverage with sufficient trajectories per prompt [Appendix C.8].
  4. Teacher reliability during training and at student-visited states [DoAj W1/Q1]. We added training dynamics showing that distillation weights decrease as students catch up with the teacher. We also tested teacher-conditioned continuations from student-visited states: switching to the teacher prompt at 25% of rollout progress improves all four tasks, raising average success from 36.7% to 43.4% [Appendix C.9, Tables A20–A21 and Figure A1].
  5. Contribution of refreshed teacher targets [DoAj W2/Q3]. We added a frozen-teacher variant. Refreshing targets improves trained/unseen success from 50.6% / 45.3% to 53.0% / 56.8%, supporting its contribution alongside RL and multi-student distillation [Appendix C.7].
  6. Evaluation uncertainty [P48c W3]. We clarified that RoboLab evaluations use four evaluation seeds and added Wilson 95% confidence intervals and McNemar’s tests. MOSAR retains the highest overall success on both splits: 53.0 ± 5.4% on trained prompts and 56.8 ± 4.8% on unseen prompts, with ± indicating confidence-interval half-widths [Appendix C.4, Tables A14–A15].
  7. Missing RL baselines in robustness evaluations [P48c W5]. We added PPO and GRPO under the same protocols. MOSAR achieves 54.2% on goal swap versus 36.5% / 34.4% for PPO/GRPO, and 51.1% under visual distractors versus 34.4% / 31.3% [Tables 3–4].

We sincerely thank the AC and all reviewers for helping us strengthen the evaluation and clarify our contributions. We hope that the rebuttal and revisions address the major concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.