acceptodds
Withdrawn from ICLR 2027

VLA-MOPD: On-Policy Distillation for Multi-Task VLA Policy Consolidation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Physical intelligence requires embodied agents to ground language in observations and execute actions while retaining useful skills. Vision-language-action (VLA) policies offer a shared interface, but task-specific post-training often produces separate specialists. Consolidation is difficult: mixed-task reinforcement learning provides sparse trajectory-level credit, while offline distillation may miss student-visited observations and endpoint distillation does not supervise intermediate action-generation states. We introduce VLA-MOPD, a VLA multi-teacher on-policy distillation protocol for consolidating flow-matching VLA specialists. After acquiring task or suite experts, a shared student collects robot rollouts and queries the instruction-routed frozen teacher at each visited observation and intermediate ordinary differential equation (ODE) state. Detached per-flow-step transition targets supervise valid action chunks through a masked local matching loss; an optional reference-policy term supports retention on other tasks. Starting from one trajectory per LIBERO task, VLA-MOPD raises mean success-once from 75.4% to 98.9%, compared with 98.15% for endpoint action distillation. It improves mean RoboTwin 2.0 success from 73.0% to 89.3%; on a Piper robot, two consolidation rounds improve average success from 70.0% to 90.0%. In a matched Spatial-adaptation study, reference matching limits the Object suite decrease to 3.0 points, versus 13.2 without the reference term.

Primary area:
reinforcement learning
Keywords:
Flow Matching, Vision-Language-Action Models, On-Policy Distillation, Multi-Task Reinforcement Learning, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_FnTFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VLA-MOPD, a multi-teacher on-policy distillation protocol that consolidates multiple VLA specialists into a single shared policy by querying task-specific teachers on student-induced observations and intermediate flow states. While the research direction of VLA policy consolidation is important, the paper currently suffers from incremental technical novelty and insufficient experimental support.

Critical strengths
  1. The paper addresses multi-task VLA capability consolidation and continual evolution, which is of strong practical significance.

  2. Supervising on student-induced states intuitively mitigates distribution shift compared to offline distillation.

  3. The evaluation spans simulated environments (LIBERO, RoboTwin 2.0) and real-world experiments on a Piper robot, demonstrating applicability across diverse settings.

Critical weaknesses
  1. The proposed methodology primarily combines existing ideas from DAgger, VLA-OPD, and Flow-OPD. The independent contribution and necessity of per-flow-step transition matching have not been fully substantiated, especially on tasks with high baseline success rates.

  2. The ablation studies rely heavily on LIBERO suites where performance is close to saturation. Here, VLA-MOPD achieves only a marginal improvement (~0.75 percentage points) over endpoint distillation.

  3. On RoboTwin 2.0 and the real-world Piper robot, the consolidated student policy consistently lags behind the specialist teachers, indicating that the method has not yet proven its ability to achieve lossless capacity consolidation across multiple specialists.

Questions for authors
  1. Compared with endpoint action distillation, VLA-OPD, and Flow-OPD, does per-flow-step transition matching yield consistent and significant independent gains on non-saturated or highly compositional tasks?

  2. What are the primary bottlenecks causing the student to slightly surpass teachers on LIBERO while lagging behind by 4–6 percentage points on RoboTwin and Piper? Is it due to task interference, dual-arm coordination challenges, long action horizons, or model capacity limits?

  3. What is the precise primary objective of this method—performance improvement, capability retention, or specialist compression? Could the authors explicitly report teacher–student capability coverage, forgetting rates, as well as the additional storage and inference overhead?

Reviewer_P58cOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes VLA-MOPD, a way to merge several task- or suite-specific π0.5 specialists into one flow-matching VLA. The student runs its own rollouts. At each visited observation, the frozen teacher assigned to that task (chosen by instruction) provides a target at every intermediate ODE step of action generation, and the student matches the teacher's Euler step means (equivalently, a δ²-weighted velocity MSE). An optional term matches a frozen copy of the initial policy on retained tasks to reduce forgetting. On LIBERO, a 1-demo-per-task student goes from 75.4% to 98.9% (teachers 98.5%, endpoint action distillation 98.15%). RoboTwin (3 tasks) goes from 73.0% to 89.3%, and a Piper robot (3 tasks) from 70.0% to 90.0%. The paper also compares against multi-task GRPO (MixRL) in optimizer steps and includes a Spatial→Object forgetting study.

Critical strengths

The setting makes sense. Many groups now end up with several RL-tuned specialist checkpoints, and turning them back into one policy is a practical problem. On-policy teacher queries are a reasonable fit for this. The student does reach close to teacher performance on all three benchmarks, including a real robot (90.0% vs. 93.3% for the teachers on Piper), and the reference term clearly reduces forgetting in the one setting where it was tested (3.0 vs. 13.2 points drop on Object). I checked the averages in all tables and they are consistent.

Critical weaknesses

The main technical claim isn't supported by the evidence. What's new relative to DAgger-style distillation is supervising the intermediate flow states instead of only the final action. The only test of this is on LIBERO: 98.9% vs. 98.15% for endpoint distillation, from a single training seed (Appx. A, seed 42). With 500 episodes per suite at around 98% success, the evaluation noise alone is about 0.6 points per suite, and both methods are already at the teachers' level (98.5%). So I don't think the 0.75-point gap tells us anything. Endpoint distillation is not run on RoboTwin or the real robot, and the paper doesn't describe how it was implemented. The other comparisons don't isolate the contribution either. MixRL learns from sparse rewards with no teacher. Its "matched rollout budget" leaves out the cost of training the PPO teachers, which the student gets for free, so the convergence plot in Fig. 2 compares two very different amounts of supervision.

It's not clear that this is consolidation of specialists in a meaningful sense, and the scale is too small to test it. According to the Table 1 caption, Spatial, Object and Goal share a single "PPO-LIBERO-130" teacher, and LIBERO-10 uses "PPO120". If LIBERO-130 refers to the union of the LIBERO suites, then the LIBERO experiment distills two checkpoints, one of which is already a multi-task policy. The "1 trajectory per task" framing also hides how much data and RL compute went into the teachers. Beyond LIBERO, the paper merges three tasks in RoboTwin and three on the real robot, and compares only against the student's initialization and the teachers themselves. There are no consolidation baselines: weight merging or task arithmetic, MergeVLA, multi-task SFT on teacher rollouts, or the closest prior methods, VLA-OPD and Flow-OPD. Interference as the number of specialists grows, which is the question I'd most want answered for a consolidation method, is not studied.

Questions for authors
  1. With a deterministic sampler, the action is fully determined by the initial noise and the velocity field. If the student matches the teacher's endpoint from the same noise, the whole chain is already pinned down. So what does matching every intermediate step buy you? I'd like to see one case where endpoint distillation clearly fails and yours doesn't. A multimodal teacher seems like the obvious candidate. Right now the only evidence is 98.9 vs 98.15, and both are already at the teacher's level.
  2. The teacher is queried at the student's ODE states, not its own. Early in training many of these states are ones the teacher's sampler would never reach. Why trust its velocity there? Endpoint distillation doesn't have this problem.
  3. You need the instruction to route to a teacher during training, and you still have it at test time. Why not just deploy the specialists with the router? The student never beats its teachers (89.3 vs 94.0 on RoboTwin, 90.0 vs 93.3 on Piper). What does the merged model do that the routed specialists can't? Unseen tasks, transfer across tasks, something else?
  4. If I read the Table 1 caption correctly, Spatial, Object and Goal all use the same PPO-LIBERO-130 checkpoint. If that checkpoint was trained on all of LIBERO, then you're mostly distilling one multi-task policy into another, not merging specialists. How many distinct teachers are there, and what was each one trained on? And what happens when two teachers disagree on similar states? That seems like the case that actually matters for consolidation, and none of your setups seem to have it.
  5. MixRL gets no teacher, while your method starts from PPO teachers that are already at 98.5%. If MixRL got the same total compute, including what went into training the teachers, would you still win?
  6. On Piper, the multi-task SFT student is at 70% and single-task SFT on the same demos reaches 93.3%. To me that looks like the multi-task SFT recipe just wasn't tuned well. Each OPD round also costs 50 real rollouts per task. Wouldn't collecting 50 more demos, or fixing the SFT recipe, get you most of the way there?
Reviewer_JwJROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

VLA-MOPD: ON-POLICY DISTILLATION FOR MULTI-TASK VLA POLICY CONSOLIDATION

The paper introduces VLA-MOPD, a multi-teacher on-policy distillation (OPD) framework for distilling multiple teachers/experts to a student model. The paper primarily introduces a method for distilling multiple experts to a single student VLA and focuses entirely on VLAs learnt on continuous actions (flow-matching action experts). The key idea is to distill multiple teachers via denser supervision. The denser supervision comes from supervising each step of the flow denoising PF-ODE via KL reg with the teacher. The student learns to copy the expert at every iterative denoising step. To preserve base capabilties, they also have a reference policy KL reg term for capability retention (Section 4.3).

The method is evaluated on both sim (Libero etc) and real robots.

Critical strengths
  1. Relevance of problem setting: Distilling multiple teachers into a single student is a very relevant setting in the present era and this work focuses on that.

  2. Denser supervision at each iterative action generation: Denser supervision wherever possible works better and this work provides denser supervision at two levels: (i) Denser supervision from teacher at each iterative generation step via a teacher on each flow denoising step. (ii) Denser supervision via soft KL in the OPD: as numerous works have shown, the denser supervision provided by soft distillation works way better than just output distillation.

  3. Evaluations both on sim and real hardware: The authors evaluate on LIBERO, RoboTwin 2.0, and three real-world manipulation tasks. The student reaches performance of the specialist teacher and attains substantial improvements over its initialization.

The paper is well written and well motivated but the weaknesses outweigh the strengths of this paper.

Critical weaknesses
  1. Limited Novelty: [W1] The method is incremental. There have been numerous work on MOPD in LLMs and it is expected to work in VLAs as well since there is no significant novel contribution in this work. The denser supervision of the flow denoising steps via a teacher has also been shown to work as the authors have noted (Diffusion OPD and Flow OPD). The work is basicallly combining existing mutli teacher OPD works with flow OPD by providing denser provision on iterative flow denoising steps. This is my biggest concern and in my opinion the biggest pitfall.

  2. Computational efficiency and teacher selection: [W2] Not completely clear how the teacher selection is done. If i understand correctly, for libero, a different teacher is trained on each of the Libero suite: Spatial, Object, Goal, Libero-10. Not clear how it is being done in the other evals. A better and more motivated work could have been to have a separate teacher for every atomic skills and to dynamically select the teacher via some end to end learned manner. Not sure if this is within the scope of this work, but that would count as a novel contribution. Libero as a benchmark is also too easy to see any benefits, which is evident since all the methods in Table 1 reach kind of similar higher numbers and its not visible if the results hold statistical significance. Holding a lot of teachers in the memory is also computationally prohibitive and expensive. Some discussion on this in the main body would be helpful.

3.Scalability comparison:[W3] any discussion about how the performance scales with the number of teachers would be useful for the community.

Overall, the main weakness is the limited novelty and contributions of this work. The work is very incremental and falls below the bar of ICLR acceptance.

Questions for authors

See weaknesses above. The main question would be to justify the signifance of this work given it is just a combination of the previous works. If the authors are able to justify the novelty, I am willing to reconsider my position.

Reviewer_TcYvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method for consolidating multiple taskor suite-specific VLAs into a shared multi-task policy through on-policy student–teacher distillation of a flow model. Specialist teachers supervise the student at its own visited robot observations and intermediate action-flow states. An optional reference-policy regularizer helps retain existing skills. Results show substantial improvements on the LIBERO and RoboTwin simulation benchmarks as well as real-world Piper robot tasks.

Critical strengths
  • Important problem enables generalized capabilities through a well-reasoned flow matching objective on on policy task-specific expert models.
  • Improvements in both sim and real settings; general performance approaches the individual experts
Critical weaknesses
  • Very limited experimental evaluation. Although 30 trials are used for the real-world experiments, it appears from the paper that only one actual model seed is evaluated; this makes it difficult to gauge the significance of the results.
  • The core conceptual insight for the paper is solid, but in the absence of more rigorous experimental results, the contribution seems incremental.
  • if these concerns were addressed I would be open to raising my square
Questions for authors
  • Is there any publicly available implementation of this algorithm?
  • Discuss / compare with Shreya Deshmukh et al., "Distillation for Efficient Multitask Manipulation Policies via Conditional Flow Matching." 2026.?
  • Could the authors add discussion and comparison with prior MOE methods  [1,2]?
  • Compare against MergeVLA.
  • How many training seeds were used for each experiment? Could you analyze the statistical significance of these experiment?

[1] Xiucheng Zhang et al., "Language-Conditioned Representations and Mixture-of-Experts Policy for Robust Multi-Task Robotic Manipulation." 2025.

[2] Zhiying Du et al., "HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies." 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.