acceptodds
Under review as a conference paper at ICLR 2027

Beyond Policy Patches: Distilling RL Specialists into Vision-Language-Action Policies

Abstract

Vision-language-action (VLA) policies can handle noncritical stages of long-horizon manipulation yet fail during short precision-critical phases. Lightweight reinforcement learning (RL) specialists can address these failures without directly applying RL to the full VLA, but retain auxiliary controllers and switching mechanisms at deployment. We study how to transfer their capabilities into a single flow-matching VLA. Direct imitation of specialist rollouts can reproduce action jitter and leave the VLA unprepared for states encountered during its own execution. We introduce REPAIR, a value-guided distillation framework that combines offline initialization with on-policy refinement. At each critical-phase observation, REPAIR queries the corresponding specialist for multiple action candidates and weights them using its critic to construct a supervision distribution. A shared weighted flow-matching objective trains the VLA first on observations from specialist-assisted rollouts, then on observations collected through its own execution. Specialists provide supervision without taking control during on-policy collection and are removed at deployment. Across six real-robot tasks with 30 evaluation trials per task, REPAIR achieves 100% success on all six tasks, with lower mean end-effector jerk than specialist-assisted execution on every task.

Primary area:
reinforcement learning
Keywords:
VLA, Reinforcement Learning, Policy distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FoqpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REPAIR distills phase-specific RL specialists into a single vision-language-action policy using critic-weighted flow matching. It first trains on observations from specialist-assisted rollouts, then refines the policy on student-visited states using newly sampled specialist action candidates. Real-robot experiments show that the resulting VLA retains specialist-level task success with smoother execution, without auxiliary controllers at deployment.

Critical strengths
  • Critic-weighted action targets and student rollouts address supervision quality and deployment-state mismatch.
  • The real-world evaluation measures both task success and motion smoothness, with specialist training included in the adaptation rollout budget.
  • The ablations support the benefits of multiple action candidates, critic weighting, and supervision on student-visited states.
Critical weaknesses
  • The closest prior distillation approaches are not directly evaluated, particularly PLD, which already addresses deployment-state coverage through targeted data collection. The recorded-action cloning and data-collection ablations support REPAIR’s components but do not establish its advantage over a complete distribution-aware distillation pipeline under matched budgets.
  • Training relies on manually defined critical phases and experimenter-supplied phase labels. The paper identifies the critical stages for each task but does not clearly explain how their boundaries are determined during specialist training and student-only rollouts. The required task knowledge and annotation effort leave scalability to new tasks uncertain.
  • Section 6 claims that online critic updates prevent performance degradation, but provides no curves or numerical results for this comparison. Since critic scores determine the distillation targets, the benefit of updating the critic on student-visited states needs quantitative support.
Questions for authors
  1. How does REPAIR compare with PLD or RLDG under matched data and interaction budgets?
  2. How are phase boundaries identified during training and data collection, and how much manual effort is required?
  3. Can you provide quantitative results comparing online critic updates, a frozen critic, and uniform candidate weighting to support the claimed prevention of performance degradation?
4 comments
AuthorsOpenReview ↗

Distinctions from PLD/RLDG and complete-pipeline comparisons

Thank you for recognizing the value of the supervision design, real-world evaluation, and ablations. Your requests helped us strengthen the baseline comparisons and document phase annotation and critic updates more clearly.

W1 & Q1: Distinctions from PLD/RLDG and complete-pipeline comparisons

This comment clarifies how REPAIR differs from PLD and RLDG in state collection and action supervision, and addresses the request for complete-pipeline comparisons under matched data and interaction budgets.

Thank you for highlighting this comparison. In the revised manuscript, we have added complete-pipeline comparisons and a recorded-action SFT control on Drawer and Table Clearing, under matched data and interaction budgets (Appendix D.1, Lines 1033–1060; D.2, Lines 1082–1107). The key methodological differences are:

MethodTraining observationsAction supervision
RLDG [1]Successful rollouts from task-specific RL expertsSFT on recorded actions
PLD [2]Base-policy probing and hybrid residual-specialist recovery rolloutsSFT on recorded actions
REPAIRSpecialist-assisted observations, then states visited by the evolving studentRe-sampled specialist candidates with critic weighting

The new Tables 7–8 report:

MethodDrawer SR (%)Table Clearing SR (%)
RLDG, complete pipeline13.3366.67
PLD, complete pipeline43.3380.00
Recorded-action SFT on REPAIR data56.6786.67
REPAIR100.00100.00

Why a single recorded-action SFT control? RLDG and PLD both clone recorded actions. For the same VLA and training buffer, they share this SFT procedure. After comparing their complete pipelines in D.1, we apply this common objective to successful REPAIR-collected trajectories in D.2.

Under the shared SFT objective, REPAIR data outperform both baselines' own data, supporting richer task-relevant state coverage near critical phases. Full REPAIR further reaches 100% on both tasks, supporting the benefit of its action supervision. These new comparisons complement the original component ablations (Tables 2–3, Lines 444–454), which hold observations fixed when comparing action targets and hold action supervision fixed when comparing collected states.

References

[1] Xu, C., Li, Q., Luo, J., and Levine, S. RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning. arXiv:2412.09858, 2024.

[2] Xiao, W., et al. Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. ICLR, 2026.

AuthorsOpenReview ↗

Phase selection, boundary annotation, and manual effort

W2 & Q2: Phase selection, boundary annotation, and manual effort

This comment explains how critical phases and their boundaries are identified in practice, what the operator does during data collection, and how much additional annotation effort the procedure requires.

Thank you for requesting these details. We have added the selection and annotation workflow to Appendix C.1 (Lines 764–774).

Selection. We execute the base SFT policy and identify recurring failure stages. Perceived difficulty can be misleading: the policy may fail to place a toy inside a drawer while reliably stacking two bowls. We therefore select phases according to observed capability gaps and leave reliably executed phases under VLA control.

Boundary annotation. An on-site operator marks phase transitions with a keypress based on task progress. During specialist training and specialist-assisted collection, this phase signal records the phase label and triggers specialist takeover. During student-only collection, the phase label identifies which specialist provides candidate actions and which critic scores them for distillation. The student alone generates the actions executed by the robot; the specialist is used only to provide training supervision.

Manual effort. Human involvement is similar in nature to FTC [3], which also uses demonstrations and human interventions. Our operator already resets the environment and initiates rollouts; phase marking is integrated into that workflow, with labels stored alongside transitions, without a separate offline annotation pass or extra personnel. For a new task, the operator inspects the base policy's failures and follows the same procedure. The new Conclusion discussion (Lines 479–481) identifies automated phase annotation as future work.

Reference

[3] Qiao, G., et al. Focus-Then-Contact: Speeding Up Robotic Contact-Rich Task Learning with Affordance-Guided Real-World Residual Reinforcement Learning. ICML, 2026.

AuthorsOpenReview ↗

Quantitative evidence for online critic updates

W3 & Q3: Quantitative evidence for online critic updates

This comment addresses whether the critic needs to be updated during online distillation by comparing training curves and final-checkpoint success rates against frozen-critic and uniform-weighting controls.

Thank you for requesting quantitative evidence. We have added running-success curves and final-checkpoint evaluations (Figure 13 and Table 9, Lines 1143–1171). Each final checkpoint is evaluated over 30 trials on Dual-Arm Peg Insertion:

VariantFinal-checkpoint success rate
Online critic updates (REPAIR)100%
Frozen critic83.33%
Uniform candidate weighting93.33%

Figure 13 shows that REPAIR reaches and sustains a 100% running success rate during training, whereas the frozen critic improves but continues to exhibit failures despite extended training. The final-checkpoint results show that specialist candidates provide useful supervision even with uniform weights. Uniform weighting's advantage over the frozen critic suggests that outdated value estimates can impair supervision; REPAIR's higher success supports updating the critic as the student visits new states.

Dear Reviewer Foqp,

Thank you again for your thoughtful review. As the discussion deadline approaches, we would appreciate your feedback on whether our revisions and additional evidence address your concerns. If you feel that we have resolved your concerns, we would sincerely appreciate your consideration of a higher score.

Thank you for your time and consideration.

Best regards,

Submission 38706 Authors

Reviewer_tVfFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to address the problem where a VLA is finetuned with RL to perform precision-critical tasks. Specifically, the paper proposes REPAIR, a method that first trains task-specific RL heads, and distill them into the VLA through a weighted flow-matching objective. The paper then illustrates that this VLA achieves similar if not better success rates than compared methods, while maintaining low jerk over the trials.

Critical strengths

None.

Critical weaknesses
  1. The writing at times are hard to understand
  • Figure 1 has limited description about (a.3) and (b).
  • In section 3, lines 146-147, is the specialist MLP trained on the specific distribution induced by the original VLA? In other words, we first run the VLA until last phase and use RL specialization; or do we simply run the whole trajectory end-to-end.
  • On line 245, why do we call this a value-tilted expert distribution when this is similar to the advantage weights?
  • It would be nice to write out LFM\mathcal{L}_\text{FM} explicitly.
  1. There is no strong motivation as to why we want to distill the RL agents but rather focus on minimizing the delays between the switch (if such delays exist).
  2. Identifying what the critical phases are and determining which transitions belong to which phase is expensive and does not seem to be scalable without some sort of LLMs. The paper should acknowledge this limitation.
Questions for authors
  1. How does this method scale or whether this method continually learns
  • In practice, how are the "phases" (or goals) decided?
  • Is there a fundamental limit as to how much the VLA can distill the RL heads, whether distilling NN tasks simultaneously vs one by one?
8 comments
AuthorsOpenReview ↗

Presentation and Formulation

This comment addresses W1 by clarifying Figure 1, the specialist's training scope, the value-tilted weighting, and the explicit flow-matching loss.

Presentation and Formulation (W1)

W1.1. What do Figure 1(a.3) and (b) show?

Thank you for suggesting a fuller explanation. We have expanded the Figure 1 caption (Lines 69–73). Panel (a.3) illustrates how cloning recorded specialist actions can reproduce jitter and leave gaps in coverage of student-visited states. Panel (b) summarizes the original supervision procedure (Section 4.2, Lines 231–240): critic-weighted specialist candidates supervise the VLA first at offline observations and then at student-only rollout states. The student retains control throughout online collection.

W1.2. Is the specialist trained only after VLA execution, or over the entire task?

The comparison in the original Section 3 includes both control scopes (Lines 143–147). In critical-phase takeover, the VLA executes the preceding stages and the specialist handles the final assembly phase, starting from VLA-induced states. In full-task takeover, the specialist controls the entire episode. REPAIR uses phase-specific specialists; full-task takeover is an analysis baseline. The original Appendix B.2 (Lines 698–701) describes phase-local training and human overrides.

W1.3. Why call the target “value-tilted” when the weights resemble advantage weights?

Energy-weighted flow matching [1] similarly formulates policy improvement through exponential Q-weighting of a base distribution; “value-tilted” follows this distributional perspective. Thank you for highlighting the connection to advantage weighting. We agree that our normalized weights in the original Eq. (3) (Lines 255–260) have an advantage-like form. For a fixed candidate set, let sq=max⁡(σq,ϵ)s_q=\max(\sigma_q,\epsilon). Then

=\frac{\exp[q_i/(\tau s_q)]}{\sum_j\exp[q_j/(\tau s_q)]}.$$ The candidate mean cancels under normalization, leaving exponential Q-weighting with effective temperature $\tau s_q$. This is consistent with the reviewer's interpretation; “value-tilted” emphasizes the resulting reweighting of the specialist's action distribution. ### W1.4. Please define the flow-matching loss explicitly. Thank you for this suggestion. We have **added the explicit definition at its first occurrence in Section 4.3 (Lines 262–266, Eq. (4))**. **References** [1] Zhang, S., Zhang, W., and Gu, Q. *Energy-Weighted Flow Matching for Offline Reinforcement Learning*. ICLR, 2025.
AuthorsOpenReview ↗

Distillation Motivation and Scalability (W2–W3)

This comment addresses W2 and W3 by explaining the motivation for distillation beyond controller switching and discussing phase annotation and scalability.

W2. Why distill the specialists instead of reducing switching delays?

Thank you for asking us to clarify the motivation. Our manuscript does not discuss switching delays. The concern is action continuity and execution quality: discontinuities at controller handoffs and jitter during specialist execution, rather than the time required to switch controllers. The original Section 3 reports jitter during specialist execution (Figure 2(c), Lines 150–157). Relatedly, ZPRL [1] reports oscillatory behavior in an action-space residual baseline and smoother execution through latent-space steering. Our existing cloning experiment removes controller switching but still reproduces jitter: fine-tuning on ten specialist trajectories reduces success from 100% under specialist takeover to 60% (18/30 trials; Section 3, Lines 163–179).

These results motivate improving the action targets and supervising student-visited states. The original ablations in Tables 2–3 (Lines 444–454) examine these two components. Even eliminating controller switching does not by itself resolve the transfer problem.

W3. Do manual phase identification and annotation limit scalability?

Thank you for raising this scalability concern. Scaling real-world RL across many tasks and robots remains an open research challenge, whether or not phase annotations are used. REPAIR studies sample-efficient VLA improvement within a human-assisted real-world RL setting, where an on-site operator already manages environment resets and rollout collection. In this setting, phase marking is integrated into the existing collection workflow: the operator marks phase transitions with a keypress, and the labels are stored alongside the collected transitions, without a separate offline annotation pass or additional personnel. We have clarified this workflow in Appendix C.1 (Lines 764–774). These annotations support specialist training and the selection of supervision during distillation. After distillation, the deployed VLA executes the complete task without phase annotations or auxiliary specialists. REPAIR thus transfers capabilities acquired through structured, human-assisted training into a single policy that no longer requires phase-based coordination at deployment.

Manual phase identification and boundary annotation nevertheless remain task-dependent training costs, and extending the approach to large collections of new tasks calls for reducing this human involvement. Our experiments demonstrate effective specialist-to-VLA transfer under limited real-world interaction budgets within this setting, while automated phase identification and annotation remain important directions for broader scaling. We have added a discussion of broader scaling and automated phase annotation to the Conclusion (Lines 479–481).

References

[1] Yu, D., Lei, K., Jiang, Z., Pan, J., and Xu, H. Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning. arXiv:2605.19919, 2026.

AuthorsOpenReview ↗

Phase Selection and Multi-Task Distillation (Q1)

This comment addresses Q1 through the phase-selection procedure, an additional sequential multi-task distillation experiment, and discussion of simultaneous distillation and capacity limits.

Q1.1. In practice, how are the phases or goals decided?

Thank you for asking us to clarify the selection procedure. We have added it to Appendix C.1 (Lines 764–767). We identify the phases requiring repair by executing the base SFT policy and examining its recurring failures. The selected phases correspond to task-specific subgoals where the base policy is unreliable. For example, in Drawer, the two selected phases are placement of the toy and final drawer closure, with the corresponding goals of depositing the toy inside the drawer and fully closing it. Stages that the base policy executes reliably remain under VLA control during specialist training. During collection, an operator marks phase transitions with a keypress based on observed task progress, and the labels are stored alongside the collected transitions.

Q1.2. Can specialists be distilled simultaneously across tasks or sequentially, and is there a capacity limit?

Thank you for asking about multi-task distillation and capacity. REPAIR primarily studies consolidating specialists for multiple critical phases of a single long-horizon task into one VLA. In principle, the same supervision objective could be extended to joint distillation across tasks, using observations from different tasks and their corresponding phase specialists and critics to train a shared VLA. In our current experimental setup, implementing joint on-policy distillation with parallel rollout collection would require simultaneous access to multiple robots and operators, increasing hardware and personnel costs. We have therefore not yet evaluated this setting.

To illustrate the feasibility of continual learning through sequential distillation, we conducted an additional example on Drawer and Table Clearing. Starting from an SFT policy shared by these two tasks, we first distilled the Drawer specialists, followed by the Table Clearing specialists. Following the replay-based continual-learning practice of revisiting earlier experience while learning subsequent tasks [1, 2], we retained all Drawer data and the corresponding RL heads during the second stage, using uniform replay sampling to keep earlier-task supervision available throughout subsequent training. The shared SFT policy initially achieved 0% success on Drawer and 13.33% on Table Clearing. After sequential distillation, the same final shared VLA achieved 100% success on both tasks (30/30 trials per task). This example supports the feasibility of incrementally incorporating specialist capabilities into a shared VLA with replay while retaining the ability to perform the earlier task.

Regarding capacity, we expect the limit to depend not only on the number of RL heads but also on student capacity, task diversity and interference, and training-data coverage. Systematically characterizing the distillation capacity limit as the number and diversity of specialists increase remains an open question.

References

[1] Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., and Wayne, G. Experience Replay for Continual Learning. NeurIPS, 2019.

[2] Chaudhry, A., et al. On Tiny Episodic Memories in Continual Learning. arXiv:1902.10486, 2019.

Reviewer_Wz4tOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a first-RL-then-distillation method to improve VLAs with online experience. The paper uses RLT as the RL specialist method, adding a critic-phase-specific module with base policy frozen. During distillation, offline and online experiences are both used to avoid OOD state visitations; besides, supervision comes from value-guided weighting of expertise action candidates, facillitating a broader and more precise supervision. The experiments achieve sample-efficient adaptation of the base policy, while reducing jerk behavior and execution time.

Critical strengths
  • Very timely contribution to the domain of sample-efficient online improvement of VLAs.
  • Simple and elegant idea. The first-RL-then-distillation method avoids unstability of RL methods that directly finetune the large base policy.
  • Strong experiment results on long-horizon manipulation tasks. Only 50 online rollouts are needed to reach 100% success rates while reducing jerk behavior and execution time compared to RL alone. This makes the method quite appealing.
  • Ablation studies clearly disentangle the contributions of each component of the method.
Critical weaknesses

I only see minor weaknesses:

  • pi0.7 is missing from the related work, which also distills RL specialist into the base policy.
  • In my view, the distillation process can be incorporated earlier during the RLT training process, which can further simplify the whole process into two stages instead of three.
Questions for authors
  • On the task Drawer, the base policy has success rate zero. Then how does RLT improve upon it? (as there are no success signals)
2 comments

We sincerely thank the reviewer for the encouraging assessment of REPAIR's timely contribution, simple design, sample efficiency, and component ablations. We appreciate the thoughtful suggestions for improving the paper.

W1. How does REPAIR relate to π0.7, which also transfers RL specialist capabilities into a generalist?

Thank you for pointing out this closely related work. We have added π0.7 and clarified the distinction in Section 2 (Lines 129–135). π0.7 [1] incorporates RL-specialist trajectories into a broader training mixture and uses episode metadata to distinguish behavior quality. REPAIR instead queries specialists for multiple action candidates and weights them with their critics, at both offline observations and student-visited states.

W2. Can distillation begin during specialist training to simplify the pipeline?

Thank you for suggesting earlier distillation. In an additional pilot reported here, we began distillation on Dual-Arm Peg Insertion after 500 RLT training steps while continuing specialist training. The student showed no success-rate improvement during the early concurrent stage; improvement began only after the specialist converged. The stages can therefore overlap, but this result supports establishing a competent specialist before distillation.

Q1. How can RLT learn on Drawer when the base policy has zero task success?

Thank you for raising this question. The original RLT training protocol uses phase-local success rewards and human interventions, with intervention actions also providing behavioral supervision (Lines 712–718). Zero end-to-end success therefore does not eliminate the local training signal. For example, an operator can complete a failed drawer-closing attempt, providing a successful segment from which the specialist learns.

References

[1] Physical Intelligence et al. π0.7: A Steerable Generalist Robotic Foundation Model with Emergent Capabilities. arXiv:2604.15483, 2026.

Thank you. I have no more concerns.

Comments on the paper

2 comments

Dear Area Chair and reviewers,

Thank you and the reviewers for the careful evaluation and constructive feedback. We have revised the manuscript, with additions and substantive changes highlighted in blue. We summarize the main updates below; detailed evidence and responses are provided in our individual replies.

  1. Comparisons with RLDG and PLD. We added complete-pipeline comparisons under matched data and interaction budgets. The results supports the value of the collected experience and the further benefit of REPAIR's action supervision (Appendix D.1–D.2, Tables 7–8).

  2. Quantitative evidence for critic updates. We added training curves and 30-trial final-checkpoint evaluations on Dual-Arm Peg Insertion. The results supports adaptation of the critic to student-visited states (Appendix E, Figure 13 and Table 9).

  3. Phase selection, annotation, and scalability. We documented how recurring base-policy failures determine phases requiring repair and how keypress-based annotation fits into existing collection. Deployment requires neither phase annotations nor auxiliary specialists. We also expanded the discussion of automated annotation and parallel specialist training (Appendix C.1, Lines 764–774; Conclusion, Lines 479–481).

  4. Sequential multi-task distillation. In an additional experiment reported in the rebuttal, we sequentially distilled Drawer and Table Clearing specialists into a shared VLA with replay. The final policy achieves 100% success on both tasks (30/30 trials each), demonstrating the feasibility of incorporating new specialist capabilities while retaining the earlier task.

  5. Distillation motivation. We clarified that the motivation is action continuity and execution quality, rather than switching delays, drawing on the original Section 3 (Lines 150–179).

  6. Presentation and related work. We expanded the Figure 1 caption, defined the flow-matching loss at its first occurrence, and added the discussion of π0.7 (Lines 69–73, 262–266, and 129–135, respectively). Our replies also clarify the original specialist-control scopes.

We hope these revisions and additional results address the reviewers' concerns and support a favorable assessment of REPAIR. We welcome further questions and discussion.

Area_Chair_8qtsOpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.