acceptodds
Under review as a conference paper at ICLR 2027

RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation

Abstract

Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher's complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student's own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student—so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student's per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied agents, LLM-based agents, knowledge distillation, robotics

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_2DehOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper presents RE-0, a framework for improving embodied Code-as-Policy agents using local corrections from imperfect teachers. It diagnoses candidate failure boundaries, requests bounded code repairs, and compares teacher interventions with student behavior through paired rollouts from the same simulator snapshot. Interventions are accepted when the lower confidence bound on their estimated benefit is positive, after which control returns to the student. RE-OPD distills this verified assistance into a standalone policy through cold-start training on teacher-generated trajectories and subsequent refinement using student execution histories. The paper also derives a conditional improvement bound that accounts for verification and projection errors.

The evaluation covers six RoboSuite tasks and includes baseline comparisons, ablations of data selection and distillation objectives, and cross-task transfer experiments. Simulation performance is evaluated using 50 seeds per task. The paper additionally reports zero-shot deployment of simulation-trained policies on a dual-arm robot across five tasks, with 20 trials per task. The deployed policies operate without teacher calls, while counterfactual verification is performed during simulation-based data collection.

Critical strengths

S1. Novel formulation of supervision from imperfect teachers.

The paper formulates teacher supervision as local counterfactual credit. Rather than assuming that the teacher is globally superior, the framework evaluates whether a bounded correction improves the current student's continuation from the same saved state. This shifts the supervision criterion from teacher success alone to the teacher's marginal contribution, providing a mechanism for exploiting locally useful corrections while rejecting unreliable guidance.

S2. An explicit theoretical connection between verified assistance and student improvement.

The analysis provides a conditional improvement bound that separates verified intervention gains from verification and projection errors. This distinguishes whether a teacher correction is beneficial from whether its benefit survives distillation into the student. Within the assumed snapshot based setting, the bound identifies the conditions under which temporary teacher assistance can translate into improved standalone performance, providing a theoretical basis for the proposed learning framework.

Critical weaknesses

W1. The experiments do not convincingly establish the method's advantage on demanding tasks.

The evaluation covers six RoboSuite task families, without an additional challenging benchmark or a systematic study of increasing task horizon and compositional complexity. Although some tasks are difficult for the base policy, the comparative evidence remains limited. Table 1 compares the student against a single shot teacher without the execution feedback and repair capabilities used during training data collection, while RFT/STaR is assessed through proxy ablations rather than complete implementations. Moreover, the experiments do not clearly establish the incremental benefit of the second stage, RE-OPD-refine, beyond the gains from the initial cold-start stage. These limitations leave unclear whether the full framework offers a meaningful advantage over simpler alternatives given comparable teacher assistance and computational budgets. This substantially weakens the empirical justification for the method's added complexity.

W2. The core mechanism relies on simulator state restoration, leaving its practical applicability insufficiently established.

Counterfactual verification requires repeatedly restoring the same saved intermediate state, not merely restarting a task. The procedure therefore cannot operate directly on physical robots without this capability. This is a structural limitation, not simply an omitted hardware experiment. Although the distilled student can be deployed, the improvement process depends on simulation followed by transfer to reality, and deployment without snapshots falls outside the theoretical guarantee. The hardware results demonstrate some transfer but do not systematically establish whether simulation verified corrections remain beneficial under perception and dynamics discrepancies. Consequently, the practical value of the improvement mechanism depends on simulator fidelity and transfer robustness that the current evaluation does not adequately characterize.

Questions for authors

Q1. Evidence on more demanding tasks

Are additional results available on a more challenging benchmark or on tasks with longer execution horizons and greater compositional complexity?

Q2. Comparisons with stronger baselines

Are direct comparisons available against more competitive baseline configurations, such as fully implemented RFT/STaR or a RL baseline?

Q3. Practical advantages of distillation over direct teacher use

What measurable advantages does the distilled student offer over directly using the same teacher with execution feedback and repair capabilities?

Reviewer_W151OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RE-0, a framework for improving Code-as-Policy agents using local corrections from a potentially unreliable teacher. It identifies candidate failure boundaries, requests bounded code repairs, and evaluates their benefit through paired counterfactual rollouts from the same simulator snapshot, returning control to the current policy after each intervention. Repairs are accepted only when their estimated advantage has a positive lower confidence bound. The accompanying RE-OPD procedure distills verified supervision into a standalone student through teacher-seat cold-start training and subsequent student-seat refinement. The paper derives a conditional per-round performance bound that accounts for verification and projection errors.

Critical strengths
  • Principled local verification: Paired counterfactual rollouts assess whether teacher corrections benefit the current student.
  • Strong cold-start results: Large gains on three challenging tasks are supported by multiple training seeds.
  • Informative ablations: Experiments distinguish data selection from weighting and transparently report failures.
Critical weaknesses
  • Unclear evidence for student-on-policy refinement. Appendix D states that the successful restack refinement involved teacher-seat recollection, leaving the contribution of student-generated failure trajectories unclear. Since lift refinement regresses, a clearer breakdown of data provenance and a strictly student-on-policy experiment are needed to substantiate the recursive improvement claim.

  • Insufficient specification of the operational verifier. The theory verifies the expected advantage of a teacher distribution, whereas the implementation describes repeated evaluation of a selected candidate. The paper should clarify this distinction and specify the candidate count, selection procedure, e-process, and error-budget allocation needed to establish the claimed simultaneous coverage.

  • Limited connection between the theoretical guarantee and standalone performance. The guarantee applies to the post-rewind augmented process and depends on an unmeasured projection error. It remains unclear how the implemented training objective controls this error or how the bound translates to standalone execution from the original task input.

Questions for authors
  1. Could you provide a data-provenance breakdown for the restack checkpoint improving from 31/50 to 40/50, including the collection policy, number of training examples from each source, and whether a student-seat-only result is available?

  2. Could you provide pseudocode for the deployed verifier, showing when teacher candidates are sampled or selected, how the confidence bound is updated, and how the total error budget is allocated?

  3. Which reported experimental performance measure corresponds to J(pi) in Theorem 1? What additional assumptions or error terms, if any, are needed to apply the bound to standalone evaluation from the original task input?

Reviewer_Y9wiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RE-0, a framework for improving embodied Code-as-Policy agents with a teacher that is not reliable on every state. A diagnostic agent locates the earliest harmful boundary in a failed student trajectory, and the teacher writes a bounded code patch at that point. Each patch is checked by paired rollouts from the same simulation snapshot against a student resample, and is admitted only if the lower confidence bound of the advantage is positive. RE-OPD then distills the admitted patches into a standalone student, and the process repeats. The paper also gives a per-round improvement bound that does not assume global teacher dominance. Experiments on six robosuite tasks with a Qwen3.8-27B student show large gains from a small number of verified rows, along with zero-shot deployment on a real dual-arm robot.

Critical strengths
  1. Assigning teacher credit locally on histories the student actually visits is well motivated for long-horizon code policies, and it removes the assumption that the teacher is better everywhere.
  2. Same-snapshot paired verification is a sound design that cancels common noise and can reject harmful patches.
  3. The method is highly data efficient, reaching strong performance with only a handful of verified samples.
Critical weaknesses
  1. The relation to recent work needs more discussion. Methods such as SCoRe (arXiv 2509.14257) and OVCSD (arXiv 2607.27937) also correct the earliest student error with a teacher, restore states from snapshots, and verify interventions before training. Clarifying how RE-0 differs from these methods would help readers better assess its contribution.
  2. The proposed distillation objective is not supported by the experiments. The authors acknowledge that credit weighting does not outperform uniform weighting, and with a black-box teacher RE-OPD reduces to weighted SFT. The results support verified data selection, but they do not test the proposed objective.
  3. The theoretical guarantee does not apply to the deployed student. Theorem 1 holds for a two-pass process with snapshot rewinds, and the projection error term is never estimated, so the bound offers little assurance about the final policy.
  4. The applicability is narrow. The method depends on resettable simulators with shared randomness and on task-specific diagnostic signals, and the claim of generalization to new robots and scenes goes beyond what the transfer results show.
Questions for authors
  1. How does RE-0 differ from SCoRe and OVCSD, and can they be included as baselines?
  2. If the weighting brings no benefit, what does RE-OPD contribute beyond SFT on verified data?
  3. How does RE-0 compare with rejection sampling or RFT under the same environment budget?
  4. How would the method extend to settings without snapshot restoration, such as real-world training?
Reviewer_2dZnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper discusses how to improve LLM-based Code-as-Policy agents using a stronger, but not necessarily reliable teacher. Their protocol consists of three steps: a diagnostic agent that finds the earliest harmful code boundary in failed student trajectories, followed by the teacher that proposes a bounded repair at that point, and then a simulator is rewound to a saved snapshot to help compare the repair against a student resample under shared randomness.

Critical strengths
  • The motivation for this paper is clear since globally strong but locally unreliable are an established issue for OPD.
  • Paired comparison using the same snapshots seems like a reasonable way to assign credit, especially with the verifier that can reject harmful patches.
Critical weaknesses
  • The title and theory are centered around credit-weighted on-policy distillation. But the teacher is black-box, so every experiment uses the trajectory-level hard estimator. The empirical support for RE-OPD as a distillation method seems very weak. The paper’s own conclusion, that the corpus rather than the objective does the work, undermines the framing of the contribution.
  • All main gains come from cold-start SFT on completion-filtered teacher trajectories, while the on-policy refinement stage has just one positive single-seed result and one regression. Its fallback claim, that LCB-verified selection is the key mechanism, is not separated from simple success filtering. The deciding ablation uses a different, weaker protocol with three training rows and three seeds, outcome filtering beats LCB on one seed, and the unverified OPD, RFT, and compute-matched baselines promised in §4.1 are missing.
Questions for authors
  • Can you run the Table 2 selection-rule comparison (LCB vs. outcome/success filtering vs. sign-only) under the Table 1 protocol, with at least three training seeds per task? If LCB clearly outperforms outcome filtering where effect sizes are large, my main concern about the selection claim would be resolved. If they match, the gate’s added value over standard RFT is unclear.
  • Baselines for unverified OPD/DAgger and RFT/STaR are described in the setup but not reported. Do these runs exist? Comparable numbers on the main protocol, ideally at matched rollout and teacher-call budget, are the most important missing evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.