- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
The paper presents RE-0, a framework for improving embodied Code-as-Policy agents using local corrections from imperfect teachers. It diagnoses candidate failure boundaries, requests bounded code repairs, and compares teacher interventions with student behavior through paired rollouts from the same simulator snapshot. Interventions are accepted when the lower confidence bound on their estimated benefit is positive, after which control returns to the student. RE-OPD distills this verified assistance into a standalone policy through cold-start training on teacher-generated trajectories and subsequent refinement using student execution histories. The paper also derives a conditional improvement bound that accounts for verification and projection errors.
The evaluation covers six RoboSuite tasks and includes baseline comparisons, ablations of data selection and distillation objectives, and cross-task transfer experiments. Simulation performance is evaluated using 50 seeds per task. The paper additionally reports zero-shot deployment of simulation-trained policies on a dual-arm robot across five tasks, with 20 trials per task. The deployed policies operate without teacher calls, while counterfactual verification is performed during simulation-based data collection.
Critical strengths
S1. Novel formulation of supervision from imperfect teachers.
The paper formulates teacher supervision as local counterfactual credit. Rather than assuming that the teacher is globally superior, the framework evaluates whether a bounded correction improves the current student's continuation from the same saved state. This shifts the supervision criterion from teacher success alone to the teacher's marginal contribution, providing a mechanism for exploiting locally useful corrections while rejecting unreliable guidance.
S2. An explicit theoretical connection between verified assistance and student improvement.
The analysis provides a conditional improvement bound that separates verified intervention gains from verification and projection errors. This distinguishes whether a teacher correction is beneficial from whether its benefit survives distillation into the student. Within the assumed snapshot based setting, the bound identifies the conditions under which temporary teacher assistance can translate into improved standalone performance, providing a theoretical basis for the proposed learning framework.
Critical weaknesses
W1. The experiments do not convincingly establish the method's advantage on demanding tasks.
The evaluation covers six RoboSuite task families, without an additional challenging benchmark or a systematic study of increasing task horizon and compositional complexity. Although some tasks are difficult for the base policy, the comparative evidence remains limited. Table 1 compares the student against a single shot teacher without the execution feedback and repair capabilities used during training data collection, while RFT/STaR is assessed through proxy ablations rather than complete implementations. Moreover, the experiments do not clearly establish the incremental benefit of the second stage, RE-OPD-refine, beyond the gains from the initial cold-start stage. These limitations leave unclear whether the full framework offers a meaningful advantage over simpler alternatives given comparable teacher assistance and computational budgets. This substantially weakens the empirical justification for the method's added complexity.
W2. The core mechanism relies on simulator state restoration, leaving its practical applicability insufficiently established.
Counterfactual verification requires repeatedly restoring the same saved intermediate state, not merely restarting a task. The procedure therefore cannot operate directly on physical robots without this capability. This is a structural limitation, not simply an omitted hardware experiment. Although the distilled student can be deployed, the improvement process depends on simulation followed by transfer to reality, and deployment without snapshots falls outside the theoretical guarantee. The hardware results demonstrate some transfer but do not systematically establish whether simulation verified corrections remain beneficial under perception and dynamics discrepancies. Consequently, the practical value of the improvement mechanism depends on simulator fidelity and transfer robustness that the current evaluation does not adequately characterize.
Questions for authors
Q1. Evidence on more demanding tasks
Are additional results available on a more challenging benchmark or on tasks with longer execution horizons and greater compositional complexity?
Q2. Comparisons with stronger baselines
Are direct comparisons available against more competitive baseline configurations, such as fully implemented RFT/STaR or a RL baseline?
Q3. Practical advantages of distillation over direct teacher use
What measurable advantages does the distilled student offer over directly using the same teacher with execution feedback and repair capabilities?