- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes RetireOPD for training multi-turn agents with reinforcement learning and on-policy distillation. It first trains a skill-conditioned teacher using environment rewards, then trains a student without skill context using a joint GRPO+OPD objective. Teacher supervision is removed when teacher–student alignment stops improving and the student reaches a target fraction of the teacher’s success rate; training then continues with GRPO alone. Experiments on ALFWorld and WebShop with Qwen2.5 models from 1.5B to 7B report improvements over continued joint training and performance beyond the trained teacher.
Critical strengths
-
Simple and practical training procedure. RetireOPD integrates naturally into a GRPO+OPD pipeline and uses two readily available signals to determine when to stop distillation. The retirement mechanism requires no additional learned controller, eliminates subsequent teacher forward passes, and produces a student that operates without privileged skill context at inference time.
-
Coherent integration of teacher guidance and reward optimization. The method assigns complementary roles to distillation and reinforcement learning: teacher supervision supports early skill acquisition, while environment rewards remain the optimization signal throughout student training. This design allows the student to benefit from teacher guidance without requiring continued imitation once the retirement criteria are met.
Critical weaknesses
-
The contribution and significance remain unclear. The method mainly adds a rule for stopping distillation to GRPO+OPD, and experiments cover only 1.5B–7B models. What practical benefit does adaptive retirement offer over a fixed-step switch? A concrete example comparing the student’s behavior on the same task before and after retirement would help explain its value.
-
The large gain needs further substantiation. On 3B/ALFWorld, success rate rises from 82.8% to 93.8%. Could the authors provide the original WandB curves and configurations for both runs?
-
Code–paper inconsistencies. Equation (6) weights trajectories equally, while the code weights tokens equally. The retirement indicator also uses a different denominator and threshold condition, producing different decisions for nonzero thresholds. These are minor concerns; convincing answers to points 1 and 2 could raise my recommendation to weak acceptance or even clear acceptance.
4: The OPD comparisons mainly cover basic OPD/OPSD and their GRPO combinations. Comparisons with more recent agentic OPD methods would help establish RetireOPD’s advantage over stronger baselines.
Questions for authors
-
What practical advantage does adaptive retirement offer over a fixed-step switch to GRPO? Could the authors provide a concrete example showing how the student’s behavior improves after retirement, and discuss whether the findings extend beyond 1.5B–7B models?
-
Could the authors provide the original WandB curves and run configurations for the 82.8% GRPO+OPD and 93.8% RetireOPD results on 3B/ALFWorld, confirming that both use the same evaluation protocol?
-
Could the authors clarify the differences between the paper and code in gap averaging and the retirement criterion? Which definitions were used for the reported results, particularly the nonzero-threshold ablations?
4: Could the authors compare RetireOPD with recent agentic OPD methods, particularly those that adapt teacher guidance during training, using the same teacher and training budget?
Kind Reminder
Dear Reviewer qiLu,
Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.
If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.
Sincerely,
RetireOPD Authors