acceptodds
Under review as a conference paper at ICLR 2027

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

Abstract

Multi-turn agents trained with reinforcement learning (RL) from verifiable rewards receive one scalar reward per trajectory, which gives little guidance for intermediate decisions. On-policy distillation (OPD) adds dense token-level supervision from a teacher; in agentic training, the teacher is conditioned on privileged information, such as task skills available only during training, so that a skill-free student can internalize them. This recipe rests on two assumptions, and we find that neither holds in agentic tasks. First, privileged information alone does not make a teacher reliable: a skill-conditioned policy that is not itself optimized for the task often fails to outperform the student it supervises. Second, the benefit of teacher supervision is stage-dependent: matching the teacher accelerates early learning but later conflicts with reward optimization and holds the student near the teacher. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over GRPO by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and it surpasses its own skill-conditioned teacher in every setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RL, OPD

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UVtyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes RetireOPD for training multi-turn agents with reinforcement learning and on-policy distillation. It first trains a skill-conditioned teacher using environment rewards, then trains a student without skill context using a joint GRPO+OPD objective. Teacher supervision is removed when teacher–student alignment stops improving and the student reaches a target fraction of the teacher’s success rate; training then continues with GRPO alone. Experiments on ALFWorld and WebShop with Qwen2.5 models from 1.5B to 7B report improvements over continued joint training and performance beyond the trained teacher.

Critical strengths
  1. Simple and practical training procedure. RetireOPD integrates naturally into a GRPO+OPD pipeline and uses two readily available signals to determine when to stop distillation. The retirement mechanism requires no additional learned controller, eliminates subsequent teacher forward passes, and produces a student that operates without privileged skill context at inference time.

  2. Coherent integration of teacher guidance and reward optimization. The method assigns complementary roles to distillation and reinforcement learning: teacher supervision supports early skill acquisition, while environment rewards remain the optimization signal throughout student training. This design allows the student to benefit from teacher guidance without requiring continued imitation once the retirement criteria are met.

Critical weaknesses
  1. The contribution and significance remain unclear. The method mainly adds a rule for stopping distillation to GRPO+OPD, and experiments cover only 1.5B–7B models. What practical benefit does adaptive retirement offer over a fixed-step switch? A concrete example comparing the student’s behavior on the same task before and after retirement would help explain its value.

  2. The large gain needs further substantiation. On 3B/ALFWorld, success rate rises from 82.8% to 93.8%. Could the authors provide the original WandB curves and configurations for both runs?

  3. Code–paper inconsistencies. Equation (6) weights trajectories equally, while the code weights tokens equally. The retirement indicator also uses a different denominator and threshold condition, producing different decisions for nonzero thresholds. These are minor concerns; convincing answers to points 1 and 2 could raise my recommendation to weak acceptance or even clear acceptance.

4: The OPD comparisons mainly cover basic OPD/OPSD and their GRPO combinations. Comparisons with more recent agentic OPD methods would help establish RetireOPD’s advantage over stronger baselines.

Questions for authors
  1. What practical advantage does adaptive retirement offer over a fixed-step switch to GRPO? Could the authors provide a concrete example showing how the student’s behavior improves after retirement, and discuss whether the findings extend beyond 1.5B–7B models?

  2. Could the authors provide the original WandB curves and run configurations for the 82.8% GRPO+OPD and 93.8% RetireOPD results on 3B/ALFWorld, confirming that both use the same evaluation protocol?

  3. Could the authors clarify the differences between the paper and code in gap averaging and the retirement criterion? Which definitions were used for the reported results, particularly the nonzero-threshold ablations?

4: Could the authors compare RetireOPD with recent agentic OPD methods, particularly those that adapt teacher guidance during training, using the same teacher and training budget?

Reviewer_qiLuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses two issues in agentic reinforcement learning, which are teachers may fail to utilize skill information, and student optimization may be constrained by continued distillation. The proposed RetireOPD first trains a teacher using reinforcement learning to utilize skill information , then trains a student without access to skills by combining reinforcement learning with on-policy distillation. Teacher supervision is adaptively terminated based on teacher–student alignment progress and relative task success rates, allowing the student to continue optimizing through reinforcement learning alone. Experiments using Qwen2.5 models on benchmarks show that RetireOPD outperforms most baselines in task success rate and accuracy.

Critical strengths

S1:Unlike fixed training step counts or preset distillation weight decay schedules, RetireOPD determines when to remove teacher supervision based on the progress of teacher-student alignment and the student’s task success rate relative to the teacher, allowing the distillation process to adapt to different training stages. S2:The student receives more intensive token-level supervision via the OPD, after the teacher is retired, the student continues to optimize task performance through reinforcement learning. This design ensures that the entire training process is not constantly constrained by the teacher’s behavior.

Critical weaknesses

W1:RetireOPD determines when to terminate distillation based on the progress of teacher-student alignment and students’ task success rates, however, a stagnation in alignment progress does not necessarily mean that teacher supervision has become ineffective. W2:The paper interprets teacher-student alignment stagnation as a signal that distillation may conflict with reinforcement learning objectives, but it has not its underlying mechanisms.

Questions for authors

Q1:Why does removing the competence condition delay retirement from step 60 to step 100? Q2:Can RetireOPD be effectively applied to other models, such as Llama, rather than being limited to Qwen2.5? Q3:Under what circumstances might teacher-student alignment lead to the premature or delayed termination of the process? Is this alignment stagnation alone sufficient to determine whether the benefits of teacher supervision have declined? Q4:Why does the stagnation in teacher-student alignment reflect a conflict between distillation objectives and reinforcement learning objectives?

1 comment
AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer qiLu,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

RetireOPD Authors

Reviewer_JK8MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents the method to improve policy distillation with reinforcement learning for language tasks. The proposed method consists of three stages: (1) teacher construction, (2) joint grpo-opd training, and (3) adaptive teacher retirement.

Critical strengths

The experimental results seem overall better than the used baselines on the used benchmarks ALFWorld and WebShop.

Critical weaknesses
  • Correctness of the paper: There are way too much false information. I just name a few here.
    • The paper’s core idea and methods are largely misleading. For example, the method claims to be based on GRPO, but the write-out in Method section is only the PPO for language models (Tuan et al. 2018) and never mentioned the newly proposed part in GRPO, which is the “normalization with std”. Also, the introduction states that the RLVR is the “framework…..” however, the stated framework is inherently what RL is doing in language tasks, it’s not related to whether it’s with verifiable rewards or not. The paper needs to be reread by authors to make sure their claims in the paper are correct.
    • More in detail:
      • Since the first sentence (L10), the sentence has little to do with whether it’s RL from verifiable rewards. Also, the whole thing is not conventionally defined as "one scalar reward per trajectory" — there are settings with more intermediate rewards. The sentence makes a false claim and needs revision.
      • The second sentence L13 is also a misleading claim. On-policy distillation is not what "adds dense token-level supervision". It’s a way to do distillation, and works that "adds dense token-level supervision" have been developed long.
      • The paper mixed the many RL in language modeling core concepts together everywhere. Such as the Figure 1 (middle), whether a framework using a "trajectory-level binary reward" or "continuous, token-level supervision" are the "environmental setting" and "reward design", they are orthogonal to whether one used GRPO or any other RL policy update method.
      • Again in the 1st sentence in INTRO (L56), the RLVR is not what defined a standard approach for language model agents on multi-turn tasks, they are completely decoupled concepts — VR is a part of reward design in what the reward function should be looked like, and multi-turn is a framework and can use any type of reward functions.
      • Again, OPD is not the way to complements this sparse reward with dense token-level supervision from a teacher on trajectories sampled by the student. OPD is only a "model distillation" way. Also, such dense token-level supervision has long existing, but the discussion of related work is very limited to recent 1 year (Zeng et al., 2026; Team, 2026; Xu et al., 2026a; Lu & Lab, 2025), ignoring the past developments share similarities.
      • Again in section 2.2 L155, “RLVR is the dominant approach for multi-turn agents” is a fairly weird claim, as one perhaps can only say “RL is one of the dominant approaches for multi-turn agents”. VR is only one variation of many possible reward designs in RL.
  • Contribution from the paper
    • This paper does not present a “new method” but a “pipeline” of combining long existing methods, change hyperparameters, and to experiment if this combination work on specific benchmarks. However, the paper seems not honestly reflect this fact, but frame all things into a “new method”.
  • Unclear method motivation:
    • What is the inspiration behind the designs of Eq7? The paper does not mention and even describes a little. Specifically, the paper needs to have a least a few sentences to describe why Eq7 and Eq8 are as the exact proposed forms.
  • Not inclusive:
    • The experiment directly draws conclusions of OPD and OPSD, but the paper never describes the difference among them.
Questions for authors
  • The paper claim prior work holds the assumption that privileged context alone makes a teacher reliable, however, I have no memory that people claim that, and the opposite should be a more general agreement. I also do not find such assumption in the referred papers (Zhang et al., 2026d; Zhou et al., 2026) in this work. Can the authors provide any published sources claim this?
  • Notations to clarify:
    • What is the used ref in Eq2, it’s not defined in the paper. The original teacher before T?
    • A notation breaks — the Δt\Delta_t in Eq4 is not used in elsewhere, how is it linked to your Eq3’s DKLD_{KL}.
  • Difference from classic works?
    • In section 3.2 Eq2, if claiming the thing is using GRPO, the paper needs to also formally write out the advantage function, especially the "std normalization" part. Otherwise, it’s the same as PPO for language model training in (Tuan et al., 2018).
    • The LOPDL_{OPD} is essential the same as teacher forcing with an assumed oracle?
    • The adaptive teacher retirement from classic works like MIXER (Ranzato et al., 2015). The “ablation study” L449’s “linear annealing from literal sounds very similar to MIXER, not sure if they are the same. Please clarify the difference between the core concept and algorithm parts, regardless of the used model type.

References:

  • Zhang et al., 2026d. Verify before you distill: Prompt-level teacher gating for on-policy distillation.
  • Zhou et al., 2026. Sage-opd: Selective agent-guided intervention for multi-turn on-policy distillation.
  • Tuan et al. 2018. Proximal Policy Optimization and its Dynamic Version for Sequence Generation
  • Ranzato et al. 2015. Sequence level training with recurrent neural networks.
1 comment
AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer JK8M,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

RetireOPD Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the reviewers and the Area Chair for their time. Below we summarize the main points of the discussion, the additional evidence provided during the rebuttal, and the revisions in the updated PDF.

Strengths Noted by the Reviewers

  • A simple and practical retirement mechanism that uses two readily available signals, needs no learned controller, and removes later teacher forward passes (UVty).

  • Adaptive retirement driven by alignment progress and relative competence, instead of a fixed step or a preset decay schedule (qiLu).

  • A coherent division of roles: teacher guidance for early skill acquisition and environment rewards as the optimization signal throughout (UVty, qiLu).

  • Consistent improvements over the baselines on ALFWorld and WebShop (JK8M, UVty).

Main Points and Our Responses

On the value of adaptive retirement (UVty, qiLu).

  • Fixed-step retirement. With Qwen2.5-3B, the best fixed retirement step differs across tasks (step 60 on ALFWorld, step 80 on WebShop), and RetireOPD selects these steps automatically, matching the best of the fixed steps tested (60/70/80) in both tasks (93.8% on ALFWorld; 85.8% WebShop). A single fixed step (e.g., step 70) reaches only 82.0% on ALFWorld. (Table 6 in A.8)

  • Student behavior after retirement. On 128 identical ALFWorld tasks, RetireOPD converts 29/37 failures to successes vs. 17/37 for GRPO+OPD, while retaining 90/91 successes (Table 5, A.7). Representative cases appear in Figs. 12–13.

  • Other model families. On Qwen3-4B-Instruct-2507, RetireOPD reaches 74.2% average ALFWorld success, compared with 66.4% for GRPO+OPD and 53.9% for GRPO. (Table 7 in A.8)

On comparisons with adaptive OPD methods (UVty). Under matched settings, RetireOPD outperforms SDAR/ATOD on ALFWorld (3B: 93.8% vs. 86.7%/87.5%; 7B: 95.3% vs. 93.8%/90.6%) and achieves higher WebShop Score and Acc at both scales.(Table 8 in A.8)

On the retirement criteria (qiLu, JK8M).

  • Why alignment stagnation indicates a conflict. A first-order analysis (A.1) shows that a non-decreasing distillation loss implies either a conflict between the reward and distillation gradients or a vanishing distillation signal. The RetireGRPO control supports the conflict reading: after stagnation, removing GRPO lets the teacher–student gap decrease further. Section 4.3 now explains how this analysis leads to the form of Eq. (7).

  • Why relative competence is needed. Without it, retirement happens earlier (step 50) and the final success rate drops from 93.8% to 89.0% (Table 2).

  • Corrections. The retirement-step labels in Table 2 are corrected, and validation-selected retirement is listed as a separate strategy.

On reproducibility and code–paper consistency (UVty). Eqs. (6)–(7) now match the code (token-weighted averaging; current-window normalization). All main experiments use δ = 0, for which both normalizations give the same retirement decisions, so the reported results are unaffected. Unsmoothed curves and run configs for 3B ALFWorld are available in the anonymous repository.

On presentation and positioning (JK8M). We clarified the distinctions between RL, verifiable rewards, and outcome-reward sparsity; expanded earlier related work (KD, policy distillation, DAgger, MIXER, scheduled sampling, PPO, Kickstarting); added GRPO, OPD, and OPSD preliminaries; and clarified that adaptive retirement is our key contribution.

Discussion Status

Reviewer UVty indicated that the concerns raised in the review were resolved. Reviewer JK8M's follow-up concerned the accuracy of several statements, the relation to earlier work, the form of Eq. (7), and a preliminary on OPD and OPSD; each is addressed in the revised PDF, as detailed in our reply to that comment.

Revisions in the Updated PDF (shown in blue)

  • Abstract and Introduction: revised statements on RL, outcome rewards and token-level supervision; Figure 1 (middle) now refers to RL w/ outcome reward. (JK8M)

  • Related Work: earlier work and algorithm-level positioning. (JK8M)

  • Preliminaries(new): Preliminaries on the setting, GRPO, OPD and OPSD. (JK8M)

  • Method: component origins in the method overview, Eq. (6)–(7) aligned with the implementation, and the rationale for Eq. (7). (JK8M, UVty, qiLu)

  • Experiments: OPD and OPSD baseline definitions, corrected Table 2, and the relation of Linear Annealing to MIXER. (JK8M, qiLu)

  • Appendix:

    • A.2: consistent notation in Algorithm 1. (UVty)
    • A.7: behavioral comparisons (Fig. 12–13) and quantitative outcome transitions (Table 5). (UVty)
    • A.8: fixed-step retirement comparison (Table 6), experiments on Qwen3 model (Table 7), and comparisons with adaptive OPD methods (Table 8). (UVty, qiLu)

We thank the reviewers again for their feedback, which has improved both the evidence and the clarity of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.