acceptodds
Under review as a conference paper at ICLR 2027

Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Long-Horizon Tasks

Abstract

Reinforcement learning is now the mainstream paradigm for training LLMs on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest. We show that this is lossy: in GRPO training of LLM agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved, and the union of problems solved across checkpoints far exceeds any single one. The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation (OPD) can mine to recover the lost coverage. Mining this pool raises two challenges: (i) which checkpoint to distill from, as no checkpoint dominates the others and complementarity to the student is problem-specific; and (ii) where to trust it, as even a well-chosen teacher is only locally superior, so indiscriminate distillation overwrites behavior the student has already mastered. We propose Agentic-OPD, which treats the best-validation checkpoint as the student and all saved checkpoints as frozen teacher candidates, using the student's evolving capability gaps to guide both teacher selection and supervision filtering. The HCN policy (Highest Coverage Next) dynamically selects teachers by weighted coverage of the student's unsolved problems, while the LAD module (Locality-Aware Distillation) confines transfer to failed student rollouts on problems where the teacher performs better. Across five agentic task domains and 24 benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to 7.4 points, with a 4B student surpassing the latest frontier models GPT-5.6 and DeepSeek-V4.1 on deep search. Agentic-OPD recasts a finished RL run as a reusable pool of policies and establishes post-RL capability recovery as a stage orthogonal to improvements in the RL algorithm itself.

Primary area:
reinforcement learning
Keywords:
LLM agents, agentic reinforcement learning, on-policy distillation, long-horizon tasks, checkpoint reuse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e3yrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agentic-OPD to recover capabilities lost during agentic RL training. Historical checkpoints serve as frozen teachers, while the best-validation checkpoint initializes the student. HCN selects teachers based on their coverage of the student’s remaining failures, and LAD filters supervision to problems where the teacher performs better and to failed student rollouts.

Critical strengths

The evaluation is broad: experiments cover five agentic domains, multiple model sizes, checkpoint-reuse baselines, and component ablations, providing evidence of the method’s applicability.

Critical weaknesses
  1. The computational trade-off is unclear. The method requires teacher-pool evaluation, repeated student evaluation, and additional teacher and student rollouts for filtering. Although the appendix discusses these costs, measured runtime or GPU usage and compute-matched comparisons are missing.

  2. Tables 1 and 2 contain an unexplained inconsistency. Table 2 specifies Qwen2.5-7B, but its GRPO scores on NQ/HotpotQA are 42.1/34.8, matching the 3B results in Table 1 rather than the 7B results of 43.2/39.2. The authors should clarify whether this reflects a reporting error or different training runs or settings, and ensure that all comparisons use consistent baselines.

Questions for authors

Could the authors report the computational costs and clarify the discrepancy between the GRPO results in Tables 1 and 2, as discussed in the Critical Weaknesses section?

3 comments
AuthorsOpenReview ↗

Response to Reviewer e3yr: measured compute and Table 1/2 correction

We thank the reviewer for the constructive comments and for pointing out these important issues.

Q1

Using Qwen2.5-7B DeepSearch as an example, the additional overhead mainly comes from three parts:

  • Training rollouts: 2,048 trajectories per update, totaling 348,160 through update 170, which is 27.2% of the GRPO training rollouts. The frozen teacher only performs a forward pass and incurs no additional environment interaction.
  • Teacher-pool evaluation: no additional overhead. The teacher pool consists of the 50 checkpoints saved every 10 GRPO steps; their Dval results were already recorded during RL training and are directly reused by HCN.
  • Student re-evaluation: once every K=5 steps, 35 times through update 170, totaling 385,525 greedy-decoding trajectories.

In total, the integration stage consumes 733,685 environment-interaction trajectories, which is 30.7% of the initial GRPO run, and about 462 GPU hours, which is 85% of GRPO. The average LAD acceptance rate is ρ=0.51\rho=0.51, so the oversampling overhead is limited. Only one teacher resides on the GPU during training; peak GPU memory is 101.9 GB, lower than 104.4 GB for GRPO. The 50 checkpoints occupy about 0.76 TB of disk, and deployment requires only the student model, so the inference cost is the same as GRPO.

GRPO has already converged, and continued training yields no meaningful improvement: under the same training-rollout budget (348,160 trajectories, about 136 GRPO steps), the total number of solved questions increases by only 0.8 percentage points, which is within normal fluctuation. Agentic-OPD improves NQ/HotpotQA by 7.2/7.3 points, respectively. We will include the GRPO training curves in the camera-ready version to further demonstrate that continued GRPO training cannot yield improvements.

Q2

Table 2 actually uses Qwen2.5-3B, but its caption incorrectly labels it as Qwen2.5-7B. We will correct this in the revised manuscript and consistently cite the 3B GRPO baseline from Table 1, ensuring that all comparisons use consistent settings.

We look forward to your response and would greatly appreciate any further comments, suggestions, or guidance you may have.

Thank the authors for their reply. Most of my concerns have been resolved.

Reviewer_vnM6OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper suggests Agentic-OPD, a method for utilizing checkpoints saved along a GRPO post-training run to improve final performance of a selected checkpoint, using on-policy distillation.

The paper identifies a gap in performance of the best selected checkpoint in RL training runs for agentic tasks, and notes that unions of historical checkpoints can provide successful trajectories for problems unsolved by the best checkpoint. The paper then suggests a multi-stage algorithm to re-use these trajectories by selecting a historical checkpoint as teacher and distilling it into the student final checkpoint.

The performance of Agentic-OPD is evaluated on a set of agentic long-horizon tasks, and compared both with out-of-the box proprietary and open-source models, as well as baselines ranging from simple GRPO training to other checkpoint-merging strategies.

Critical strengths

What I find most compelling about this paper is the core idea: re-using training time data and checkpoints which are usually discarded to improve final performance is the right direction to go as tasks become more complex and training becomes more expensive.

I also appreciate the preliminary study performed by the authors (as well as some of the analysis in Section 4.3) to validate the intuition and motivations behind Agentic-OPD. These help with constructing the storyline of the paper and make the original claim even more compelling.

Critical weaknesses

My main concern with the paper is that the results seem lacking: given the high computational overhead of the suggested method (over-sampling new batches, student and teacher forward passes, distillation computation) the gains over GRPO seem minimal (and possibly not statistically significant for the tested models, as standard deviations or statistical tests of the results are not reported). The paper strengthens this observation in Table 2: simple, no-training merging methods reach virtually the same performance as the highly computational Agentic-OPD.

In addition, OPSD results seem to be an unfair comparison - the privileged information is run on top of the base model, and not using the GRPO-obtained best checkpoint as in Agentic-OPD. A more fair comparison would be to provide the privileged information to the same checkpoint distilled with selected historical teachers.

Another weakness is that the paper organization is lacking; hard to follow the written logic in the preliminary study and the method section (pointers to later sections and figures; notation and terms are not properly defined). Sections 2 and 3.3 are particularly lacking in terms of notation definitions.

Questions for authors
  1. A central finding driving the motivation for agentic-OPD is the complementarity of historical checkpoints, solving tasks different to the ones solved by the current checkpoint (Section 2). Have the authors attempted to hypothesize why this happens? Is this an artifact of training dynamics or hyperparameter choices?
  2. Why use a specific teacher to train at each step, and not just select teachers per-rollout given the sampled batch for the student update? I.e. sample the batch, then find solutions for each problem in it from previous checkpoints (if they exist) and use them as teaching signal. This could potentially save computation of HCN and make training more efficient.
  3. If greedy decoding is used for all validation (Section 2), why would we obtain both successful and unsuccessful rollouts for the teacher and student at each selected problem (Section 3.3)? Or is greedy decoding only used for the study in Section 2 and not later in the paper?
  4. Following up on the previous question, if the student has both successful and failed rollouts, why not just pick the successful ones as learning targets? Why use teacher rollouts (which are potentially off-policy and misleading given we are using a forward pass of the teacher on a student-generated trajectory)?
  5. Figure 1 caption and notation is confusing - the (i) notation is described in the caption but does not appear in the figure itself; also the plot is unclear on what the “current checkpoint” is. While the notation is somewhat defined later in sec. 2, could the authors improve the description of this figure and its significance?
  6. What is the computational overhead of agentic-OPD, both in terms of additional computation post-RL and memory costs of maintaining a large number of checkpoints? Given these costs, is the method feasibly usable with models larger than 4B?
  7. For the un-trained closed- and open-sourced models in Table 1: are results obtained by the authors independently or collected from reported benchmarks by the original vendors and model authors? In general, some of these results seem off - do the older, small Qwen base models really outperform modern flagship models on the DeepSearch tasks?
  8. Why are older backbone model used, and not the most recently available ones (eg. Qwen3.5 instead of Qwen2.5 and Qwen3)?
  9. Given the large (>60% gap) between best checkpoint and union of previous checkpoints in Fig.1, how do the authors explain the relatively small gap between GRPO and Agentic-OPD in Table 1?
  10. Have the authors considered comparisons to other capability retention methods? Using the same budget as consumed by the additional computation required in Agentic-OPD, could additional fine-tuning on failed problems (while retaining previously solved problems) outperform the Agentic-OPD, possibly avoiding the loss of best-checkpoint capabilities (Fig 4 (a))?
3 comments
AuthorsOpenReview ↗

Response to Reviewer vnM6: complementarity, compute overhead, and RFT baseline

We thank the reviewer for the constructive comments.

Q1

We attribute this mainly to RL optimization dynamics rather than a single hyperparameter. Prior work (Temporal Sampling for Forgotten Reasoning in LLMs) observed forgetting under RL/SFT; we extend it to Agentic RL. A likely mechanism is gradient interference: single- and multi-hop subtasks have competing optimization directions, so updates for some abilities degrade others (Imbalanced Gradients in RL Post-Training of Multi-Task LLMs ). A full theory is still lacking.

Q2

Each teacher checkpoint's validation performance was recorded during RL. OPD only needs one student inference pass on the validation set (~1s) to select a teacher, so HCN overhead is negligible. Per-rollout teacher selection would require re-solving each problem with every historical checkpoint, which is more expensive.

Q3

Greedy decoding is used only for the checkpoint-level analysis in Section 2. Section 3.3 uses sampled rollouts to estimate student/teacher success rates (temperature 1.0, following Search-R1).

Q4

A successful student rollout already shows the capability; distilling it again would introduce unnecessary updates. Rethinking On-Policy Distillation shows OPD is effective only when the teacher provides experience the student lacks. We instead run a teacher forward pass on the student's own trajectory to provide token-level supervision.

To verify, we add an RFT baseline: starting from Qwen2.5-3B-Instruct and the best RL checkpoint, we collect successful historical trajectories on seven DeepSearch datasets and perform RFT (rejection fine-tuning).

DatasetNQHotpotQAPopQATriviaQA2WikiMuSiQueBamboogleAvg.
GRPO (Best RL Checkpoint)42.134.839.255.535.813.928.035.6
RFT (Historical Successful Trajectories)41.336.142.558.935.613.832.037.2
Agentic-OPD45.437.943.260.834.314.632.838.4

RFT averages 37.2, below Agentic-OPD (38.4), and underperforms the best RL checkpoint on NQ/2Wiki/MuSiQue. This is due to distribution shift and compounding errors in offline imitation (Lauffer et al., 2026, ICML). We therefore perform on-policy distillation on the student's failed prefixes.

Q5

We will revise Figure 1: TiT_i is the checkpoint at step ii, and U(i)U_(i) is the cumulative set of problems solved by at least kk checkpoints among the first ii; U(i)U_(i) is the historical union through step ii. We will also fix the caption.

Q6

For Qwen2.5-7B DeepSearch, integration through update 170 uses ~0.73M trajectories (30.7% of GRPO) and 462 GPU hours (~85%); average LAD acceptance ρ=0.51\rho=0.51. Peak memory is 101.9 GB (GRPO: 104.4 GB); 50 checkpoints use ~0.76 TB disk, and deployment needs only the student. GRPO has converged, so this compute recovers historical capabilities rather than repeating saturated RL optimization; camera-ready will include GRPO curves. We validate up to 7B; memory does not scale linearly with checkpoints.

Q7

Closed/open-source results in Table 1 were independently evaluated by us with a protocol consistent with AgentGym-RL; after accounting for hardware and variance, they broadly match. Small Qwen base models outperforming some flagship models on DeepSearch mainly reflects adaptation to the tool-use/search protocol, not general capability ranking.

Q8

Qwen3.5 was not released when this work was done; due to the submission timeline, we will add newer backbones in future work.

Q9

The historical union is potentially recoverable capability, not the gain from one teacher. LAD selects only problems where the teacher has a local advantage and the student fails; distillation can also cause degradation: Figure 4(a) recovers 60 and forgets 45, net +15. Thus the union and final gain are not linearly related.

Q10

We compare with TIES, DARE-TIES, Model Soups, and Temporal Sampling; none outperforms Agentic-OPD. For fine-tuning on failed problems under the same extra compute, we add an RFT baseline on historical successful trajectories: average 37.2, below Agentic-OPD (38.4); failed-problem RFT is still being added.

We look forward to your response and would greatly appreciate any further comments, suggestions, or guidance you may have.

I'd like to thank the authors for taking the time to respond and address my questions.

Some of my concerns have been resolved, however some central points remain: specifically, the compute required by Agentic-OPD over GRPO is still significant, and the minor gains compared to non-training merging methods may not be worth the cost.

In light of the authors' responses, and as I am still unconvinced of the overall empirical significance of the results, I will raise my score to 2: weak rejection.

Reviewer_boPKOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates capability forgetting during reinforcement learning (GRPO) of multi-turn LLM agents. Through a preliminary study across five agentic domains, the authors show that the union of problems solved by intermediate checkpoints along a single RL run is substantially larger than the set solved by the single best validation checkpoint. To recover these lost capabilities into a single model without increasing deployment cost, the paper proposes Agentic-OPD, which initializes a student from the best RL checkpoint and distills from the pool of frozen historical checkpoints. Specifically, Highest Coverage Next (HCN) periodically selects the historical checkpoint that covers the largest weighted set of validation problems currently missed by the student, while Locality-Aware Distillation (LAD) restricts token-level KL distillation to student-generated turns on failed student rollouts for training problems where the selected teacher achieves a higher empirical rollout accuracy than the student. Experiments across five domains (DeepSearch, Tool-use, WebShop, SciWorld, and ALFWorld) show consistent improvements over the initial GRPO checkpoint and standard model-merging baselines.

Critical strengths
  1. The empirical characterization of temporal forgetting in multi-turn agentic RL is well-motivated and clearly presented. Showing that the historical union of solved tasks (even when requiring tasks to be solved by at least two or three checkpoints, U>=2 and U>=3 in Figure 1) far exceeds any single checkpoint across five diverse agentic environments provides a strong motivation for reusing intermediate checkpoints from a completed RL run.

  2. Framing historical checkpoint consolidation as a selective weak-to-strong distillation problem is intuitive and practical. Pairing validation-guided teacher routing (HCN) with problem- and rollout-level filtering (LAD) consistently improves over the converged GRPO initialization across five agentic domains and outperforms training-free weight merging methods such as Model Soups and DARE-TIES while keeping single-model inference cost at deployment.

Critical weaknesses
  1. There is a substantial gap between the large historical union motivating the paper (Figure 1) and the actual net recovery achieved during consolidation, alongside a potential mismatch in distilling only on failed student trajectories. In Figure 4(a), while the student recovers 60 previously solved validation questions from historical teachers, it simultaneously loses 45 questions that the initial student had already solved, yielding a net gain of only 15 questions (+0.75 percentage points on the 2,000-question validation set). This indicates that even with LAD filtering, overwriting already mastered behaviors remains severe. Methodologically, LAD discards all successful teacher trajectories after computing qt(x) and only forces the student to match teacher token distributions conditioned on the student's own failed trajectories (Eqs. 12 and 13). In multi-turn agentic tasks, once a student rollout makes an early wrong search query or tool call, the subsequent environment observations diverge from the states where the teacher actually succeeded, so querying the teacher's logits on off-track student prefixes may provide weak or noisy supervision compared to leveraging the teacher's successful rollouts.

  2. The computational overhead of the consolidation loop is not accounted for when comparing against GRPO and merging baselines, and there is a data mismatch between Table 1 and Table 2. As noted in Section 3.4 and Appendix A.3, Agentic-OPD requires evaluating all ~100 historical checkpoints on Dval, periodically re-evaluating the student on Dval every K steps, and running 2BG/rho online multi-turn rollouts per update (since both the student and the activated teacher must draw G rollouts per problem under DAPO-style oversampling when only a small fraction rho of rollouts pass the qt(x) > s(x) filter). However, the paper does not report the wall-clock time or total rollout count of this stage, nor does it compare against continuing GRPO (or running GRPO with the same DAPO oversampling budget) for an equivalent number of environment interactions. In addition, Table 2 states that it reports Qwen2.5-7B results on NQ and HotpotQA (GRPO: 42.1 / 34.8; Agentic-OPD: 45.4 / 37.9), but these exact numbers match the Qwen2.5-3B block in Table 1 rather than the Qwen2.5-7B block (which reports 43.2 / 39.2 for GRPO and 50.4 / 46.5 for Agentic-OPD).

Questions for authors
  1. How much total rollout compute (in environment episodes or GPU hours, including teacher rollouts for qt(x) and validation evaluations for HCN) does Agentic-OPD consume relative to the initial GRPO run, and what is the typical acceptance rate rho of the LAD filter in Eq. 11? How does Agentic-OPD compare against simply continuing GRPO with DAPO-style dynamic sampling under the exact same additional rollout budget?

  2. In Figure 4(a), the student loses 45 already-solved questions while recovering 60 (+15 net). Why does LAD still cause regressions on 75% as many questions as it recovers, and have you tested also distilling (or applying SFT/RL) on the teacher's successful rollouts rather than only evaluating teacher logits on failed student prefixes?

  3. Could you clarify the discrepancy between Table 1 and Table 2 regarding Qwen2.5-7B vs. Qwen2.5-3B on NQ and HotpotQA (where Table 2 is labeled Qwen2.5-7B but lists the exact numbers from the Qwen2.5-3B block of Table 1)? Also, in Table 1, were the proprietary and large open-source models (e.g., GPT-5.6, Claude Sonnet 5) evaluated with the exact same 4-turn Wikipedia search tool and prompt format as your trained models?

2 comments
AuthorsOpenReview ↗

Response to Reviewer boPK: compute overhead, GRPO convergence, and RFT baseline

We thank the reviewer for the constructive comments and for pointing out these important issues.

Q1

Taking Qwen2.5-7B DeepSearch as an example, the initial GRPO training involved approximately 2.39M environment interaction trajectories and 539 GPU hours. By update 170 of the Agentic-OPD integration stage, it required approximately 0.73M trajectories (30.7% of GRPO) and 462 GPU hours (approximately 85% of GRPO). The average LAD acceptance rate was ρ=0.51\rho=0.51, indicating limited oversampling overhead. Peak GPU memory usage was 101.9 GB, lower than GRPO's 104.4 GB. The 50 checkpoints occupied approximately 0.76 TB of disk space, while deployment required only the student model.

GRPO had already converged, and continued training yielded no meaningful improvements. In contrast, Agentic-OPD achieved gains of 7.2 and 7.3 points on NQ and HotpotQA, respectively. We will include the GRPO training curves in the camera-ready version to further demonstrate that our method can continue improving the performance of an already-converged Agentic-RL model.

Q2

In Figure 4(a), Agentic-OPD recovers 60 questions and forgets 45, resulting in a net gain of 15 questions. The ratio 45/60 = 75% represents the number of forgotten questions relative to the number of recovered questions; it does not indicate a 75% overall performance drop. LAD performs filtering at three levels: question, rollout, and token, to reduce, but not completely eliminate, cross-task interference caused by parameter updates.

For the SFT baseline that directly uses successful teacher trajectories, we use Qwen2.5-3B-Instruct as the base model. Starting from the best RL checkpoint, we collect successful trajectories from historical checkpoints across seven DeepSearch datasets and use them for SFT. The results are shown below (Exact Match, %):

DatasetNQHotpotQAPopQATriviaQA2WikiMuSiQueBamboogleAvg.
GRPO42.134.839.255.535.813.928.035.6
RFT41.336.142.558.935.613.832.037.2
Agentic-OPD45.437.943.260.834.314.632.838.4

SFT underperforms the best RL checkpoint on NQ, 2Wiki, and MuSiQue, achieving an average score of 37.2. It does not consistently outperform the best RL checkpoint and remains below Agentic-OPD (38.4). This is due to distribution shift and compounding errors in offline imitation of multi-turn agents: once the student deviates from a successful trajectory, subsequent observations quickly move outside the training distribution, causing single-step errors to accumulate along the trajectory (Lauffer et al., 2026, ICML, Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections).

Therefore, we adopt on-policy distillation, which provides token-level supervision on the student's own trajectories, rather than requiring the student to imitate successful teacher trajectories.

Q3

Table 2 actually uses Qwen2.5-3B, but its caption incorrectly identifies the model as Qwen2.5-7B. We will correct this in the revised manuscript. All external models are evaluated using the same scoring rules and evaluation protocol.

We look forward to your response and would greatly appreciate any further comments, suggestions, or guidance you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.