acceptodds
Under review as a conference paper at ICLR 2027

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

10 views

Abstract

Reinforcement learning(RL) with verifiable rewards constructs trajectory-level advantage, yet often fails to credit the few pivotal decisions that drive outcomes in long-horizon multi-turn agentic RL. Some recent works introduce privileged self-distillation into credit assignment for RL, offering denser supervision, but it still remains unclear how such a local signal should express sequential credit. We therefore propose AgentOPSD, a critic-free recursive turn-level credit assignment for agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence, recursively updates a Bayesian belief state in log-odds space. This provides a principled reweighting scheme that transforms sparse outcome supervision into turn-level credit signals and identifies pivotal turns by the marginal revision between consecutive states while remaining fully compatible with standard policy optimization, requiring neither additional rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA with three Qwen model scales (Qwen2.5-3B/7B and Qwen3-1.7B). AgentOPSD improves over GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B, and ablations attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, agent skills, on-policy distillation, llm agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_teXROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AgentOPSD, a credit-assignment method for multi-turn LLM agents trained with GRPO. It reweights the advantage at each turn using a self-distillation signal. For each turn, it measures how much a training-only retrieved “skill” in the prompt increases the policy’s likelihood of the action taken. It then accumulates these per-turn gaps into a recursive “success belief” in log-odds space, initialized with the group success rate. Experiments on ALFWorld, WebShop, and Search-QA with Qwen2.5-3B/7B and Qwen3-1.7B show that AgentOPSD achieves the best or near-best performance on most aggregate metrics.

Critical strengths
  1. Turn-level credit in critic-free agentic RL is an important open issue. Combining it with privileged self-distillation signals is timely. Also, the methodology is simple and drop-in. It adds only a cheap advantage reweighting on top of GRPO. It needs no critic and no extra rollouts.
  2. The baseline suite is comprehensive, covering training-free baselines, GRPO with and without skills, and six self-distillation variants, all using the same set of skills. This makes the comparison of how the supervision signal is utilized reasonably fair.
Critical weaknesses
  1. The evaluation sets are small. ALFWorld percentages imply roughly 128 evaluation tasks, so one task is about 0.8 points. The 7B gaps to StepOPSD (88.4) and Skill-GRPO* (88.3) are about one task, and per-category numbers rest on 8–30 episodes each.
  2. The reweighting is too small to plausibly explain the reported gains. Under b=0.2b = 0.2 and λ=0.5\lambda = 0.5, each turn's advantage is scaled by a factor in [0.9,1.1][0.9, 1.1], at most ±10% around GRPO, but the method improves on GRPO by 8 points on ALFWorld-7B. Removing a single component (e.g., the B0B_0 prior) drops performance to 78.9, which is below GRPO itself. A ±10% modulation is unlikely to produce swings of this size, which points to run-to-run variance rather than the mechanism.
Questions for authors

Questions: See weaknesses.

Minor issues:

  1. The abstract contains a broken phrase ("requiring neither additional rollouts").
  2. At the beginning of Section 2.1, Eq. (1) should be placed after "At turn k, it samples...".
Reviewer_cwD7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new algorithm for RL with verifiable rewards. The main setting is a multi-turn agent interaction where teh reward is only given at terminal success/failure. A standard baseline is GRPO, which rewards the complete interaction trajectory. More advanced techniques use a teacher/student approach, where the teacher has more "hints" to solve the task (here using skills, to the best of my understanding). The question this paper tackles is how to integrate the teacher signal into the RL training. The main idea is to observe the difference between token probabilities without the hint P(a|history) and with the hint P(a|history, hint), which the authors use an an approximation for P(a|history, success). Using this idea and Bayes rules, the propose to approximate which actions contribute to the success in the multi-turn interaction, and weigh those more in the update. The actual update rule (Sec. 2.3-2.4) is a rather complex (and not very clear) function of the individual contributions at each step that involves discounting, clipping, and various normalizations. I was not able to understand the motivation for this update rule or why it actually works.

The results show improved performance versus baselines that (to my understanding) use the same hints for the teacher, just use them differently in the update rule. The performance gains seem most significant in the Qwen3 1.7B model, while in Qwen2.5-3B-Instruct the gains are less significant. In any case, since standard errors are not reported, it is hard to assess the significance of the results.

Minor note: First paragraph of Sec 2.1 : Eq (1) should be after the action description.

Critical strengths
  1. The idea of using the teacher to approximate the Bayesian belief of success is elegant and (to my knowledge) novel.

  2. Looks like this works on benchmarks.

Critical weaknesses
  1. The gap between the clean algorihtmic idea and the actual implementation with its many hyperparameters is very large, and it's hard to tell how much parameter tuning has to do with the results. Figure 4 aims to investigate parameter sensitivity, but I'm not sure what the consequence from it is.

  2. The writing is technical, heavily AI-assisted, and Sections 2.3 and 2.4 lack motivation. It would help if the motivation behind the actual implementation was clearer. How did you come up with these ideas? What was the reasoning? Or was it pure trial and error that ended up working well?

Questions for authors

see above.

Reviewer_bP6WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper targets the problem of turn-level, credit assignment in long-horizon agentic RL domain, especially in sparse reward settings. Canonical GRPO assigns every atomic turn of a trajectory the same advantage, which flattens out turns that carries critical learning signals. AgentOPSD re-weights the per-turn advantage using a skill-conditioned self-teacher.

Based on my understanding, the paper's methodology and novelty primarily resides in four steps:

  • Sum the token-level teacher–student log-probability gap within each turn to get the turn evidence eke_k.
  • Accumulate this evidence in log-odds, starting from the group success rate, to get a running "success belief" BkB_k.
  • Score each turn by how much it changes the belief, δBk\delta B_k, and align the sign with the verifier outcome.
  • Standardize within the trajectory and apply a bounded multiplier to the GRPO advantage.

There is one critical underlying assumption the paper builds arounds, i.e., the skill-conditioned policy approximates/aligns with the success-conditioned policy.

Critical strengths
  • Clear, relevant problem formulation. Uniform credit assignment in long-horizon agent RL is a real limitation of GRPO typed methods that propagate trajectory level advantage across tokens, and the paper's motivation is clearly built upon this problem space.
  • Simple, efficient, and safe methodology design by construction. In terms of actual compute cost, the method only adds another teacher forward pass per trajectory. The designed multiplier is bounded and strictly positive, so it does not risk flipping the GRPO update direction. The method also resolves back to GRPO with λ=0\lambda=0, which makes its integration into existing framework and pipeline.
  • Intuitive modeling of history dependence. Because evidence is accumulated in log-odds, the same local evidence counts more while the outcome is uncertain and less once the belief has saturated. This feature nicely captures the idea that a local gap alone is not sequential credit.
Critical weaknesses
  1. The reported gain is hard to separate from the skill prior. The paper is well developed overall. My central question is the real gain (and loss) of using skills as the self-teacher's privileged information. Table 1 (ALFWorld, Qwen2.5-7B) suggests most of the improvement over GRPO comes from the skills themselves, not from the turn-level Bayesian belief design:
  • Regarding AgentOPSD's +7.9 margin over GRPO, simply supplying skills at inference time already yields +7.1. A turn-level baseline using the same skills gives +7.2.
  • The method-specific margin over the best skill-using baseline is small on ALFWorld: +0.7 (7B), 0. (3B, same as SDAR), +1.0 (1.7B). It only shows margin on WebShop and Search-QA on 3B model.
  1. Risk of "boxed-in" nudging toward the skill's patterns. eke_k fundamentally measures how much the hint likes a turn, not how much success likes it. The underlying assumption is the hint approximates/(directionally) agrees with the success, which is not always guaranteed in agentic RL settings.
  • In successful trajectories, hint-conforming turns get more credit. In failed ones, hint-deviating turns get more blame.
  • So the reshaping systematically pushes/nudges the policy toward the skill's stereotyped patterns and risks not rewarding strategies that beat the hint.
  1. Scalability and verifiability of the skill source. The method is, in some sense and at varying degrees, dependent on a curated, environment-specific skill bank, and the retrieval quality is not measured.
  • There is no analysis of what happens when skills are noisy, incomplete, wrong, or missing for new environments.
  • This concern limits the method's claim to be a general credit-assignment approach for agentic RL.
Questions for authors

Q1. "Placebo" skill. What happens to AgentOPSD if c+c^+ is replaced with a shuffled or irrelevant skill? This can potentially measures how much of the performance gain depends on skill content vs. the reshaping mechanism. Q2. Random-multiplier control. What if wkw_k is replaced by a random per-turn multiplier with the same range? Does the ordering of turns given by ΔBk\Delta B_k matter? Q3. Boxed-in behavior. How does AgentOPSD perform on task types or environments not covered by an available skill prior? Can you report action diversity or entropy compared with GRPO? Q4. Skill quality and scale. How sensitive is performance gain vs. skill hint bank quality (i.e., being noisy, partial, or even wrong)? What is the cost of preparing/acquiring the necessary skill hint for each environment? How would the method apply (or not apply) where no curated, verifiable skills exist compared to baseline? Is the gain still significant?

Reviewer_c4sTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AgentOPSD, a recursive self-distillation method for turn-level credit assignment in agentic reinforcement learning. It uses teacher–student probability gaps to estimate the importance of each turn and recursively updates a Bayesian-inspired success belief. The resulting signals are used to reshape GRPO advantages without an additional critic or extra rollouts. Experiments on ALFWorld, WebShop, and Search-QA show improvements over GRPO and several self-distillation baselines.

Critical strengths
  1. Interesting and well-motivated approach.

Turn-level credit assignment is an important problem in long-horizon agentic RL. The recursive belief update is a clever way to incorporate historical information into advantage estimation.

  1. Solid experimental evaluation.

The paper evaluates the method across three environments and different model scales. The comparisons with self-distillation baselines and the component ablations provide useful evidence for the proposed design.

Critical weaknesses
  1. Strong dependence on retrieved skill quality.

AgentOPSD relies on skill-conditioned teacher–student probability gaps for credit assignment. However, the quality and relevance of retrieved skills are not systematically evaluated. Incorrect or incomplete skills may produce misleading supervision, especially for complex tasks.

  1. Limited validation of the Bayesian interpretation.

The method assumes that skill-conditioned behavior approximates successful behavior. This assumption is important but not directly validated. It is also unclear whether larger belief revisions actually correspond to more important decisions, rather than changes caused by the recursive weighting mechanism.

Questions for authors

Q1. Skill Quality

Would higher-quality skills lead to better credit assignment and policy performance? Could the authors provide an ablation with different levels of skill quality or relevance?

Q2. Reliability of the Self-Teacher

For complex tasks, the model may fail to decompose the task correctly, even with retrieved skills. In such cases, the teacher may provide noisy or misleading guidance. How does AgentOPSD handle unreliable skills and avoid incorrect credit assignment?

Q3. Bayesian Credit Assignment

The Bayesian interpretation relies on the assumption that skill-conditioned behavior approximates success-conditioned behavior. How is this assumption validated? Also, belief revision can occur due to evidence decay even without new evidence. How can the authors verify that these revisions reflect actual turn-level contribution

Reviewer_uZNnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses turn-level credit assignment in multi-turn agentic RL. GRPO broadcasts a single sequence-level advantage to every token; the authors instead use an on-policy self-distillation signal: the same policy scores the student's sampled action with and without a training-only retrieved skill in context, and the per-token log-prob gaps are summed within each turn to form a turn-level "evidence" eke_k. This evidence is accumulated recursively in log-odds space with a decay γ\gamma, starting from the GRPO group success rate as a prior B0B_0. The per-turn belief change ΔBk=σ(ℓk)−σ(ℓk−1)\Delta B_k = \sigma(\ell_k) - \sigma(\ell_{k-1}), multiplied by sign(Aseq)\mathrm{sign}(A_{seq}) and standardized within the trajectory, becomes a bounded multiplier in [1−b,1+b][1-b, 1+b] that reshapes the GRPO advantage. No critic and no extra rollouts are needed; the overhead is one teacher forward pass. Experiments on ALFWorld, WebShop and Search-QA with Qwen2.5-3B/7B and Qwen3-1.7B report gains over GRPO and over several self-distillation baselines (SDAR, RLSD, Skill-SD, StepOPSD), with 89.1% on ALFWorld at 7B. Ablations attribute the gains to turn-level aggregation, the recursive update, the outcome sign, and the prior anchor.

Critical strengths
  1. The problem is real and well motivated: uniform credit from GRPO is a known weakness in long-horizon settings, and using a self-distillation gap as a cheap, critic-free per-turn signal is a sensible entry point.
  2. The method is simple, inexpensive, and a strict superset of GRPO (Props. 1-3: bounded, sign-preserving, recovers GRPO at λ=0\lambda=0). It is easy to implement and reproduce.
  3. The comparison against other self-distillation methods is information-controlled: all methods see the same retrieved skills and differ only in how the gap enters training. This is the right experimental design for isolating the contribution.
  4. The component ablation (granularity / recursion / sign / prior) is cleanly structured.
  5. The horizon-sensitivity analysis in Fig. 1(b) is a nice angle that connects back to the motivation.
Critical weaknesses
  1. Statistical reliability of the main results (major).* Table 1 reports no number of seeds and no std / confidence intervals. The paper itself provides a noise floor: in Sec. 3.4, λ=0.01\lambda=0.01, where the multiplier lies in [0.998,1.002][0.998, 1.002], i.e. essentially GRPO scores 83.6, while GRPO in Table 1 scores 81.2, a 2.4-point gap between two near-identical configurations. Similarly, the "drop B0B_0" ablation reaches 78.9, below GRPO, even though the multiplier is strictly positive and confined to [0.8,1.2][0.8, 1.2] and therefore cannot plausibly degrade GRPO by that much; run-to-run variance is the more likely explanation. The γ\gamma sweep (87.5 / 82.0 / 85.2) is non-monotone with a 7-point spread, which the text describes as "not particularly sensitive". This reading is not supported. The "ten of twelve vs. SDAR" claim includes a tie (84.4 vs. 84.4 on 3B ALFWorld) and several sub-1-point margins. At least 3 seeds with std and a significance test for the headline claims are needed.

  2. Hyperparameters tuned on the evaluation set.* The sweep in Sec. 3.4 uses the same evaluation tasks as Table 1, and the selected values (λ=0.5\lambda=0.5, γ=0.95\gamma=0.95, ϵhigh=0.24\epsilon_{high}=0.24) are then reported as main results. Baselines "share the SDAR setting" and receive no comparable tuning. Moreover, λ\lambda is only swept downward ({0.5,0.25,0.1,0.01}\{0.5, 0.25, 0.1, 0.01\}); no value in (0.5,1](0.5, 1] is tested, so "λ=0.5\lambda=0.5 is best" is not established.

  3. Signal scale and belief saturation are never discussed (major).* eke_k is a sum of per-token gaps over the turn, not a mean. Fig. 4 shows the per-token gap δˉ\bar\delta is negative throughout training (roughly −0.05-0.05 to −0.2-0.2; about −0.08-0.08 for ALFWorld 7B). With tens to hundreds of think+action tokens per turn, eke_k is on the order of −5-5 to −30-30 nats. Starting from logit(B0)=0\mathrm{logit}(B_0)=0 for a 50% group, a single e1≈−15e_1 \approx -15 gives B1≈3×10−7B_1 \approx 3\times10^{-7}; with γ=0.95\gamma=0.95 the steady-state accumulator is ≈e/(1−γ)≈−300\approx e/(1-\gamma) \approx -300, so the belief never recovers. The consequence would be ΔB1≈−0.5\Delta B_1 \approx -0.5 and ΔBk>1≈0\Delta B_{k>1} \approx 0; after within-trajectory standardization and clipping, the mechanism collapses to "multiply the first turn by 0.8 and the rest by slightly more than 1," which has nothing to do with identifying pivotal turns. I cannot be certain this is what happens in practice, but it follows directly from the numbers the paper reports. The authors must show the distribution of eke_k, BkB_k, ΔBk\Delta B_k and wkw_k as a function of turn index, and the fraction of turns that receive non-trivial credit.

  4. The Bayesian framing is largely cosmetic.*

  • Assumption (A1), that the skill-conditioned branch approximates p(ak∣sk,C)p(a_k \mid s_k, C), is never verified. The skill is a task-level text retrieved by keyword matching, identical across all turns, with no direct link to success. Fig. 4 shows the teacher assigns lower probability to the student's actions on average, which sits uneasily with the claim that the teacher represents success-associated behavior.
  • Assumption (A2) requires ρk→0\rho_k \to 0, yet B0B_0 is the group success rate, which approaches 0.9 late in ALFWorld training.
  • Eq. (3) uses ¬C\neg C in the denominator while Eq. (7) uses the marginal; these are different quantities and the text switches between them.
  • Props. 1-3, 5 and 6 are essentially trivial. The authors concede BkB_k is only "relative support," not a calibrated probability; the title and abstract should be toned down accordingly.
  • More fundamentally, after sign(Aseq)\mathrm{sign}(A_{seq}), within-trajectory z-scoring and clipping, only the ranking of ΔBk\Delta B_k within a trajectory survives; the prior, the absolute belief, and the Bayes-factor interpretation are all washed out by the normalization.
  1. No evidence that the credit is actually correct.* Nothing in the paper shows that turns with large ∣ΔBk∣|\Delta B_k| are in fact pivotal. A Monte-Carlo estimate of per-turn value change on a small subset (VinePPO-style) correlated with ΔBk\Delta B_k, or at minimum qualitative examples, would be needed. Without it, "identifies pivotal turns" is an untested assertion.

  2. Baselines and fairness.*

  • OPSD reaches 0.0-0.1 accuracy on 3B Search-QA. That is a collapsed run, not a property of the method, and it raises doubts about how carefully the other baselines were tuned.
  • The headline "improves over GRPO" conflates credit assignment with privileged information: AgentOPSD uses training-only skills, GRPO does not. The fair reference point is Skill-GRPO*, which reaches 88.3 at 7B.
  • The SkillBank comes from SkillRL; the paper should state whether the skills were built from trajectories on the same task distribution (or the same tasks) as the evaluation set.
Questions for authors
  1. Please report the distributions of eke_k, ΔBk\Delta B_k, wkw_k by turn index and the fraction of trajectories in which BkB_k saturates. Why sum rather than average the token gaps? Was a length-normalized variant tried?
  2. Results for λ>0.5\lambda > 0.5?
  3. Std over multiple seeds for Tables 1 and 2?
  4. Any direct check that ΔBk\Delta B_k correlates with counterfactual per-turn value?
  5. Why does OPSD collapse on Search-QA, and was it tuned?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.