acceptodds
Under review as a conference paper at ICLR 2027

ProjOPD: Conflict Projection for On-Policy Distillation in Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning typically provides only a single scalar reward at the end of an episode, so the many intermediate decisions within a trajectory all share one signal. Recent work supplements this sparse supervision with on-policy self-distillation, but existing approaches mainly study how to construct the teacher signal or how to weight individual tokens, and pay little attention to whether the auxiliary update is compatible with the task objective. We identify an optimization-level conflict between the two: a context teacher expresses nothing more than the local preference of the same policy under an additional prompt, and its gradient can have a negative inner product with the policy gradient; adding the two directly then increases the RL objective to first order, i.e. the auxiliary signal cancels the descent direction dictated by the environment reward. We therefore propose ProjOPD, which builds both logit gradients explicitly on a shared sparse support, removes from the distillation gradient the component opposing the policy gradient within each environment turn, retains only its orthogonal part, and back-propagates through a detached surrogate. The projection is applied per environment turn—a turn corresponds to one complete environment action, whereas deciding token by token ignores how the tokens of a single action jointly point. The construction guarantees to first order that the auxiliary signal cannot cancel the descent direction of the policy gradient, and requires no additional rollouts. In addition, we propose a cheaper teacher construction: the teacher is simply the current policy conditioned on a plan context that the policy itself generates; it only re-scores the tokens the student has already sampled and never re-executes the environment. At test time the planner and the teacher are removed, so inference cost matches an ordinary actor. We evaluate ProjOPD on ALFWorld, WebShop and Search-QA with Qwen2.5-3B and 7B, where it attains the best result on all three benchmarks at 7B; ablations attribute the gains to the conflict projection and to the turn-level projection granularity.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, on-policy distillation, gradient conflict, credit assignment, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7qZBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies agentic RL with a sparse outcome reward, supplemented by on-policy self-distillation (OPD). The teacher is the same policy with a self-generated "plan context" (constraints plus a to-do list) prepended to the prompt, and it re-scores tokens the student already sampled. The authors argue the OPD gradient can have a negative inner product with the GRPO gradient and so cancel the reward-driven update. Their fix, ProjOPD, works on a sparse top-64 + sampled-token + residual support in logit space. For each environment turn, it removes the component of the OPD gradient that opposes the policy gradient. Proposition 1 gives a first-order non-interference guarantee. Experiments on ALFWorld, WebShop and Search-QA (Qwen2.5-3B/7B) show gains over GRPO and recent self-distillation baselines, and ablations suggest an online planner beats static or hindsight-skill contexts.

Critical strengths
  1. The method is simple, cheap (about 3% overhead, no extra rollouts) and free at inference. 2, The appendix is unusually honest about limits: the guarantee holds in logit space, not parameter space, and the KL/entropy terms are ignored. The implementation details (thresholds, edge cases, prompts) are thorough.
Critical weaknesses
  1. The key empirical claims are not statistically supported.
  • Everything rests on one run, with no seeds or confidence intervals, on a 140-task ALFWorld set where one episode is about 0.7 points.
  • The headline projection gain (86.7 → 90.0) is about 4–5 episodes.
  • Orthogonal projection versus simply zeroing conflicting gradients differs by 0.9 points (about 1 episode), which the authors themselves call indistinguishable.
  • The "+2.86 over 30B", "+3.57 hindsight skills" and "+6.43 online vs static" claims are all differences of 4–9 episodes. On Search-QA the margins are −0.3 (3B) and +0.7 (7B).
  1. Baseline comparisons are weak. Baseline numbers are copied from other papers, with different training budgets, prompts, seeds and no planner. Ablation rows are given as per-category numbers such as 99.0 or 80.2 that cannot be k/n for categories of about 20–35 tasks, so the evaluation protocol may differ from that of the baselines.
Questions for authors
  1. Please report multiple seeds with confidence intervals for the main and ablation results, and ‖gD‖/‖gR‖ and the distribution of cosines on conflicting turns.
  2. How does the projection compare with raw OPD at tuned λ, with token-level and episode-level projection, and with simple cosine gating?
  3. Is the 140-task ALFWorld set used for checkpoint selection? Why do per-category numbers not match integer counts?
Reviewer_YAvkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to combine outcome-based agentic reinforcement learning with planner-conditioned on-policy self-distillation. The current policy generates a plan from the initial observation, and a plan-conditioned teacher re-scores the student's own trajectories. ProjOPD constructs policy-gradient and reverse-KL gradients on a shared sparse logit support, projects conflicting auxiliary gradients at the environment-turn level, and implements the resulting direction through a detached surrogate. The planner and teacher are removed at inference. The paper evaluates Qwen2.5-3B and 7B on ALFWorld, WebShop, and Search-QA, with component and context-source ablations on 3B ALFWorld.

Critical strengths
  1. The sparse-logit projection and detached surrogate offer a practical way to control auxiliary supervision. Proposition 1 establishes a local geometric property, and the method adds no environment rollouts or inference-time planner. The training overhead reported in Appendix D.4 is modest in the measured setting.
  2. The component ablations provide useful evidence about the method. With teacher construction held fixed, Table 2 reports 90.0 for the full method, 86.7 for raw OPD, and 87.6 for conflict-drop. The context ablation also compares online plans, static plans, and hindsight skills. These comparisons help assess the contributions of gradient projection and planner conditioning.
Critical weaknesses
  1. The central experimental comparisons are difficult to reconstruct consistently. (a) Section 3.3 reports baseline results taken from the original papers without providing evidence that their experimental settings align with those used here. Comparisons across hardware or software environments remain unconvincing without evidence that these differences do not affect the reported metrics. (b) The seven displayed 7B Search-QA scores in Table 1 average to approximately 49.2, compared with the reported macro-average of 49.7. (c) Table 2 gives full/drop/raw scores of 90.0/87.6/86.7, but Section 3.4 describes the 87.6 versus 86.7 comparison as retaining versus dropping the orthogonal component. (d) Appendix C.2 specifies a 15-turn horizon for WebShop, while Table 7 lists 50 turns. This discrepancy directly affects the comparability of the reported results. These issues make the size and comparability of the claimed gains uncertain.
  2. The theory does not establish that gradient conflicts between RLVR and OPSD are necessarily harmful. Proposition 1 guarantees non-opposition to an unclipped grouped-logit proxy, but this local property does not imply improved task performance. RLVR uses coarse outcome rewards and faces credit-assignment limitations, so a conflicting auxiliary gradient could still provide useful supervision. The paper needs stronger justification for treating such conflicts as detrimental to learning.
  3. (Minor) (a) Figure 1 and Table 1 are crowded and difficult to read. Larger text and clearer separation of environments in Table 1 would improve presentation. (b) I suggest renaming ProjOPD to ProjOPSD, since Section 2.2 uses the same policy as teacher and student and thus studies on-policy self-distillation.
Questions for authors
  1. What evidence shows that the baseline results taken from prior papers are comparable to ProjOPD? Please clarify representative factors such as the number of training update steps and reward setting. If compute is limited, citing the original results is acceptable, but the authors should document any configuration differences. Reproducing the two strongest competitive baselines under the same configuration as ProjOPD would materially strengthen the comparison.
  2. Beyond the local guarantee in Proposition 1, what theoretical or empirical evidence supports treating RLVR–OPSD gradient conflicts as harmful to task learning? How does the analysis account for the possibility that a conflicting OPSD gradient provides useful supervision when RLVR's coarse outcome rewards lead to imperfect credit assignment?
  3. Please address the inconsistencies identified in Weakness 1(b)–(d) and provide the corresponding corrections or clarifications.
Reviewer_E7fROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes ProjOPD, an on-policy self-distillation method that resolves conflict between the gradient of the unbiased RL task and the biased self-distillation term. The method is applied to two LLMs, Qwen 2.5 3B and Qwen 2.5 7B and evaluated on three tasks / environments: ALFWorld, WebShop, and Search-QA. On a subset of the evaluation tasks, the method outperforms various baselines.

Critical strengths

The core insight that the gradients w.r.t RL and OPD can be conflicting is correct and noteworthy. Beyond this, I find it hard to identify key strengths in this work, which is why I am leaning towards recommending rejection, based on the following.

Critical weaknesses

The biggest weakness is, without a doubt, the clarity of the manuscript. Various symbols and concepts are not properly introduced or defined, which makes it very hard to follow the paper. At many points I had to resort to guessing what the authors mean exactly. I have several questions for the authors below, which originate from this lack of clarity.

The presentation of the results is underwhelming and lacks a proper analysis. The main result (lines 316 - 322) is a pure read off from table 1, instead of explaining why these results occur.

While the subsequent ablation offers some form of analysis, it seems to be only partially about ablating crucial components of the method: ProjOPD is introduced as a method for resolving gradient conflict, it is therefore confusing that the ablation focusses on teacher teacher construction and look-ahead planning.

Figure 1 is not readible on paper.

Questions for authors

Line 099: Is the multi-turn environment a partially observable Markov decision process?

Line 100: What is hth_t? A scalar? A vector? Please properly define the symbol.

Line 101: Similarly, what is yty_t exactly?

Figure 1: What exactly does the planner do? Does it always output todo lists and constraints as stated int he capture of the Figure? Is the planner another LLM? Is it the same LLM with a particular prompt?

Line 132: What is the executor? I think the notion of a planner and executor has not been introduced at this point?

Line 145: What does z=P(o0)z = \mathcal{P}(o_0) mean? Is z a vector? If yes from what space? What dimensionality?

Line 171: What exactly is St,l\mathcal{S}_{t,l}? What is res\\{res\\}?

Line 187: What is gRg_R? It is an undefined symbol at this point in the manuscript.

Equation 6: What is the function AA?

Can you please help me parse the sentence starting on line 6 and ending on 209?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.