acceptodds
Under review as a conference paper at ICLR 2027

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5–15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, on-policy self-distillation, gradient projection, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7DorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses gradient conflict between GRPO and on-policy distillation in agent training. CORD keeps one gradient intact and projects the other onto its orthogonal complement at each optimizer mini-batch. The method is evaluated as a drop-in modification of three existing joint training recipes on ALFWorld and WebShop, with additional analysis of protection direction, gating, mixing coefficient, and aggregation choices.

Critical strengths

The diagnostic chain is complete and well presented, connecting measured gradient conflict to lost first-order progress. The ablation design is systematic, negative results are reported honestly, and the configuration-level findings, such as the need to choose protection direction and gating jointly, are practically useful for practitioners working on joint agent training.

Critical weaknesses

W1: Limited novelty, and the core assumption behind the method is never examined. Conflict-aware optimization is a well-established technique in multi-task learning, and PCGrad already demonstrated gradient surgery in multi-task RL back in 2020. The core update rule here is geometrically equivalent to A-GEM and PCGrad, which the authors acknowledge. The contribution is therefore best described as a transfer of an existing technique to a new setting. Such a transfer can be valuable, but the paper never completes its most important step, namely checking whether the prerequisites of the technique hold in RL. In supervised multi-task learning, gradients are exact gradients of real losses, so a conflict test is a reliable statement about the objectives. In RL this is fundamentally different. A GRPO gradient is a high-variance estimate from a handful of sampled trajectories, and the sign of the inner product in a single mini-batch may reflect nothing but sampling noise. This raises three issues the paper never discusses.

  1. If the conflict test is driven by noise, the projection removes a useful component of the teacher signal along a random direction while protecting a direction that did not deserve protection. The paper reports no variance of the conflict statistic and no estimate of the false projection rate.

  2. Each projection introduces a one-sided bias on the distillation objective, and these biases accumulate over 150 iterations into a systematic drift. Proposition 1 only guarantees first-order progress on the sampled surrogate objective, not on expected return.

  3. The distillation here is self-distillation, so both signals are model-generated and neither carries ground truth. Protecting RL by default encodes the assumption that task reward is more trustworthy than teacher guidance, an assumption that only holds when rewards are dense and informative. The paper never tests where this assumption breaks.

In short, the paper shows the method happens to work on two clean benchmarks. It does not show the method is sound for RL.

W2: The experimental evidence is not strong enough to support the claims.

  1. All training comparisons use a single training seed, and four evaluation passes only capture decoding variance, which cannot rule out training noise, and this is exactly what the largest gain of 15.9 points would need.

  2. The method is extremely sensitive to configuration, with a 24-point gap between switch settings of the same method, a fragility consistent with performing precision surgery on noisy gradients that also undermines the plug-and-play claim.

  3. On WebShop the GRPO+OPSD recipe gets worse on both metrics after adding CORD, and the authors report this honestly but offer no diagnosis.

  4. The only gradient-level baseline is PCGrad, which is also a projection-based method and therefore the most similar to CORD. Other families of multi-objective optimization methods, as well as alternative strategies for combining RL and supervised signals that do not operate on gradients directly, are not compared against, which weakens the evidence that the observed gains come from the proposed combination rule itself.

Questions for authors

See Weaknesses W1 and W2. I would be willing to raise my score if the authors can address these concerns.

Reviewer_BfyzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CORD, a conflict-aware gradient combination framework for jointly optimizing reinforcement learning (RL) and on-policy distillation (OPD) in LLM agents. By identifying and mitigating gradient conflicts through objective-preserving projection, CORD provides a flexible mechanism supported by theoretical analysis. Experiments on ALFWorld and WebShop, integrating CORD into three existing training methods (Skill-SD, GRPO+OPSD, and SDAR), demonstrate performance improvements across several settings, with additional analyses examining key design choices and their effects.

Critical strengths

1、The paper addresses gradient conflicts between RL and distillation, an important issue in jointly optimizing multiple learning objectives.

2、The projection mechanism is formally characterized through a constrained optimization formulation, with a closed-form solution and first-order analysis of the protected objective.

3、CORD introduces a straightforward gradient combination mechanism that can be integrated into existing joint training frameworks without changing their original learning objectives.

Critical weaknesses

1、The methodological novelty relative to existing gradient projection methods is limited.

2、The interpretation of the largest performance improvement requires further clarification.

3、The relationship between gradient conflict, practical parameter updates, and performance improvements could be better established.

Questions for authors
  1. The core projection mechanism in CORD closely resembles the asymmetric gradient projection used in A-GEM. Although the manuscript discusses its differences from A-GEM and PCGrad, the methodological novelty beyond selecting the protected objective and projection gate remains unclear. A clearer explanation of CORD's distinct contributions relative to existing gradient projection methods would strengthen its methodological justification.

  2. The largest performance improvement is observed for GRPO+OPSD on ALFWorld (+15.9 percentage points). However, Appendix B.2 indicates that the gradient of its k1 auxiliary objective is independent of teacher probabilities when teacher scores are detached and token weights are uniform. This raises an important question regarding the actual role of teacher guidance in this configuration and its contribution to the observed improvement.

  3. Figure 1 illustrates the existence of gradient conflicts, but the diagnostics use a different model configuration from the main experiments. It would be informative to examine the gradient conflicts in the three joint training methods evaluated in Table 1 and investigate whether differences in conflict severity help explain the performance improvements achieved by CORD.

  4. CORD requires separate gradient computations for the RL and distillation objectives, introducing additional computational overhead. While Appendix C.6 reports actor-update overhead and estimates the increase in overall training time, direct end-to-end runtime comparisons are still lacking. More importantly, whether CORD maintains its performance advantages under matched computational budgets remains an open question.

  5. The experimental results reveal substantial sensitivity to the choice of protection direction and projection gate, with certain configurations even leading to notable performance degradation. This raises concerns about the robustness and practical applicability of CORD. Clearer selection principles or practical guidelines for configuring CORD on previously unseen tasks would help address these concerns.

1 comment

Thank you for the detailed and thoughtful responses. I appreciate the additional theoretical analysis, experimental results, and clarifications, particularly regarding the gradient interaction between RL and OPD and the computational overhead.

My main remaining concern is about the methodological novelty of CORD. As clarified in the response, its core projection operator is shared with A-GEM, and the primary distinction lies in the choice of the protected gradient and its application to joint RL-OPD training. While I agree that this is a meaningful adaptation to the considered setting, I am not yet fully convinced that the proposed formulation establishes a sufficiently distinct algorithmic contribution beyond existing gradient projection approaches.

Overall, the rebuttal has addressed several of my concerns and improved my understanding of the method. However, this remaining concern about the methodological novelty prevents me from changing my original score. I appreciate the authors' efforts and believe the work provides useful insights into conflict-aware optimization for joint RL and distillation.

Reviewer_fxcqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies joint reinforcement learning and online self-distillation for language-model agents. GRPO provides task-reward signals, while online self-distillation provides dense token-level guidance. The authors observe that the gradients from these two objectives can conflict, reducing the local optimization progress of the RL objective. The paper proposes CORD (Conflict-Aware Optimization for Joint RL and Distillation). Given the RL gradient u and the weighted distillation gradient v=λ∇LOPDv=\lambda\nabla L_{OPD}, CORD selects one gradient to protect and projects the other gradient, reducing the local optimization progress of the RL objective. The paper analyzes the effects of the distillation coefficient, token-level loss, and gradient aggregation. Experiments are conducted with Qwen2.5-3B-Instruct on ALFWorld and WebShop, applying CORD to three joint-training recipes: Skill-SD, GRPO+OPSD, and SDAR. The results show improvements in most tested combinations.

Critical strengths
  1. The paper addresses gradient conflicts between sparse task-reward signals and dense self-distillation signals in agent reinforcement learning, which is a natural issue in joint training. 2.The experiments cover protection direction, projection gate, distillation loss, mixing coefficient, gradient aggregation, and PCGrad, showing that CORD depends on the specific optimization configuration rather than being universally effective.
Critical weaknesses

1.The method has limited novelty. The core operation in CORD is closely related to the single-half-space projection used in A-GEM and the gradient modification used in PCGrad. The paper does not demonstrate clearly different theoretical properties or optimization guarantees compared with existing methods.

2.The main performance experiments use only one training seed, making it difficult to support claims about training-level stability.

Questions for authors

1.Could the authors provide results from at least 3–5 independent training seeds for the main Base–CORD comparisons, together with confidence intervals and statistical tests at the training-seed level?

2.Could the authors evaluate CORD across more model scales, different teacher constructions, dynamic skill banks, and out-of-distribution tasks to support its positioning as a general joint-optimization framework?

3.Could the authors provide a clearer explanation of the interaction among the protection direction, projection gate, and \lambda? At present, these differences are mainly demonstrated empirically, without a unified theoretical explanation or testable mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.