- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses gradient conflict between GRPO and on-policy distillation in agent training. CORD keeps one gradient intact and projects the other onto its orthogonal complement at each optimizer mini-batch. The method is evaluated as a drop-in modification of three existing joint training recipes on ALFWorld and WebShop, with additional analysis of protection direction, gating, mixing coefficient, and aggregation choices.
Critical strengths
The diagnostic chain is complete and well presented, connecting measured gradient conflict to lost first-order progress. The ablation design is systematic, negative results are reported honestly, and the configuration-level findings, such as the need to choose protection direction and gating jointly, are practically useful for practitioners working on joint agent training.
Critical weaknesses
W1: Limited novelty, and the core assumption behind the method is never examined. Conflict-aware optimization is a well-established technique in multi-task learning, and PCGrad already demonstrated gradient surgery in multi-task RL back in 2020. The core update rule here is geometrically equivalent to A-GEM and PCGrad, which the authors acknowledge. The contribution is therefore best described as a transfer of an existing technique to a new setting. Such a transfer can be valuable, but the paper never completes its most important step, namely checking whether the prerequisites of the technique hold in RL. In supervised multi-task learning, gradients are exact gradients of real losses, so a conflict test is a reliable statement about the objectives. In RL this is fundamentally different. A GRPO gradient is a high-variance estimate from a handful of sampled trajectories, and the sign of the inner product in a single mini-batch may reflect nothing but sampling noise. This raises three issues the paper never discusses.
-
If the conflict test is driven by noise, the projection removes a useful component of the teacher signal along a random direction while protecting a direction that did not deserve protection. The paper reports no variance of the conflict statistic and no estimate of the false projection rate.
-
Each projection introduces a one-sided bias on the distillation objective, and these biases accumulate over 150 iterations into a systematic drift. Proposition 1 only guarantees first-order progress on the sampled surrogate objective, not on expected return.
-
The distillation here is self-distillation, so both signals are model-generated and neither carries ground truth. Protecting RL by default encodes the assumption that task reward is more trustworthy than teacher guidance, an assumption that only holds when rewards are dense and informative. The paper never tests where this assumption breaks.
In short, the paper shows the method happens to work on two clean benchmarks. It does not show the method is sound for RL.
W2: The experimental evidence is not strong enough to support the claims.
-
All training comparisons use a single training seed, and four evaluation passes only capture decoding variance, which cannot rule out training noise, and this is exactly what the largest gain of 15.9 points would need.
-
The method is extremely sensitive to configuration, with a 24-point gap between switch settings of the same method, a fragility consistent with performing precision surgery on noisy gradients that also undermines the plug-and-play claim.
-
On WebShop the GRPO+OPSD recipe gets worse on both metrics after adding CORD, and the authors report this honestly but offer no diagnosis.
-
The only gradient-level baseline is PCGrad, which is also a projection-based method and therefore the most similar to CORD. Other families of multi-objective optimization methods, as well as alternative strategies for combining RL and supervised signals that do not operate on gradients directly, are not compared against, which weakens the evidence that the observed gains come from the proposed combination rule itself.
Questions for authors
See Weaknesses W1 and W2. I would be willing to raise my score if the authors can address these concerns.
Thank you for the detailed and thoughtful responses. I appreciate the additional theoretical analysis, experimental results, and clarifications, particularly regarding the gradient interaction between RL and OPD and the computational overhead.
My main remaining concern is about the methodological novelty of CORD. As clarified in the response, its core projection operator is shared with A-GEM, and the primary distinction lies in the choice of the protected gradient and its application to joint RL-OPD training. While I agree that this is a meaningful adaptation to the considered setting, I am not yet fully convinced that the proposed formulation establishes a sufficiently distinct algorithmic contribution beyond existing gradient projection approaches.
Overall, the rebuttal has addressed several of my concerns and improved my understanding of the method. However, this remaining concern about the methodological novelty prevents me from changing my original score. I appreciate the authors' efforts and believe the work provides useful insights into conflict-aware optimization for joint RL and distillation.