- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies agentic RL with a sparse outcome reward, supplemented by on-policy self-distillation (OPD). The teacher is the same policy with a self-generated "plan context" (constraints plus a to-do list) prepended to the prompt, and it re-scores tokens the student already sampled. The authors argue the OPD gradient can have a negative inner product with the GRPO gradient and so cancel the reward-driven update. Their fix, ProjOPD, works on a sparse top-64 + sampled-token + residual support in logit space. For each environment turn, it removes the component of the OPD gradient that opposes the policy gradient. Proposition 1 gives a first-order non-interference guarantee. Experiments on ALFWorld, WebShop and Search-QA (Qwen2.5-3B/7B) show gains over GRPO and recent self-distillation baselines, and ablations suggest an online planner beats static or hindsight-skill contexts.
Critical strengths
- The method is simple, cheap (about 3% overhead, no extra rollouts) and free at inference. 2, The appendix is unusually honest about limits: the guarantee holds in logit space, not parameter space, and the KL/entropy terms are ignored. The implementation details (thresholds, edge cases, prompts) are thorough.
Critical weaknesses
- The key empirical claims are not statistically supported.
- Everything rests on one run, with no seeds or confidence intervals, on a 140-task ALFWorld set where one episode is about 0.7 points.
- The headline projection gain (86.7 → 90.0) is about 4–5 episodes.
- Orthogonal projection versus simply zeroing conflicting gradients differs by 0.9 points (about 1 episode), which the authors themselves call indistinguishable.
- The "+2.86 over 30B", "+3.57 hindsight skills" and "+6.43 online vs static" claims are all differences of 4–9 episodes. On Search-QA the margins are −0.3 (3B) and +0.7 (7B).
- Baseline comparisons are weak. Baseline numbers are copied from other papers, with different training budgets, prompts, seeds and no planner. Ablation rows are given as per-category numbers such as 99.0 or 80.2 that cannot be k/n for categories of about 20–35 tasks, so the evaluation protocol may differ from that of the baselines.
Questions for authors
- Please report multiple seeds with confidence intervals for the main and ablation results, and ‖gD‖/‖gR‖ and the distribution of cosines on conflicting turns.
- How does the projection compare with raw OPD at tuned λ, with token-level and episode-level projection, and with simple cosine gating?
- Is the 140-task ALFWorld set used for checkpoint selection? Why do per-category numbers not match integer counts?