- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents AHEAD, a step-aware self-distillation method for RL training of multi-turn LLM agents. For each failed trajectory, an external LLM analyzer (Claude Opus 4.7) compares it with a successful rollout from the same group, marks up to five error steps, and writes a corrective hint for each. The policy then rescores its own tokens with the next environment observation appended at every step and the hint added at error steps. The resulting log-probability gap reweights the negative GRPO advantage by at most 10% during the first 50 training steps. On ALFWorld, AHEAD raises success rate over GRPO from 76.6 to 93.8 at 7B and from 43.0 to 65.6 at 1.7B, and it outperforms the self-distillation baselines RLSD, SDAR and Skill-SD at these two scales.
Critical strengths
-
The paper is clearly written, and the idea of giving the teacher environment feedback at every step and a corrective hint only at error steps is simple and well motivated.
-
Against GRPO rerun in the same codebase, the gains on ALFWorld are large and hold on the unseen split, which is not used for any selection.
Critical weaknesses
-
The novelty is limited. AHEAD is on-policy self-distillation whose teacher context adds the next observation and LLM-written hints, and its advantage reweighting follows RLSD [1] directly. Using privileged feedback to credit individual agent steps is also not new. LEAP [2] corrects student trajectories with a privileged LLM on the same ALFWorld and WebShop benchmarks, and the paper itself calls SWEET-RL [3] the closest work, yet compares against neither. The remaining contribution, deciding which privileged information to show at which step, is incremental.
-
Hard to scale. AHEAD calls an external LLM API (Claude Opus 4.7) on failed trajectories during training, with the full failed trajectory and a successful rollout in every call. Since RL usually needs many prompts and rollouts, this cost grows quickly with the training scale. The paper reports only call counts for the first 50 of 150 training steps (1.3k to 2.1k per run, Table 10), and gives no API cost or wall-clock time. A self-hosted Qwen3.6-27B analyzer performs close to Opus 4.7 (64.8 vs. 65.6, Table 2), but its compute cost is not reported either. It is therefore unclear whether AHEAD remains practical at a larger scale.
[1] Yang C, Qin C, Si Q, et al. Self-distilled RLVR[J]. arXiv preprint arXiv:2604.03128, 2026.
[2] Choudhury S, Sodhi P. Better than your teacher: LLM agents that learn from privileged AI feedback[C]//International Conference on Learning Representations. 2025.
[3] Zhou Y, Jiang S, Tian Y, et al. SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks[J]. arXiv preprint arXiv:2503.15478, 2025.
Questions for authors
-
Can you add GiGPO [1], LEAP [2] and SWEET-RL [3] as baselines, together with the LLM-free control that uses the successful rollout directly as the teacher's privileged information?
-
Can you report the API cost and wall-clock overhead of the analyzer and the teacher pass relative to GRPO, beyond the 1.3k to 2.1k analyzer calls per run in Table 10?
-
Why are erroneous or redundant steps in successful trajectories left untreated, given that the "all trajectories" ablation gives them no analyzer hints and the reweight can never change the sign of the advantage?
[1] Feng L, Xue Z, Liu T, et al. Group-in-group policy optimization for LLM agent training[J]. Advances in Neural Information Processing Systems, 2025, 38.
[2] Choudhury S, Sodhi P. Better than your teacher: LLM agents that learn from privileged AI feedback[C]//International Conference on Learning Representations. 2025.
[3] Zhou Y, Jiang S, Tian Y, et al. SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks[J]. arXiv preprint arXiv:2503.15478, 2025.