acceptodds
Under review as a conference paper at ICLR 2027

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic Reinforcement Learning

Abstract

Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same privileged information uniformly to every step, ignoring an asymmetry between steps: routine steps need little additional guidance, while critical error steps need explicit corrective direction. We propose AHEAD, a step-aware self-distillation method that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method modifies only the GRPO advantage. On ALFWorld, WebShop, and Search-based QA at three model scales, AHEAD improves over GRPO ( points on ALFWorld and on WebShop success rate at 7B) and over self-distillation baselines on ALFWorld at 7B and 1.7B. It also reaches a given success rate in fewer training steps and solves tasks in fewer interaction steps than GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, LLM Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FKhJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents AHEAD, a step-aware self-distillation method for RL training of multi-turn LLM agents. For each failed trajectory, an external LLM analyzer (Claude Opus 4.7) compares it with a successful rollout from the same group, marks up to five error steps, and writes a corrective hint for each. The policy then rescores its own tokens with the next environment observation appended at every step and the hint added at error steps. The resulting log-probability gap reweights the negative GRPO advantage by at most 10% during the first 50 training steps. On ALFWorld, AHEAD raises success rate over GRPO from 76.6 to 93.8 at 7B and from 43.0 to 65.6 at 1.7B, and it outperforms the self-distillation baselines RLSD, SDAR and Skill-SD at these two scales.

Critical strengths
  1. The paper is clearly written, and the idea of giving the teacher environment feedback at every step and a corrective hint only at error steps is simple and well motivated.

  2. Against GRPO rerun in the same codebase, the gains on ALFWorld are large and hold on the unseen split, which is not used for any selection.

Critical weaknesses
  1. The novelty is limited. AHEAD is on-policy self-distillation whose teacher context adds the next observation and LLM-written hints, and its advantage reweighting follows RLSD [1] directly. Using privileged feedback to credit individual agent steps is also not new. LEAP [2] corrects student trajectories with a privileged LLM on the same ALFWorld and WebShop benchmarks, and the paper itself calls SWEET-RL [3] the closest work, yet compares against neither. The remaining contribution, deciding which privileged information to show at which step, is incremental.

  2. Hard to scale. AHEAD calls an external LLM API (Claude Opus 4.7) on failed trajectories during training, with the full failed trajectory and a successful rollout in every call. Since RL usually needs many prompts and rollouts, this cost grows quickly with the training scale. The paper reports only call counts for the first 50 of 150 training steps (1.3k to 2.1k per run, Table 10), and gives no API cost or wall-clock time. A self-hosted Qwen3.6-27B analyzer performs close to Opus 4.7 (64.8 vs. 65.6, Table 2), but its compute cost is not reported either. It is therefore unclear whether AHEAD remains practical at a larger scale.

[1] Yang C, Qin C, Si Q, et al. Self-distilled RLVR[J]. arXiv preprint arXiv:2604.03128, 2026.

[2] Choudhury S, Sodhi P. Better than your teacher: LLM agents that learn from privileged AI feedback[C]//International Conference on Learning Representations. 2025.

[3] Zhou Y, Jiang S, Tian Y, et al. SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks[J]. arXiv preprint arXiv:2503.15478, 2025.

Questions for authors
  1. Can you add GiGPO [1], LEAP [2] and SWEET-RL [3] as baselines, together with the LLM-free control that uses the successful rollout directly as the teacher's privileged information?

  2. Can you report the API cost and wall-clock overhead of the analyzer and the teacher pass relative to GRPO, beyond the 1.3k to 2.1k analyzer calls per run in Table 10?

  3. Why are erroneous or redundant steps in successful trajectories left untreated, given that the "all trajectories" ablation gives them no analyzer hints and the reweight can never change the sign of the advantage?

[1] Feng L, Xue Z, Liu T, et al. Group-in-group policy optimization for LLM agent training[J]. Advances in Neural Information Processing Systems, 2025, 38.

[2] Choudhury S, Sodhi P. Better than your teacher: LLM agents that learn from privileged AI feedback[C]//International Conference on Learning Representations. 2025.

[3] Zhou Y, Jiang S, Tian Y, et al. SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks[J]. arXiv preprint arXiv:2503.15478, 2025.

Reviewer_6HBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AHEAD extends RLSD-style advantage reweighting by using step-aware privileged information and restricting reweighting to failed trajectories. Environment feedback is supplied to a self-teacher at all steps, with analyzer-generated corrective hints at selected errors. A bounded log-probability-based multiplier adjusts GRPO update magnitudes while preserving their signs, and decays to zero intervention early in training. Experiments cover ALFWorld, WebShop, and search-based QA across three model scales, with component ablations, analyzer substitutions, and held-out room-layout evaluation.

Critical strengths
  1. A simple intervention addresses an important bottleneck. The algorithm changes the advantage rather than adding a separate training objective. Successful trajectories remain untouched, and inference does not require the analyzer or privileged context.
  2. The main ALFWorld evidence is strong. The 7B and 1.7B improvements persist on Unseen-134, which uses the checkpoint selected on Val-128. Appendix F.1 therefore answers the concern that the method is evaluated only on a seen validation subset.
  3. The paper includes unusually relevant controls. Tables 2 and 3 test alternate analyzers and regenerate baseline skills with the same analyzer model. Table 4 ablates components, while Appendix F.4 crosses PI source with failed-only versus all-trajectory application. These controls materially strengthen the case beyond merely comparing with GRPO.
  4. The scope of novelty is candid. The paper explicitly retains RLSD's reweighting mechanism and changes what the teacher sees and where the signal applies. This is clearer and more defensible than presenting the weighting formula as a new optimizer.
Critical weaknesses

W1. The main table contains concrete protocol discrepancies, beyond an unspecified lack of reruns. Appendix C specifies fixed category counts of 31/12/24/15/23/23 on Val-128, with one episode per task. Several inherited SDAR rates cannot occur under those denominators: 94.7% Pick at 7B lies between 29/31=93.5% and 30/31=96.8%; 68.2% Cool and 78.9% Pick2 are also incompatible with their stated counts of 23. The AHEAD and rerun-GRPO rows pass the same integer-count check. Borrowing results is disclosed in Appendix D, so this does not imply fabrication, but Table 1 cannot be interpreted throughout as a matched fixed-set comparison. AHEAD Table 5 also lists 2,400 WebShop training tasks, while the cited SDAR experiment reports 1,000. Please reconcile this documented difference before attributing the WebShop gap to the method. See SDAR, Section 3.

W2. The central alternative explanation is not tested. Every hint that affects training is generated using a successful rollout from the same group. Appendix A itself identifies direct use of that successful rollout as teacher PI, without an analyzer, as an untested control. Holding the failed-only weighting, environment feedback, and schedule fixed, this is the most direct way to ask whether explicit error localization adds value beyond access to richer task-specific supervision. The AUROC diagnostic in F.3 is not a substitute: error labels select the steps receiving richer PI, and the resulting larger gap is then used to recover those same labels. It establishes a differentiated signal, not independent correctness of the error localization.

W3. The strongest existing reruns and the small intervention require a clearer uncertainty account. Table 3 is valuable, but it changes baseline skills as well as rerunning methods. SDAR falls from 85.9 to 73.4 at 7B and from 53.9 to 30.5 at 1.7B; Skill-SD falls from 52.3 to 25.0 at 1.7B. These changes do not establish an implementation error, but prevent treating the newer-generator rerun as a clean, uniformly strengthened version of the original comparator. A matched closest-method run with original skills, fixed task IDs, and the same unseen evaluation is needed. The [0.9, 1.1] intervention ending at step 50 can plausibly have large downstream effects; a focused paired repeat would provide evidence about whether the decisive margin is repeatable. Repeated selection of checkpoint maxima on Val-128 does not supply that uncertainty estimate.

W4. Sample-efficiency and stability interpretations need narrower wording. Figure 3 counts optimizer steps, not elapsed time or total compute. Table 10 already reports the number of analyzer calls, and Appendix H.3 already supplies the assembled teacher context; these details should be credited. Remaining costs include analyzer input/output tokens and PI scoring, for which end-to-end measurements are not reported. Sign preservation bounds the per-token multiplier but does not prove stable optimization or correct causal credit. Similarly, the all-failure-group limitation is already acknowledged in Appendix A; it limits the method's cold-start scope rather than constituting an undisclosed flaw.

Questions for authors
  1. Which task IDs, category denominators, and training sets produced the inherited rows? How should the impossible-under-Val-128 category fractions and the WebShop training-task difference be reconciled?

  2. How does AHEAD compare with a closest self-distillation baseline rerun with its original skills under the fixed Val-128/Unseen-134 protocol? What explains the large Table 1-to-3 shifts?

  3. Under the same weighting and schedule, does direct use of the already available same-group successful rollout match the analyzer-generated hints? The paper already identifies this control as open.

  4. Can a focused paired repeat establish the main matched margin, and can measured token/elapsed-time overhead supplement the analyzer counts already provided?

Reviewer_8Yg3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AHEAD, a self-distillation method for multi-turn agentic RL that uses environment feedback and LLM-generated corrective hints to reweight GRPO advantages on failed trajectories. The hints use successful same-group rollouts as references, while the bounded reweighting decays during training and requires no privileged information or external analyzer at inference.

Critical strengths

AHEAD uses environment feedback and targeted hints to refine training updates without adding inference-time overhead, making the approach practical at deployment. Improvements over GRPO on ALFWorld and WebShop, including unseen ALFWorld layouts, provide evidence of useful learning gains beyond the checkpoint-selection set.

Critical weaknesses
  1. Every effective hint uses a successful same-group rollout, but the missing direct-reference PI baseline leaves the added value of LLM-based diagnosis and correction insufficiently isolated.
  2. Unseen-layout baseline comparisons cover only GRPO, while comparisons with stronger distillation methods use best Val-128 checkpoints without repeated-seed results, limiting evidence for robust superiority.
Questions for authors
  1. Could the successful same-group rollout be used directly as teacher PI, without an analyzer, keeping environment feedback, failure-only reweighting, and the decay schedule unchanged?
  2. Could AHEAD and the strongest same-generator baselines be compared on Unseen-134 using Val-128-selected checkpoints? For a representative model scale, please also report results across training seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.