acceptodds
Under review as a conference paper at ICLR 2027

Learning from Decision Contrasts: History-Aligned Self-Distillation for Agentic Reinforcement Learning

Abstract

Agentic on-policy self-distillation (OPSD) mitigates coarse credit assignment in outcome-based reinforcement learning (RL) by using privileged information (PI) at the trajectory or step level to guide self-teacher rescoring. However, both levels have limitations: trajectory-level supervision can weaken as student response prefixes grow, while step-level PI may be inappropriate for the target turn. Our key observation is that turns with similar interaction histories exhibit substantial contrasts in actions and outcomes, providing a natural basis for localized and informative PI construction. We propose History-Aligned Self-Distillation (HASD), a unified framework that leverages history-aligned decision contrasts to provide fine-grained and contextually appropriate OPSD supervision. HASD first groups turns by their recent observation histories and aggregates empirical action outcomes to identify those with the highest and lowest outcomes. It then constructs two PI views from these action sets with opposite high/low label assignments to produce comparative OPSD signals that mitigate the negative tendency of one-sided signals. Finally, because the comparative signal is not directly verified against task outcomes, an agreement gate retains it only when its turn-level ordering agrees with GRPO advantages. Across 1.7B, 3B, and 7B models, HASD ranks first in seven of nine settings and exceeds the strongest baselines by up to 14.1, 9.3, and 1.7 points on ALFWorld, WebShop, and Search-QA, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agentic RL, On-Policy Self-Distillation, History Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gEvROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HASD, an auxiliary self-distillation loss for GRPO-based agent training. Turns from different rollouts of the same task with similar recent histories are grouped, and the actions in each group are split into high- and low-outcome sets based on episode success. The current policy then acts as a self-teacher: it scores the student’s responses under two prompts that list these actions with opposite HIGHER/LOWER labels, and the difference of the two log-probabilities serves as a token-level signal. A gate keeps the signal only where it agrees with the GRPO advantages. Experiments on ALFWorld, WebShop and Search-QA with 1.7B–7B models show consistent gains over GRPO and recent self-distillation baselines, especially for smaller models.

Critical strengths
  1. The paper is well written and easy to follow. The motivation, method and design choices are clearly explained.
  2. The empirical gains are substantial, especially on ALFWorld and WebShop with smaller backbones (+10–12 points over the strongest of GRPO/SERL at 1.7B), and they hold over three training seeds (Table 7).
  3. The design choices are well motivated and supported by analysis. In particular, the paper gives a simple theoretical argument for why one-sided self-distillation signals are biased and shows empirically that the comparative construction removes this bias. The analysis of the history window also provides useful insight into the method’s behavior.
Critical weaknesses
  1. The multi-seed study (Appendix E.4, Table 7) is a welcome addition. However, it covers only GRPO and SERL and excludes Search-QA. The strongest competitors in Table 1, SDAR and StepOPSD, are reported from a single run. On Search-QA, HASD’s margins are small (0.5–1.7 points at 1.7B/3B), and at 7B it is slightly below SDAR and RLSD. Repeating these comparisons with multiple seeds (at least for SDAR and StepOPSD, including Search-QA) would clarify which improvements are robust. It would also help to vary evaluation-time sampling (evaluation seeds, temperature), since currently only the training seed is varied while the evaluation configuration is fixed.
  2. The paper does not state which baseline numbers were reproduced by the authors and which were taken from the original papers (the caption of Table 1 suggests some are taken from prior work). Taking results from prior work is reasonable, but marking them explicitly, along with the evaluation protocol used for reproduced runs, would make the comparison clearer. Relatedly, ϵH\epsilon_H and ϵC\epsilon_C are selected on the same Search-QA split used to report final results (Table 6); clarifying this, or using a held-out split, would be helpful.
  3. The method is interesting, but it remains unclear how it extends to more complex benchmarks, where recurring states are harder to identify, and to larger backbones.
Questions for authors
  1. Could the authors report multi-seed results (mean ± std or confidence intervals) on Search-QA, and for the competitors closest in performance (SDAR and StepOPSD) on all three environments? Varying evaluation-time sampling (evaluation seeds, temperature) in addition to the training seed would also be informative.
  2. GRPO and HASD are trained for the same 150 steps and compared at the final checkpoint. Could the authors provide training curves for both methods over a longer horizon (e.g., 300+ steps)? This would show whether GRPO simply converges more slowly and eventually closes the gap, or whether HASD reaches a higher final performance.
  3. Fig. 4 analyzes the history window on ALFWorld with Qwen3-1.7B. Could the authors provide the same analysis for other backbone–environment combinations?
  4. RLCSD (arXiv 2606.11709) contrasts teacher–student gaps under correct and incorrect hints to cancel hint-induced shifts, which is closely related to HASD’s comparative PI construction. I encourage the authors to cite it and discuss the key differences (e.g., label reversal over the same action evidence vs. an incorrect hint).
1 comment
Reviewer_gEvROpenReview ↗

Response to Rebuttal

I thank the authors for the thorough rebuttal. The additional results address most of my concerns. While the new experiments do not establish that HASD outperforms GRPO on every benchmark (e.g., on ALFWorld the gap narrows with longer training), they clearly show that HASD converges faster. The added multi-seed comparisons, the new benchmark and backbone results, and the other clarifications make the paper considerably stronger and more informative. I am therefore raising my score.

Reviewer_PvkUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposed a framework for improving local supervision in fine-tuning multi-turn LLM agents using on-policy distillation, named History-Aligned Self-Distillation (HASD). It works by first grouping decision turns with similar recent observation histories and identifying actions associated with the highest and lowest empirical episode outcomes. It then constructs two privileged-information views with reversed high/low action labels, using the difference between their self-teacher scores as a comparative signal for distillation. They evaluate the proposed method on a variety of benchmarks including ALFWorld, WebShop, and Search-QA across three model backbones.

Critical strengths
  • The idea of grouping actions based on similar interaction histories is an intuitive way to make privileged information more relevant when constructing local supervision
  • extensive benchmark results across several settings show superior performance gains, showing the effectiveness of the proposed components
Critical weaknesses
  • The proposed method uses the final task success or failure to estimate how good an action is, but an episode or a trajectory's outcome depends on many actions, not necessarily the action being evaluated. Thus, it raises a question about how reliably the constructed higher/lower label distinguishes better and worse local actions, rather than the actions that happened to occur in successful and failed episodes.

  • The paper's clarity in presenting core technical concepts can be enhanced. E.g., it is difficult to understand what the actual constructed PI looks like in previous related work until I read the appendix, which makes it clear PI construction involves a designed prompt. Another example is "negative tendency of one-sided signals", which is presented as a limitation this work aims to overcome. But its meaning is not clearly explained when first introduced. Also, the important implementation details behind eq 4 are deferred to the appendix, and the final paragraph in sec 4.1 is hard to immediately understand how G' is obtained from G. Also, the response prefixes discussed in fig 1a are not clarified: Does it accumulate across turns or refer to tokens within a turn? Also, the contribution can be made clearer in the intro; The last para in the intro reads like a restatement of the results.

  • The paper argues that one-sided OPSD signals tend to have a negative mean; however, a negative avg does not automatically mean the student receives incorrect supervision, so why might having a negative avg score be harmful for learning? Furthermore, fig 5 compares comparative rescoring with one-sided rescoring; I think another useful baseline would be simply subtracting the mean from the one-sided scores before using them for training (maybe with some scaling?). That would remove the negative mean without introducing a second PI-conditioned teacher view. Knowing this may help better confirm whether the comparative construction is genuinely useful.

  • (minor) the constructed PI describes which actions were associated with better or worse outcomes, but eq 10 applies the resulting supervision to every generated response token (including reasoning), so it naturally raises the question why the relative preference between two actions needs to provide supervision for all tokens, rather than just supervising non-action tokens. I think such a comparison would be informative to examine where the major benefit comes from.

I'm glad to raise my rate when concerns are addressed.

Questions for authors
  • it would be useful to compare the proposed agreement gate with randomly retaining the same proportion of eligible groups, to distinguish the benefit of the used gating from that of simply applying less auxiliary supervision, in addition to the grpo objective.
  • In Sec 4.1, it uses a representative turn to decide group assignment, but why not require all within-group turns to satisfy the similarity threshold? Could the authors examine whether stricter pairwise matching, or matching that also incorporates actions, would change the quality of the constructed PI as well as the final performance?
  • Since this work cares about supervision quality at individual decision turns, can the authors also compare against recent GRPO variants that also target credit assignment issues in long-horizon settings?
Reviewer_gPKQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between privileged information supervision and the current decision context in multi-turn LLM agent reinforcement learning. It proposes HASD, which groups decisions by their recent observation histories and aggregates success signals associated with actions to construct high and low outcome action sets. The auxiliary signals are retained only when their turn-level scores exhibit positive within-group agreement with GRPO advantages. Across three model backbones and benchmark settings, HASD ranks first among the baselines. The paper also provides ablations, leave-one-trajectory-out analysis, multi-seed evaluations, and training-cost analysis.

Critical strengths
  1. The paper addresses an important and practically relevant supervision problem in agent reinforcement learning. HASD combines observation-history matching, action–outcome evidence, and reward-based filtering to construct local supervision from existing on-policy trajectories, without requiring a separate teacher model. The algorithm and prompt templates are described in sufficient detail to make the approach understandable.
  2. The empirical results are convincing on ALFWorld and WebShop. HASD achieves substantial performance improvements on these benchmarks, while component ablations support the contributions of its individual designs. The three-seed evaluations and leave-one-trajectory-out experiments provide additional evidence that the gains are robust and are not solely attributable to target-trajectory evidence.
Critical weaknesses

1.The multi-seed study omits SDAR, which outperforms HASD in some main-table settings. A controlled comparison with related local credit-assignment methods, particularly GiGPO where setups overlap, would better establish the advantage of HASD. 2. HASD needs matched decisions with different actions and outcomes, so many turns receive no additional supervision. For eligible turns, episode success is only a noisy proxy for the contribution of an individual action. The paper does not show which contexts are excluded or when these action labels are reliable, limiting the scope of its claims. 3. HASD matches actions by normalized text, so function calls with different or equivalent arguments may be treated as unrelated. This could make their outcome estimates unreliable.

Questions for authors

When successful rollouts are rare or tool calls have mostly unique arguments, few turns may qualify for HASD’s extra supervision. Does HASD still improve over GRPO in these settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.