- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HASD, an auxiliary self-distillation loss for GRPO-based agent training. Turns from different rollouts of the same task with similar recent histories are grouped, and the actions in each group are split into high- and low-outcome sets based on episode success. The current policy then acts as a self-teacher: it scores the student’s responses under two prompts that list these actions with opposite HIGHER/LOWER labels, and the difference of the two log-probabilities serves as a token-level signal. A gate keeps the signal only where it agrees with the GRPO advantages. Experiments on ALFWorld, WebShop and Search-QA with 1.7B–7B models show consistent gains over GRPO and recent self-distillation baselines, especially for smaller models.
Critical strengths
- The paper is well written and easy to follow. The motivation, method and design choices are clearly explained.
- The empirical gains are substantial, especially on ALFWorld and WebShop with smaller backbones (+10–12 points over the strongest of GRPO/SERL at 1.7B), and they hold over three training seeds (Table 7).
- The design choices are well motivated and supported by analysis. In particular, the paper gives a simple theoretical argument for why one-sided self-distillation signals are biased and shows empirically that the comparative construction removes this bias. The analysis of the history window also provides useful insight into the method’s behavior.
Critical weaknesses
- The multi-seed study (Appendix E.4, Table 7) is a welcome addition. However, it covers only GRPO and SERL and excludes Search-QA. The strongest competitors in Table 1, SDAR and StepOPSD, are reported from a single run. On Search-QA, HASD’s margins are small (0.5–1.7 points at 1.7B/3B), and at 7B it is slightly below SDAR and RLSD. Repeating these comparisons with multiple seeds (at least for SDAR and StepOPSD, including Search-QA) would clarify which improvements are robust. It would also help to vary evaluation-time sampling (evaluation seeds, temperature), since currently only the training seed is varied while the evaluation configuration is fixed.
- The paper does not state which baseline numbers were reproduced by the authors and which were taken from the original papers (the caption of Table 1 suggests some are taken from prior work). Taking results from prior work is reasonable, but marking them explicitly, along with the evaluation protocol used for reproduced runs, would make the comparison clearer. Relatedly, and are selected on the same Search-QA split used to report final results (Table 6); clarifying this, or using a held-out split, would be helpful.
- The method is interesting, but it remains unclear how it extends to more complex benchmarks, where recurring states are harder to identify, and to larger backbones.
Questions for authors
- Could the authors report multi-seed results (mean ± std or confidence intervals) on Search-QA, and for the competitors closest in performance (SDAR and StepOPSD) on all three environments? Varying evaluation-time sampling (evaluation seeds, temperature) in addition to the training seed would also be informative.
- GRPO and HASD are trained for the same 150 steps and compared at the final checkpoint. Could the authors provide training curves for both methods over a longer horizon (e.g., 300+ steps)? This would show whether GRPO simply converges more slowly and eventually closes the gap, or whether HASD reaches a higher final performance.
- Fig. 4 analyzes the history window on ALFWorld with Qwen3-1.7B. Could the authors provide the same analysis for other backbone–environment combinations?
- RLCSD (arXiv 2606.11709) contrasts teacher–student gaps under correct and incorrect hints to cancel hint-induced shifts, which is closely related to HASD’s comparative PI construction. I encourage the authors to cite it and discuss the key differences (e.g., label reversal over the same action evidence vs. an incorrect hint).
Response to Rebuttal
I thank the authors for the thorough rebuttal. The additional results address most of my concerns. While the new experiments do not establish that HASD outperforms GRPO on every benchmark (e.g., on ALFWorld the gap narrows with longer training), they clearly show that HASD converges faster. The added multi-seed comparisons, the new benchmark and backbone results, and the other clarifications make the paper considerably stronger and more informative. I am therefore raising my score.