- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
The paper proposes HiSD for on-policy self-distillation in multi-turn agents. At the step level, it adjusts distillation weights using accumulated and remaining teacher–student divergence. At the token level, it selects an adaptive support based on the current divergence and applies forward-KL supervision. The method is evaluated on ALFWorld, Search-QA, and WebShop across three model sizes.
Critical strengths
Clear motivation and a sensible design. The paper addresses both state drift across interactions and the distributional coverage of token-level supervision. Step weighting and adaptive support selection target different aspects of the problem and appear complementary.
Broad evaluation and useful mechanism analyses. The experiments cover three tasks and three model sizes, with ablations of components, weighting strategies, support selection, and hyperparameters. The analysis of long-horizon tasks is also informative: HiSD improves over SDAR by approximately 16 points on average on Pick2.
Critical weaknesses
-
The public implementation differs from the method definition. The paper requires at least two support tokens, while the scripts allow one; after renormalization, singleton support produces zero distillation loss and gradient. The paper also weights trajectories equally, whereas the code averages over interaction steps, giving longer trajectories greater weight. These differences change the supervision mechanism and optimization objective. The authors should clarify which implementation produced the reported results.
-
The step-weighting ablation may not control distillation strength. The public code omits the base coefficient in the Relative Position branch but includes it in full HiSD. With the default coefficient of 0.01, this omission increases the positional variant’s coefficient by a factor of 100 relative to its intended value. If Table 2 uses this implementation, its lower performance could reflect excessive distillation rather than an inferior weighting strategy, weakening the evidence for the proposed design.
Questions for authors
-
Were the reported results obtained with the paper’s minimum support size of two and equal trajectory weighting, or with the code’s allowance for singleton support and equal step weighting? Could the authors provide the actual run configurations and explain the effects of these differences?
-
Did the Relative Position ablation in Table 2 use the corresponding public-code branch? If so, could the authors restore the multiplier and repeat the comparison at matched distillation strength?
2 comments
Thank you!
Dear Reviewer 5H4i,
Thank you for re-examining our updated code and log. We are glad our clarifications addressed your main concerns, and we appreciate your willingness to raise your rating to 3.
Please let us know if any specific concerns remain.
With sincere thanks,
The Authors
I used AI to discuss the paper, check the code, and polish the review. I evaluated its suggestions and made the final assessment and recommendation myself. After reviewing the updated code and training log, I consider my main concerns addressed and am willing to raise my rating to 3.