- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes HAIR, which uses a reasoning harness to solve difficult problems and reconstructs successful executions into standalone trajectories for supervised updates interleaved with GRPO.
Critical strengths
A compelling approach to learning from problems that native sampling fails to solve.
Critical weaknesses
See Questions For Authors
Questions for authors
-
I am curious whether retaining trial-and-error and self-verification changes the model’s response length, especially on easier problems. Have the authors observed any interesting trends?
-
Could the authors share the length limit used for SFT trajectories and how particularly long reconstructed traces are handled?
5 comments
1. Effect on response length
Thank you for this thoughtful question. We indeed observe that HAIR increases response length relative to GRPO. This trend is evident in both the synthesized SFT data and the training rollouts from the final model snapshots. The latter are summarized below:
| Statistic | GRPO | HAIR |
|---|---|---|
| Mean response length (tokens) | 41,723 | 47,206 |
| Responses reaching the generation length limit | 14.4% | 33.8% |
Our inspection of the trajectories suggests that additional self-verification accounts for part of this increase: the model spends more tokens checking its intermediate reasoning and candidate solutions. This trend is consistent with HAIR’s primary objective of improving performance on challenging problems, where retaining trial-and-error and self-verification can support more thorough reasoning.
2. SFT length limit and handling of long trajectories
Our SFT trajectory length limit is 128k tokens, compared with a total length limit of 64K tokens during RL. We empirically found that allowing trajectories longer than 64K tokens in SFT yields greater performance gains during subsequent RL.
Our interpretation is that, because HAIR trajectories are sampled from the current policy, the successful trajectory primarily teaches the model how to structure its reasoning policy, including coordinate trial-and-error and self-verification, rather than memorize entire trajectories. Longer trajectories can therefore provide useful training signals that translate into improved performance under the shorter RL response budget.
Regarding particularly long reconstructed traces, none of the synthesized trajectories exceed 128k tokens, as their length is bounded by the context limit of the final policy completion step.
Thank you for your thoughtful review and constructive suggestions. We address your concerns below.