- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors provide an improved method of on-policy self-distillation named HEIR. The core method is that for each question, the student generates four rollouts independently, filters them by correctness, and chooses the shortest one. The rollout is used as student traj for distillation. If the shortest one is shorter than the original reference solution at least 5%, they use it to replace the hint of the teacher. The authors conduct experiments on Qwen3 models across several math benchmarks. They report improvements over OPSD, with up to 3.34 points in Avg@12. Their ablation shows that group winner selection contributes +1.82 points, while reference replacement adds another +0.87 points on Qwen3-4B.
Critical strengths
-
The proposed method is simple and easy to understand. It does not require changing the original OPSD training objective.
-
The authors provide some useful ablation studies. For example, Table 2 separates the improvement from group winner selection and reference replacement. They also investigate how the length of the original reference affects the results.
Critical weaknesses
-
Looks like the presentation of this paper is poor. For example, (1) the experiment section includes ten subsections, which is very defensive. (2) The first three paragraphs (for example, line 34 to line 51) introduce many basic settings of OPSD, which should appear in a later section, while the paragraph begins at line 53 is hard to understand: the first sentence of this paragraph is a negation, and the author didn't explain what is a "2x2" in line 72. Seems like the author assume the reader familiar with the work even before reading it. (3) Lack of explanation of figure 1. Hard for me to understand the meaning and differentce between left panel and right panel. (4) Poor presentation of table 1. What is the meaning of bold and underline? I didn't see any in paper.
-
Intuitively, the proposed method assumes a shorter solution is better. For example, it uses the shortest solution generated by model itself, compares it with the original reference, and replaces the original reference when the model-generated solution is shorter. However, I think this initial idea is somewhat heuristic. Could the authors explain more about why they do this? If shorter is better, why not just use the model itself to shorten the answer (using some prompt like "here is a solution, plz make it short and elegant"), or even just provide the final answer instead of the full solution?
-
In the original OPSD setting, the number of training examples is far beyond what is needed for a short training run. If each prompt is used only once, I doubt that the "reference-conditioned anchoring" (line 43) would cause much harm, since the model never sees the same reference again.
-
The idea is not novel to me at all. For example, filtering a correct solution generated by the model itself is adapted in the very original SDPO paper (in their "no env feedback" setting). The only improvement is that this paper adopted the shortest solution and filtered it to be shorter than the reference solution. This is heuristic and incremental, I cant think it a fair contribution.
-
The improvement across the proposed method and the OPSD baseline in Table 1 is marginal, and the author neither explained why it is marginal nor ran multi-seed experiments on this table 1 comparison (although some other setting have).
-
The authors even not provide any information about the training sets. What is the dataset? They just give some benchmark names.
Questions for authors
See Weakness.