acceptodds
Under review as a conference paper at ICLR 2027

HEIR: When the Student Inherits the Teacher's Seat Pareto-Competitive Reference Selection for On-Policy Self-Distillation of Reasoning LLMs

Abstract

On-policy self-distillation improves reasoning models by training on student rollouts while a teacher receives a privileged reference solution. That reference is usually fixed, even after the student begins producing better teaching examples. We introduce HEIR (Hint-Earning by Iterative Rollouts), a simple reference-handoff rule: sample a group of student rollouts, verify them, and replace the reference only when the shortest correct rollout clears a relative length margin. The admitted rollout conditions the teacher and supplies the student trajectory; the distillation objective itself is unchanged. Across Qwen3-1.7B/4B/8B and three competition-math benchmarks, HEIR improves a tuned fixed-reference baseline on every 4B and 8B Avg@12 result, with gains up to 3.34 points while shortening responses. A matched 4B comparison separates group selection from reference handoff: winner selection contributes 1.82 macro-average points over standard distillation, and handoff adds another 0.87 points (95% CI: 0.31–1.38) at essentially identical cost. The handoff gain is negligible for compact references but grows to 1.45 points for padded ones; it remains positive after non-verbatim and answer-masked controls and under injected verifier false positives. These results identify the operative regime precisely: a capable student, a verifiable task, and a reference whose form has become a constraint rather than an asset.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
on-policy self-distillation, reasoning language models, verifier-guided learning, reference handoff, mathematical reasoning, efficient reasoning, self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_V8bYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors provide an improved method of on-policy self-distillation named HEIR. The core method is that for each question, the student generates four rollouts independently, filters them by correctness, and chooses the shortest one. The rollout is used as student traj for distillation. If the shortest one is shorter than the original reference solution at least 5%, they use it to replace the hint of the teacher. The authors conduct experiments on Qwen3 models across several math benchmarks. They report improvements over OPSD, with up to 3.34 points in Avg@12. Their ablation shows that group winner selection contributes +1.82 points, while reference replacement adds another +0.87 points on Qwen3-4B.

Critical strengths
  • The proposed method is simple and easy to understand. It does not require changing the original OPSD training objective.

  • The authors provide some useful ablation studies. For example, Table 2 separates the improvement from group winner selection and reference replacement. They also investigate how the length of the original reference affects the results.

Critical weaknesses
  • Looks like the presentation of this paper is poor. For example, (1) the experiment section includes ten subsections, which is very defensive. (2) The first three paragraphs (for example, line 34 to line 51) introduce many basic settings of OPSD, which should appear in a later section, while the paragraph begins at line 53 is hard to understand: the first sentence of this paragraph is a negation, and the author didn't explain what is a "2x2" in line 72. Seems like the author assume the reader familiar with the work even before reading it. (3) Lack of explanation of figure 1. Hard for me to understand the meaning and differentce between left panel and right panel. (4) Poor presentation of table 1. What is the meaning of bold and underline? I didn't see any in paper.

  • Intuitively, the proposed method assumes a shorter solution is better. For example, it uses the shortest solution generated by model itself, compares it with the original reference, and replaces the original reference when the model-generated solution is shorter. However, I think this initial idea is somewhat heuristic. Could the authors explain more about why they do this? If shorter is better, why not just use the model itself to shorten the answer (using some prompt like "here is a solution, plz make it short and elegant"), or even just provide the final answer instead of the full solution?

  • In the original OPSD setting, the number of training examples is far beyond what is needed for a short training run. If each prompt is used only once, I doubt that the "reference-conditioned anchoring" (line 43) would cause much harm, since the model never sees the same reference again.

  • The idea is not novel to me at all. For example, filtering a correct solution generated by the model itself is adapted in the very original SDPO paper (in their "no env feedback" setting). The only improvement is that this paper adopted the shortest solution and filtered it to be shorter than the reference solution. This is heuristic and incremental, I cant think it a fair contribution.

  • The improvement across the proposed method and the OPSD baseline in Table 1 is marginal, and the author neither explained why it is marginal nor ran multi-seed experiments on this table 1 comparison (although some other setting have).

  • The authors even not provide any information about the training sets. What is the dataset? They just give some benchmark names.

Questions for authors

See Weakness.

Reviewer_uMabOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a method for training reasoning models: on-policy self-distillation. Under this setting, a student model solves problems independently, while a teacher model guides it token by token. The teacher outperforms the student because it has access to a reference solution, which the student cannot see. In existing practice, this reference solution remains fixed throughout training. The authors identify a clear issue: reference solutions are usually written by humans and tend to be wordy, with unnecessary explanations. After training for some time, the student can actually produce shorter and correct answers. However, when the teacher still relies on the long reference to guide generation, it drags the student back to that original style and length. The authors define this effect as "reference anchoring." The authors propose HEIR to address the challenge. The key idea is the reference solution must defend its position through performance. For each problem, the student generates candidate answers, and a verifier checks if they are correct. Among the correct ones, the shortest answer was selected. And if it was shorter than the reference, it replaces the old reference. This new answer then serves both as the teacher's reference and as the target trajectory for the student. If there is no candidates satisfy, the original reference is kept. The training objective remains unchanged; only the reference provided to the teacher is updated. The paper claims three main contributions:

(1) The update rule directly plugs into existing self-distillation pipelines.

(2)Controlled experiments separate the effect of best-of-N selection from reference replacement, measuring each component's gain.

(3)The method was evaluated across different models, verifier choices, and noisy verification settings.

Experiments were carried out on Qwen3 at multiple scales across three math benchmarks: AIME24, AIME25, and HMMT25 (30 problems each, evaluated with 12 samples per problem). On 4B and 8B models, HEIR consistently outperforms the fixed-reference baseline across all benchmarks, gaining up to 3.34 points while generating shorter solutions. On the 1.7B model, gains are limited, which the authors attribute to the small model rarely producing valid replacements. Ablation results show that while candidate selection provides most of the improvement, updating the reference provides an additional gain of about 0.87 points. And moreover when references are rewritten to be either compact or redundant, more redundant references yield larger replacement gains, supporting the anchoring hypothesis. The appendix provides supplementary results on additional benchmarks, compute costs, diversity metrics, theoretical analysis, and significance tests.

Critical strengths

The paper separates two factors that are easily conflated: selecting the shortest correct solution from G trajectories as the student's target, and inserting this solution into the teacher's context. By designing a dedicated baseline with a fixed reference and the best trajectory in the group, combined with a 2×2 factorial setup (Table 2, Table 3), the reported gain of +0.87 is reliably attributable by design. Most self-distillation studies skip this step and directly credit gains from best-of-G selection to their proposed method.

The reference redundancy experiment (Table 9) directly tests the "reference anchoring" hypothesis. Rewriting the same set of references into three length levels (70%, 100%, and 130%) to check whether handoff gains shift accordingly provides clearer mechanistic evidence than merely reporting an overall score. If these findings are reliable, this is the most convincing part of the paper.

The paper is quite measured in its claims about effect sizes. The authors clearly report the net improvement as +0.87, acknowledge that training cost increases to roughly three times when G=4, state the negative outcome on the 1.7B model alongside a slight drop in diversity, and offer a practical check in Section 5.10 to help users decide whether to apply the method.

Critical weaknesses
  1. Several evaluation metrics across the tables cannot occur under the evaluation protocol stated in the paper. Section 5.1 specifies that each benchmark contains 30 problems with 12 samples per problem. Section 5.6 emphasizes that every score uses the exact same 12 samples, and Appendix J states that Pass@12 is a 0/1 binary metric over 30 problems. Under this setup, Pass@12 and Maj@12 must be integer multiples of 100/30, and Avg@12 must be an integer multiple of 100/360. All 54 entries in Table 1 satisfy this condition. However, Tables 5, 6, 14, 20, and 23 report many Pass@12 and Maj@12 values that fall between valid grid points, such as 85.0, 91.3, 92.92, and 93.61. In Table 22, Pass@32 = 97.50 corresponds to 29.25 problems. In Table 11, among the 18 Pass@12 and Maj@12 values across three non-Qwen3 models, 12 cannot be obtained from k/30 even with rounding. The paper mentions no aggregation method that explains these values, while Appendix J states that the main tables present single-run point estimates.

  2. The confidence intervals in Table 16 and the curves in Figures 8 and 9 cannot be derived from the declared data. Table 16 states that 1,000 paired bootstrap iterations were run over 30 problems, with all single-method interval half-widths falling between ±1.9 and ±2.2 points. Taking 8B OPSD on AIME24 as an example, Avg@12 = 76.39 combined with Pass@12 = 90.00 means that exactly 3 problems were completely incorrect. Even under the distribution with minimal variance across the remaining 27 problems, problem-level resampling yields a standard error of at least 4.68 points, meaning the interval half-width should exceed ±9 points. Even if the 360 generations were treated as independent Bernoulli trials, the half-width would still be ±4.4 points. Paired resampling narrows the difference interval, but it cannot narrow the marginal distribution of an individual method. The caption of Figure 8 states that slices at k = 12 match corresponding columns in Table 1. However, values read from the plot for 8B on AIME24 at k = 12 are 85.47 / 90.61, whereas Table 1 reports 90.00 / 93.33. The curves start at roughly 20% at k = 1, yet within the same sample pool, Pass@1 and Maj@1 are defined to equal Avg@12 (76.39). At k = 32, the curves converge precisely to the Pass@12 values from Table 1 rather than the Pass@32 values reported in Table 22. In fact, curves across all six panels can be fitted exactly by the analytical formula A(1 − r^k).

  3. Unresolved repetitions and contradictions exist between the main text, appendix, and figures. The pair of numbers 724 and 591 appears simultaneously as the student output lengths at step 0 and step 160 during 8B training in Table 15, the evaluation lengths for OPSD and HEIR after 8B training in Table 22, and the converged length for 4B HEIR in Figure 4. Meanwhile, Figure 12 marks 8B OPSD at 700. In Figure 10, the reference solution of 612 tokens and the winner solution of 475 tokens in a single-problem example match the full dataset averages in Table 15. In Table 2, for the six entries shared with Table 1, the three-seed means match the single-seed point estimates from Table 1, and every standard deviation is ±.28, which corresponds to exactly one sample (1/360). In Tables 4 and 24, the standard deviation of ±.61 for HumanEval+ is exactly 1/164, and ±.20 for MATH-500 is exactly 1/500. Sections 5.5 and 5.9 state that in late-stage 8B training, the verification rate is 0.74 and the winner trajectory rate is 0.62. Appendix I, however, states that 0.74 and 0.41 are the values at step 0, which increase to 0.91 and 0.78 at step 160. Because both sections cite the identical set of numbers (612, 591, and 0.34), they can hardly represent distinct training runs. Training steps are reported as 100 in Figure 6, 160 in Table 15, and 200 in Table 18. Figure 6 shows 1,175 cumulative winner updates within 100 steps, implying roughly 20 prompts per step, whereas the 6.48k tokens per step in Table 18 correspond to only two or three prompts. Appendix A states that OPSD uses a fixed-teacher mode, while Appendix O notes a default EMA decay of 0.99 throughout the paper. Furthermore, the polynomial x^3 − 3x^2 + 2x − 1 in Figure 10 has a discriminant of −23 and only one real root, violating the problem statement of real roots r, s, t. Appendix H mentions sampling 50 AIME24 problems, though this benchmark contains only 30 problems. Resolving these issues requires the authors to provide run identifiers, problem-level raw outputs, and generation scripts.

  4. Core results conflict with known model behaviors, and critical experiments cannot be reproduced. The paper reports an Avg@12 of 76.39 to 78.89 for Qwen3-8B on AIME24, with average evaluation lengths of 724 tokens for OPSD and 591 tokens for HEIR. According to the Qwen3 technical report, this model scores 76.0 with thinking enabled and 29.1 without thinking. The OPSDC paper cited by the authors reports that its base model averages 14,170 reasoning tokens on AIME24. The paper does not state whether thinking mode was enabled, nor does it define how tokens were counted, making it impossible to assess how this combination of scores and lengths was achieved. In Table 11, Llama-3-8B achieves an Avg@12 of 23.00 to 34.50 on AIME24, substantially higher than single-digit scores reported publicly. In addition, the paper omits the name and size of the training dataset, the source of reference solutions, the learning rate, and the batch size. Section 5.4 describes reference rewriting solely as keeping the answer and reasoning content, without specifying the model, prompts, or equivalence verification methods used. Section 5.3 directs readers to Appendix C for details on non-verbatim and answer-masked baselines, yet Appendix C merely repeats the same numbers.

  5. Data reported in the paper suggest that the practical value of online replacement remains unproven. Table 9 shows that simply rewriting original references into compact versions offline increases the score of the baseline with a fixed reference and the best trajectory in the group from 56.26 to 57.00. This gain accounts for 85% of the +0.87 improvement achieved by HEIR on original references, while yielding even shorter evaluation lengths (565 versus 590 tokens). The difference between 57.00 ± .25 and 57.13 ± .28 is only 0.13, which is well within seed variation. The authors acknowledge in the abstract and Section 5.10 that the gain from handoff on compact references is negligible. However, they do not include one-time offline reference compression as a baseline in the main tables to compare costs and benefits, nor do they report single-trajectory OPSD on compact references. Because rewriting alters both phrasing and length, the dose-response relationship in Table 9 cannot be attributed to length alone. Even if all preceding numerical issues are resolved, the net gain attributed to online handoff is only +0.87 points, while an alternative approach requiring no online verification or replacement logic achieves nearly the same effect.

Questions for authors
  1. How were Pass@12, Maj@12, and Pass@32 calculated in Tables 5, 6, 14, 20, 22, and 23? Under a setup with 30 problems and 12 samples per problem, these metrics can only take integer multiples of 100/30, yet values such as 85.0, 92.92, 93.61, and 97.50 appear in the tables. If these tables used a protocol different from Section 5.1, please clarify this table by table, and explain why Appendix J states that the main tables present single-run point estimates.

  2. What data were used to generate the confidence intervals in Table 16 and the curves in Figures 8 and 9? Please provide the bootstrap code, the per-problem correct counts for each cell, and the raw Pass@k and Maj@k values used for plotting. When resampling over 30 problems, the interval half-width for 8B OPSD on AIME24 should not be smaller than roughly 9 points, whereas the table reports ±2. While the caption of Figure 8 states that slices at k = 12 match Table 1, values read from the plot for 8B on AIME24 are 85.47 / 90.61, and the curves start around 20% at k = 1.

  3. Was Qwen3 thinking mode enabled during training and evaluation? Do the token counts in Tables 15 and 22 include the thinking content? If so, please explain how an average output of 591 to 724 tokens achieves over 76% on AIME24. If not, please report the full sequence lengths, and clarify which counting definition applies to Figure 4, Figure 12, and the claim of shorter outputs.

  4. Please provide the run identifier, model size, and training step count for each table and figure, along with individual seed scores for all multi-seed cells in Tables 2, 4, and 24. Specifically, please explain whether 724 / 591 shared between Table 15 and Table 22 and the 591 for 4B in Figure 4 come from the same run; whether 0.74 / 0.62 in Section 5.9 and 0.74 / 0.41 in Appendix I describe the same run; why training steps are listed as 100 in Figure 6, 160 in Table 15, and 200 in Table 18; whether the OPSD baseline uses a fixed teacher (Appendix A) or an EMA decay of 0.99 (Appendix O); and why the three-seed means for the six cells shared between Table 1 and Table 2 match the single-seed point estimates exactly, with every standard deviation being ±.28.

  5. What are the name and size of the training dataset, the source of the reference solutions, the learning rate, and the batch size? Which model and prompts were used to rewrite the compact and redundant references in Table 9, and how was it verified that the reasoning content remained unchanged? How were the non-verbatim and answer-masked hints in Tables 3 and 8 constructed? What were the exact version, untrained baseline scores, and evaluation decoding settings for Llama-3-8B in Table 11?

Reviewer_okpFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes HEIR, which is a modificiation to on-policy self-distillation. At every iteration, several student solutions are sampled. HEIR has two components: First, it uses the shortest correct student rollout as the student's trainable tokens; second, if that solution is shorter than the teacher's current reference solution, then it replaces the teacher's reference. Results show that HEIR results in improved accuracy and shorter student outputs.

Critical strengths

Overall the empirical results are quite thorough. In particular,

  1. The performance gains from HEIR seem quite consistent across datasets and models.
  2. The paper provides quite extensive ablations on the different components of the method.
  3. The paper quantifies error with confidence intervals and multiple seeds.
Critical weaknesses

I find the writing style of this paper confusing, difficult to parse, and in some places contradictory.

  • The method is not well motivated by the paper.
    • In the intro, it is difficult to parse out why the length of the teacher's reference solution is of particular importance
    • Later on, the authors do not (as far as I can tell) motivate the choice of short reference solutions, despite this being a key design choice of the proposed method
    • The introduction and motivation for the paper focus on the teacher hint selection, and do not motivate the additional component of selecting the student solution for training. However, in their results (Table 2), the latter seems to account for a large part of the performance gains.
  • Furthermore, if I am interpreting Table 9 correctly, simply rewriting the initial reference solution to be shorter, combined with their "student winner selection", seems to recover nearly all the gains of full HEIR (Table 9).
  • Equation 3 selects the teacher reference solution from a list that includes the original reference y* but Algorithm 1 does not include y* in the candidate set
  • In the introduction, phrases such as '2x2 design', 'margin-gating', and 'full-minus-fixed gain' are used without context/definition.

There are other more minor issues:

  • The solid/dashes lines in Figure 2 are difficult to tell apart.
  • Two rewards (r_1, r_2) are defined in 4.1 but only r_1 is used; r_2 is not referred to anywhere else in the paper

There are substantial issues with the writing/communication, and lack of motivation/justification for the main method. Thus I recommend a weak rejection, and will consider improving my score if these issues are well addressed in the rebuttal.

Questions for authors
  1. What is the main justification for biasing towards shorter-length reference solutions?
  2. Why do the authors also use these solutions for the distillation trajectories? Can the authors explain the results in Table 9, which seem to indicate that this component, along with rewriting the original reference solution, is fairly sufficient?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.