acceptodds
Under review as a conference paper at ICLR 2027

LBSD: Local-Branch Selective Self-Distillation

Abstract

On-policy self-distillation (OPSD) mitigates the coarse-grained supervision issue of reinforcement learning with verifiable rewards (RLVR) with long reasoning trajectories by introducing dense token-level targets. However, standard OPSD methods largely treat all tokens alike, overlooking that only a subset of positions provides strong signals for exploration or correction. We introduce LBSD, a Local-Branch Selective self-Distillation framework that concentrates supervision on informative parts of the reasoning process. LBSD extends single-trajectory OPSD by selectively expanding critical intermediate states into local on-policy branches. It uses student entropy and teacher-student discrepancy to identify informative branches and token positions for learning. This design provides broader and more targeted feedback. Moreover, we propose abstract solution strategies as privileged teacher information instead of full reference reasoning trajectories, reducing the risk of answer leakage. Experiments across four competition mathematics benchmarks and multiple Qwen3 model scales demonstrate consistent improvements over representative RLVR and self-distillation baselines. Ablation studies further support the effectiveness of local branching and distribution-informed selection. These findings suggest that effective self-distillation depends not only on providing dense targets, but also on identifying where and along which local continuations those targets should be applied.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Model Reasoning, On-Policy Self-Distillation, Local Branching, Selective Distillation, Privileged Information, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UuzuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a variant of on policy self distillation (on policy distillation where the teacher is shown privileged context) wherein partial student rollouts are supervised from several branching points across a single student rollout.

Critical strengths
  • At face value, the numerical results are impressive.
  • The idea of forking at several key positions is intuitive and well motivated (although not totally novel).
  • The paper is generally clean and well presented.
Critical weaknesses
  • The leakage numbers aren't significant, and why does the base model have any leakage?
  • The quoted OPSD numbers differ from the paper, making the comparison more favorable for the authors.
  • The method costs more (it generates more tokens during training) but this was never equalized it seems?
  • Confidence intervals should really be added for your numbers. Especially with such small benchmarks/effects.
Questions for authors
  • How does OPSD do when you give it the same strategy?
  • What happens when you equalize training steps?
3 comments

We thank the reviewer for the constructive feedback. We address the concerns about leakage, OPSD comparability, training cost, and statistical uncertainty below.

[W1] Leakage analysis and the base model's non-zero leakage. We agree that the leakage-rate differences in the original analysis are small and should not be treated as conclusive on their own. The analysis uses the relatively short training schedule in our main experiments, with responses truncated at 1,024 tokens. We also consider the response-style statistics reported in the paper: answer density, solution density, boxed frequency, and final-answer frequency. OPSD shows higher answer/solution densities and boxed frequency than LBSD, indicating a stronger shift toward answer-oriented generation. These statistics complement, rather than replace, direct leakage measurement. We further conducted an AIME26 leakage experiment with four OPSD rollouts per prompt and unchanged training steps. Under the same manual inspection protocol, the OPSD leakage rate rises to 3.75%, compared with 1.25% in the original AIME26 analysis. The base model's non-zero rate may reflect exposure to public benchmark problems during pretraining.

[W2] Difference between our OPSD results and those in the original paper. The OPSD results in our main table come from our reproduction rather than the original paper. The protocols differ in three respects: we train on DeepMath rather than OpenThought; we evaluate with Avg@36 rather than Avg@12; and we report a fixed 100-step checkpoint rather than the best checkpoint selected from evaluations every 20 steps. Only AIME25 and HMMT25 overlap between the evaluation sets. In particular, a fixed-checkpoint score can be lower than the best-checkpoint score.

[W3, Q2] Additional rollout cost and training-budget comparison. We added OPSD-4r to examine whether LBSD's gains can be explained by generating more tokens. OPSD-4r uses four rollouts per prompt to bring its training-token budget to the level of LBSD. It is trained for 100 steps, with other settings and checkpoint selection following the original OPSD protocol. Results are evaluated with Avg@12 (%):

ModelMethodAIME25AIME26HMMT25HMMT26
1.7BOPSD-4r39.1740.2825.8330.05
1.7BLBSD40.8346.9426.1129.80
4BOPSD-4r69.4466.9443.0641.67
4BLBSD69.4467.2245.2842.17
8BOPSD-4r71.1170.5645.5645.20
8BLBSD73.8973.6148.8948.48

LBSD outperforms OPSD-4r in 10 of the 12 model–benchmark pairs, including all four Qwen3-8B benchmarks. Thus, the gains are not explained by additional rollout count alone.

[W4] Confidence intervals. We conducted a paired bootstrap analysis with 10,000 resamples. For each resample, we sampled benchmark problems with replacement, retained all 12 responses per problem, and used identical problem indices for LBSD and OPSD-4r. The table reports LBSD minus OPSD-4r (percentage points) and 95% percentile confidence intervals:

ModelAIME25AIME26HMMT25HMMT26
1.7B+1.67 [-3.06, 7.22]+6.67 [1.39, 12.78]+0.28 [-2.22, 2.78]-0.25 [-4.04, 3.03]
4B+0.00 [-4.44, 4.17]+0.28 [-2.50, 2.78]+2.22 [-1.11, 5.83]+0.51 [-2.78, 3.79]
8B+2.78 [-0.28, 6.11]+3.06 [-1.94, 8.61]+3.33 [0.28, 6.67]+3.28 [~0.00, 7.07]

The intervals exclude zero for AIME26 (1.7B) and HMMT25 (8B), while several others remain inconclusive. We therefore interpret the results as a favorable overall trend, not as statistically significant gains in every setting.

[Q1] OPSD with the same abstract strategy. We also trained OPSD using the same abstract strategies as LBSD while keeping the remaining settings unchanged from the paper. Results are Avg@12 (%):

ModelAIME25AIME26HMMT25HMMT26
1.7B39.1742.5025.2828.54
4B68.0666.3945.5641.41
8B70.5569.4446.1145.45

LBSD outperforms strategy-conditioned OPSD in 11 of 12 settings (except Qwen3-4B on HMMT25). This suggests that the improvements are not due solely to the abstract strategy; local branching and selective supervision provide additional benefits.

Reviewer_UuzuOpenReview ↗

Reply

Thank you for the detailed response. I am still considering all the replies and re-reading the parts of the paper I had concerns with, but I will certainly raise my score as several of my concerns have been addressed. I will potentially raise it further in the future after considering the responses more.

Thank you for your thoughtful response and for taking the time to reconsider our paper. We appreciate your careful evaluation of the additional experiments. We would also like to ask whether you have any remaining concerns or questions regarding our method or experimental results. We would be happy to provide further clarification and address any outstanding issues.

Reviewer_nqShOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds on on-policy self-distillation (OPSD), in which a model is trained on its own reasoning traces to match a teacher copy of itself that is shown privileged information about the solution. The authors identify three limitations of OPSD: it supervises only the single trajectory the student happens to sample, it weights all tokens equally, and it gives the teacher the full reference solution, which can leak the answer. LBSD is their proposal for addressing all three.

First, I see it picking a few intermediate reasoning steps where the student is uncertain or disagrees most with the teacher. It re-samples alternative continuations from just before those steps and distills on them as well as on the original path, weighting each group of branches by how much it reduces teacher–student disagreement. Second, within every path it keeps only the tokens with the highest combined student-entropy and teacher–student-KL score. Third, it conditions the teacher on an abstract solution strategy rather than the full reference solution.

To validate this, the authors train Qwen3-1.7B, 4B and 8B with LoRA on hard DeepMath problems. They evaluate on AIME 2025/2026 and HMMT 2025/2026 (Avg@36) against GRPO, OPSD, RLSD and RLCSD. In their results, LBSD has the best four-benchmark average at every scale.

Critical strengths
  1. A clearly specified extension of OPSD that improves consistently across three model scales. I found the method easy to follow and the code is released. Empirically, LBSD has the best four-benchmark average at all three scales (1.7B, 4B, 8B) and ranks first in 10 of 12 benchmark cells. Seeing the same ordering at three independent scales is more convincing to me than any single number. It is also achieved at a wall-clock cost comparable to RLSD and RLCSD.

  2. A useful framing of self-distillation design, with concrete evidence that the form of privileged information matters. I appreciate that the paper separates three questions that are usually bundled together: which continuations to supervise, which tokens to distill, and what privileged information the teacher should see. With full reference solutions, OPSD roughly doubles how often the student talks about answers and solutions relative to the base model, whereas LBSD's student stays close to the base model.

Critical weaknesses
  1. I can't tell whether local branching is what produces the gain. The paper's central claim is that LBSD wins by extending supervision to selected local branches and concentrating it on informative tokens. But compared with OPSD, LBSD changes several things at once: a) It gives the teacher an abstract strategy instead of the full reference; b) It adds branching and token selection; c) It uses a different training recipe; d) It generates several times more rollouts.

The paper's own Qwen3-4B ablations suggest the strategy context accounts for more of the improvement (about 1.1 of the 1.6 points over OPSD) than branching does. Each of these comes from a single run, and the OPSD row comes from a different code base and step count. The comparison that would settle this is missing: OPSD with the same strategy and training recipe, spending the same rollout budget on independent full trajectories.

  1. I can't judge how large the improvement really is. a) No base-model reference: Under the OPSD paper's essentially matched protocol, several baselines score at or below the untrained model: GRPO at 1.7B and 4B, RLCSD at 1.7B, and GRPO-8B on HMMT25. b) Reduced baseline budgets: RLSD and RLCSD were run well below the budgets in their own papers, and every method is reported at its final checkpoint despite step budgets ranging from 60 to 500. c) Single runs: All results are single training runs. The per-scale margins of 1–2 points are only about two to three standard errors of evaluation noise, and the defaults were tuned on AIME25, which is also a test benchmark.
Questions for authors
  1. On Qwen3-4B: OPSD with the same strategy context and 75-step recipe, spending the same rollout budget on independent full trajectories rather than local branches. This is the single comparison that would most directly show whether branching, rather than the strategy context or extra samples, drives the gain.

  2. On Qwen3-4B: three training seeds each for LBSD, LBSD without branching and OPSD, with paired significance tests. If LBSD's margin holds across seeds, I would be much more confident it is real.

1 comment

We thank the reviewer for the constructive feedback. We address the concerns about performance attribution, baseline comparison, and statistical reliability below. All new evaluations use Avg@12, rather than the Avg@36 protocol in the submission.

[W1, Q1] Contribution of local branching, privileged strategy, and rollout budget. We agree that these factors should be examined separately. We added two OPSD controls at all three model scales: OPSD-4r uses four rollouts to match LBSD's training-token expenditure, while OPSD-w-strategy replaces the full reference solution with the same abstract strategy used by LBSD. The results (Avg@12, %) are:

ModelMethodAIME25AIME26HMMT25HMMT26Avg.
1.7BOPSD-4r38.3341.1123.0628.2832.70
OPSD-w-strategy39.1742.5025.2828.5433.87
4BOPSD-4r65.5666.9441.9440.4053.71
OPSD-w-strategy68.0666.3945.5641.4155.36
8BOPSD-4r71.1170.5644.7243.4357.46
OPSD-w-strategy70.5569.4446.1145.4557.89

OPSD-w-strategy has a higher average than OPSD-4r at all three scales, consistent with the reviewer's observation that privileged context contributes to performance. We do not attribute the entire gain over standard OPSD to branching. To examine branching more directly, we compared LBSD with LBSD w/o Branch while keeping the remaining method configuration unchanged. Across three Qwen3-4B training seeds, removing branching lowers the four-benchmark average from 56.11 to 54.72 (table below). This provides additional evidence for the contribution of branching beyond the strategy context.

[W2] Missing base-model reference. We evaluated the three Qwen3 models with Avg@12. The results (%) are:

Base modelAIME25AIME26HMMT25HMMT26Avg.
Qwen3-1.7B36.3938.8924.7227.7831.95
Qwen3-4B64.4463.8943.0639.6552.76
Qwen3-8B68.0666.9444.4444.7056.04

Both additional OPSD variants exceed their corresponding base-model average at all three scales. These results provide the requested reference for assessing absolute post-training gains.

[W2, Q2] Run-to-run variability and statistical significance. We conducted three independent Qwen3-4B training runs for OPSD, LBSD w/o Branch, and full LBSD. Results are mean ± standard deviation across runs (Avg@12, %):

MethodAIME25AIME26HMMT25HMMT26Avg.
OPSD67.13±0.7065.18±0.5842.41±1.3841.83±0.8154.14±0.14
LBSD w/o Branch67.96±0.6466.57±0.4243.03±0.6041.33±0.5354.72±0.14
LBSD69.45±0.7467.50±1.2144.82±0.8042.68±1.1056.11±0.38

LBSD improves the four-benchmark mean by 1.97 points over OPSD and 1.39 over LBSD w/o Branch. The per-run averages are 54.05/54.07/54.30 (OPSD), 54.88/54.60/54.70 (w/o Branch), and 56.54/55.82/55.97 (LBSD). Even the lowest LBSD run exceeds the highest run of either comparison, and LBSD has the highest three-run mean on all four benchmarks. This alleviates the single-run concern, although aggregate scores alone do not establish statistical significance for these three-way comparisons. We also acknowledge the concern about AIME25-based tuning; the multi-seed advantage extends to AIME26 and both HMMT benchmarks.

Reviewer_aVGiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LBSD, which can improve on-policy self-distillation by changing where the student explores, which tokens receive supervision, and what information the teacher receives. LBSD selects intermediate reasoning states using student entropy and teacher–student disagreement, samples alternative continuations, and selectively distills their teacher distributions. Instead of full reference solutions, the teacher receives abstract solution strategies, extracted offline by Qwen3.5-27B. Experiments on three Qwen3 models and four competition-mathematics benchmarks showed the effectiveness of the proposed method.

Critical strengths
  • LBSD achieves consistent improvements across the four tested Qwen models and four benchmarks.
  • Detailed ablation study supports the design of each component and the combination of them.
  • The idea of selective token supervision is interesting and achieved promising results compared to uniform token supervision.
Critical weaknesses
  • The experiments are all conducted on Qwen model series, so that it is unclear whether the conclusion can be extended to other models.
  • There is no statistical significance test in the results, especially for the 1-2 percent gains in table 1 and 3.
Questions for authors

Please refer to my comments in the weakness part.

1 comment

We thank the reviewer for the constructive feedback. We address the concerns about model-family generalization and statistical reliability below.

[W1] Generalization beyond the Qwen model family. We agree that evaluation on Qwen3 alone does not establish whether LBSD transfers to other model families. We therefore evaluated OPSD, RLCSD, and LBSD on Llama3-8B using the same four mathematics benchmarks. The results (%) are:

MethodAIME25AIME26HMMT25HMMT26Avg.
OPSD66.9468.8943.8944.7056.11
RLCSD65.8368.3341.9441.1654.32
LBSD68.0670.0046.3944.7057.29

LBSD achieves the highest four-benchmark average, exceeding OPSD by 1.18 points and RLCSD by 2.97 points. Relative to OPSD, it improves on three benchmarks and ties on HMMT26; relative to RLCSD, it improves on all four. The gain over OPSD is 2.50 points on HMMT25. These results suggest that the benefit of LBSD is not limited to Qwen3.

[W2] Statistical uncertainty of the 1–2 point gains. We agree that small improvements should be accompanied by uncertainty estimates. We conducted two complementary analyses using Avg@12, rather than the Avg@36 protocol in the submission.

First, we trained OPSD, LBSD without branching, and full LBSD with three independent seeds on Qwen3-4B. Results are mean ± standard deviation across training runs (Avg@12, %):

MethodAIME25AIME26HMMT25HMMT26Avg.
OPSD67.13±0.7065.18±0.5842.41±1.3841.83±0.8154.14±0.14
LBSD w/o Branch67.96±0.6466.57±0.4243.03±0.6041.33±0.5354.72±0.14
LBSD69.45±0.7467.50±1.2144.82±0.8042.68±1.1056.11±0.38

LBSD improves the four-benchmark mean by 1.97 points over OPSD and 1.39 points over the no-branch ablation. Its per-run averages (56.54, 55.82, 55.97) all exceed the best runs of OPSD (54.30) and LBSD w/o Branch (54.88). This supports the consistency of the gains across training runs.

Second, we performed a paired bootstrap comparison of LBSD and OPSD-4r, which uses four rollouts per prompt to control for additional sampling. For this comparison, we selected the best-performing checkpoints following the checkpoint-selection protocol of the original OPSD paper. We drew 10,000 bootstrap samples of benchmark problems with replacement, kept all 12 responses for each sampled problem, and used identical sampled problem indices for both methods. The table reports LBSD minus OPSD-4r in percentage points, with 95% percentile confidence intervals:

ModelAIME25AIME26HMMT25HMMT26
Qwen3-1.7B+1.67 [-3.06, 7.22]+6.67 [1.39, 12.78]+0.28 [-2.22, 2.78]-0.25 [-4.04, 3.03]
Qwen3-4B+0.00 [-4.44, 4.17]+0.28 [-2.50, 2.78]+2.22 [-1.11, 5.83]+0.51 [-2.78, 3.79]
Qwen3-8B+2.78 [-0.28, 6.11]+3.06 [-1.94, 8.61]+3.33 [0.28, 6.67]+3.28 [~0.00, 7.07]

The intervals exclude zero for AIME26 on Qwen3-1.7B and HMMT25 on Qwen3-8B; several other comparisons remain inconclusive. Together, the multi-seed and bootstrap results indicate consistent overall improvements, with stronger statistical evidence in some benchmark settings than others.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.