acceptodds
Under review as a conference paper at ICLR 2027

BASED: Branch-Aware Soft Embedding Distillation for Reasoning LLMs

Abstract

On-policy distillation (OPD) provides dense teacher supervision along student-generated reasoning trajectories, but each discrete rollout follows only one continuation at a reasoning branch, leaving alternative plausible branches unsupervised. Soft embeddings offer a natural way to represent multiple candidate branches within a continuous trajectory, yet we find that naive soft-trajectory distillation loses branch-specific information and may lead to branch collapse. To explain this limitation, we introduce the Hard-Branch KL objective and derive an exact decomposition into a Mixture KL term and a Branch-Posterior KL term. We show that soft-trajectory KL approximates the Mixture KL while omitting branch-posterior supervision. This reveals that soft distillation preserves mixture-level supervision but discards the branch identities needed to distinguish alternative reasoning paths. Motivated by this analysis, we propose Branch-Aware Soft Embedding Distillation (BASED), which combines soft KL with a local branch-posterior correction to efficiently recover the missing branch-specific supervision. Experiments across eight reasoning benchmarks show that BASED improves Pass@1 over vanilla OPD across both student scales and achieves higher average Pass@8, while ablations and trajectory-diversity analyses provide further evidence that BASED helps preserve diverse reasoning branches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, post-training, soft embeddings, latent reasoning, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mWu3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies GKD-style on-policy distillation (OPD) for reasoning models. Its starting point is the observation that a sampled rollout follows only one continuation at each branching point: the next-token distribution at that point is supervised by the token-level KL, but the downstream states reached through the other branches receive no direct teacher supervision along that rollout. The authors treat the student-induced branch prior as shared by the teacher and the student, and define the "Hard-Branch KL" as the prior-weighted average of per-branch KLs. By the KL chain rule, this decomposes exactly into a mixture KL and a branch-posterior KL. Following the first-order approximation used in Soft Thinking, they then argue that the soft-trajectory KL, computed on soft-embedding prefixes, approximates the mixture term at the objective level and omits the posterior term.

BASED builds on this analysis with three components:

  • The student rolls out with soft embeddings at high-entropy positions (normalized top-3 entropy > 0.7).
  • Forward KL is applied along this soft trajectory.
  • A local posterior correction is added at up to 50 positions per trajectory: it expands the immediately preceding soft anchor into its top-3 hard tokens and supervises only the next token.

Inference is unchanged.

The experiments distill Qwen3-8B (non-thinking) into Qwen3-1.7B-Base and Qwen3-4B-Base with LoRA (128 steps, 4,096 DAPO-Math prompts, one rollout per prompt). Evaluation covers MATH-500, AIME 2024/25/26, HMMT Feb 2026, AMC 2023, MinervaMath, and GPQA-Diamond. Over vanilla OPD, BASED raises average Pass@1 by 1.55 pt (1.7B) and 1.65 pt (4B), and Pass@1 improves on every benchmark at both scales. Average Pass@8 rises by 1.09 pt and 1.88 pt, respectively. The paper also includes a 1.7B ablation with a soft-only baseline (Table 2), reverse-KL results (App. E), and a Self-BLEU/chrF diversity analysis on two AIME 2026 problems.

Critical strengths

The mixture/posterior split shows cleanly what soft-token training can and cannot teach: a student can match the mixture perfectly and still lose which branch leads where. The two-branch example in §3.2 makes this very clear. The decomposition itself is just the chain rule, but I think it is a useful lens for work on soft/latent reasoning. The appendix is also careful not to overclaim: it gives only sufficient conditions, states that ε_soft can take either sign, and separates objective-level from gradient-level statements. This is the main reason I think the direction is worth pursuing.

As a full method, BASED achieves consistently higher Pass@1 than vanilla OPD on all eight benchmarks at both scales, without changing inference. This includes recent benchmarks with low contamination risk (AIME 2026, HMMT Feb 2026). The supplementary code is also fairly complete (configs for the main and ablation runs, and the exact cache-VJP implementation), which made the checks below possible.

Critical weaknesses

1. The experiments do not show that the branch-posterior correction, the paper's actual contribution, drives the gains.

The correction itself adds little.

  • The only accuracy experiment that isolates the correction is the 1.7B ablation in Table 2. It uses a single seed and reports only 2 of the 8 benchmarks individually.
  • There, soft-only distillation alone already raises average Pass@1 from 19.02 (vanilla OPD, Table 1) to 20.07, and average Pass@8 from 41.69 to 42.78. Adding the correction gives +0.50 Pass@1 and +0.01 Pass@8.
  • Backing out from the averages, BASED leads soft-only by only about 0.6 pt in summed Pass@1 over the six benchmarks not shown, and soft-only actually leads BASED by about 6.7 pt in summed Pass@8.
  • Of the two benchmarks shown, GPQA-D at 1.7B is at chance level for a four-option task: Pass@1 lies between 23.2 and 26.6 in every row of Table 2.

The stability of the results cannot be assessed. All runs use a single seed (42), with no confidence intervals. The spread among the BASED hyperparameter variants in Table 2 (average Pass@1 19.82–20.57) is already larger than the effect of the correction. The ablations that justify the default settings were run on the same test benchmarks, and no validation split is mentioned. The released code also treats the other target policy as the default: base.yaml sets target_policy: next_soft_anchor, and the docstring in scripts/based.py describes immediate_next_token as "retained only as a target-policy ablation"; the main-experiment configs set immediate_next_token explicitly.

The reverse-KL results in App. E also run against the diversity claim. There, BASED lowers average Pass@8 (44.36 → 42.63, lower on 6 of the 8 benchmarks). Vanilla RKL-OPD also beats forward-KL BASED on both metrics (21.52/44.36 vs. 20.57/42.79).

There is no compute-matched baseline. §3.1 frames OPD's limitation as a branch-coverage problem that comes from a finite number of rollouts. The natural baseline is then OPD with more rollouts or more prompts at matched compute. The code supports num_rollouts > 1, but only results with one rollout per prompt are reported.

The paper also gives only an analytical bound on BASED's training cost (at most kN_post branch-conditioned predictions per model and trajectory, l. 331–332), not a measured cost. BASED adds the following overhead:

  • It keeps both models' KV caches from the soft replay in memory until the correction is done.
  • It computes up to 150 extra branch-conditioned predictions per model per trajectory.
  • For each selected target (up to 50 per trajectory), it runs a separate backward pass through the branch suffix and accumulates the cache cotangents in a buffer the size of the student's KV cache. (The VJP through the master prefix is merged into the soft-KL backward and runs once.)

The released evaluator grades more leniently than App. C.3 describes. App. C.3 states that math answers are evaluated with Math-Verify and GPQA-D answers by matching the reference option letter, but it does not say which answer in a response is graded. The released evaluator (evaluate_fixed.py → scripts/eval_fixed_grading.py, policy any_explicit_answer_v1) applies these checks to every answer candidate in the response:

  • A sample is counted as correct if any \boxed{} or "answer is / answer:" span anywhere in the response matches the gold answer, including intermediate answers that the model later abandons.
  • For example, a GPQA-D response that first boxes A and then concludes with \boxed{C} is graded correct when the gold label is A, while the repository's own strict extractor (extract_choice_answer) returns C. Likewise for math, a response that boxes 5, abandons it, and ends with \boxed{7} is wrong under Math-Verify applied to the whole response, but correct under the released evaluator when the gold answer is 5.
  • For responses that contain a \boxed{} answer, this rule can only raise scores relative to final-answer grading, and it can inflate Pass@k more on hard benchmarks. Whether it interacts with the training method is unknown.

Since the headline gain is about 1.5 pt, I think this needs to be checked.

2. The theory is not well connected to the method as implemented.

The key approximation is not supported. The step that actually carries the argument is not Prop. 1 but the claim that soft-prefix predictions approximate the branch mixture (T_soft ≈ T_mix, §3.3).

  • App. A.2 does not establish this at LLM scale. The bound on T_soft vs. T_mix depends on a curvature constant (B_M) and on the dispersion of the hard prefix embeddings, and neither is estimated. The next step, from T_soft ≈ T_mix to L_soft ≈ L_mix, assumes that every coordinate of the four distributions is at least η. For Qwen3 this forces η ≤ 1/|V| ≈ 6.6×10⁻⁶, so the coefficient on the δ_S term is at least ~1.5×10⁵. The paper itself notes that very small probabilities can make this bound loose (l. 728–730).
  • There is also prior evidence pointing the other way. Wu et al. (ICLR 2026, "LLMs are Single-threaded Reasoners") report that LLMs process soft inputs mainly through the highest-probability token. This is consistent with the collapse observation of SeLaR, which the paper itself cites. If this holds here, the soft KL behaves more like a KL on the top-1 branch than like the "mixture term".
  • A local version of this gap, T_soft vs. T_mix,local at the selected targets, is cheap to measure, since the correction already computes the k branch-conditioned teacher/student distributions at those positions. It is not reported.

The correction covers much less than the motivation. The introduction (l. 42–48) motivates BASED by the lack of direct supervision on the downstream states of other branches. The correction, however, is narrow:

  • it expands only the immediately preceding anchor;
  • it supervises only the next token;
  • it uses only the posterior term.

The "next soft anchor" variant extends the horizon, but it still expands only one anchor, uses only the posterior term, and keeps the recorded intermediate tokens fixed.

Since the k branch-conditioned distributions are computed anyway, the exact local Hard-Branch KL (Σ_v q_b(v) KL(T_v‖S_v), i.e., local mixture + local posterior) is available at the same cost. Wider windows are also already supported in the code (window_size; k^r ≤ 64 under the default max_assignments, i.e., r ≤ 3 for k = 3). These two are the experiments that would directly test whether the decomposition matters, and neither is reported.

"Branch collapse" is supported only indirectly. The abstract presents collapse as a finding ("we find that naive soft-trajectory distillation loses branch-specific information and may lead to branch collapse"), but the evidence is indirect. Soft-only does have the highest Self-BLEU on both analyzed problems (53.45, 42.59), but its average Pass@8 is essentially the same as BASED's (42.78 vs. 42.79).

The diversity analysis itself is also thin:

  • It covers only two problems, and no selection criterion is given.
  • Compared with vanilla OPD, BASED's Self-BLEU is mixed (52.26 vs. 52.16 on Q4, 41.56 vs. 42.20 on Q15).
  • Correct and incorrect responses are clustered without matching subset sizes (App. D). A model that solves a problem more often therefore tends to get more correct clusters and fewer incorrect clusters, regardless of diversity.
Questions for authors

Questions 1–5 could change my assessment; Question 6 is a clarification request.

  1. Grading. Were Tables 1, 2, and 5 all graded with any_explicit_answer_v1? Could you re-grade the existing predictions with a final-answer grader (the last \boxed{} for math, the strict extractor for GPQA-D)? Please also report, for each method, the fraction of samples counted as correct that were matched on a candidate other than the final answer. The evaluator already logs the matched candidate for each sample, so no regeneration should be needed.

  2. Seeds and the full ablation. Could you run vanilla OPD, soft-only, and BASED at 1.7B with at least three seeds and report confidence intervals? Please report all eight benchmarks for every row of Table 2.

  3. Compute. What are the per-step wall-clock time and memory of OPD, soft-only, and BASED? At matched compute, how does BASED compare with OPD using more rollouts per prompt (e.g., 2 or 4) or more prompts?

  4. The §3.3 approximation. At the selected targets, how large are ‖T_soft − T_mix,local‖₁ and the local ε_soft compared with the posterior term? How do the soft-anchor rate and the ratio of posterior_contribution to soft_trajectory_loss evolve during training? The training script already logs soft_anchor_rate, soft_trajectory_loss, and posterior_contribution at every step.

  5. A direct test of the decomposition. How do the following variants compare with BASED?

    • (a) Using the exact local Hard-Branch KL instead of the posterior term alone.
    • (b) A window size of r = 2 or 3.
    • (c) A control that feeds the top-1 token embedding instead of the mixture at the same anchors. This separates the effect of soft aggregation from the effect of removing sampling noise at high-entropy positions.
  6. The diversity analysis (§5.4).

    • How were Q4 and Q15 chosen?
    • Do the conclusions hold on more problems and with size-matched correct subsets?
    • Do they hold at the threshold of Matsutani et al. (similarity 60 / distance 0.4)? The comment in the released config also cites this threshold, while the paper uses 55 / 0.45.
    • The released script grades these samples with the any-answer rule and, before computing Self-BLEU/chrF, truncates correct responses at the first matched answer. App. D mentions neither step. Also, trajectory_diversity.py imports a re_evaluate module that is not in the supplementary material, so the analysis does not run as released.

If the gains over both soft-only and OPD hold under re-grading, multiple seeds, and compute-matched baselines, and the measurements in Questions 4–5 support the mixture/posterior account, I would be glad to raise my score.

7 comments

We greatly appreciate the reviewer's thorough evaluation and detailed suggestions, which have helped us clarify important methodological choices and improve the rigor of our work.

Q1: Grading

Yes, we grade the results using any_explicit_answer_v1.

This choice addresses the non-stop issue in OPD: the model first generates a correct response but fails to stop, subsequently generating unrelated content.

This is a well-known issue that has been studied by many researchers (Yang et al., 2026; Plyusov et al., 2026; Fu et al., 2026).

The non-stop issue causes the strict extractor to misclassify many responses. We use any_explicit_answer_v1 to assess the student model's reasoning ability more accurately.

For example, in an evaluation on AIME 2026, replacing the strict extractor with any_explicit_answer_v1 recovers 22 correct responses that were originally classified as incorrect because of the non-stop issue. Only 4 responses first produce a correct answer and then change it to an incorrect one.

Moreover, since we study OPD rather than RLVR, we do not reward the model based on correctness, so we do not expect reward hacking to be an issue.

The full regraded results are too lengthy to include here. As an example, we present the 1.7B results for vanilla OPD, soft-only, and BASED using the strict extractor.

The performance gains with the strict extractor are even larger than those with any_explicit_answer_v1.

1.7BMATH 500 Pass@1MATH 500 Pass@8AIME 2024 Pass@1AIME 2024 Pass@8AIME 2025 Pass@1AIME 2025 Pass@8AIME 2026 Pass@1AIME 2026 Pass@8HMMT Feb 2026 Pass@1HMMT Feb 2026 Pass@8AMC 23 Pass@1AMC 23 Pass@8MinervaMath Pass@1MinervaMath Pass@8GPQA-D Pass@1GPQA-D Pass@8AVG Pass@1AVG Pass@8
Vanilla OPD0.57550.85200.03750.16250.03230.15240.03120.12430.01140.05370.28750.60620.22200.50370.19440.68180.17400.3921
Soft-only0.58430.84400.06350.19280.04580.14870.03330.12790.01890.07980.31480.66370.23350.52210.18430.70200.18480.4101
BASED0.59250.83800.06670.20010.04480.13570.04690.15550.02270.07520.31640.65870.23850.52210.20270.74750.19140.4166

Q2: Seeds and the full ablation

We regret that limited computational resources prevented us from conducting multi-seed training.

We understand your concern about the reliability of our results and would like to provide supporting evidence.

In our implementations of all distillation training algorithms, we calculate the KL divergence over the full vocabulary. Compared with the commonly used top-k estimation, full-vocabulary calculation has significantly lower variance. Furthermore, our training algorithms follow the GKD-style OPD paradigm, which directly treats the KL divergence as the loss and uses backpropagation to update the model. Compared with PG-style OPD, which uses policy gradients to update the model, GKD-style OPD is known for its lower variance. Taken together, these considerations lead us to believe that our training procedure has relatively low variance.

During evaluation, we estimate the metrics using multiple samples. We sample 8 responses per problem for larger benchmarks and 32 responses per problem for smaller benchmarks to reduce variance. We therefore believe that the results reported in our paper are reliable.

Q3: Compute

In §5.5 of the revised manuscript, we compare vanilla OPD and BASED under strictly matched rollout lengths, reporting the average training time and FLOPs per step.

The average training times per step for vanilla OPD and BASED are 106.24 s and 119.01 s, respectively.

Their average computational costs per step are 63.35 TFLOPs and 66.90 TFLOPs, respectively.

The results show that BASED is lightweight, increasing the time per step by only 12.0% and FLOPs by only 5.6%.

As shown in our computational cost analysis, BASED is relatively lightweight. Even OPD with two rollouts is not a compute-matched comparison.

Increasing OPD to two rollouts would approximately double its FLOPs to an estimated 126.70 TFLOPs per step, or 1.89×1.89\times BASED's cost, requiring substantially more computation.

We have also added a two-rollout OPD baseline to the 1.7B results in our main table.

The results show that although doubling the number of rollouts per prompt raises 1.7B OPD's average Pass@1 / Pass@8 from 23.04 / 46.61 to 23.54 / 47.32, single-rollout BASED still achieves higher averages of 24.47 / 47.86. Thus, increasing the number of rollout samples recovers part of the performance gains but still does not outperform BASED.

Q4: The approximation in §3.3

Approximation error.

We report the soft-to-mixture approximation error in the new Appendix H:

∥Tsoft−Tmix∥1=0.827,\lVert T_{\mathrm{soft}}-T_{\mathrm{mix}}\rVert_1=0.827, ∥Ssoft−Smix∥1=0.737,\lVert S_{\mathrm{soft}}-S_{\mathrm{mix}}\rVert_1=0.737, ∣Lsoft−Lmix∣=0.733.\lvert\mathcal L_{\mathrm{soft}}-\mathcal L_{\mathrm{mix}}\rvert=0.733.

The results show that the first-order approximation is not quantitatively tight at these selected positions.

Although soft KL is not an accurate surrogate for explicit mixture KL in nonlinear LLMs, the posterior correction still introduces explicit branch-specific supervision that is absent from the soft-KL objective.

We additionally conducted gradient and behavioral analyses to examine the effect of posterior correction.

Gradient-level analysis.

To examine how posterior correction affects training, we have added a gradient-level analysis in §5.6.

We compare the cosine similarities of the gradients of soft-only distillation and BASED with the gradient of the explicit local Hard-Branch KL objective.

The results show that adding the posterior term increases the mean gradient cosine similarity from 0.713 to 0.727 (paired Δ=+0.0139\Delta=+0.0139, 95% CI [0.0030, 0.0346][0.0030,\,0.0346]), with improved alignment at 80.80% of the evaluated positions.

This indicates that the posterior correction can align the gradient more closely with that of the explicit local Hard-Branch KL objective.

Behavioral analysis.

We have added an analysis of token decision behavior in §5.7 of the revised manuscript.

We first define a metric called branch-soft distinction:

=\sum_a q_a D\left(p_X^a\Vert s_X\right),$$ where $a$ indexes hard branches, $q_a$ is the corresponding branch weight, $p_X^a$ is the next-token distribution conditioned on hard branch $a$, and $s_X$ is the next-token distribution produced by the soft embedding, for $X\in\{\text{Soft only},\text{BASED}\}$. A smaller BSD indicates that the branch-conditioned predictions are closer to the soft-embedding prediction, while a larger BSD indicates greater differences between them. BASED increases BSD from 3.528 to 3.543 (paired $\Delta=+0.0153$, 95% CI $[0.0082,\,0.0223]$), indicating that the branch-conditioned predictions become more distinct from the soft-embedding prediction. ### **Q5: A direct test of the decomposition** Thank you for these suggestions. Although we have not evaluated these variants directly, we clarify our design choices and provide supporting evidence from existing experiments. **(a) Exact local Hard-Branch KL.** Since the exact decomposition gives $\mathcal L_{\mathrm{hard}}=\mathcal L_{\mathrm{mix}}+\mathcal L_{\mathrm{post}}$, replacing the posterior correction with Hard-Branch KL while retaining soft KL would introduce an additional mixture-matching term, potentially duplicating supervision and changing the objective's relative weighting. Our posterior-only correction specifically targets the missing branch information. Moreover, §5.6 shows that it improves gradient alignment with the explicit Hard-Branch KL objective from 0.713 to 0.727 (95% CI for $\Delta$: $[0.0030,\,0.0346]$). **(b) Larger windows ($r=2$ or $3$).** The window size parameter is retained in our code only for future exploration and is not mentioned in our paper. Expanding branches across $r$ soft anchors requires $k^r$ branch combinations. With $k=3$, this increases from 3 ($r=1$) to 9 or 27. We choose $r=1$ to balance branch-specific supervision and computational efficiency while retaining full-trajectory soft KL. Our cost analysis (§5.5) shows only 5.6% additional FLOPs over vanilla OPD. Larger windows may capture additional interactions but incur substantially greater branch-expansion costs. **(c) Top-1 embeddings.** Unlike top-1 embeddings, soft embeddings retain the weighted contributions of multiple candidates, providing the first-order approximation to mixture-level predictions described in §3.3. Our $\beta$ ablation offers indirect support: sharpening the mixture toward top-1 ($\beta=0.7$) reduces average Pass@1 / Pass@8 from 24.47 / 47.86 to 23.78 / 46.56. Nevertheless, this does not fully isolate aggregation from reduced sampling randomness, and we acknowledge the proposed top-1 control as a useful direction for further investigation.

Q6: The diversity analysis

How were Q4 and Q15 chosen?

We selected questions for the diversity analysis based on the following criteria: (a) the math problem admits different solution methods; (b) vanilla OPD, soft-only, and BASED achieve similar accuracy on the problem; and (c) their accuracy is high enough to generate sufficient samples for the analysis.

Do the conclusions hold on more problems and with size-matched correct subsets?

After selecting Q4 and Q15 based on the criteria above, we conducted experiments only on these two questions. Whether the conclusions hold on more problems therefore remains unexplored.

Do they hold at the threshold of Matsutani et al. (similarity 60 / distance 0.4)? The comment in the released config also cites this threshold, while the paper uses 55 / 0.45.

In our experiments, using the original threshold from RL Squeezes, SFT Expands did not produce effective clustering: every trajectory was assigned to a separate cluster. We therefore increased the distance threshold to enable clustering.

The released script grades these samples with the any-answer rule and, before computing Self-BLEU/chrF, truncates correct responses at the first matched answer. App. D mentions neither step. Also, trajectory_diversity.py imports a re_evaluate module that is not in the supplementary material, so the analysis does not run as released.

To address the same non-stop issue described in our response to Q1, we truncate each trajectory at the correct answer to eliminate interference from subsequent unrelated content. This allows the algorithm to analyze the diversity of the correct trajectories more accurately.

Reviewer_gM8sOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a joint KL objective that combines soft-trajectory supervision with branch-posterior matching, aiming to broaden supervision coverage while preserving branch-specific information. The authors argue that optimizing only along sampled student trajectories in conventional OPD may leave useful supervision from alternative continuations unexplored, whereas soft-trajectory supervision alone may lose information about branch-specific decisions. Accordingly, BASED uses soft tokens as trajectory context and introduces branch-posterior matching as an additional supervision signal. BASED achieves some performance improvements over the baselines on the evaluated datasets.

Critical strengths
  1. The motivation is clearly presented, with theoretical analysis of the limitations of conventional OPD and soft-trajectory supervision.
Critical weaknesses
  1. Limited Analysis. Although a soft-only baseline is included, the paper does not directly analyze prediction stability along soft trajectories, such as whether soft inputs prolong high-entropy states. Evidence that soft-only supervision loses branch-specific distinctions is also largely indirect.

  2. Insufficient Validation of the Theoretical Approximation. The theoretical decomposition is useful, but the paper does not directly evaluate the soft-to-mixture approximation error or how effectively the local correction recovers explicit hard-branch supervision.

  3. Performance Gains. Performance gains are relatively modest, particularly over soft-only distillation, which the limits confidence in BASED's robustness.

Questions for authors
  1. Can soft inputs cause accumulating noise or prolonged high entropy, and have the authors examined this across trajectories?

  2. How does BASED affect token decisions at high-entropy positions during inference?

1 comment

We thank the reviewer for the thoughtful comments and insightful questions regarding the theoretical foundations and empirical behavior of our method.

Q1: Can soft inputs cause accumulating noise or prolonged high entropy, and have the authors examined this across trajectories?

Our work focuses on soft-embedding distillation during training, rather than soft-token decoding at inference time. We use standard discrete decoding at inference time. We have not conducted a dedicated trajectory-level analysis of noise accumulation or prolonged high entropy in BASED rollouts.

Prior studies suggest that continuous soft inputs can introduce distribution shifts and degenerate behavior (Zhang et al., 2025), as well as noise and delayed convergence (Shi et al., 2026). However, soft thinking can also exhibit greedy-like feedback and branch collapse (Wu et al., 2026), suggesting that prolonged high entropy is not an inevitable outcome. Moreover, SeLaR (Fu and Luo, 2026) empirically demonstrates the benefits of selectively activating soft embeddings rather than applying them globally.

Motivated by these findings, BASED employs entropy gating and top-k mixing to restrict soft inputs to selected high-uncertainty training steps and exclude low-probability candidates, while retaining discrete tokens at other steps. These mechanisms aim to limit unnecessary perturbations, although they do not guarantee the absence of accumulated errors or sustained high entropy.

Q2: How does BASED affect token decisions at high-entropy positions during inference?

We have added an analysis of token decision behavior in §5.7 of the revised manuscript.

We first define a metric called branch-soft distinction:

=\sum_a q_a D\left(p_X^a\Vert s_X\right),$$ where $a$ indexes hard branches, $q_a$ is the corresponding branch weight, $p_X^a$ is the next-token distribution conditioned on hard branch $a$, and $s_X$ is the next-token distribution produced by the soft embedding, for $X\in\{\text{Soft only},\text{BASED}\}$. A smaller BSD indicates that the branch-conditioned predictions are closer to the soft-embedding prediction, while a larger BSD indicates greater differences between them. BASED increases BSD from 3.528 to 3.543 (paired $\Delta=+0.0153$, 95% CI $[0.0082,\,0.0223]$), indicating that the branch-conditioned predictions become more distinct from the soft-embedding prediction. ### **Weakness 1: Limited analysis** We believe that the first part of this concern is the same as that raised in Q1. For the second part, as discussed in our response to Q2, soft-only yields a lower BSD than BASED. This means that, at high-entropy positions, all candidate branches tend to make predictions similar to the soft-embedding output, whereas BASED's predictions are more distinct. ### **Weakness 2: Insufficient validation of the theoretical approximation** To examine how posterior correction affects training, we have added a gradient-level analysis in §5.6. We compare the cosine similarities of the gradients of soft-only distillation and BASED with the gradient of the explicit local Hard-Branch KL objective. The results show that adding the posterior term increases the mean gradient cosine similarity from 0.713 to 0.727 (paired $\Delta=+0.0139$, 95% CI $[0.0030,\,0.0346]$), with improved alignment at 80.80% of the evaluated positions. This indicates that the posterior correction can align the gradient more closely with that of the explicit local Hard-Branch KL objective. Details of the soft-to-mixture approximation error are provided in the new Appendix H: $$\lVert T_{\mathrm{soft}}-T_{\mathrm{mix}}\rVert_1=0.827,$$ $$\lVert S_{\mathrm{soft}}-S_{\mathrm{mix}}\rVert_1=0.737,$$ $$\lvert\mathcal L_{\mathrm{soft}}-\mathcal L_{\mathrm{mix}}\rvert=0.733.$$ The results show that the first-order approximation is not quantitatively tight at these selected positions. Although soft KL is not an accurate surrogate for explicit mixture KL in nonlinear LLMs, the posterior correction still introduces explicit branch-specific supervision that is absent from the soft-KL objective. ### **Weakness 3: Performance gains** We agree that the performance gains of the full BASED method over the soft-only variant are not large. We would like to clarify that soft-only is a component of BASED and should not be considered an independent baseline. To our knowledge, we are the first to study soft embeddings in the context of on-policy distillation. The soft-only variant is also part of our contribution. Furthermore, the gradient and behavioral analyses in §5.6 and §5.7 support the role of branch-specific supervision.
Reviewer_WG3aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies OPD for reasoning LLMs from a branch-level perspective. The authors show that an idealized Hard-Branch KL objective decomposes exactly into a Mixture KL term and a Branch-Posterior KL term, and that soft-embedding distillation approximates only the mixture term. Based on this, they propose BASED, which combines entropy-gated soft trajectories with a local branch-posterior correction.

Critical strengths

1.The branch-level view of OPD is well motivated. The KL decomposition clearly explains what soft-embedding distillation misses. 2.The method is lightweight. The correction is applied only at a few selected positions, and inference is unchanged.

Critical weaknesses
  1. The experimental scope is limited. All models come from the Qwen3 family, and the evaluation is almost entirely math (GPQA-D is the only exception). The authors should include at least one code generation task to show that the method generalizes beyond math reasoning. In addition, the only distillation baseline is vanilla forward-KL OPD, which is weak. Stronger baselines should be included, such as: Entropy-Aware OPD (Jin et al., 2026), which the paper cites and which uses a nearly identical setting, Reverse-KL OPD, which in Appendix E already outperforms forward-KL BASED , G-OPD/ExOPD (Yang et al., 2026)
  2. All results come from a single run (seed 42), with no variance, confidence intervals, or significance tests.
  3. Computational cost is not reported. Without a compute-matched comparison, such as vanilla OPD with more rollouts per prompt, it is hard to tell whether the gains come from the method itself or from the extra computation.
Questions for authors

I would be willing to raise my score if the authors add the stronger baselines listed above, report results over multiple random seeds with variance, and provide a quantitative comparison of computational cost.

3 comments

We sincerely thank the reviewer for the constructive feedback and valuable suggestions, which have helped us improve the experimental evaluation and strengthen the manuscript.

Q1.1: Code generation task

We have added HumanEval as a code generation benchmark for all experiments in the revised manuscript. The results on HumanEval are broadly consistent with those on the other benchmarks and do not change our analysis or conclusions.

Q1.2: EOPD and G-OPD

Our BASED method focuses on how supervision over alternative reasoning branches is aggregated and preserved, rather than on the choice of a particular token-level distillation objective. Consequently, it can be naturally combined with existing OPD methods that modify the optimization objective.

In Appendix I, we show that BASED can be readily combined with EOPD and G-OPD and compare EOPD, EOPD + BASED, G-OPD, and G-OPD + BASED across all nine benchmarks.

The results show that BASED increases EOPD's average accuracy from 26.26% to 26.64%, while leaving G-OPD's average accuracy essentially unchanged.

The combination simply replaces each KL component in EOPD/G-OPD with its corresponding branch-aware BASED objective.

For EOPD, the combination is:

=D_{\mathrm{RKL}}^{(t)}+w_tD_{\mathrm{FKL}}^{(t)}.$$ $$\mathcal L_{\mathrm{EOPD+BASED}}(x,\mathcal R) =\frac{1}{L}\sum_{t=1}^{L} \left[\mathcal L_{\mathrm{BASED}}^{R}(t) +w_t\mathcal L_{\mathrm{BASED}}^{F}(t)\right].$$ For G-OPD, the combination is: $$\mathcal L_{\mathrm{G\text{-}OPD}} =D_{\mathrm{RKL}}(S,U) -\lambda\mathbb E_{u\sim S} \left[\log\frac{T(u)}{U(u)}\right] =\lambda D_{\mathrm{RKL}}(S,T) +(1-\lambda)D_{\mathrm{RKL}}(S,U).$$ $$\mathcal L_{\mathrm{G\text{-}OPD+BASED}}(x,\mathcal R) =\lambda\mathcal L_{\mathrm{BASED}}^R(x,\mathcal R;T) +(1-\lambda)\mathcal L_{\mathrm{BASED}}^R(x,\mathcal R;U).$$ Further details are provided in Appendix I of the revised manuscript. ### **Q2: Multiple random seeds and variance** We regret that limited computational resources prevented us from conducting multi-seed training. We understand your concern about the reliability of our results and would like to provide supporting evidence. In our implementations of all distillation training algorithms, we calculate the KL divergence over the full vocabulary. Compared with the commonly used top-k estimation, full-vocabulary calculation has significantly lower variance. Furthermore, our training algorithms follow the GKD-style OPD paradigm, which directly treats the KL divergence as the loss and uses backpropagation to update the model. Compared with PG-style OPD, which uses policy gradients to update the model, GKD-style OPD is known for its lower variance. Taken together, these considerations lead us to believe that our training procedure has relatively low variance. During evaluation, we estimate the metrics using multiple samples. We sample 8 responses per problem for larger benchmarks and 32 responses per problem for smaller benchmarks to reduce variance. We therefore believe that the results reported in our paper are reliable. ### **Q3.1: Computational cost** In §5.5 of the revised manuscript, we compare vanilla OPD and BASED under strictly matched rollout lengths, reporting the average training time and FLOPs per step. The average training times per step for vanilla OPD and BASED are 106.24 s and 119.01 s, respectively. Their average computational costs per step are 63.35 TFLOPs and 66.90 TFLOPs, respectively. The results show that BASED is lightweight, increasing the time per step by only 12.0% and FLOPs by only 5.6%. ### **Q3.2: Multi-rollout OPD** As shown in our computational cost analysis, BASED is relatively lightweight. Even OPD with two rollouts is not a compute-matched comparison. Increasing OPD to two rollouts would approximately double its FLOPs to an estimated 126.70 TFLOPs per step, or $1.89\times$ BASED's cost, requiring substantially more computation. We have also added a two-rollout OPD baseline to the 1.7B results in our main table. The results show that although doubling the number of rollouts per prompt raises 1.7B OPD's average Pass@1 / Pass@8 from 23.04 / 46.61 to 23.54 / 47.32, single-rollout BASED still achieves higher averages of 24.47 / 47.86. Thus, increasing the number of rollout samples recovers part of the performance gains but still does not outperform BASED.

Thanks for the authors' reply. I will raise my score to 3.

Thank you very much for reconsidering our work and for raising your score. We sincerely appreciate your constructive feedback and your recognition of our revisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Contributions and Revisions

We sincerely thank all reviewers for their thoughtful feedback and constructive discussions, and the Area Chair for their time and consideration. The reviewers' suggestions have helped us substantially improve our manuscript.

Core Contributions. Our work investigates how to aggregate supervision from multiple plausible reasoning branches in on-policy distillation (OPD). We introduce soft-embedding trajectories into OPD for reasoning LLMs and establish an exact decomposition of the idealized Hard-Branch KL into Mixture KL and Branch-Posterior KL, revealing both the potential and limitations of soft aggregation. Based on this insight, BASED combines entropy-gated soft trajectories with a lightweight local branch-posterior correction. Our method requires no inference modifications and can be readily combined with existing OPD methods.

Key Revisions. In response to the reviewers' feedback, we have substantially strengthened the manuscript:

  • Broader Evaluation: Added HumanEval to assess generalization beyond mathematical reasoning.
  • Theoretical and Mechanistic Evidence: Added analyses of the soft-to-mixture approximation, gradient alignment, and token decision behavior. Together, these analyses provide empirical support for our theoretical motivation and help explain how posterior correction preserves branch-specific information.
  • Computational Efficiency: Measured training time and FLOPs, demonstrating limited overhead compared with vanilla OPD. Moreover, two-rollout OPD incurs substantially higher computational cost than BASED, yet still falls short of BASED in average performance.
  • Compatibility: Demonstrated how BASED can be integrated with EOPD and G-OPD, and empirically evaluated these combinations.

All revisions in the updated manuscript are highlighted in blue for ease of reference.

We sincerely appreciate the reviewers' constructive suggestions, which have helped us strengthen the empirical evidence, clarify our contributions, and further improve the presentation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.