- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies GKD-style on-policy distillation (OPD) for reasoning models. Its starting point is the observation that a sampled rollout follows only one continuation at each branching point: the next-token distribution at that point is supervised by the token-level KL, but the downstream states reached through the other branches receive no direct teacher supervision along that rollout. The authors treat the student-induced branch prior as shared by the teacher and the student, and define the "Hard-Branch KL" as the prior-weighted average of per-branch KLs. By the KL chain rule, this decomposes exactly into a mixture KL and a branch-posterior KL. Following the first-order approximation used in Soft Thinking, they then argue that the soft-trajectory KL, computed on soft-embedding prefixes, approximates the mixture term at the objective level and omits the posterior term.
BASED builds on this analysis with three components:
- The student rolls out with soft embeddings at high-entropy positions (normalized top-3 entropy > 0.7).
- Forward KL is applied along this soft trajectory.
- A local posterior correction is added at up to 50 positions per trajectory: it expands the immediately preceding soft anchor into its top-3 hard tokens and supervises only the next token.
Inference is unchanged.
The experiments distill Qwen3-8B (non-thinking) into Qwen3-1.7B-Base and Qwen3-4B-Base with LoRA (128 steps, 4,096 DAPO-Math prompts, one rollout per prompt). Evaluation covers MATH-500, AIME 2024/25/26, HMMT Feb 2026, AMC 2023, MinervaMath, and GPQA-Diamond. Over vanilla OPD, BASED raises average Pass@1 by 1.55 pt (1.7B) and 1.65 pt (4B), and Pass@1 improves on every benchmark at both scales. Average Pass@8 rises by 1.09 pt and 1.88 pt, respectively. The paper also includes a 1.7B ablation with a soft-only baseline (Table 2), reverse-KL results (App. E), and a Self-BLEU/chrF diversity analysis on two AIME 2026 problems.
Critical strengths
The mixture/posterior split shows cleanly what soft-token training can and cannot teach: a student can match the mixture perfectly and still lose which branch leads where. The two-branch example in §3.2 makes this very clear. The decomposition itself is just the chain rule, but I think it is a useful lens for work on soft/latent reasoning. The appendix is also careful not to overclaim: it gives only sufficient conditions, states that ε_soft can take either sign, and separates objective-level from gradient-level statements. This is the main reason I think the direction is worth pursuing.
As a full method, BASED achieves consistently higher Pass@1 than vanilla OPD on all eight benchmarks at both scales, without changing inference. This includes recent benchmarks with low contamination risk (AIME 2026, HMMT Feb 2026). The supplementary code is also fairly complete (configs for the main and ablation runs, and the exact cache-VJP implementation), which made the checks below possible.
Critical weaknesses
1. The experiments do not show that the branch-posterior correction, the paper's actual contribution, drives the gains.
The correction itself adds little.
- The only accuracy experiment that isolates the correction is the 1.7B ablation in Table 2. It uses a single seed and reports only 2 of the 8 benchmarks individually.
- There, soft-only distillation alone already raises average Pass@1 from 19.02 (vanilla OPD, Table 1) to 20.07, and average Pass@8 from 41.69 to 42.78. Adding the correction gives +0.50 Pass@1 and +0.01 Pass@8.
- Backing out from the averages, BASED leads soft-only by only about 0.6 pt in summed Pass@1 over the six benchmarks not shown, and soft-only actually leads BASED by about 6.7 pt in summed Pass@8.
- Of the two benchmarks shown, GPQA-D at 1.7B is at chance level for a four-option task: Pass@1 lies between 23.2 and 26.6 in every row of Table 2.
The stability of the results cannot be assessed. All runs use a single seed (42), with no confidence intervals. The spread among the BASED hyperparameter variants in Table 2 (average Pass@1 19.82–20.57) is already larger than the effect of the correction. The ablations that justify the default settings were run on the same test benchmarks, and no validation split is mentioned. The released code also treats the other target policy as the default: base.yaml sets target_policy: next_soft_anchor, and the docstring in scripts/based.py describes immediate_next_token as "retained only as a target-policy ablation"; the main-experiment configs set immediate_next_token explicitly.
The reverse-KL results in App. E also run against the diversity claim. There, BASED lowers average Pass@8 (44.36 → 42.63, lower on 6 of the 8 benchmarks). Vanilla RKL-OPD also beats forward-KL BASED on both metrics (21.52/44.36 vs. 20.57/42.79).
There is no compute-matched baseline. §3.1 frames OPD's limitation as a branch-coverage problem that comes from a finite number of rollouts. The natural baseline is then OPD with more rollouts or more prompts at matched compute. The code supports num_rollouts > 1, but only results with one rollout per prompt are reported.
The paper also gives only an analytical bound on BASED's training cost (at most kN_post branch-conditioned predictions per model and trajectory, l. 331–332), not a measured cost. BASED adds the following overhead:
- It keeps both models' KV caches from the soft replay in memory until the correction is done.
- It computes up to 150 extra branch-conditioned predictions per model per trajectory.
- For each selected target (up to 50 per trajectory), it runs a separate backward pass through the branch suffix and accumulates the cache cotangents in a buffer the size of the student's KV cache. (The VJP through the master prefix is merged into the soft-KL backward and runs once.)
The released evaluator grades more leniently than App. C.3 describes. App. C.3 states that math answers are evaluated with Math-Verify and GPQA-D answers by matching the reference option letter, but it does not say which answer in a response is graded. The released evaluator (evaluate_fixed.py → scripts/eval_fixed_grading.py, policy any_explicit_answer_v1) applies these checks to every answer candidate in the response:
- A sample is counted as correct if any
\boxed{}or "answer is / answer:" span anywhere in the response matches the gold answer, including intermediate answers that the model later abandons. - For example, a GPQA-D response that first boxes A and then concludes with
\boxed{C}is graded correct when the gold label is A, while the repository's own strict extractor (extract_choice_answer) returns C. Likewise for math, a response that boxes 5, abandons it, and ends with\boxed{7}is wrong under Math-Verify applied to the whole response, but correct under the released evaluator when the gold answer is 5. - For responses that contain a
\boxed{}answer, this rule can only raise scores relative to final-answer grading, and it can inflate Pass@k more on hard benchmarks. Whether it interacts with the training method is unknown.
Since the headline gain is about 1.5 pt, I think this needs to be checked.
2. The theory is not well connected to the method as implemented.
The key approximation is not supported. The step that actually carries the argument is not Prop. 1 but the claim that soft-prefix predictions approximate the branch mixture (T_soft ≈ T_mix, §3.3).
- App. A.2 does not establish this at LLM scale. The bound on T_soft vs. T_mix depends on a curvature constant (B_M) and on the dispersion of the hard prefix embeddings, and neither is estimated. The next step, from T_soft ≈ T_mix to L_soft ≈ L_mix, assumes that every coordinate of the four distributions is at least η. For Qwen3 this forces η ≤ 1/|V| ≈ 6.6×10⁻⁶, so the coefficient on the δ_S term is at least ~1.5×10⁵. The paper itself notes that very small probabilities can make this bound loose (l. 728–730).
- There is also prior evidence pointing the other way. Wu et al. (ICLR 2026, "LLMs are Single-threaded Reasoners") report that LLMs process soft inputs mainly through the highest-probability token. This is consistent with the collapse observation of SeLaR, which the paper itself cites. If this holds here, the soft KL behaves more like a KL on the top-1 branch than like the "mixture term".
- A local version of this gap, T_soft vs. T_mix,local at the selected targets, is cheap to measure, since the correction already computes the k branch-conditioned teacher/student distributions at those positions. It is not reported.
The correction covers much less than the motivation. The introduction (l. 42–48) motivates BASED by the lack of direct supervision on the downstream states of other branches. The correction, however, is narrow:
- it expands only the immediately preceding anchor;
- it supervises only the next token;
- it uses only the posterior term.
The "next soft anchor" variant extends the horizon, but it still expands only one anchor, uses only the posterior term, and keeps the recorded intermediate tokens fixed.
Since the k branch-conditioned distributions are computed anyway, the exact local Hard-Branch KL (Σ_v q_b(v) KL(T_v‖S_v), i.e., local mixture + local posterior) is available at the same cost. Wider windows are also already supported in the code (window_size; k^r ≤ 64 under the default max_assignments, i.e., r ≤ 3 for k = 3). These two are the experiments that would directly test whether the decomposition matters, and neither is reported.
"Branch collapse" is supported only indirectly. The abstract presents collapse as a finding ("we find that naive soft-trajectory distillation loses branch-specific information and may lead to branch collapse"), but the evidence is indirect. Soft-only does have the highest Self-BLEU on both analyzed problems (53.45, 42.59), but its average Pass@8 is essentially the same as BASED's (42.78 vs. 42.79).
The diversity analysis itself is also thin:
- It covers only two problems, and no selection criterion is given.
- Compared with vanilla OPD, BASED's Self-BLEU is mixed (52.26 vs. 52.16 on Q4, 41.56 vs. 42.20 on Q15).
- Correct and incorrect responses are clustered without matching subset sizes (App. D). A model that solves a problem more often therefore tends to get more correct clusters and fewer incorrect clusters, regardless of diversity.
Questions for authors
Questions 1–5 could change my assessment; Question 6 is a clarification request.
-
Grading. Were Tables 1, 2, and 5 all graded with
any_explicit_answer_v1? Could you re-grade the existing predictions with a final-answer grader (the last\boxed{}for math, the strict extractor for GPQA-D)? Please also report, for each method, the fraction of samples counted as correct that were matched on a candidate other than the final answer. The evaluator already logs the matched candidate for each sample, so no regeneration should be needed. -
Seeds and the full ablation. Could you run vanilla OPD, soft-only, and BASED at 1.7B with at least three seeds and report confidence intervals? Please report all eight benchmarks for every row of Table 2.
-
Compute. What are the per-step wall-clock time and memory of OPD, soft-only, and BASED? At matched compute, how does BASED compare with OPD using more rollouts per prompt (e.g., 2 or 4) or more prompts?
-
The §3.3 approximation. At the selected targets, how large are ‖T_soft − T_mix,local‖₁ and the local ε_soft compared with the posterior term? How do the soft-anchor rate and the ratio of
posterior_contributiontosoft_trajectory_lossevolve during training? The training script already logssoft_anchor_rate,soft_trajectory_loss, andposterior_contributionat every step. -
A direct test of the decomposition. How do the following variants compare with BASED?
- (a) Using the exact local Hard-Branch KL instead of the posterior term alone.
- (b) A window size of r = 2 or 3.
- (c) A control that feeds the top-1 token embedding instead of the mixture at the same anchors. This separates the effect of soft aggregation from the effect of removing sampling noise at high-entropy positions.
-
The diversity analysis (§5.4).
- How were Q4 and Q15 chosen?
- Do the conclusions hold on more problems and with size-matched correct subsets?
- Do they hold at the threshold of Matsutani et al. (similarity 60 / distance 0.4)? The comment in the released config also cites this threshold, while the paper uses 55 / 0.45.
- The released script grades these samples with the any-answer rule and, before computing Self-BLEU/chrF, truncates correct responses at the first matched answer. App. D mentions neither step. Also,
trajectory_diversity.pyimports are_evaluatemodule that is not in the supplementary material, so the analysis does not run as released.
If the gains over both soft-only and OPD hold under re-grading, multiple seeds, and compute-matched baselines, and the measurements in Questions 4–5 support the mixture/posterior account, I would be glad to raise my score.
7 comments
Q4: The approximation in §3.3
Approximation error.
We report the soft-to-mixture approximation error in the new Appendix H:
The results show that the first-order approximation is not quantitatively tight at these selected positions.
Although soft KL is not an accurate surrogate for explicit mixture KL in nonlinear LLMs, the posterior correction still introduces explicit branch-specific supervision that is absent from the soft-KL objective.
We additionally conducted gradient and behavioral analyses to examine the effect of posterior correction.
Gradient-level analysis.
To examine how posterior correction affects training, we have added a gradient-level analysis in §5.6.
We compare the cosine similarities of the gradients of soft-only distillation and BASED with the gradient of the explicit local Hard-Branch KL objective.
The results show that adding the posterior term increases the mean gradient cosine similarity from 0.713 to 0.727 (paired , 95% CI ), with improved alignment at 80.80% of the evaluated positions.
This indicates that the posterior correction can align the gradient more closely with that of the explicit local Hard-Branch KL objective.
Behavioral analysis.
We have added an analysis of token decision behavior in §5.7 of the revised manuscript.
We first define a metric called branch-soft distinction:
=\sum_a q_a D\left(p_X^a\Vert s_X\right),$$ where $a$ indexes hard branches, $q_a$ is the corresponding branch weight, $p_X^a$ is the next-token distribution conditioned on hard branch $a$, and $s_X$ is the next-token distribution produced by the soft embedding, for $X\in\{\text{Soft only},\text{BASED}\}$. A smaller BSD indicates that the branch-conditioned predictions are closer to the soft-embedding prediction, while a larger BSD indicates greater differences between them. BASED increases BSD from 3.528 to 3.543 (paired $\Delta=+0.0153$, 95% CI $[0.0082,\,0.0223]$), indicating that the branch-conditioned predictions become more distinct from the soft-embedding prediction. ### **Q5: A direct test of the decomposition** Thank you for these suggestions. Although we have not evaluated these variants directly, we clarify our design choices and provide supporting evidence from existing experiments. **(a) Exact local Hard-Branch KL.** Since the exact decomposition gives $\mathcal L_{\mathrm{hard}}=\mathcal L_{\mathrm{mix}}+\mathcal L_{\mathrm{post}}$, replacing the posterior correction with Hard-Branch KL while retaining soft KL would introduce an additional mixture-matching term, potentially duplicating supervision and changing the objective's relative weighting. Our posterior-only correction specifically targets the missing branch information. Moreover, §5.6 shows that it improves gradient alignment with the explicit Hard-Branch KL objective from 0.713 to 0.727 (95% CI for $\Delta$: $[0.0030,\,0.0346]$). **(b) Larger windows ($r=2$ or $3$).** The window size parameter is retained in our code only for future exploration and is not mentioned in our paper. Expanding branches across $r$ soft anchors requires $k^r$ branch combinations. With $k=3$, this increases from 3 ($r=1$) to 9 or 27. We choose $r=1$ to balance branch-specific supervision and computational efficiency while retaining full-trajectory soft KL. Our cost analysis (§5.5) shows only 5.6% additional FLOPs over vanilla OPD. Larger windows may capture additional interactions but incur substantially greater branch-expansion costs. **(c) Top-1 embeddings.** Unlike top-1 embeddings, soft embeddings retain the weighted contributions of multiple candidates, providing the first-order approximation to mixture-level predictions described in §3.3. Our $\beta$ ablation offers indirect support: sharpening the mixture toward top-1 ($\beta=0.7$) reduces average Pass@1 / Pass@8 from 24.47 / 47.86 to 23.78 / 46.56. Nevertheless, this does not fully isolate aggregation from reduced sampling randomness, and we acknowledge the proposed top-1 control as a useful direction for further investigation.Q6: The diversity analysis
How were Q4 and Q15 chosen?
We selected questions for the diversity analysis based on the following criteria: (a) the math problem admits different solution methods; (b) vanilla OPD, soft-only, and BASED achieve similar accuracy on the problem; and (c) their accuracy is high enough to generate sufficient samples for the analysis.
Do the conclusions hold on more problems and with size-matched correct subsets?
After selecting Q4 and Q15 based on the criteria above, we conducted experiments only on these two questions. Whether the conclusions hold on more problems therefore remains unexplored.
Do they hold at the threshold of Matsutani et al. (similarity 60 / distance 0.4)? The comment in the released config also cites this threshold, while the paper uses 55 / 0.45.
In our experiments, using the original threshold from RL Squeezes, SFT Expands did not produce effective clustering: every trajectory was assigned to a separate cluster. We therefore increased the distance threshold to enable clustering.
The released script grades these samples with the any-answer rule and, before computing Self-BLEU/chrF, truncates correct responses at the first matched answer. App. D mentions neither step. Also, trajectory_diversity.py imports a re_evaluate module that is not in the supplementary material, so the analysis does not run as released.
To address the same non-stop issue described in our response to Q1, we truncate each trajectory at the correct answer to eliminate interference from subsequent unrelated content. This allows the algorithm to analyze the diversity of the correct trajectories more accurately.
We greatly appreciate the reviewer's thorough evaluation and detailed suggestions, which have helped us clarify important methodological choices and improve the rigor of our work.
Q1: Grading
Yes, we grade the results using
any_explicit_answer_v1.This choice addresses the non-stop issue in OPD: the model first generates a correct response but fails to stop, subsequently generating unrelated content.
This is a well-known issue that has been studied by many researchers (Yang et al., 2026; Plyusov et al., 2026; Fu et al., 2026).
The non-stop issue causes the strict extractor to misclassify many responses. We use
any_explicit_answer_v1to assess the student model's reasoning ability more accurately.For example, in an evaluation on AIME 2026, replacing the strict extractor with
any_explicit_answer_v1recovers 22 correct responses that were originally classified as incorrect because of the non-stop issue. Only 4 responses first produce a correct answer and then change it to an incorrect one.Moreover, since we study OPD rather than RLVR, we do not reward the model based on correctness, so we do not expect reward hacking to be an issue.
The full regraded results are too lengthy to include here. As an example, we present the 1.7B results for vanilla OPD, soft-only, and BASED using the strict extractor.
The performance gains with the strict extractor are even larger than those with
any_explicit_answer_v1.Q2: Seeds and the full ablation
We regret that limited computational resources prevented us from conducting multi-seed training.
We understand your concern about the reliability of our results and would like to provide supporting evidence.
In our implementations of all distillation training algorithms, we calculate the KL divergence over the full vocabulary. Compared with the commonly used top-k estimation, full-vocabulary calculation has significantly lower variance. Furthermore, our training algorithms follow the GKD-style OPD paradigm, which directly treats the KL divergence as the loss and uses backpropagation to update the model. Compared with PG-style OPD, which uses policy gradients to update the model, GKD-style OPD is known for its lower variance. Taken together, these considerations lead us to believe that our training procedure has relatively low variance.
During evaluation, we estimate the metrics using multiple samples. We sample 8 responses per problem for larger benchmarks and 32 responses per problem for smaller benchmarks to reduce variance. We therefore believe that the results reported in our paper are reliable.
Q3: Compute
In §5.5 of the revised manuscript, we compare vanilla OPD and BASED under strictly matched rollout lengths, reporting the average training time and FLOPs per step.
The average training times per step for vanilla OPD and BASED are 106.24 s and 119.01 s, respectively.
Their average computational costs per step are 63.35 TFLOPs and 66.90 TFLOPs, respectively.
The results show that BASED is lightweight, increasing the time per step by only 12.0% and FLOPs by only 5.6%.
As shown in our computational cost analysis, BASED is relatively lightweight. Even OPD with two rollouts is not a compute-matched comparison.
Increasing OPD to two rollouts would approximately double its FLOPs to an estimated 126.70 TFLOPs per step, or 1.89× BASED's cost, requiring substantially more computation.
We have also added a two-rollout OPD baseline to the 1.7B results in our main table.
The results show that although doubling the number of rollouts per prompt raises 1.7B OPD's average Pass@1 / Pass@8 from 23.04 / 46.61 to 23.54 / 47.32, single-rollout BASED still achieves higher averages of 24.47 / 47.86. Thus, increasing the number of rollout samples recovers part of the performance gains but still does not outperform BASED.