acceptodds
Under review as a conference paper at ICLR 2027

Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

Abstract

On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher–reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD with three SFT references spanning Qwen and Olmo model families. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released upon publication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Distillation, cross-teacher distillation, style bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rFd8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a practical issue in on-policy distillation: whether the student is learning actual capabilities or merely imitating the teacher’s style. It identifies the style mismatch problem in cross-teacher OPD, analyzes its underlying behavior, and proposes a corresponding mitigation method. The problem is clearly defined and well motivated, and the proposed approach achieves consistent improvements across multiple benchmarks and model settings.

Critical strengths

The paper addresses a realistic and practically relevant problem in OPD: when the SFT reference and the OPD teacher come from different models, the student may learn stylistic differences rather than actual capabilities. The problem is concrete and clearly defined, and the paper provides a useful insight into how cross-teacher mismatch can affect OPD supervision. The proposed solution is also reasonable and well aligned with the identified issue.

Critical weaknesses

1.Limited task settings. The experiments focus mainly on single-turn reasoning and code-generation tasks. It remains unclear whether the proposed method can generalize to more realistic long-horizon agentic settings, where the student interacts with an environment and the teacher must supervise evolving states and tool-use decisions.

2.Engineering-heavy solution. Although the underlying insight is elegant and practically relevant, the proposed solution is relatively engineering-oriented and less conceptually elegant. The paper would benefit from a clearer theoretical justification and comparisons with simpler alternatives for mitigating cross-teacher style mismatch.

Questions for authors

Could the authors evaluate whether the proposed method also works in agentic settings, especially on tasks with clear outcome rewards such as SWE? If effective, this would make the method more practically valuable, since such settings involve not only long-horizon reasoning but also tool use, environment interaction, and potential effects from the agent harness. It would be particularly interesting to see whether the method can distinguish stylistic mismatch from genuine policy differences in these more complex settings.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer rFd8

W1 / Q1. Agentic settings and tool use.

We added a multi-turn tool-use experiment. The teacher is Qwen3-30B-A3B-Thinking-2507 and the student is Klear-AgentForge-8B, an agentic tool-use model. Training uses student rollouts on ToolMind tool-use tasks, including multi-turn ones, for 150 updates with the paper's settings. We evaluate on BFCL v3 (4,441 items in 17 categories), whose multi-turn tasks run stateful, executable environments over many turns and grade the final state.

BFCL v3BaseLightning OPDLightning OPD 2.0
Non-live85.281.585.2
Live81.279.981.7
Multi-turn50.131.053.9
Overall72.264.173.6

2.0 matches or improves over the base model in every group, by 1.4 points overall and 3.8 points on multi-turn tasks, while Lightning OPD falls below the base model in every group. Here the main style mismatch is tool-call formatting. Lightning OPD imitates the teacher's format and breaks the student's native one, whereas 2.0 removes this recurring disagreement and keeps the tool-use signal.

Full SWE-style training does not fit in the rebuttal period. We have added this experiment to Appendix F.11 (Table 19) and list SWE settings as future work in the Limitations (Appendix G).

W2. Engineering-heavy solution, theoretical justification and simpler alternatives.

The method rests on one statistical idea. Regress the teacher–reference disagreement d=ℓT−ℓRd = \ell^T - \ell^R on features zz that cannot encode whether a step is correct (token identity, coarse position and surprisal), then subtract the prediction. The lookup tables are group means, the prior masses are standard shrinkage for rare groups, and cross-fitting is the standard double-ML device. The whole estimator runs in under 30 seconds on one CPU core.

Theory. We have added a formal analysis of the training signal with proofs in Appendix D (Propositions D.1 to D.3).

  1. Frozen-replay OPD is log-probability regression. With a detached advantage, the update is exactly −∇12,E[(target−log⁡πθ)2]-\nabla \tfrac{1}{2}\\,\mathbb{E}[(\text{target} - \log \pi_\theta)^2]. Lightning OPD regresses onto ℓT\ell^T, and 2.0 onto ℓR\ell^R plus a residual bounded by 1.
  2. What the recurring part does. With d=b(z)+vd = b(z) + v and b(z)=E[d∣z]b(z) = \mathbb{E}[d \mid z], the Lightning OPD update splits exactly into g=gv+gbg = g_v + g_b. Because zz contains the sampled token, gbg_b is the gradient of the expected "style score" Ey∼πθ[b(z)]\mathbb{E}_{y \sim \pi_\theta}[b(z)], which moves the student toward the teacher's recurring lexical preferences whatever the content of the step. Subtracting b(z)b(z) keeps only gvg_v, the part that varies with the specific problem and step.
  3. Why cross-fitting. Folds are formed by prompt, so a row's estimate does not depend on its own disagreement. An in-sample table would subtract part of each row's own signal.

Simpler alternatives. A moment-matched global affine control (same mean and scale, no conditioning on zz) is 1.2 points (Qwen3-4B) and 1.1 points (Klear-8B) below 2.0 on AIME24 (Table 6). On Qwen3-4B, 2.0 also beats IW-OPD by 3.5 points and TA-OPD by 6.1 points on the math average, and both on each of AIME24, AIME25 and HMMT25 (Table 1).

We have added this regression view to the method section (Section 3.4, "Regression view").

Reviewer_rFd8OpenReview ↗

Thanks for response

I maintain my score. please add the discussion to camera-ready.

Thanks very much for your suggestions that helped us strengthen our work. We are grateful for your time and support.

Reviewer_h4UKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-teacher on-policy distillation, where the OPD teacher differs from the model that generated the SFT demonstrations. It proposes Lightning OPD 2.0, which estimates recurring teacher–reference disagreement using cross-fitted token and context lookup tables and subtracts it from the distillation signal. Experiments on mathematical reasoning and code generation show improvements over Lightning OPD across three student models.

Critical strengths
  1. The paper is well organized and easy to follow, with clear explanations of the motivation, method, and experimental setup.
  2. The motivation is practically relevant, as SFT data may have mixed or unknown provenance, making teacher consistency difficult to ensure.
  3. The proposed correction is intuitive and lightweight, preserving the offline training workflow while showing consistent gains across the evaluated settings.
Critical weaknesses

Please quantify the additional runtime, peak memory, and storage costs of reference scoring, cross-fitted estimation, and corrected-cache construction, and report the end-to-end overhead relative to Lightning OPD under matched hardware and data conditions.

Questions for authors
  1. Please justify the equal weighting of the token and context lookup predictions when estimating recurring bias.
  2. Have the authors considered periodically refreshing rollouts with the current student, rescoring them with the teacher, and updating the bias estimates? It remains unclear whether estimates from the initial SFT replay remain effective as the student evolves.
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer h4UK

W1. Overhead of reference scoring, cross-fitted estimation and corrected-cache construction.

2.0 adds 13.3% wall time and 12.6% GPU-hours end to end over 150 updates, almost all from a one-time reference forward pass before training. We measured every stage on the main Qwen3-4B-SFT setting (17,398 cached responses, 70.2M response tokens), on one 8× H200 node for the GPU stages and one CPU allocation for the builder. Both methods share the same replay and teacher scores.

StageLightning OPDLightning OPD 2.0Memory (2.0)Storage
Teacher scoring9.5 min (1.26 GPU-h)same56.9 GiB weights / GPU415 MB cache (both methods)
Reference scoring (once per replay)—+8.7 min (+1.16 GPU-h)7.5 GiB weights / GPU281 MB reference scores (float32)
Cross-fitted estimation (5 folds)—+6 s on 1 CPU core4.3 GiB RAM—
Corrected-cache construction—+21 s on 1 CPU core10.0 GiB RAM460 MB corrected cache (+45 MB, +11%)
Training, per update22.42 s22.50 s (+0.4%)50.6 GB / GPU, same for both—
End-to-end, 150 updates69.8 min (9.31 GPU-h)79.1 min (10.48 GPU-h)
Relative overhead+13.3% wall time, +12.6% GPU-h
  • Training is unchanged. 2.0 only replaces the cached teacher score with the corrected one, so its training step is the same as that of Lightning OPD.
  • Reuse across teachers. Reference scores do not depend on the teacher. For a new teacher on the same replay, the only extra cost is the builder: under 1% wall time and 0 GPU-h.
  • Upper bound. End-to-end time covers scoring, the builder, training, checkpointing and export. Rollout generation is identical for both methods and excluded, so +13.3% is an upper bound.
  • Storage. Even with the reference scores kept for reuse, the extra files total about 0.74 GB, under a tenth of one bf16 checkpoint of the 4B student (8 GB).

We have added this table to Appendix F.8 (Table 15).

Q1. Equal weighting of the token and context lookups.

We did not tune this weighting. Equal weights are the simplest combination and keep the estimate on the scale of a single lookup. To check whether it matters, we retrained the Qwen3-4B-SFT 2.0 model with token:context weights of 2:1 and 1:2, all else identical. The 1:1 and Lightning OPD rows are from Table 1. Pass@1 uses 64 samples per problem.

Token : contextAIME24AIME25HMMT25Avg
2 : 164.653.538.152.1
1 : 163.555.036.751.7
1 : 261.056.936.051.3
Lightning OPD59.651.634.648.6

All three weightings are within 0.4 points of each other on average, 2.7 to 3.5 points above Lightning OPD, and above it on every benchmark. Their training targets correlate at 0.989 per token.

Using both lookups is what matters. In the component ablation (Table 2), the token-only and context-only variants are below the full method in every column, by up to 2.2 points. Token identity captures lexical preferences, and position and reference surprisal capture where the disagreement occurs. The out-of-fold fit favors the token lookup on the Qwen cache and the context lookup on the Klear cache, while equal weighting is within 0.004 R2R^2 of the best fit on both. Equal weights are therefore a robust default when the better lookup is not known in advance.

We have added these results to Appendix F.9 (Table 16).

AuthorsOpenReview ↗

Rebuttal for Reviewer h4UK, continued

Q2. Periodic refresh of rollouts, teacher scores and bias estimates.

We ran exactly this, and the estimates from the initial SFT replay are sufficient. In the main Qwen3-4B-SFT run (150 updates), at each refresh point we sample fresh rollouts from the current student, rescore them with the teacher and the reference, rebuild the corrected cache, and continue with the optimizer state carried over. The refreshed runs branch off the paper's run at update 30 or 50, so each comparison isolates the effect of refreshing (Pass@1, 64 samples per problem).

RunBias estimatesAIME24AIME25HMMT25Avg
Lightning OPD 2.0, no refreshinitial SFT replay63.555.036.751.7
Refresh at 50, 100refit on fresh rollouts62.154.035.850.6
Refresh at 30, 60, 90, 120refit on fresh rollouts60.855.435.450.6
Refresh at 50, 100initial SFT replay (frozen)63.856.536.552.2
  • Refreshing brings no meaningful gain. With refit estimates, both refresh frequencies are 1.1 points below the no-refresh run on average. Refreshing only the rollouts changes the average by +0.5 points.
  • Re-estimating the bias brings no gain. On the same refreshed rollouts, refit estimates are 1.6 points worse than frozen ones.

Why the initial estimates stay valid. 2.0 keeps the student close to its reference, so the recurring bias barely moves. During the refresh runs, the refit and initial estimates correlate at 0.89–0.97 per token, and the training targets at 0.97 or more. After training, we drew fresh rollouts on 2,000 training prompts and compared the frozen estimates with estimates refit on each policy's own rollouts:

Rollouts fromR2R^2 of frozen estimatesToken-table corr., frozen vs. refitMean ∣log⁡π−log⁡πR∣\lvert \log \pi - \log \pi_R \rvert per token
SFT reference (control)13.9%0.98—
Lightning OPD 2.0 (trained)11.0%0.950.05
Lightning OPD (trained)2.3%0.600.24

Lightning OPD drifts five times as far from the reference, and its frozen tables decorrelate (0.60). 2.0 avoids this because its objective is anchored to the reference and its teacher signal is clipped. A single estimate from the initial replay therefore suffices, which keeps the method fully offline. We have added both experiments to Appendix F.10 (Tables 17 and 18).

Thank you for the authors' detailed response. My concerns have been adequately addressed, and I am satisfied with the clarifications provided. Therefore, I maintain my positive score.

Thanks very much for your suggestions that helped us strengthen our work. We are grateful for your time and support.

Reviewer_f6cTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study offline on-policy distillation (OPD) when the distillation teacher may differ from the SFT demonstration generator. They propose Lightning OPD 2.0, which uses cached student rollouts to estimate recurring teacher–reference disagreement as a proxy for style bias. Cross-fitted lookup tables based on token identity and coarse contextual features provide an offset that is subtracted from the token-level supervision before applying the Lightning OPD update. Experiments demonstrate consistent improvements over Lightning OPD across three student models on mathematical reasoning and code generation benchmarks.

Critical strengths

The method is reasonable and intuitive, and it seems straightforward to applied to Lightning OPD.

The authors demonstrate good results with Lightning OPD 2.0 across different student models on mathematical reasoning and code generation benchmarks.

Critical weaknesses

It is not sure whether the recurring disagreement contains only style, as it might also have repetitive reasoning errors.

If the method can reduce the style bias across teachers, the paper would benefit from more cross-teacher experiments (apart from Table 8), by fixing the same student model and training on different teacher models, demonstrating the effectiveness of Lightning OPD 2.0. Or training the same student model but on different mixes of OPD teachers. Such evaluation is missing.

Please see further questions.

Questions for authors

Since AIME and HMMT are a small benchmark, it would be great to see Pass@1 and Pass@8, as well as error bars.

Have authors evaluated on other benchmarks such as OlympiadBench, AMC, GPQA?

How to efficiently choose correct smoothing prior masses as it seems to be sensitive to results?

Are the cross-tokenizer scores intended to represent a normalized teacher distribution over student tokens or an unnormalized surrogate?

It seems that reference model is the same initialized model for both math and code, it would be interesting to understand what kind of topics did SFT demonstrations cover.

6 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer f6cT

W1. Could the recurring disagreement also contain recurring reasoning errors?

We agree it is not guaranteed to be pure style. But 2.0 keeps the teacher's signal against repetitive reasoning (redundant "Wait" re-checks, cycling between alternatives), the main recurring error a token-level estimate could absorb. Otherwise its students would stay repetitive and truncate more. They do not (AIME24, AIME25, HMMT25, 64 samples per problem):

StudentMethodRepetitive (%)Truncated (%)
Qwen3-4B-SFTSFT reference12.825.7
Lightning OPD0.926.8
Lightning OPD 2.02.317.9
Klear-Reasoner-8B-SFTSFT reference0.517.2
Lightning OPD0.517.7
Lightning OPD 2.00.416.0
Olmo-3-7B-Think-SFTSFT reference2.83.1
Lightning OPD5.87.0
Lightning OPD 2.01.92.2

A response is repetitive if at least half of the word 8-grams in its last 1,000 words occur more than once. 2.0 truncates less than Lightning OPD for all three students and is never more repetitive than the SFT reference.

Why it is kept. The estimate is a group mean over token identity and (position, reference surprisal) bins, fitted on other prompts. It removes only the part of a token's penalty shared by all its uses, so if the teacher penalizes a redundant "Wait" more than a needed one, that difference stays. It explains only 14% (Qwen) and 10% (Klear) of the held-out disagreement variance. A context-free penalty on a token can be lost, as we now state in Appendix G.

We have added these statistics to Appendix F.7 (Table 14).

W2. More teachers for a fixed student, and mixtures of teachers.

Both use Qwen3-4B-SFT as the student, 150 updates, unchanged settings, and Pass@1 over 64 samples per problem.

(a) Fixed student, different teachers (all share the student's tokenizer).

TeacherMethodAIME24AIME25HMMT25Avg
—SFT reference57.552.434.948.3
Qwen3-30B-A3B-Thinking-2507Lightning OPD59.651.634.648.6
Lightning OPD 2.063.555.036.751.7
Polaris-4B-PreviewLightning OPD60.655.234.250.0
Lightning OPD 2.061.360.840.254.1
QwQ-32BLightning OPD56.750.029.245.3
Lightning OPD 2.061.054.235.450.2
P1-30B-A3BLightning OPD53.548.832.144.8
Lightning OPD 2.062.154.838.851.9
PromptCoT-2.0-SelfPlay-30B-A3BLightning OPD58.154.032.748.3
Lightning OPD 2.061.054.036.950.6

2.0 beats Lightning OPD for every teacher, by 2.4 to 7.1 points on average, and on 14 of the 15 teacher–benchmark pairs (one tie). Lightning OPD falls below SFT for two teachers, while 2.0 is above SFT for all five.

(b) Mixed teachers. We mix Qwen3-30B-A3B-Thinking-2507 (TT) and P1-30B-A3B (PP) as ℓmix=w,ℓT+(1−w),ℓP\ell^{\text{mix}} = w\\,\ell^T + (1-w)\\,\ell^P. 2.0 corrects the mixed score once, refitting b^\hat{b} on ℓmix−ℓR\ell^{\text{mix}} - \ell^R.

T : PMethodAIME24AIME25HMMT25Avg
1 : 0Lightning OPD59.651.634.648.6
Lightning OPD 2.063.555.036.751.7
2 : 1Lightning OPD59.651.732.547.9
Lightning OPD 2.060.854.835.250.3
1 : 1Lightning OPD57.351.229.846.1
Lightning OPD 2.062.556.037.151.9
1 : 2Lightning OPD54.653.333.147.0
Lightning OPD 2.059.255.237.550.6
0 : 1Lightning OPD53.548.832.144.8
Lightning OPD 2.062.154.838.851.9

Without retuning, 2.0 beats Lightning OPD at every ratio and on every benchmark. Mixing does not help Lightning OPD, which stays below SFT (48.3) at all mixed ratios since the mixed score carries both teachers' recurring disagreement.

We have added both experiments to Appendix F.3 (Table 9) and F.4 (Table 10).

AuthorsOpenReview ↗

Rebuttal for Reviewer f6cT, continued

Q1. Pass@1, Pass@8 and error bars.

Every number in the paper averages 64 samples per problem, which largely removes sampling variance. Brackets give 95% problem-level bootstrap CIs (10,000 resamples). Pass@8 uses the unbiased estimator of Chen et al. (2021).

Pass@1

StudentMethodAIME24AIME25HMMT25Avg
Qwen3-4B-SFTSFT reference57.5 [45.1, 69.5]52.4 [38.7, 66.1]34.9 [22.1, 48.3]48.3 [40.7, 56.1]
Lightning OPD59.6 [47.0, 71.8]51.6 [37.9, 65.3]34.6 [21.7, 48.5]48.6 [40.9, 56.5]
IW-OPD60.8 [48.1, 72.7]49.9 [36.4, 63.8]33.8 [21.1, 47.3]48.2 [40.5, 56.0]
TA-OPD56.0 [43.3, 68.4]50.0 [36.5, 63.6]30.6 [17.8, 44.3]45.6 [38.0, 53.5]
Lightning OPD 2.063.5 [50.6, 75.9]55.0 [41.1, 68.7]36.7 [23.5, 50.7]51.7 [43.9, 59.6]
Klear-Reasoner-8B-SFTSFT reference81.3 [69.1, 91.6]77.5 [64.8, 89.0]62.2 [49.4, 74.3]73.6 [66.5, 80.3]
Lightning OPD80.6 [68.8, 90.7]77.2 [64.6, 88.6]62.9 [49.9, 75.4]73.6 [66.4, 80.4]
IW-OPD81.9 [70.3, 91.7]77.6 [64.9, 88.9]62.3 [50.0, 74.2]73.9 [67.0, 80.5]
TA-OPD82.7 [70.9, 92.7]76.9 [64.4, 88.3]62.0 [49.4, 74.1]73.9 [66.9, 80.5]
Lightning OPD 2.082.4 [70.8, 92.1]77.7 [65.3, 88.8]63.8 [51.4, 75.6]74.6 [67.7, 81.1]
Olmo-3-7B-Think-SFTSFT reference70.6 [57.7, 82.3]58.4 [45.4, 71.1]37.4 [23.5, 51.8]55.5 [47.8, 63.1]
Lightning OPD71.4 [58.4, 83.2]57.7 [43.7, 71.6]37.6 [24.3, 51.7]55.6 [47.7, 63.5]
Lightning OPD 2.072.4 [59.6, 84.2]61.2 [47.4, 74.3]40.3 [26.7, 54.2]58.0 [50.2, 65.6]

Pass@8

StudentMethodAIME24AIME25HMMT25Avg
Qwen3-4B-SFTSFT reference81.6 [68.3, 93.2]74.9 [60.2, 88.5]57.7 [41.2, 73.6]71.4 [62.8, 79.5]
Lightning OPD83.2 [70.5, 94.1]77.1 [63.2, 89.6]59.7 [44.2, 74.6]73.4 [65.3, 80.9]
IW-OPD82.8 [69.6, 94.1]77.0 [63.4, 89.4]58.2 [43.1, 73.0]72.7 [64.7, 80.3]
TA-OPD81.0 [67.8, 92.6]73.8 [59.0, 87.3]50.1 [33.8, 66.3]68.3 [59.9, 76.5]
Lightning OPD 2.084.6 [72.1, 95.1]77.5 [63.3, 90.3]58.7 [42.6, 74.3]73.6 [65.5, 81.4]
Klear-Reasoner-8B-SFTSFT reference91.5 [81.5, 99.1]88.6 [76.9, 97.6]86.6 [75.2, 96.0]88.9 [82.8, 94.1]
Lightning OPD92.2 [82.2, 99.4]89.0 [77.8, 97.8]84.9 [72.3, 95.5]88.7 [82.4, 94.1]
IW-OPD92.6 [82.6, 99.6]89.2 [78.2, 98.4]87.9 [77.0, 96.6]89.9 [84.0, 94.9]
TA-OPD92.3 [82.3, 99.5]89.2 [78.2, 98.0]84.9 [71.8, 95.8]88.8 [82.4, 94.3]
Lightning OPD 2.092.6 [82.6, 99.7]89.8 [78.6, 98.9]87.6 [76.1, 96.7]90.0 [84.0, 95.1]
Olmo-3-7B-Think-SFTSFT reference86.0 [73.3, 96.5]80.8 [67.5, 92.9]59.1 [43.6, 74.2]75.3 [67.4, 82.8]
Lightning OPD86.6 [75.2, 96.2]77.5 [63.2, 90.2]58.4 [41.8, 74.4]74.2 [66.0, 81.8]
Lightning OPD 2.087.4 [76.4, 96.4]81.3 [68.0, 93.1]64.1 [48.9, 78.8]77.6 [70.0, 84.8]

2.0 has the highest average Pass@1 and Pass@8 for all three students, and the highest Pass@1 on 8 of the 9 student–benchmark pairs. We have added both tables to Appendix F.5 (Tables 11 and 12).

Q2. Other benchmarks.

We evaluated the Table 1 checkpoints on AMC23 (40 problems, 16 samples), GPQA-Diamond (198 problems, 4 samples) and OlympiadBench (675 English open-ended math problems, 1 sample).

StudentMethodAMC23GPQA-DiamondOlympiadBenchAvg
Qwen3-4B-SFTSFT reference87.844.968.066.9
Lightning OPD93.649.671.371.5
Lightning OPD 2.093.150.572.071.9
Klear-Reasoner-8B-SFTSFT reference98.165.478.180.5
Lightning OPD98.067.077.580.8
Lightning OPD 2.098.667.278.781.5

2.0 beats SFT on every benchmark for both students and has the highest average. We have added these results to Appendix F.6 (Table 13).

Q3. Choosing the smoothing prior masses.

The priors shrink each lookup cell toward the fold mean, which matters mainly for the ~3% of tokens in sparse token-id cells. Too small a prior makes their corrections noisy and too large leaves their bias in place, so halving and doubling both hurt (Table 7).

Choosing them needs no training. Cross-fitting gives each candidate an out-of-fold R2R^2 on held-out prompts, which both failure modes lower. It is cheap on a CPU and stable: choosing on 4 folds and scoring on the 5th picks the same setting in all 5 folds. One setting (token 0.25, context 2) is used for all experiments.

We have added this analysis to Appendix F.2.

AuthorsOpenReview ↗

Rebuttal for Reviewer f6cT, continued 2

Q4. Cross-tokenizer scores: normalized distribution or surrogate?

An unnormalized surrogate, by design, which does not affect the update. For a student token yy, ℓT\ell^T is the teacher log-probability of yy's bytes along one forced teacher-token path Φ(y)\Phi(y). It is exact for the 95.9% of Olmo-cache tokens that map to a single teacher token, and a lower bound for the rest (multi-digit numbers).

Why this suffices. The implied distribution p~(y∣h)∝exp⁡(ℓT(y∣h))\tilde{p}(y \mid h) \propto \exp(\ell^T(y \mid h)) has a normalizer that depends only on the prefix hh. Like a baseline, it shifts the advantage of every token at that position equally and leaves the expected update unchanged.

Φ(y)\Phi(y) depends only on token identity, so the systematic surrogate error recurs per token and the token lookup of 2.0 removes it. In the cross-tokenizer Olmo–Qwen setting, 2.0 beats Lightning OPD by 2.4 (math) and 2.8 (code) points (Table 1).

We have added this explanation to Appendix A.2.

Q5. Topics covered by the SFT demonstrations.

All three SFT references were trained on math and code data.

  • Qwen3-4B-SFT. Qwen3-4B-Base trained on OpenThoughts3-1.2M questions (850K math, 250K code, 100K science) with all responses regenerated by Qwen3-8B.
  • Klear-Reasoner-8B-SFT. Public, Qwen3-8B-Base trained on long-CoT math and code data distilled from DeepSeek-R1-0528.
  • Olmo-3-7B-Think-SFT. Public, trained on Dolci-Think-SFT: math, code and science traces (mainly OpenThoughts3) plus instruction-following, chat, safety and multilingual data.

We have added this description to Appendix E.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.