acceptodds
Under review as a conference paper at ICLR 2027

SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

Abstract

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Distillation, LLM Post Training, Long Context Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LeqDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the transferring reasoning abilities from long- to short-context teacher to student models. The focus on on-policy-ness while staying tokenizer agnostic. They show this functioning on several open models specifically intern-s2 seems to provide compelling improvements over closed source variants.

Critical strengths
  • The method is simple, and easy to understand. Its more testing a novel paradigm and introducing technical novelties
  • The paper is veryt practitioner-friendly: it documents a failure modes of OPD well, e.g. unintended growth of length (or the opposite; truncation) and repetition loops.
  • the paper covers quite a lot of models and benches and provides useful ablations (rollout lengths, super relevant for reasoning)
Critical weaknesses
  • the main critcism is that tokenizer-agnostic is a bit of a stretch since most of the models have basically the same tokenizer (Lexical overlap ρ is 0.90–0.98), stemming form chinese-english-heavy settings. and maybe unsurprisingly the less similar tokenizer Gemma4 doesnt perform well here.
  • generally this reads a bit more like a tech/practitioner report than a fundamental research paper. depending on definitions of novelty, its limited here.
  • there are quite some critical confounds: checkpoints are selected based on the exact same test set (AIME), Gemma uses different rollout lengths than the other models, Deepseekv4 is both teacher and judge for proofbench, etc.
Questions for authors

Mostly:

  • What is Gemma result with 32k rollouts?
  • Can you mitigate the mentioned confounds, e.g. re-select checkpoints on a held-out set? Use different teacher and judge, etc.
1 comment

W1 / Q1:"Tokenizer-agnostic" is a stretch; Gemma at 32K rollouts

Gemma at 32K

SettingProofBench@4AnswerBench@8
Gemma-4-26B-A4B base25.568.75
6K rollout34.267.5
32K rollout, ref-KL 1.535.5670.16

32K is slightly better than 6K on both benchmarks. The paper used 6K because we held hyperparameters fixed across students for comparability, and at 32K the ref-KL coefficient of 1.0 used elsewhere is unstable on Gemma; 32K requires raising it to 1.5. The 6K setting let us keep a single coefficient across all students. The result above shows the conclusion is not an artifact of the shorter rollout length.

On "tokenizer-agnostic". We will soften the claim to state that the method is validated on high-overlap, Chinese–English-heavy pairs, with Gemma as the lowest-overlap case.

W2:Reads more like a practitioner report; limited novelty

We accept this is a framing disagreement. Our intended contribution is the empirical characterization (supervision coverage, cost, and length dynamics in long-context cross-tokenizer OPD), rather than the alignment rule itself, and we will reframe the paper accordingly.

W3 / Q2:Confounds: checkpoint selection, judge == teacher

Checkpoint selection. We re-selected the checkpoint for Qwen3.5-35B-A3B on Omni-MATH, which is disjoint from every reported benchmark. The held-out criterion selects the same training step, and scores move by less than evaluation noise (~1 pp across seeds), so the contamination did not materialize. We will adopt the held-out criterion in the paper.

Judge == teacher. We re-scored the same saved Section 5.4 proofs under an independent judge (Gemini-3.1-Pro) alongside a same-family one (DeepSeek-V4.1-Flash):

RowDeepSeek-V4.1-FlashGemini-3.1-Pro
Intern-S2-Preview (base)32.56 ±1.4334.64 ±1.35
+ DS-OPD, ref-KL 0.544.11 ±0.3046.79 ±0.57

The two judges agree on the base-to-OPD gain to within 0.59 pp, within evaluation noise, so we do not detect self-preference.

Reviewer_harVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a few practical challenges related to applying on-policy distillation (OPD) in scenarios where the teacher is a long-context model and the student is a short-context model (which potentially has a different tokenizer). These challenges include tokenizer mismatch (which is solved through a token alignment step), teacher-student distribution mismatch (which is solved with a student-reference KL term), and response length explosion and training instability (which are solved by combining the reference KL with some special token masking). The experiments distill a mathematical proof-reasoning teacher into several student models and evaluate mostly on math benchmarks.

Critical strengths
  1. The empirical results show clear improvements over the base students across several models (Table 1), and SimpleOPD outperforms the two OPD baselines included in the paper (EOPD and G-OPD) on most benchmarks.
  2. The generalization from math to other scientific tasks is promising. Intern-S2-OPD improves on HiPhO and FrontierScience-Research after its math distillation training. It would be interesting to see if this holds for the other student models too.
Critical weaknesses
  1. The method, as described in Section 2, consists of aligning the tokenizers and then distilling the teacher knowledge in the aligned token space. However, this token alignment problem has already been explored, even specifically for cross-tokenizer OPD [1], which is cited in the paper but not compared against. There is no empirical evidence that the alignment in 2.1 is better than the one proposed in [1], or even a discussion for why it might be.
  2. There is potentially a “training on the test set” issue in the way the authors pick checkpoints for their algorithm. Section 3.3 says that the best checkpoint is selected based on the average score of AIME@4 and AnswerBench@1. But AIME and AnswerBench are two of the benchmarks used to evaluate the different methods. How should we trust the comparisons on these two benchmarks in that case? Shouldn't they be excluded from the comparisons?
  3. A similar issue is that DeepSeek-V4-Flash is used both as a judge (it reads the proofs and evaluates them for ProofBench) and as a teacher whose knowledge is distilled into Intern-S2-Preview in Section 5.4. There is established research on LLMs favoring their own answers [2], so how can we trust this evaluation? The paper claims that a stronger teacher helps, but the reported gap in section 5.4 is small enough that it could be due to using the same model as a judge and teacher.
  4. Some of the method's most central components are not formally defined, and they do not appear in Section 2 (Method) at all. Specifically, I could not find a precise definition of the reference KL or the special token masking, which are only described in the results section (4.1). The paper says it masks the OPD loss on the </think> and <|im_end|> tokens (while the abstract says it masks the advantages). These </think> and <|im_end|> tokens are too specific to Qwen too. What about the other student models, which use different special tokens? Is there an equation that describes this loss? Regarding the reference KL, how is it different from the reference-based constraint in [3]? and Is it added to the loss or to the advantage?

[1]: Sun et al., SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation, NeurIPS 2026

[2]: Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024

[3]: Luo et al., Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models, arXiv April 2026

Questions for authors

My questions follow the order of weaknesses.

  1. Can you compare the proposed alignment to the one in [1] from weakness 1?
  2. How sensitive are the results to the choice of checkpoint? Or maybe put differently, how do the AIME and AnswerBench results change if you choose different checkpoints?
  3. Can you re-evaluate the student in section 5.4 with a judge other than DeepSeek-V4-Flash?
  4. Can you give the exact equations for the reference KL and the special token masking, and include further details on the design of these two components?
3 comments

W1 / Q1:Comparison with the alignment in SimCT

We implemented SimCT's Minimal Aligned Unit (MAU) supervision space in our codebase, with each MAU scored by the length-normalized teacher continuation log-probability (SimCT Eq. 7). It is exposed as a single flag, so the teacher request, chat templates, termination masking, and loss are shared with SimpleOPD and only the supervision space differs.

Both were run under an identical pipeline (SU-01 teacher, Intern-S2-Preview student, same data and hyperparameters). ProofBench@4 judged by deepseek-v4-flash; AIME25@8 and AnswerBench@8 complete:

MethodProofBench@4AIME25@8AnswerBench@8
SimpleOPD46.0795.8378.63
SimCT42.9895.4277.50

SimCT's alignment brings no accuracy gain over exact shared-token matching in this setting, which is the empirical evidence the reviewer asked for. We will add this comparison to the paper.

W2 / Q2:Checkpoint selection on the reported benchmarks

The reviewer is right that selecting on AIME@4 + AnswerBench@1 and then reporting those same benchmarks is contaminated.

We re-selected the checkpoint for Qwen3.5-35B-A3B using Omni-MATH, which is disjoint from every reported benchmark. The held-out criterion selects the same training step. The comparison is as follows (both at 32k rollout length):

SettingProofBench@4AnswerBench@8AIME25@8Average
Paper result42.3980.1696.6761.25
Omni-MATH selected42.9878.7595.3362.25

The reported scores are unchanged to within evaluation noise. The contamination therefore did not materialize: an independent selection set picks the same checkpoint, so the AIME and AnswerBench comparisons are unaffected.

W3 / Q3:DeepSeek-V4-Flash as both judge and teacher

We re-scored the same saved proofs (no regeneration, so no sampling noise enters the comparison) under two same-generation judges: one from the teacher's own family, one fully independent:

RowDeepSeek-V4.1-Flash (same family)Gemini-3.1-Pro (independent)
Intern-S2-Preview (base)32.56 ±1.4334.64 ±1.35
+ DS-OPD, ref-KL 0.544.11 ±0.3046.79 ±0.57

The two judges agree on the base-to-OPD gain to within 0.59 pp, smaller than the 4-evaluation standard deviation. We therefore do not detect the self-preference effect of Panickssery et al.; the Section 5.4 conclusion is unchanged.

W4 / Q4:Formal definitions of reference KL and special-token masking

The two regularizers act at different levels of the objective, which we will state explicitly in the method section.

Token-level distillation signal. All our settings are pure distillation (the task reward is zero), so the per-token advantage is the masked reverse KL between student and teacher:

At=− mt⋅(log⁡πθ(yt)−ℓ~tϕ)A_t = -\, m_t \cdot \left( \log \pi_\theta(y_t) - \tilde{\ell}_t^{\phi} \right)

where ℓ~tϕ\tilde{\ell}_t^{\phi} is the aligned teacher log-probability (the student's own log-probability at positions the alignment leaves unmatched), and mt∈{0,1}m_t \in \{0,1\} is the mask that excludes termination tokens. These advantages enter a PPO clipped surrogate with ε = 0.2.

Sequence-level reference regularization. The student-reference KL is a separate penalty added to the objective rather than a modification of the advantage:

L=LPPO+βref⋅KL(πθ ∥ πref)L = L_{\mathrm{PPO}} + \beta_{\text{ref}} \cdot \mathrm{KL}\left( \pi_\theta \,\|\, \pi_{\text{ref}} \right)

where βref\beta_{\text{ref}} is the reference-KL coefficient and the divergence is computed with a low-variance estimator. The abstract's statement that masking applies to the advantages is correct for the distillation term; we will make clear that the reference KL is a distinct penalty.

On token specificity. The masked strings are configured per student, not hard-coded to Qwen; we will list them per model in an appendix.

Relation to Luo et al. That work studies length inflation and stabilization for same-tokenizer OPD. In contrast, our work tackles a more complex reasoning scenario: transferring proof-reasoning capabilities from a long-context reasoning teacher to a short-context student. This introduces combined challenges of tokenizer mismatch, teacher-student distribution mismatch, and response length explosion. Our reference KL interacts with the alignment mask to remain active at exactly the positions where teacher supervision is withheld, and we additionally mask the advantages of special termination tokens. Together, these strategies mitigate excessive generation length and frequent truncation, thereby fostering steady length growth in this complex cross-tokenizer reasoning setting.

Reviewer_harVOpenReview ↗

Response to authors

I thank the authors for taking the time to address my comments and questions. While the response provides some new experiments, I don’t think it really addresses the original concerns. I also found several inconsistencies in the reported numbers that make me less confident in how carefully verified these last-minute experiments were.

W1 (comparison to SimCT): on two of the reported datasets, the difference between SimCT and the proposed method is 0.41 (AIME25) and 0.37 (AnswerBench). The authors’ responses to W2 and W3 clearly state that differences of this size are within evaluation sampling noise, but here they are presented here as empirical evidence that SimpleOPD outperforms SimCT. On datasets of 30 (AIME25) and 400 (AnswerBench) problems respectively, this is indeed within sampling noise and does not establish an empirical advantage. It is also not clear to me what numbers are reported here. Why are the Intern-S2-Preview student numbers here (46.26 ProofBench, 66.62 AnswerBench@8) different from the corresponding numbers in the response to reviewer KUUy (44.50 and 80.10) and from the paper itself (79.10 AnswerBench@8)?

W2: Like I mentioned in the previous point, I can accept that this is within sampling noise, but that only shows that most of the reported advantages of 1-2 points across the paper are within sampling noise. Also, I don’t know what the average column is reporting, because it is not the average of the other columns.

W3: Why is the judge here DeepSeek-V4.1-Flash instead of DeepSeek-V4-Flash (which is what the original paper reported)? Also, my original concern was about the claim in Section 5.4 that a stronger teacher helps. The new experiment only re-judges the base student and the DeepSeek-distilled student, but not the student distilled from SU-01. So it does not tell us whether the gap between the two teachers holds under an independent judge.

W4: I appreciate the clarification, but I think it confirms my point that the proposed loss is not clearly distinguishable from the standard OPD literature. The “student-reference KL” is the standard KL penalty, the same as the reference-KL term used in GRPO training and in Luo et al.

The response also mentions “Our reference KL interacts with the alignment mask to remain active at exactly the positions where teacher supervision is withheld” as a point of comparison with Luo et al. But that is not a special property of the proposed reference KL. If I understand correctly, the only new piece is masking the termination-token advantages.

AuthorsOpenReview ↗

Response to Reviewer harV

We apologize for the confusion. The inconsistent numbers in W1/W2 were caused by a GPT-OSS judge outage that we failed to notice; the affected runs were mis-scored, and we mistakenly reported those incorrect results. All affected numbers have been recomputed and updated in the rebuttal.

For W3, DeepSeek officially discontinued the DeepSeek-V4-Flash API. We therefore self-deployed V4-Flash and re-scored the saved proofs; V4.1-Flash was used only as an additional same-family judge. The updated results are included in the rebuttal.

We also want to clarify that we do not claim SimpleOPD outperforms SimCT. Our claim is only that SimCT’s alignment does not bring a significant accuracy gain over exact shared-token matching in this setting. The small differences on AIME25 and AnswerBench are within evaluation noise.

Reviewer_KUUyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the transfer of mathematical proof capabilities from a long-context reasoning teacher to different students via on-policy distillation (OPD). SimpleOPD converts the student’s output into shared text and applies teacher log-probability supervision only to tokens where the teacher and student tokenizers cover the same text intervals. It also incorporates student-reference KL divergence and termination token advantage masking to control length growth during training. The paper evaluates mathematical reasoning performance on Qwen3, Qwen3.5, Intern-S2-Preview, GLM-4.7-Flash, and Gemma-4, and reports results for scientific reasoning benchmarks.

Critical strengths
  1. The token alignment rules are simple and can be directly applied to teacher–student pairs using different tokenizers. In Table 1, Intern-S2-Preview’s ProofBench@4 score improved from 21.70 to 44.50, and other cross-tokenizer students also showed improvements.
  2. The paper diagnoses the training challenges of long-context OPD: direct OPD is accompanied by longer outputs, repetitions, and truncation, whereas student-reference KL significantly mitigates truncation. This observation and the corresponding fix are valuable for practical training.
Critical weaknesses
  1. The paper does not report a complete set of benchmark results for direct OPD under the same settings. The stabilization configurations and training budgets used for EOPD and G-OPD are also insufficiently specified.
  2. The paper does not sufficiently explain how its alignment design relates to existing cross-tokenizer OPD methods, and it lacks a direct comparison with a relevant method such as SimCT [1]. SimCT constructs a common supervision space that includes multi-token aligned units, whereas this paper supervises sampled tokens only at exact-match spans. The practical value of this simpler design remains unclear without evidence of its performance and computational cost relative to existing methods in the long-context setting. If the authors can adequately address the above concerns, I would consider raising my score.
Questions for authors
  1. Could you provide the complete benchmark results for direct OPD, so they can be compared with existing results? What stabilization configurations, training budgets, and checkpoint selection rules were used for EOPD and G-OPD, respectively?
  2. How does the performance and training overhead of the method compare to that of SimCT or related baselines? Do these results support the design choice of supervising only exact-match positions in long-context scenarios?
  3. What is the complete training objective that includes student-reference KL and termination-token masking? Does KL regularization still cover misaligned tokens and masked termination tokens?
1 comment

W1 / Q1:Complete direct-OPD results; EOPD and G-OPD specifications

Baseline configurations

Both baselines share our training setup: same teacher, data, and optimizer; 100 rollout iterations with batch size 64 prompts and 4 sampled responses per prompt; constant learning rate 1e-6; evaluation every 25 iterations; and same termination-token masking.

G-OPD

G-OPD differs from direct-OPD in one respect: reward scaling factor α = 2/3. Yang et al. (arXiv:2602.12125) show that standard OPD is the special case of dense KL-constrained RL where reward and KL regularization carry equal weight, and introduce α to control their relative weight:

At=α⋅(ℓtteacher−ℓtref)+(ℓtref−log⁡πθ(yt))A_t = \alpha \cdot \left( \ell_t^{\text{teacher}} - \ell_t^{\text{ref}} \right) + \left( \ell_t^{\text{ref}} - \log \pi_\theta(y_t) \right)

Thus α = 2/3 weights reward against KL in a 2:3 ratio. The reference policy is the student's own pre-distillation model.

EOPD

EOPD uses 6K rollout length and adds a forward-KL term to the standard reverse-KL objective. The forward-KL term applies over the teacher's 16 highest-probability tokens at positions where teacher token-level entropy exceeds 0.8. Both terms are weighted at 1.0, and student-reference KL is disabled, making this row directly comparable to direct-OPD.

Both configurations will be added to the paper.

Direct-OPD results

For Intern-S2-Preview at 32K rollout length, with best checkpoint selected and evaluated under the same protocol:

SettingProofBench@4AnswerBench@8AIME25@8
Intern-S2-Preview (base)21.7076.0388.33
Direct OPD38.3077.0095.00
OPD + student-reference KL44.5080.1095.00

Compared with direct OPD, SimpleOPD adds +6.20 on ProofBench and +3.10 on AnswerBench, with AIME25 unchanged. This isolates the contribution of SimpleOPD from that of direct OPD itself.

W2 / Q2:Relation to existing cross-tokenizer OPD; comparison with SimCT

We implemented SimCT in our codebase: its Minimal Aligned Unit (MAU) supervision space, with each MAU scored by length-normalized teacher continuation log-probability (SimCT Eq. 7). It is exposed as a single flag, so teacher request, chat templates, termination masking, and loss are shared with SimpleOPD; only the supervision space differs.

We then ran SimpleOPD and SimCT under this identical pipeline (SU-01 teacher, Intern-S2-Preview student, same data and hyperparameters) and compared downstream accuracy.

Results

ProofBench@4 judged by deepseek-v4-flash; AIME25@8 complete:

MethodProofBench@4AIME25@8AnswerBench@8
SimpleOPD46.0795.8378.63
SimCT42.9895.4277.50

Conclusion

SimCT brings no accuracy gain over exact shared-token matching in this setting. Its additional machinery is therefore unnecessary for long-context cross-tokenizer OPD, supporting our simpler design. We will add this comparison to the paper.

Q3:Complete training objective; does KL cover misaligned and masked tokens?

The two regularizers act at different levels of the objective, which we will state explicitly in the method section.

Token-level distillation signal

All settings are pure distillation (task reward is zero), so the per-token advantage is the masked reverse KL between student and teacher:

At=− mt⋅(log⁡πθ(yt)−ℓ~tϕ)A_t = -\, m_t \cdot \left( \log \pi_\theta(y_t) - \tilde{\ell}_t^{\phi} \right)

where ℓ~tϕ\tilde{\ell}_t^{\phi} is the aligned teacher log-probability (the student's own log-probability at positions the alignment leaves unmatched), and mt∈{0,1}m_t \in \{0,1\} is the mask that excludes termination tokens. These advantages enter a PPO clipped surrogate with ε = 0.2.

Sequence-level reference regularization

Student-reference KL is a separate penalty added to the objective rather than a modification of the advantage:

L=LPPO+βref⋅KL(πθ∥πref)L = L_{\mathrm{PPO}} + \beta_{\text{ref}} \cdot \mathrm{KL}(\pi_{\theta} \Vert \pi_{\mathrm{ref}})

where βref\beta_{\text{ref}} is the reference-KL coefficient, and the divergence uses a low-variance estimator. The abstract's statement that masking applies to advantages is correct for the distillation term; we will clarify that reference KL is a distinct penalty.

Does KL regularization still cover misaligned and masked termination tokens?

Yes. The mask mtm_t removes the distillation signal at those positions, but reference KL is evaluated at every response position without masking. Misaligned and masked termination tokens therefore receive no teacher supervision while remaining regularized toward the reference policy. This is deliberate: it prevents length drift precisely where teacher supervision is withheld.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.