acceptodds
Under review as a conference paper at ICLR 2027

SpanShift: Cross-Tokenizer Policy-Shift Distillation via Raw-Text Rewards

Abstract

Cross-tokenizer distillation enables knowledge transfer across model families, typically through vocabulary mapping or text alignment. Policy-shift distillation focuses on the changes acquired during teacher post-training, but its token-level implementations rely on a shared vocabulary. We introduce SpanShift, which expresses the teacher's policy shift from reinforcement learning (RL) as a reward on student-generated text. The reward is the pre-RL/post-RL log-likelihood difference for the same response, capturing the teacher's change in preference without projecting its distributions into the student vocabulary. Scoring successive text prefixes yields reward increments that sum to the response reward. Their reward-to-go trains each student token using rewards from its current step and subsequent continuation. When the tokenizer is shared, dense supervision over candidate next tokens replaces the sampled current-token term while retaining future reward. We evaluate SpanShift on mathematical reasoning and code generation across three cross-tokenizer student families and a shared-tokenizer setting. SpanShift improves the mathematics and code scores over the strongest baselines by +0.6 and +1.2 on average across tokenizers, and by +1.7 and +1.7 with a shared tokenizer. Ablations show gains from the teacher contrast, reward-to-go, and dense current-token supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cross-tokenizer distillation, on-policy distillation, policy-shift distillation, knowledge distillation, large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_5SMzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SpanShift, which defines the log-likelihood difference between post-RL and pre-RL teachers on the same student-generated text as a teacher policy-shift reward. In the cross-tokenizer setting, each student-token prefix is decoded, re-encoded by the teacher tokenizer, and scored. Differences between consecutive prefix scores define immediate rewards, whose suffix sums provide reward-to-go for student training. In the shared-tokenizer setting, dense top-k current-token supervision replaces the sampled current reward while retaining future rewards. Experiments cover English mathematics and code, three cross-tokenizer student families, and one shared-tokenizer setting.

Critical strengths
  1. The method turns teacher policy shift into scalar feedback on student text, avoiding explicit vocabulary projection and providing a clear prefix-difference and reward-to-go decomposition.
  2. The evaluation includes three cross-tokenizer student families, a shared-tokenizer control, component ablations, per-benchmark results, and measured GPU costs. The main arithmetic is internally consistent.
Critical weaknesses
  1. The novelty boundary and the interpretation of the probability object could be clearer. Feng et al. (2026) already use the sequence log-ratio between pre-RL and post-RL teachers as policy-shift supervision on student trajectories [1], while prior cross-tokenizer work also performs likelihood scoring on aligned text or text chunks. The main novelty here therefore appears more precisely to be canonical teacher re-encoding of every student text prefix, using raw-text policy shift to avoid explicit token mapping; policy shift itself, the telescoping prefix difference, and reward-to-go are not new basic mechanisms. This interface remains practically useful, but the paper should distinguish the value of this combination from foundational methodological novelty more cautiously. Equation (2) also does not discuss how its canonical sequence probability differs from a marginal text-event probability; span has no independent role in the per-student-token objective, and EOS, template-boundary, and affected-suffix rescoring conventions need clarification. [1] Feng et al., “Weak-to-Strong Generalization via Direct On-Policy Distillation,” arXiv:2607.05394, 2026.
  2. The main gains are small and statistically unsupported. The average cross-tokenizer mathematics advantage is only 0.6, and SpanShift is 0.3 below the strongest Mistral mathematics baseline. No complete student-training seeds, standard deviations, confidence intervals, or significance tests are reported, while 30-question AIME sets receive equal aggregate weight with much larger benchmarks.
Questions for authors
  1. Relative to Direct-OPD's pre-RL versus post-RL sequence log-ratio and prior cross-tokenizer text-likelihood scoring, is the main novelty the combination of these ideas into a per-student-prefix raw-text policy-shift interface? Please distinguish more explicitly among a new supervision signal, a new credit-assignment mechanism, and a new cross-tokenizer implementation.
  2. Equation (2) defines π(y∣x)\pi(y\mid x) as the probability of the canonical sequence π(τT(y)∣x)\pi(\tau_T(y)\mid x). How should this quantity be interpreted relative to a marginal over all teacher-token sequences that decode to yy, and could the distinction affect the main conclusions?
  3. How do the spans in the title and Figure 2 relate to the per-student-token training objective? Please also specify EOS, prompt-response boundary, and suffix-rescoring conventions.
Reviewer_VUBeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on distillation across different tokenizers by analyzing the shifts in a teacher model's preferences before and after RL. The proposed method, SpanShift, constructs intrinsic rewards via the difference in log-likelihoods assigned by a pre-trained teacher model and its post-trained version to text generated by a student model. The method designs reward-to-go as the training rewards, The authors evaluate cross-tokenizer and shared-tokenizer transfer across mathematical reasoning and code generation tasks, and conduct ablation studies to analyze the contribution of individual components.

Critical strengths

The idea is interesting and avoids tokenizer mismatch, offering an elegant approach for knowledge transfer across different model families.

The usage of reward-to-go as an intrinsic reward is reasonable and clearly motivated.

The paper is well-written, and the author analyzes the computational cost in the appendix (although I prefer to put it in the main body).

Critical weaknesses

I am most concerned about the training cost.

As shown in Tables 1 and 17, for the Llama-based mathematical reasoning task, SpanShift consumes 229.6 GPU hours, while 160.8 GPU hours required by BPM (slightly higher with matching the tokenizer), improving the performance from 46.2 to 47.8. It remains unclear whether this advantage persists if BPM is trained further using the same total computational budget. Also, the authors do not fully justify the practical benefits of introducing an additional teacher (teacher before rl). Is it available? Do we need to consider it as an additional training cost as well? If so, this process is super expensive compared with the on-policy distillation process. This might be a trade-off problem. Do we really need to sacrifice computational cost in such a case?

Another question is the change in teacher preference during RL. It does not match the needs of student training. Is it only helpful in the intersected queries where the teacher is not able (or less able) to master before RL and is good at after RL? How to address the gap where the teacher is able to master before RL but the student is not. Not sure which set can improve more, but I think both of them are important for distilling students.

Additionally, can we use teacher pairs before SFT and after SFT? It seems that any teacher pair is okay if there is a preference change.

Questions for authors
  • When keeping the total GPU-hours (including both teacher scoring and student training) constant, does SpanShift still outperform BPM or other strong single-teacher baselines? Could you provide a plot showing performance as a function of the total computational budget?
  • How much do the main results fluctuate when using different random seeds for student training? Are the average gains of 0.6 points (math) and 1.2 points (code) consistent across runs?
  • How much training cost do we need for teacher scoring? Specifically, how much computation is saved through prefix caching, and does this benefit?
Reviewer_t7exOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpanShift to transfer the policy shift between a pre-RL and post-RL teacher to a student model with a different tokenizer. Instead of aligning the teacher and student vocabularies, the authors use the difference in teacher log-likelihoods on student-generated text as a reward. They then compute reward-to-go from successive text prefixes to extract token-level supervision. For shared-tokenizer settings, they can also combine dense current-token supervision with future rewards.

The authors evaluate SpanShift on mathematical reasoning and code generation across three cross-tokenizer student families and one shared-tokenizer setting. Compared to the strongest baselines, they report average improvements of +0.6 and +1.2 points on math and code for cross-tokenizer transfer, and +1.7 points on both tasks with a shared tokenizer.

Critical strengths
  1. The idea of expressing policy-shift supervision directly on raw text is simple and reasonable. It avoids explicit vocabulary mapping or token alignment and provides a relatively straightforward way to transfer the teacher's post-training improvements across model families.

  2. The paper compares against relevant recent baselines, including cross-tokenizer OPD and policy-shift extensions of existing methods. The component ablations are also useful for understanding the contributions of the teacher contrast and reward-to-go.

Critical weaknesses
  1. Novelty is limited. The main idea largely combines existing work on cross-tokenizer distillation with policy-shift distillation. The reward-to-go formulation also follows standard policy-gradient credit assignment. In fact, the authors acknowledge in Appendix A.2 that reward-to-go and response-level rewards produce the same expected on-policy gradient, and their advantages are identical with exact value baselines. While the proposed combination is reasonable, I think the actual methodological contribution is quite narrow.

  2. The empirical improvements are extremely marginal and inconsistent. The average cross-tokenizer gains are only +0.6 points on math and +1.2 on code. Looking at the individual benchmark results in Tables 10-12, SpanShift does not outperform the strongest baseline on MATH-500 or LiveCodeBench for any of the three cross-tokenizer students. In other words, it loses all six of these comparisons. The shared-tokenizer gains are somewhat larger, but these use an additional dense supervision mechanism. Overall, the results do not demonstrate a particularly compelling improvement over existing approaches.

  3. Following up on that point, the paper does not provide convincing evidence that these small gains are reliable and worthwhile. The main results are reported as point estimates without confidence intervals or a systematic analysis of student-training variance. Table 14 examines variation across three source RL runs, but only for the Llama mathematics setting. For improvements of around one percentage point, I think this is an important omission. The method also introduces additional teacher scoring and training costs, so I would have liked a clearer assessment of whether the small gains justify this overhead.

Questions for authors

See weaknesses.

Reviewer_DF8TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SpanShift, a method for transferring an RL-trained teacher’s policy shift to students with different tokenizers. It scores student-generated text using the log-likelihood difference between frozen post-RL and pre-RL teacher checkpoints, avoiding vocabulary mapping or token alignment. Differences between successive prefix scores yield reward increments, which are accumulated from the current step onward to provide token-level reward-to-go weights. When teacher and student share a tokenizer, SpanShift replaces the sampled current-token reward with dense supervision over candidate tokens—implemented using the student’s top-64 candidates—while retaining rewards from the sampled continuation.

Validation covers mathematical reasoning and code generation with Llama, Gemma, and Mistral students in cross-tokenizer settings and a Qwen student in a shared-tokenizer setting. Relative to the strongest baseline within each student and domain, SpanShift reports average gains of 0.6 points in mathematics and 1.2 in code across the three cross-tokenizer students, and 1.7 points in both domains with a shared tokenizer. Ablations examine the contributions of the teacher contrast, reward-to-go, and dense supervision. Additional analyses assess reward-ranking quality, sensitivity to different teacher RL training runs, general-capability retention, and computational cost.

Critical strengths
  1. A concise and logically consistent interface for transferring post-RL strategies across different tokenizers. SpanShift leverages both pre-RL and post-RL teacher models to score the same student-generated text, eliminating the need for vocabulary mapping or token alignment. Even when appended text alters the teacher’s tokenization, the prefix-based differencing ensures that the sum of individual reward increments equals the total reward for the complete response. This provides a unified interface for transferring post-training strategy shifts from a teacher model to a student model—even when they employ different tokenizers—and represents the most significant methodological contribution of this work.
  2. The experimental setup includes targeted alternatives and component-wise ablation studies. By employing the same teacher model pairs used in SpanShift, the "SimCT + policy shift" and "BPM + policy shift" baselines help verify that the observed gains stem from the proposed interface itself, rather than simply from the use of two distinct teacher checkpoints. Further ablations—examining the use of only the post-RL teacher, length normalization, and various reward distribution schemes—provide additional justification for the chosen design. Supported by evaluations across three student model families (with different tokenizers) and two task domains, these results offer substantial evidence for the practical utility of the proposed method; however, the modest performance gains warrant further validation through repeated experiments and uncertainty estimation.
Critical weaknesses

The main gains lack uncertainty estimates. Across tokenizers, the average improvements over the strongest baselines are 0.6 points in mathematics and 1.2 in code; Mistral mathematics is 0.3 points below its strongest baseline. Without repeated student-training runs or confidence intervals, it is difficult to judge whether these differences are reliable. The three source-RL runs provide some reassurance, but cover only Llama mathematics and do not establish the robustness of the main comparisons. This limits confidence in the central empirical claim.

The mechanism behind the reward-to-go improvement remains unclear. Appendix A.2 correctly shows that, for a fixed policy, response-level and reward-to-go weighting yield identical advantages with exact value baselines. The observed gains therefore do not establish a different expected policy-gradient objective. Value approximation, optimization behavior, and stochastic variation may contribute, but no value-prediction or gradient-variance diagnostics are reported. The ablation supports effectiveness in the tested settings without identifying why it works better.

The performance–cost advantage is not established under matched compute. In the Llama mathematics setting reported in Table 17, SpanShift requires 229.6 GPU-hours, compared with 147.2 for SimCT and 160.8 for BPM. Equal generated-token budgets thus do not imply equal training costs. SpanShift is cheaper than BPM + policy shift and substantially reduces scorer-to-actor traffic, both meaningful advantages. However, the cost comparison omits cross-tokenizer OPD, a particularly competitive baseline. Given the modest score gains, compute-matched evaluations would help determine whether SpanShift offers a better practical trade-off.

Questions for authors
  1. Reliability of the reported gains — clarification. Were the main comparisons repeated with different student-training seeds? If so, how variable were the differences against the strongest baselines? Please also clarify whether evaluation uncertainty was estimated at the problem level, accounting for within-problem dependence among the eight sampled responses. These details would help establish whether the small cross-tokenizer gains are robust.

  2. Interpretation of the RTG ablation — clarification of existing evidence. Appendix A.2 shows that, for a fixed policy, response-level and RTG weighting yield identical advantages with exact value baselines. Do the existing runs provide evidence supporting improved value fitting or reduced gradient variance as an explanation for RTG’s gains? Are comparable value-fitting or advantage-estimation errors available, accounting for the different value targets? If not, should the result be understood as an empirical optimization benefit whose mechanism remains unresolved?

  3. Performance–cost comparison with cross-tokenizer OPD — clarification. What are the total GPU-hours, including teacher scoring, for cross-tokenizer OPD in the Llama mathematics setting used in Table 17? If existing timing and checkpoint records allow this comparison, does SpanShift retain its advantage at comparable cumulative compute? This information would affect my assessment of the performance–cost trade-off, since cross-tokenizer OPD is a competitive baseline but is absent from the cost table. The question concerns existing experimental records, not a request for a new benchmark study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.