acceptodds
Under review as a conference paper at ICLR 2027

When Tokenizers Diverge, Reasoning Can Transfer: Cross-Tokenizer On-Policy Distillation

Abstract

On-policy distillation provides dense teacher guidance on student-generated trajectories, but token-level distribution matching typically assumes a shared vocabulary. We present Cross-Tokenizer On-Policy Distillation (CT-OPD) for transferring mathematical reasoning across model families. Decode-Verified Chunk Alignment (DVCA) constructs corresponding text spans despite differences in token boundaries. The Cross-Tokenizer Rank-Divergence Score (CRDS) then combines rank-guided direction, log-probability magnitude, and entropy-based confidence to produce a student-token advantage. This auxiliary signal is annealed while the outcome-reward advantage remains active throughout training. Using a Qwen2.5-32B-SPEAR teacher, CT-OPD improves over outcome-only RL on all five evaluated benchmarks for both Youtu-Llm-2B-Base and Llama-3.1-8B-Instruct, with AIME24 gains of 6.66 and 3.55 percentage points, respectively. It achieves the best student result on nine of the ten model–benchmark combinations. Component ablations support the combined use of rank, magnitude, and confidence weighting, highlighting the importance of both correspondence and scoring in cross-tokenizer distillation.

Primary area:
reinforcement learning
Keywords:
Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K6KMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Cross-Tokenizer On-Policy Distillation (CT-OPD) for transferring mathematical reasoning across models with different tokenizers. It aligns teacher and student spans using DVCA and builds an auxiliary policy-gradient signal from token rank, log-probability, and teacher entropy. With a Qwen2.5-32B-SPEAR teacher, CT-OPD improves over outcome-only RL by 3.59 and 3.63 percentage points on average for two student models across five math benchmarks. The paper also includes ablations on the scoring design, alignment method, and annealing schedule.

Critical strengths
  1. CT-OPD improves over outcome-only RL on all five benchmarks for both students, and gives the best student result in nine of ten model–benchmark settings. It also improves over DSKD by 1.30 and 1.96 points on average. The results are consistent across the evaluated settings, although repeated-run uncertainty is not reported.
  2. DVCA handles a concrete cross-tokenizer alignment issue by matching jointly decoded spans. It also keeps the original student action positions, excludes tool observations from policy targets, and preserves outcome supervision for unmatched actions. Table 3 reports 99.7–99.8% token coverage with exact text matching.
  3. The training setup keeps the outcome advantage active and anneals only the auxiliary teacher signal. Fixed-weight CRDS variants also outperform the no-CRDS baseline. This supports combining teacher guidance with outcome-based RL, although the specific linear schedule is less clearly motivated.
Critical weaknesses
  1. The comparison to prior work does not fully isolate the contribution of CT-OPD. Section 2 discusses Niu et al. (2026) as a closely related chunk-level cross-tokenizer method, but there is no direct comparison in Table 1. It is also unclear whether GOLD and DSKD use the same outcome reward and similar training budgets. A matched chunk-based OPD baseline would make it easier to tell how much of the gain comes from the proposed scoring and credit assignment.
  2. The cross-tokenizer rank comparison remains insufficiently supported. Equation 9 only shows rank invariance for a fixed candidate set, while Equation 10 compares mean raw ranks across different vocabularies, tokenizations, and positions. The log-probability magnitude is also tokenization-dependent. The raw-gap ablation is useful, but more controlled evidence is needed to show that rank provides a reliable direction signal across tokenizers.
  3. The alignment ablation mainly shows that the tested byte-prefix implementation can fail. It does not clearly establish that DVCA is better than a correctly implemented byte-based aligner. Appendix B notes that using the underlying token bytes would avoid the illustrated byte-count error. A raw-byte baseline would therefore be more informative. Also, the 100% text-match rate is partly expected from DVCA’s acceptance rule, so coverage is the more meaningful metric.
Questions for authors
  1. Are GOLD, DSKD, and the other distillation baselines trained with the same outcome reward as CT-OPD, and are the training and tuning budgets comparable? A direct comparison with Niu et al. (2026), with alignment and outcome supervision kept as similar as possible, would also help clarify the incremental contribution.
  2. What supports comparing mean raw ranks across different vocabularies and tokenizations? It would be useful to report how often the rank-based direction disagrees with the raw log-probability gap, and whether the result is sensitive to rank normalization or tokenization granularity.
  3. How does DVCA compare with a byte-based aligner that reads the underlying token bytes directly? Such a comparison would help separate the benefit of DVCA from simply avoiding the isolated-decoding error in the current byte-prefix baseline.
Reviewer_9ohdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Cross-Tokenizer On-Policy Distillation (CT-OPD) to transfer reasoning across models with different tokenizers. It introduces DVCA for token alignment and CRDS for constructing distillation advantages using token ranks, log-probability gaps, and teacher entropy, which are combined with outcome-based RL.

Critical strengths

S1. The method enables on-policy distillation between teacher and student models with different tokenizers, expanding the range of applicable model combinations.

S2. The paper constructs distillation advantages using token ranks, log-probability gaps, and teacher entropy without requiring full vocabulary distribution matching, which reduces computational overhead.

S3. The paper provides chunk-level teacher feedback for intermediate reasoning steps and combines it with outcome-based RL, gradually reducing teacher guidance during training.

Critical weaknesses

W1. The paper argues that the sign of the log-probability difference cannot be trusted due to calibration differences between models. However, it uses the absolute value of the same difference as the magnitude of the advantage, even though calibration differences can distort not only the sign but also the magnitude. Therefore, the proposed scoring method logically contradicts the problem identified by the paper itself.

W2. The paper determines the direction of the advantage by comparing the average ranks of tokens corresponding to the same text in the teacher and student. However, different tokenizers have different token representations and competing candidate sets, so a higher rank in the teacher does not necessarily indicate a stronger preference for the corresponding text. Therefore, it is unclear whether cross-tokenizer rank comparison provides a valid learning direction.

W3. The paper compares the average log-probabilities of tokens corresponding to the same text. However, average log-probability depends on the number of tokens produced by each tokenizer, so it can yield different values even for the same text. In contrast, summing token log-probabilities, as in GOLD, has a clearer probabilistic justification because it computes the likelihood of the corresponding text according to the chain rule. Therefore, it is unclear why average log-probability should be used instead of existing approaches such as GOLD.

W4. The proposed method combines outcome-based RL with a distillation advantage, but the paper does not specify whether the same outcome-based RL was applied to the GOLD and DSKD baselines. Therefore, it is unclear whether the performance improvements come from the new scoring method or from the addition of outcome-based RL.

Questions for authors

Q1. Figure 2 shows that CT-OPD reaches its peak performance at a certain point during training and then declines, while the results reported in Table 1 closely match these peak values. However, the paper does not specify whether the best checkpoint was used, how checkpoints were selected, or whether the same selection criterion was applied to all baselines. Furthermore, Table 4 presents a hyperparameter sweep on the AIME24 test set, and the best-performing configuration appears to be reported in Table 1. This raises concerns about potential evaluation bias due to hyperparameter selection on the test set. Therefore, the checkpoint and hyperparameter selection procedures need to be clarified.

Reviewer_9BMHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CT-OPD distills reasoning across tokenizer families by aligning shared decoded chunks and converting teacher feedback into student-token advantages. Its score uses ranks for direction, log-probability differences for magnitude, and teacher entropy for weighting, with teacher guidance gradually annealed while outcome rewards remain active. Across two students and five benchmarks, it consistently improves over GRPO and achieves the better student results.

Critical strengths
  • The method tackles a relevant problem with a clear alignment and policy-update pipeline.
  • Gains over RL are consistent across both student configurations.
Critical weaknesses
  • Token ranks and mean log-probabilities are not established as comparable across vocabularies and segmentations.
  • Comparisons with the closest OPD methods are absent. The paper mentioned a few related works, while did not directly compare with them.
Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.