- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Cross-Tokenizer On-Policy Distillation (CT-OPD) for transferring mathematical reasoning across models with different tokenizers. It aligns teacher and student spans using DVCA and builds an auxiliary policy-gradient signal from token rank, log-probability, and teacher entropy. With a Qwen2.5-32B-SPEAR teacher, CT-OPD improves over outcome-only RL by 3.59 and 3.63 percentage points on average for two student models across five math benchmarks. The paper also includes ablations on the scoring design, alignment method, and annealing schedule.
Critical strengths
- CT-OPD improves over outcome-only RL on all five benchmarks for both students, and gives the best student result in nine of ten model–benchmark settings. It also improves over DSKD by 1.30 and 1.96 points on average. The results are consistent across the evaluated settings, although repeated-run uncertainty is not reported.
- DVCA handles a concrete cross-tokenizer alignment issue by matching jointly decoded spans. It also keeps the original student action positions, excludes tool observations from policy targets, and preserves outcome supervision for unmatched actions. Table 3 reports 99.7–99.8% token coverage with exact text matching.
- The training setup keeps the outcome advantage active and anneals only the auxiliary teacher signal. Fixed-weight CRDS variants also outperform the no-CRDS baseline. This supports combining teacher guidance with outcome-based RL, although the specific linear schedule is less clearly motivated.
Critical weaknesses
- The comparison to prior work does not fully isolate the contribution of CT-OPD. Section 2 discusses Niu et al. (2026) as a closely related chunk-level cross-tokenizer method, but there is no direct comparison in Table 1. It is also unclear whether GOLD and DSKD use the same outcome reward and similar training budgets. A matched chunk-based OPD baseline would make it easier to tell how much of the gain comes from the proposed scoring and credit assignment.
- The cross-tokenizer rank comparison remains insufficiently supported. Equation 9 only shows rank invariance for a fixed candidate set, while Equation 10 compares mean raw ranks across different vocabularies, tokenizations, and positions. The log-probability magnitude is also tokenization-dependent. The raw-gap ablation is useful, but more controlled evidence is needed to show that rank provides a reliable direction signal across tokenizers.
- The alignment ablation mainly shows that the tested byte-prefix implementation can fail. It does not clearly establish that DVCA is better than a correctly implemented byte-based aligner. Appendix B notes that using the underlying token bytes would avoid the illustrated byte-count error. A raw-byte baseline would therefore be more informative. Also, the 100% text-match rate is partly expected from DVCA’s acceptance rule, so coverage is the more meaningful metric.
Questions for authors
- Are GOLD, DSKD, and the other distillation baselines trained with the same outcome reward as CT-OPD, and are the training and tuning budgets comparable? A direct comparison with Niu et al. (2026), with alignment and outcome supervision kept as similar as possible, would also help clarify the incremental contribution.
- What supports comparing mean raw ranks across different vocabularies and tokenizations? It would be useful to report how often the rank-based direction disagrees with the raw log-probability gap, and whether the result is sensitive to rank normalization or tokenization granularity.
- How does DVCA compare with a byte-based aligner that reads the underlying token bytes directly? Such a comparison would help separate the benefit of DVCA from simply avoiding the isolated-decoding error in the current byte-prefix baseline.