- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies cross-tokenizer on-policy distillation and asks whether recovering supervision on spans without a 1:1 token alignment actually helps. Across three heterogeneous teacher-student pairs on math and code, strict 1:1 positions cover most student tokens despite low static vocabulary overlap. At these positions, a student-selected top-16 subset of the shared vocabulary already holds most of both models' probability mass. Reverse KL on strict positions beats the evaluated cross-tokenizer baselines (also with a larger teacher), while adding a span MSE on mismatch groups lowers accuracy.
Critical strengths
It establishes useful strong-baseline results: plain reverse KL on strict positions over the shared vocabulary cut down to a student top-16 subset keeps at least 96% of the full-vocabulary gain. The larger-teacher results point the same way. This gives a much simpler default and raises the bar for new cross-tokenizer OPD methods.
Critical weaknesses
- The coverage-vs-reliability conclusion rests on one mismatch objective. In gradient terms, log-prob MSE on sampled span paths is a single-sample estimate of span-level reverse KL, while the strict term is an exact KL over the shared vocabulary. The drop may therefore come from mixing in a noisy estimator rather than from unreliable mismatch targets. The gradient diagnostics do not separate the two: they are taken on strict-only checkpoints, where a rising span-to-strict ratio is not surprising since only the strict loss is being minimized, and the split-half control uses the same exact estimator on both halves. Because this is the paper’s main message, the evidence currently supports only a narrower claim: this span loss does not help on top of strict OPD.
Questions for authors
Does the same sampled-path MSE also hurt when applied at strict positions in place of the exact KL? Or does a lower-variance mismatch objective (e.g., SimCT-style units) help on top of the same strict loss? If the first hurts or the second helps, the problem is the estimator rather than coverage.
Clarification: is the rising norm ratio mainly the strict gradient shrinking? Absolute norms, and the same diagnostics on a lambda>0 run, would make Section 5 much more convincing; and were the baselines tuned at all, and why does SimCT fall below shared-vocabulary OPD here when its own paper reports the opposite?