- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SpanShift, which defines the log-likelihood difference between post-RL and pre-RL teachers on the same student-generated text as a teacher policy-shift reward. In the cross-tokenizer setting, each student-token prefix is decoded, re-encoded by the teacher tokenizer, and scored. Differences between consecutive prefix scores define immediate rewards, whose suffix sums provide reward-to-go for student training. In the shared-tokenizer setting, dense top-k current-token supervision replaces the sampled current reward while retaining future rewards. Experiments cover English mathematics and code, three cross-tokenizer student families, and one shared-tokenizer setting.
Critical strengths
- The method turns teacher policy shift into scalar feedback on student text, avoiding explicit vocabulary projection and providing a clear prefix-difference and reward-to-go decomposition.
- The evaluation includes three cross-tokenizer student families, a shared-tokenizer control, component ablations, per-benchmark results, and measured GPU costs. The main arithmetic is internally consistent.
Critical weaknesses
- The novelty boundary and the interpretation of the probability object could be clearer. Feng et al. (2026) already use the sequence log-ratio between pre-RL and post-RL teachers as policy-shift supervision on student trajectories [1], while prior cross-tokenizer work also performs likelihood scoring on aligned text or text chunks. The main novelty here therefore appears more precisely to be canonical teacher re-encoding of every student text prefix, using raw-text policy shift to avoid explicit token mapping; policy shift itself, the telescoping prefix difference, and reward-to-go are not new basic mechanisms. This interface remains practically useful, but the paper should distinguish the value of this combination from foundational methodological novelty more cautiously. Equation (2) also does not discuss how its canonical sequence probability differs from a marginal text-event probability; span has no independent role in the per-student-token objective, and EOS, template-boundary, and affected-suffix rescoring conventions need clarification. [1] Feng et al., “Weak-to-Strong Generalization via Direct On-Policy Distillation,” arXiv:2607.05394, 2026.
- The main gains are small and statistically unsupported. The average cross-tokenizer mathematics advantage is only 0.6, and SpanShift is 0.3 below the strongest Mistral mathematics baseline. No complete student-training seeds, standard deviations, confidence intervals, or significance tests are reported, while 30-question AIME sets receive equal aggregate weight with much larger benchmarks.
Questions for authors
- Relative to Direct-OPD's pre-RL versus post-RL sequence log-ratio and prior cross-tokenizer text-likelihood scoring, is the main novelty the combination of these ideas into a per-student-prefix raw-text policy-shift interface? Please distinguish more explicitly among a new supervision signal, a new credit-assignment mechanism, and a new cross-tokenizer implementation.
- Equation (2) defines as the probability of the canonical sequence . How should this quantity be interpreted relative to a marginal over all teacher-token sequences that decode to , and could the distinction affect the main conclusions?
- How do the spans in the title and Figure 2 relate to the per-student-token training objective? Please also specify EOS, prompt-response boundary, and suffix-rescoring conventions.