SEAM: Shared-Event Alignment for Cross-Tokenizer On-Policy Distillation

On-policy distillation across tokenizers requires predictions in different vocabularies to express a common observation. Shared token identities provide incomplete and asymmetric correspondence, while broader alignment interfaces must still determine how predictions are conditioned at token boundaries. We introduce Shared-Event Alignment via Marginalization (SEAM), which matches local next-byte and control-event distributions through the models’ native outputs. At each student boundary, compatible tokens either extend beyond the query or end there and contribute a native continuation. Marginalizing these routes on both sides preserves earlier native histories and requires no shared token inventory or auxiliary prediction head. We derive the probability decomposition, native-logit gradients, and a bound for thresholded teacher lookahead. Across three teacher–student pairs and five mathematics/programming benchmarks, SEAM achieves the highest average score among the evaluated methods, with gains of 0.47–1.09 percentage points over the strongest baselines. Distribution diagnostics reveal changes in supervision even at common boundaries. A complementary phi-4 (14B) comparison reports a higher mean for the full construction than for teacher-only covering. The results support shared events with explicit local conditioning as a useful interface for cross-tokenizer supervision.