acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation

Abstract

On-policy distillation (OPD) transfers knowledge between language models through teacher supervision on student-generated trajectories. With different tokenizers, a single teacher token may require multiple student tokens to generate, creating intermediate states where the event is entered but not yet completed. Existing cross-tokenizer methods align tokens or text spans to construct comparable prediction targets. We study a complementary problem after partial generation: once the student produces a prefix of a teacher token, multiple next tokens may complete the same remaining bytes, but the teacher only specifies the required completion rather than how probability should be divided among these valid continuations. We introduce Event-Set Completion Distillation (ESCD), which complements cross-tokenizer probability alignment with completion-set supervision. ESCD aggregates prefix-related teacher events and supervises the total probability of byte-compatible one-step student completions, avoiding tokenizer-dependent probability splits among individual tokens. The method reuses student trajectories and predictions, requiring neither additional rollouts nor changes to the student vocabulary. Experiments demonstrate consistent gains in mathematics, code, and scientific reasoning across model families and tokenizers, extending to large-scale MoE distillation from a 1T teacher to a 35B student. Local analyses show that retaining completion sets better matches the reference supervision, while one-step completion covers over 99% of observed compatible teacher mass after partial event entry in the studied tokenizer pairs. These findings support event entry and event completion as complementary supervision targets for cross-tokenizer knowledge transfer. Code will be released on GitHub.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ovvtOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Summary

Paper proposes new technique to do on-policy distillation from a teacher to a student, where the 2 models have different tokenization logics.

Process

Continuing with the paper’s example of generating “Bread”, where student generates a “Br” token, and teacher has multiple tokens in its vocab starting with “Br”, but not specifically “Br” itself:

  1. Let the generated seq before “Bread” be ‘x’. Get possible teacher tokens set that complete “Br”, and their probabilities
  2. aggregate common prefix teacher tokens and uses their sum to supervise the total probability of the byte-compatible one-step completion set. For example, both ead and eads satisfy the residual byte constraint ead without requiring a prescribed probability split.
  3. Collect all native full-vocabulary student content tokens whose byte realizations begin with the required residual string
  4. Apply a teacher-mass-weighted negative log-likelihood loss on the student probability of this completion set + root (‘x’) losses.

Related and Novelty

  1. Token Mappings and Alignment (e.g., FuseLLM, DSKD, MultiLevelOT, and GOLD): construct comparable prediction targets through token mappings, sequence alignment, or distribution matching across vocabularies.
  2. Text/Byte-Level Representation and Scoring (e.g., ALM, CTLS, BLD, SimCT, and BPM): align chunks or compute likelihood scores over text/byte segments.
  3. These methods focus primarily on building comparable prediction targets or aligning tokens/spans rather than explicitly supervising the remaining byte constraint after partial event entry without forcing tokenizer-dependent probability splits.
  4. ESCD doesn’t enforce forced token-level splits. It formalizes the event-completion gap as aggregate set-probability supervision objective.

Observations

  1. Good metrics compared to BPM, SimCT for OPD tasks.
  2. Outperforms number of baselines on a variety of tasks

Required Theory

  1. On-policy Post training: training on student model’s own generated trajectories with RL (as opposed to off-policy, which uses a teacher model’s logits, or final token sequence to do distillation on student - SFT)
  2. On-policy Distillation: grading student model’s trajectories with a teacher and training on that with distillation instead of RL to get over the sparse reward issue with RL.
  3. “Event Completion Gap” - (Fig 2 in the paper explains this well) in context of OPD with diff tokenizations, a single teacher token might split across several student tokens - so, even if the student starts with the right characters, it might cross the teacher-equivalnet token boundaries incorrectly. the teacher supervision only specifies what bytes must eventually be completed, not how probability should be divided or allocated among these individual student completion tokens.
Critical strengths
  1. Very detailed analysis of different aspects of the study, including reliance on one-step rewards, vocab overlap, example walk-through, etc
  2. Good way to address tokenizer boundary mismatches via byte-level prefix aggregation and completion sets instead of rigid token-to-token mappings.
  3. Validated across multiple model architectures (dense and MoE), scales, and domains (mathematics, programming, and advanced physics/science).
Critical weaknesses
  1. auxiliary child supervision loss is bounded to one-step completions of teacher vocab tokens. Residuals requiring multiple tokens to finish ("Deeper" completions) have no auxiliary child loss

Did not find any serious and obvious flaw/missed opportunity in this paper

Questions for authors
  1. How sensitive is ESCD's final performance to the exclusion criteria used when filtering out boundary-crossing tokens or handling very short residuals (such as single-byte newlines)?
  2. I think the one-step limitation of the child loss can be extended to multi-step paths. I know the one-step covers most of the trajectories, but I believe there’s scope to research that.
  3. Can you provide concrete metrics on the overhead introduced by prefix aggregation and byte-index lookups?
5 comments

We thank the reviewer for the careful assessment and constructive suggestions. We address the three questions below with additional analyses, supplementary training results, and computational overhead measurements.

1. Sensitivity to boundary rules and short residuals

The completion rule in Section 3.2 requires a student token to complete all residual bytes, while allowing it to extend beyond the residual boundary. For example, both Memory and MemoryWarning satisfy the residual constraint Memory. This means that they satisfy the same byte-prefix requirement, without implying that their full continuations are semantically equivalent. The current implementation does not discard residuals merely because they contain only one byte; single-byte newline residuals can receive child supervision.

Appendix C.3 already examines boundary-crossing completions. In the original Qwen and GLM subsets, these tokens account for 2.10% and 0.63% of the completion probability, respectively. Removing them produces gradients with cosine similarities of 0.9832 and 0.9887 to the full-set gradients. The original analysis also includes short residuals: 45 of the 48 Qwen events have one-byte residuals, predominantly newlines.

To extend this analysis, we generated responses to 32 questions—16 mathematics and 16 code questions—using the original Qwen3.5-2B student with thinking enabled and a maximum output length of 27,648 tokens. We analyzed the full generated trajectories, including responses reaching the length limit. Under each teacher setting, we examined 4,096 valid positions without updating model parameters.

Analysis on full-length trajectoriesQwen3-32B teacherGLM-Z1-9B teacher
Share of full-set completion probability assigned to boundary-crossing tokens1.61%0.57%
Child supervision weight removed by excluding all one-byte residuals>99.99%>99.99%
Child supervision weight removed by excluding only one-byte newline residuals>99.99%59.38%

The removal fractions are calculated using teacher-event probability weights, relative to the total weight of valid child supervision. They do not represent fractions of the overall training loss. These results show that boundary-crossing tokens carry a small share of completion probability in the analyzed trajectories, whereas one-byte residuals account for most of the weighted child supervision.

2. One-step limitation and multi-step extension

Thank you for this suggestion. ESCD retains root supervision but applies no child loss when one-step completion is unavailable. In our full-length code analysis (16 trajectories; 2,048 valid states per teacher setting), one-step completion covers 98.93% and 98.64% of visited compatible teacher-event mass under Qwen3-32B and GLM-Z1-9B, respectively. Coverage is weighted by teacher probability mass over visited compatible representative events after filtering and prefix aggregation.

Two-step extension. We implemented exhaustive two-step supervision for residuals without one-step completions, retaining the original root and one-step objectives. We sum all valid two-token path probabilities and apply a negative log-likelihood weighted by the original teacher-event mass, without event sampling or path pruning. Unlike the one-step objective, which reuses trajectory logits, this requires predictions after alternative first-token branches.

For Qwen3-32B→Qwen3.5-2B, the first four code trajectories required 559,919 branches after sharing computations for identical position/first-token pairs. On 24 GPUs (eight each for teacher, rollouts, and student training), computation of the added objective exceeded 100 minutes without completing the first training step. This measures the cost of our current exhaustive implementation, not average step time or the minimum cost of optimized methods. We therefore have no complete two-step training results. High one-step coverage does not rule out downstream benefits from deeper completion; we will clarify this limitation and discuss efficient extensions as future work.

Supplementary training comparison. These experiments use code-only data, whereas the main experiments use a 1:1 mathematics/code mixture. All models are evaluated with decoding seeds 42 and 43. Evaluation sampling changes can affect Teacher/Student scores relative to the main tables; distilled scores may additionally reflect the training-data difference.

LiveCodeBench (%, ↑)

MethodSeed 42: pass@1Seed 43: pass@1Avg pass@1pass@2
Teacher60.4460.9960.7166.48
Original Student13.7412.0912.9115.38
BPM27.4724.1825.8231.87
Root-only29.1225.8227.4734.07
Root + Single23.6324.7324.1832.42
ESCD-RKL28.0225.8226.9231.87
ESCD (ours, FKL)30.7728.5729.6734.62

TACO (%, ↑)

MethodSeed 42: pass@1Seed 43: pass@1Avg pass@1pass@2
Teacher53.3653.7153.5357.60
Original Student8.836.367.6010.25
BPM22.2619.4320.8528.62
Root-only21.9117.3119.6129.68
Root + Single16.6118.7317.6726.86
ESCD-RKL24.7324.0324.3832.16
ESCD (ours, FKL)26.8625.4426.1533.57

Avg pass@1 uses the unrounded results of both seeds; pass@2 is the fraction of problems for which either solution passes. Bold indicates the best distillation result. ESCD-FKL is our final method; ESCD-RKL uses a smoothed reverse-KL root objective with the same child objective.

ESCD leads among distillation configurations across both benchmarks and both decoding seeds, supporting consistency under the tested sampling conditions. Root + Single falls below Root-only in all four pass@1 comparisons. Section 3.3 offers a possible explanation: Single increases completion-set probability while additionally favoring one token within the set, although the teacher byte event does not specify this allocation.

This agrees with Section 5's local gradient diagnostic: COUF for Root/Single/Event is 0.7383/0.4323/0.8495 under Qwen and 0.8848/0.7493/0.9341 under GLM. These observations support preserving the completion set. However, checkpoints come from different training steps, so the comparisons do not isolate the cause of performance differences.

3. Computational overhead of prefix aggregation and byte-index lookups

Thank you for requesting concrete overhead measurements. We conducted paired measurements of complete training steps with child supervision disabled and enabled in the same implementation. The measured increment covers the entire child-supervision path, including prefix aggregation, byte-index lookups, completion-set probability computation, and backpropagation.

We evaluated Qwen3-32B→Qwen3.5-2B on eight H100 GPUs with a batch size of four. The workload comprised 24 full-length trajectories, with 12 mathematics and 12 code trajectories, totaling 620,227 output tokens. The maximum output length was 27,648 tokens per trajectory. For each batch, both modes were warmed up separately before timing, yielding six paired measurements.

Both modes used the same trajectories, teacher predictions, and student weights. We set the learning rate to zero to keep the weights fixed while still executing the full forward pass, backward pass, gradient synchronization, and optimizer step. Both modes used identical CPU offloading settings, with transfer time included in the measurements.

ConfigurationMean time per step (s)Peak allocated memory per GPU (GiB)
Root-only: child supervision disabled15.8952.80
ESCD: child supervision enabled17.2352.80

Enabling child supervision increases the mean time per complete training step by approximately 8.4%, while peak allocated GPU memory remains unchanged at the reported precision. This increment reflects the cost of the full child-supervision path. Timing excludes trajectory generation, teacher pre-scoring, and one-time initialization. For memory, we report the maximum peak allocation across all training GPUs.

We will include these measurement details in the revised manuscript and limit the overhead claim to the tested model pair, long-sequence workload, and memory configuration.

Reviewer_BDUYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the token alignment problem in On-Policy Distillation (OPD) between teachers and students using different tokenizers and proposes Event-Set Completion Distillation (ESCD). The paper point out that when one teacher token is represented by multiple student tokens, existing methods may require unnecessary probability allocation among valid completion tokens. To address this, they propose an objective that learns the total probability of valid completion sets rather than individual token probabilities, combined with the existing root-level alignment loss (Eqs. 7–10). The method requires no additional rollouts or vocabulary changes. Experiments show improvements over existing methods across mathematics, code generation, and scientific reasoning benchmarks with various teacher–student pairs (Tables 1–2), including distillation from a 1T teacher to a 35B student. Gradient alignment analysis and completion coverage experiments further support the effectiveness of the method (Tables 3–4).

Update 11/20

Thank you for the additional experiments comparing different distillation objectives. I appreciate the authors' efforts to address my concerns.

  1. Disentangling the contributions of ESCD and the distillation objective.: The improvement from ESCD-FKL to ESCD-CSD(T,S) is substantial and comparable to the gain from BPM-FKL to ESCD-FKL. This suggests that the choice of distillation objective plays an important role in the reported performance. A comparison between BPM-CSD(T,S) and ESCD-CSD(T,S) would provide stronger evidence that ESCD offers improvements independently of the distribution-matching objective.

  2. Sensitivity to the distillation objective: The reason for the relatively poor performance with standard RKL remains unclear. Interestingly, both FKL and CSD(T,S) appear to favor mode-covering behavior, while the mode-seeking RKL performs substantially worse. This raises the question of whether ESCD is particularly effective with mode-covering objectives. Further analysis of this interaction would strengthen the paper.

  3. Overall assessment and rating update: Despite these remaining concerns, I believe the core contribution of ESCD (cross-tokenizer alignment and completion supervision in on-policy distillation) has been reasonably validated under the controlled FKL setting, particularly on reasoning and code-generation tasks. I am therefore willing to increase my rating from 2 (Weak Reject) to 3 (Weak Accept), provided that the authors clearly acknowledge the following points in the revised manuscript:

  • Separation of contributions: Cross-tokenizer alignment/completion supervision and the distribution-matching objective are distinct design components. I encourage the authors to include a separate paragraph discussing distillation objectives in the Related Work section. The main contribution of ESCD is demonstrated through improvements over BPM under the same FKL objective.

  • Additional gains from advanced objectives: Combining ESCD with CSD(T,S) yields further improvements beyond ESCD-FKL. However, the current results do not establish that ESCD-CSD(T,S) outperforms BPM-CSD(T,S).

  • Sensitivity to divergence choice: Standard RKL performs substantially worse than FKL and CSD(T,S) in the reported ESCD experiments. The interaction between completion supervision and the choice of divergence remains insufficiently understood and should be acknowledged as a limitation.

Overall, I appreciate the extensive additional experiments and the authors' constructive engagement during the discussion.

Critical strengths
  1. Very strong empirical results as shown in Table 1. Results with both avg and pass metric seems very proper.

  2. The loss can be computed using per-token probabilities already available in standard OPD, without additional rollouts or vocabulary changes, resulting in negligible computational overhead.

  3. Table 3 shows that the proposed completion-set supervision is closer to the reference gradient, while Table 4 empirically confirms that most of the observed compatible teacher mass can be handled by one-step completion.

Critical weaknesses
  1. This is my major concern: In the reasoning domain, token diversity is very limited, so it is not sufficient to evaluate the general capability of the proposed cross-tokenizer method. Including instruction-following tasks with benchmarks such as MT-Bench, AlpacaEval, and Arena-Hard is necessary, as token diversity is much greater in these tasks.

  2. The experiments only focus on reasoning tasks, which are very challenging for small student models, so there is still a large performance gap compared to the teacher.

  3. As shown in Eq. 7, this paper uses the forward KL loss, while MiniLLM (ICLR 2024), an early work on on-policy distillation, proposed reverse KL. Why did you choose forward KL? Also, subsequent works such as DistiLLM (ICML 2024), ABKD (ICML 2025), and CSD (ICLR 2026) have shown that simply changing the loss function can significantly affect performance. Are the performance gains over the baseline maintained with reverse KL? Can the proposed method also be applied to logit-based distribution matching methods such as CSD?

  4. SFT on teacher-generated samples is also a natural baseline in the cross-tokenizer setting, but it is missing. I understand that the rollout costs are different, but I would like to see this baseline included for comparison within a reasonable computational budget.

Questions for authors

See Weaknesses

6 comments

We thank the reviewer for the constructive suggestions regarding evaluation scope, distillation objectives, and baseline comparisons. We have added instruction-following experiments, a teacher-generated SFT baseline, and supplementary training results with different root objectives. We address the concerns below.

1. Instruction-following evaluation and the teacher–student performance gap

We agree that the original mathematics, code, and science experiments primarily evaluate reasoning and are insufficient to support conclusions about general instruction-following capabilities. To broaden the evaluation, we drew on CSD's task settings and model choices and added a cross-family, cross-tokenizer experiment with Gemma-2-9B-IT → Qwen2.5-1.5B-Instruct.

Training prompts are first-turn user prompts from Tülu 3. SFT, BPM, and ESCD start from the same student checkpoint. Evaluation covers IFEval, MT-Bench, AlpacaEval 2, and Arena-Hard v0.1, using three decoding seeds: 42, 43, and 44. The following table reports averages across these seeds. DeepSeek is used as the judge for the latter three benchmarks. AlpacaEval 2 reports raw win rate, and Arena-Hard reports weighted win rate.

MethodIFEval strict prompt-level accuracy (%)MT-Bench scoreAlpacaEval 2 raw win rate (%)Arena-Hard weighted win rate (%)
Teacher69.507.0229.9030.20
Original Student39.004.374.722.86
SFT41.844.485.964.67
BPM41.534.465.474.34
ESCD42.584.546.544.54

Higher is better for all metrics. Bold denotes the highest score among SFT, BPM, and ESCD in each column.

The results for each benchmark and decoding seed are provided below.

IFEval: strict prompt-level accuracy (%, ↑)

MethodSeed 42Seed 43Seed 44Avg
Teacher66.7370.6171.1669.50
Original Student42.3338.8235.8639.00
SFT41.5942.1441.7741.84
BPM40.8543.6240.1141.53
ESCD42.1441.0444.5542.58

MT-Bench: score (↑)

MethodSeed 42Seed 43Seed 44Avg
Teacher7.017.047.017.02
Original Student4.474.414.244.37
SFT4.444.504.504.48
BPM4.404.444.534.46
ESCD4.694.444.494.54

AlpacaEval 2: raw win rate (%, ↑)

MethodSeed 42Seed 43Seed 44Avg
Teacher29.4430.9329.3229.90
Original Student4.104.845.224.72
SFT5.345.347.205.96
BPM5.714.606.095.47
ESCD6.346.466.836.54

Arena-Hard v0.1: weighted win rate (%, ↑)

MethodSeed 42Seed 43Seed 44Avg
Teacher28.3831.9630.2530.20
Original Student2.882.393.322.86
SFT4.944.464.624.67
BPM4.194.584.264.34
ESCD5.164.314.164.54

Averages are calculated from unrounded results.

ESCD's average scores exceed those of the original student and BPM on all four tasks. Compared with SFT, ESCD is slightly better on IFEval, MT-Bench, and AlpacaEval 2, and slightly worse on Arena-Hard. These results provide additional evidence of applicability beyond reasoning tasks, while showing that the relative benefits of different supervision methods depend on the task.

We agree that a substantial teacher–student gap remains. We study distillation from a larger model to a smaller model across model families and tokenizers. A stronger teacher provides an opportunity for knowledge transfer, but the extent to which the student can absorb these capabilities may be constrained by its capacity, training data, and optimization conditions. Based on our practical observations, OPD within a given budget may not fully close this gap. The current experiments also do not establish a performance ceiling for the student.

Some work constructs RL-improved teachers from the same base model and transfers their capabilities back to the original model, as in ExOPD's domain-expert merging setting [1]. Such experiments provide an empirical reference for performance attainable at the same model scale. Our setting additionally involves compressing capabilities from a larger, heterogeneous teacher into a smaller student. We acknowledge that students can outperform teachers on particular tasks, but this does not imply that every cross-family compression setting should match the teacher. We will report both improvements over the initial student and baselines and the remaining teacher–student gap, and add these considerations to the discussion.

2. Choice of forward KL and supplementary reverse-KL results

Thank you for raising this question. We agree that the distillation objective can substantially affect training and that its choice and scope should be explained.

First, on-policy trajectory generation and distribution matching are separate design choices: the former determines that training prefixes are generated by the student, while the latter determines how teacher and student predictions are compared at those prefixes. Using student-generated trajectories does not require reverse KL. GKD [2] also studies trajectory sources and divergence choices separately.

We use FKL primarily to retain BPM's root objective and maintain consistency in the baseline comparison. BPM [3] compares FKL, JSD, and RKL, with FKL achieving the best six-benchmark average results for all three teacher–student pairs. This provides an empirical basis for our choice, but does not imply that FKL is universally better than RKL.

Second, Eq. 7 describes the root-alignment term. The full ESCD objective additionally includes child completion supervision:

LESCD=Lroot+Lchild.\mathcal{L}_{\mathrm{ESCD}} = \mathcal{L}_{\mathrm{root}} + \mathcal{L}_{\mathrm{child}}. Lchild=−∑gMglog⁡(∑u∈C(rg)pθ(u∣hg)).\mathcal{L}_{\mathrm{child}} = -\sum_g M_g \log(\sum_{u\in\mathcal{C}(r_g)} p_\theta(u\mid h_g)).

Here, MgM_g is the teacher-event weight, C(rg)\mathcal{C}(r_g) is the set of valid student tokens completing residual bytes rgr_g, and hgh_g is the corresponding visited child state. The root term matches distributions in the projected common coordinate space. The child term supervises total completion-set probability without prescribing the allocation among individual tokens. The event construction and child supervision can be retained when replacing the root objective; ESCD therefore does not require an FKL root objective.

To provide supplementary evidence, we compare ESCD-FKL and ESCD-RKL in the Qwen3-32B → Qwen3.5-2B code-training setting, alongside the teacher, original student, and full BPM. Both ESCD configurations retain the same child objective and its weight.

ESCD-RKL here uses smoothed reverse KL. Let qˉ\bar q and pˉ\bar p denote the projected teacher and student distributions. Its root objective is:

Lroot,RKL=DKL(pˉ∥(0.9qˉ+0.1pˉ)).\mathcal{L}_{\mathrm{root,RKL}} = D_{\mathrm{KL}}(\bar p \Vert (0.9\bar q + 0.1\bar p)).

Evaluation uses decoding seeds 42 and 43, generating one solution per problem for each seed.

LiveCodeBench (%, ↑)

MethodSeed 42: pass@1Seed 43: pass@1Avg pass@1pass@2
Teacher (Qwen3-32B)60.4460.9960.7166.48
Original Student (Qwen3.5-2B)13.7412.0912.9115.38
BPM27.4724.1825.8231.87
ESCD-RKL28.0225.8226.9231.87
ESCD-FKL (ours)30.7728.5729.6734.62

TACO (%, ↑)

MethodSeed 42: pass@1Seed 43: pass@1Avg pass@1pass@2
Teacher (Qwen3-32B)53.3653.7153.5357.60
Original Student (Qwen3.5-2B)8.836.367.6010.25
BPM22.2619.4320.8528.62
ESCD-RKL24.7324.0324.3832.16
ESCD-FKL (ours)26.8625.4426.1533.57

Avg pass@1 is calculated from unrounded results. Pass@2 is the fraction of problems for which at least one of the two solutions passes. Bold denotes the best result among the compared distillation configurations.

Both ESCD configurations improve over the original student. Relative to BPM, the smoothed RKL version has higher Avg pass@1 on both tasks, matches LiveCodeBench pass@2, and improves TACO pass@2. The FKL version performs better in this comparison. This provides supplementary evidence of improvements with an alternative root objective. However, BPM uses its default FKL objective, and the configurations use checkpoints from different training steps. These results therefore do not constitute a controlled comparison under the same divergence and training budget, and should not be directly generalized to unsmoothed reverse KL.

3. Teacher-generated SFT baseline and computational budget

We agree that SFT on teacher-generated responses is a natural and important baseline for cross-tokenizer distillation. We have included it in the common evaluation in Section 1.

SFT uses the same Tülu 3 prompt source. Gemma-2-9B-IT generates responses, which are then tokenized with the student's tokenizer for training. The resulting dataset contains 49,471 training examples and 500 validation examples. Cross-entropy loss is applied only to assistant responses and end-of-sequence tokens. Training uses 32 GPUs, 10 epochs, a global batch size of 128, a learning rate of 1e-5, 10% warmup followed by cosine decay, and a maximum sequence length of 8,192 tokens. A checkpoint is saved after each epoch.

OPD training for BPM and ESCD uses 49,472 prompts, a global batch size of 64, and a constant learning rate of 5e-7. Generation uses instruction mode with temperature 1, top-p 0.95, top-k 20, and presence penalty 1.5. Input and output lengths are each capped at 4,096 tokens. Both SFT and OPD use training seed 42 and a gradient-clipping threshold of 1.0.

We separately record the resources used for SFT data generation, SFT training, and ESCD training through the reported checkpoint:

StageNumber of GPUsGPU-hours
Teacher generation of SFT data324.77
SFT training3230.75
Total SFT data generation and training—35.52
ESCD training through the reported checkpoint2423.99

These figures describe the actual resource usage of the reported runs. SFT includes offline data generation and training, whereas ESCD is measured through the checkpoint corresponding to the reported results. Because the workflows and stopping points differ, we treat these figures as resource records, not as a strictly budget-matched comparison or a general efficiency claim.

As shown in Section 1, SFT improves over the original student on all four tasks and achieves the highest average Arena-Hard score among the compared trained students. ESCD has slightly higher average scores on the other three tasks. Teacher-generated text supervision is therefore a competitive baseline, and the benefits of different supervision methods are task-dependent.

SFT, RL, and OPD can also be combined according to the initialization, task objective, and computational budget. SFT can provide an initialization for subsequent OPD or RL. OPD supplies teacher-distribution supervision at student-visited states and can be combined with task rewards, as investigated in prior work [2]. The Kimi setting in our paper likewise applies OPD after SFT initialization.

The purpose of this supplementary experiment is to include a teacher-generated SFT baseline and compare performance and resource usage under the reported settings. ESCD is a supervision mechanism for cross-tokenizer post-training. The ordering of training stages and the benefits of combining it with SFT or RL require task- and budget-specific controlled evaluation.

References

[1] Yang, W., et al. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv:2602.12125, 2026.

[2] Agarwal, R., et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR, 2024. arXiv:2306.13649.

[3] Wang, H., et al. Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization. arXiv:2607.22334, 2026.

Reviewer_DMdpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies cross-tokenizer on-policy distillation (OPD), where one teacher token may span several student generation steps. It identifies the event-completion gap: after the student generates a strict byte prefix of a teacher token (e.g., Br of Bread), the residual bytes (ead) constrain which student tokens validly complete the event, but the teacher does not specify how probability should be split among multiple valid completions (e.g., ead vs. eads). The proposed Event-Set Completion Distillation (ESCD) aggregates prefix-related teacher events into representative prefix constraints weighted by pooled teacher mass, and adds a teacher-mass-weighted NLL on the total probability of the one-step completion set at the naturally visited child state, on top of a BPM-style byte-aligned root-projection forward KL (dropping BPM's interior/spanning supervision). The method reuses the on-policy trajectory and existing student logits: no counterfactual rollouts, no extra teacher queries, no vocabulary changes.

Experiments distill Qwen3.5-2B from Qwen3-32B and GLM-Z1-9B on AIME 2024–2026, HMMT-26, LiveCodeBench, and TACO against GOLD, X-Token, SimCT, and BPM, re-implemented in a unified framework with shared data, rollout settings, and budget (each method samples its own trajectories); and extend to MoE distillation — Qwen3.5-397B-A17B → Qwen3-30B-A3B-Thinking (13.2×) and Kimi-K2.7-Code → Qwen3.6-35B-A3B (28.6×) — on FrontierScience Olympiad-100 and the new PHYRD-40 benchmark, compared against BPM. ESCD matches or exceeds the best baseline on every reported metric, with the largest gains in code generation (e.g., TACO pass@2 from BPM's 18.7% to 47.3% under the Qwen3-32B teacher) and +3.0–5.1 points on the MoE science benchmarks. Mechanism analyses (a local-gradient fidelity diagnostic, and coverage statistics showing over 99% one-step completion availability on visited mass) support the design.

Critical strengths
  1. The entry/completion distinction identifies a real over-specification in token-allocation objectives such as BPM's conditional per-token targets, and ESCD's set-level NLL is a natural, essentially free relaxation: it reuses student logits at visited states, needs no counterfactual rollouts and no teacher queries beyond the root-level pass, and requires no vocabulary changes. That the gains persist from the 2B controlled setting up to 1T-teacher MoE distillation makes the result practically relevant for real cross-family distillation pipelines.

  2. All baselines are re-implemented in one framework with identical data, rollout settings, and budget (four-way comparison at 2B scale; BPM at MoE scale), which makes the consistent ordering across two teachers, six benchmarks, and two MoE pairs credible. I also value the honest analysis of scope and failure modes — the one-step coverage statistics, the boundary-crossing analysis, and the documentation of OPD degradation in the Kimi→Qwen3.6 setting (C.6) — which increases my trust in the reported results.

Critical weaknesses
  1. The comparison of Root vs. Single vs. Event targets is evaluated only through COUF, a local-gradient diagnostic whose reference gradient is itself a byte-event marginal — the quantity the Event loss approximates. The diagnostic is not vacuous (Single scores below Root), but trained ablations under identical budgets would give much cleaner evidence: root-only, root + single-completion child loss, full ESCD, and the Exact/Minimal/Full boundary variants the authors themselves mention in C.3. Since ESCD's delta over full BPM combines adding the child loss with removing BPM's interior/spanning supervision, such ablations would also help attribute the gains more precisely.

  2. The math benchmarks have 30–33 problems, so several pass@8 deltas correspond to roughly one solved problem (e.g., +3.4 on AIME25/26 pass@8 under the Qwen3-32B teacher; +3.0 on HMMT-26 pass@8 under the GLM teacher), and no training-seed variance is reported. The large-scale results rely on two LLM-graded benchmarks with one sample per problem at temperature 1.0: +3.0 on Olympiad-100 corresponds to about 3 of 100 problems, and the PHYRD-40 gains are shifts in a 0–100 rubric mean over 40 problems. I do not have per-problem scores, so this is a concern about the reported protocol rather than a computed significance test — but some variance reporting would meaningfully strengthen the cross-scale claims.

Questions for authors
  1. Could you provide ablations trained under identical budgets isolating the child objective — (a) root-only, (b) root + single-completion child loss, (c) full ESCD — ideally also alongside full BPM from Table 1, so that the effect of dropping interior/spanning supervision is separated from the effect of the child loss? (As you note in C.3, Exact/Minimal/Full boundary variants would also be informative.)

  2. Table 6 reports checkpoint/evaluation intervals but not the total number of training steps, how the evaluated checkpoint was selected (final vs. best), or the number of seeds. Could you clarify these, and report run-to-run variance if available?

  3. For FrontierScience Olympiad-100 and PHYRD-40: did you audit DeepSeek-V4 Pro Preview's grading against human judgment on a subsample? And how stable are the reported scores if the temperature-1.0, one-sample-per-problem protocol is repeated?

Suggestion for future work (no response needed): extending completion supervision beyond one step for the residual ~0.6–1.0% of visited mass, and studying interactions with RL-style objectives, would broaden the contribution.

5 comments

We thank the reviewer for the positive assessment of the problem formulation, method, and experimental analysis, and for the constructive suggestions regarding trained ablations, checkpoint selection, and evaluation reliability. We address these points below.

1. Trained ablations and the roles of different supervision objectives

We agree that COUF measures local gradient agreement with a specified reference objective and cannot replace downstream comparisons after training. To provide complementary training evidence, we conducted supplementary code-task experiments with the following configurations:

  • BPM: retains the full BPM objective, including supervision at root, interior, and spanning positions.
  • Root-only: retains only the root-alignment objective used by ESCD.
  • Root + Single: adds supervision for one valid completion token per eligible event to the same root objective. We select the valid token with the shortest byte length, breaking ties by token ID. Selection is independent of student probabilities.
  • ESCD: retains the same root objective and supervises the total probability of all valid one-step completion tokens.

We additionally report ESCD-RKL, which uses a smoothed reverse-KL root objective while retaining the same child completion-set objective. ESCD with the FKL root objective is the final method used in the paper.

These supplementary experiments use code-only training data, whereas the main experiments use a 1:1 mixture of mathematics and code data. All models are evaluated with decoding seeds 42 and 43. Changes in evaluation sampling can affect Teacher and original Student scores relative to the main tables; distilled-model scores may additionally reflect the different training-data composition.

LiveCodeBench (%, ↑)

MethodSeed 42: pass@1Seed 43: pass@1Avg pass@1pass@2
Teacher60.4460.9960.7166.48
Original Student13.7412.0912.9115.38
BPM27.4724.1825.8231.87
Root-only29.1225.8227.4734.07
Root + Single23.6324.7324.1832.42
ESCD-RKL28.0225.8226.9231.87
ESCD (ours, FKL)30.7728.5729.6734.62

TACO (%, ↑)

MethodSeed 42: pass@1Seed 43: pass@1Avg pass@1pass@2
Teacher53.3653.7153.5357.60
Original Student8.836.367.6010.25
BPM22.2619.4320.8528.62
Root-only21.9117.3119.6129.68
Root + Single16.6118.7317.6726.86
ESCD-RKL24.7324.0324.3832.16
ESCD (ours, FKL)26.8625.4426.1533.57

All scores are percentages, with higher values indicating better performance. Bold denotes the best result among the compared distillation configurations. Avg pass@1 is calculated from the unrounded results of both seeds; pass@2 is the fraction of problems for which at least one of the two solutions passes.

ESCD achieves the highest pass@1 among the compared distillation configurations on both benchmarks under both decoding seeds, and also leads in Avg pass@1 and pass@2. This provides additional evidence of consistency under the tested evaluation sampling conditions.

Regarding the contributions of different supervision components, Root-only and full BPM have different rankings across the two benchmarks: Root-only has higher Avg pass@1 on LiveCodeBench but lower Avg pass@1 on TACO. ESCD exceeds Root-only on both benchmarks, by 2.20 and 6.54 percentage points, respectively. These observations provide training-level comparisons relevant to distinguishing the removal of BPM’s interior/spanning supervision from the addition of completion-set supervision.

Root + Single has lower pass@1 than Root-only on both benchmarks under both decoding seeds, consistent with the local gradient diagnostic. The decomposition in Section 3.3 offers a possible explanation: beyond increasing completion-set probability, Single favors the selected token within the set, although the teacher byte event does not prescribe this allocation. The design of the child objective therefore matters.

For the Exact/Minimal/Full boundary variants, the supplementary results do not yet include complete training comparisons. The frozen-trajectory analysis in Appendix C.3 measures local objective and gradient changes and cannot establish downstream performance sensitivity. We will distinguish these forms of evidence in the revision.

2. Training steps, checkpoint selection, and seeds

Thank you for pointing out this reporting omission. The original manuscript did not clearly distinguish training seeds from evaluation seeds. We clarify these settings below.

Training. All methods use the same fixed training seed, with training trajectories generated through temperature sampling. We did not conduct independent training repeats with different seeds.

Evaluation in the main experiments. All methods and their checkpoints are evaluated using the same fixed decoding seed and the specified sampling temperature. This setting is consistent with the evaluation protocols of baselines such as BPM and ensures common evaluation conditions across methods. We agree that a single decoding seed cannot fully characterize sensitivity to sampling randomness.

Supplementary evaluation. To provide additional evidence, the supplementary code experiments retain the same training seed and evaluate each model’s fixed checkpoint using two decoding seeds, 42 and 43. We report pass@1 for each seed, their average, and pass@2 from the combined samples. ESCD leads the compared distillation configurations on both benchmarks under both decoding seeds, supporting consistency under the tested sampling conditions. These experiments assess sensitivity to evaluation sampling; they do not estimate variance across training seeds.

For training termination and checkpoint selection, we evaluate saved checkpoints throughout training. Approximately five consecutive checkpoints without noticeable improvement, or a decline in overall performance, serve as practical stopping criteria. We report one earlier checkpoint selected based on overall performance across the benchmarks reported in the paper. Consequently, the actual stopping step and selected checkpoint step may differ across methods.

In the revised manuscript, we will report the total training steps and selected checkpoint step for each experiment, list training and evaluation seeds separately, and explicitly document the checkpoint-selection procedure.

3. Repeated sampling, problem-level results, and grading reliability on the science benchmarks

We understand the concern about one-sample evaluation and automated grading. We performed repeated-sampling checks at fixed model weights and selected five different checkpoints for sampling comparisons on the complete benchmarks. These tests covered all problems in Olympiad-100 and PHYRD-40, using temperature 1.0 and a fixed seed. We observed small changes in aggregate scores in these checks.

Here, five refers to the number of checkpoints, not five independent training runs or five repeated samples from a single checkpoint. These observations characterize repeatability under the fixed evaluation settings and variation across the tested checkpoints; they do not quantify sampling variance across different decoding seeds. We will distinguish fixed-weight repeated sampling from cross-checkpoint comparisons and document the corresponding evaluation settings in the revision.

We also provide problem-level comparisons to explain the aggregate improvements reported in the paper. The model settings are:

  • Qwen: Qwen3.5-397B-A17B → Qwen3-30B-A3B-Thinking, using direct OPD.
  • Kimi: Kimi-K2.7-Code → Qwen3.6-35B-A3B, with BPM and ESCD sharing the same SFT initialization.

Olympiad-100: retained correct answers and newly solved problems

SettingBPM correctESCD correctBoth correctESCD onlyBPM onlyBoth incorrectNet gain (pp)
Qwen48/10051/100465247+3.00
Kimi69/10072/100666325+3.00

ESCD retains 95.83% (46/48) and 95.65% (66/69) of the problems correctly answered by BPM, while newly solving five and six problems, respectively. After accounting for regressions, both settings gain three correctly answered problems. Thus, the observed improvements accompany retention of most previously correct answers.

PHYRD-40: problem-level rubric score changes

PHYRD-40 assigns each response a rubric score from 0 to 100 and reports the mean score, rather than accuracy. Here, Δ denotes ESCD minus BPM.

SettingBPM meanESCD meanMean ΔMedian ΔSD of ΔImprovedUnchangedDecreased
Qwen46.6049.60+3.00+3.054.1822126
Kimi69.2074.30+5.10+4.856.0924106

ESCD improves the mean score by 3.00 and 5.10 points. Scores increase on 22/40 and 24/40 problems, respectively, and both median differences are positive. The observed score improvements therefore extend to more than half of the physics problems.

The standard deviations describe variation in score differences across problems, not variance across repeated runs. These statistics help explain the aggregate gains but do not establish statistical significance.

Regarding agreement between DeepSeek-V4 Pro Preview and human judgment, the current supplementary results do not include human-audit agreement statistics. Repeated sampling and problem-level comparisons cannot replace that audit, so we do not claim that agreement with human grading has been validated. We will provide the problem-level results and evaluation details and clarify the scope of the single-sample main results and automated scoring.

4. Future directions: multi-step completion and interaction with RL objectives

We appreciate these suggestions. Marginalizing over multi-token student paths that complete the residual bytes is a natural extension. The approximately 0.6%–1.0% uncovered mass reported in the paper pertains to the tokenizer pairs and trajectories in the original analysis and should not be treated as a general estimate for other model combinations.

We have attempted an exhaustive two-step extension that retains the root and one-step objectives and adds supervision for residuals without one-step completions. In a full-length code-trajectory experiment with Qwen3-32B→Qwen3.5-2B, the first four trajectories required 559,919 branches after sharing computations for identical positions and first tokens. On 24 GPUs, computation of the added objective exceeded 100 minutes without completing the first training step. We therefore do not yet have complete two-step training results. This observation concerns the current exhaustive implementation and does not establish the efficiency of optimized multi-step approaches.

Further work includes efficient multi-step probability computation and evaluation of the trade-offs among coverage, computational cost, and downstream gains. Combining event-completion supervision with RL-style objectives could also incorporate both local teacher constraints and task-level reward signals. Its effectiveness and optimization stability require systematic evaluation. We will include these directions in the future-work discussion.

Comments on the paper

3 comments

We thank all three reviewers for reading our paper carefully and offering specific, constructive suggestions. We also thank the Area Chair for coordinating the review and discussion. The feedback helped us extend our analysis of the mechanism to training experiments, broaden our evaluation, and examine how cross-tokenizer alignment interacts with different distillation objectives. We summarize the main results and analyses from the discussion below.

1. Training ablations on completion-set supervision (Reviewers DMdp and ovvt)

We compared three training configurations on code tasks: Root-only, Root + Single, and full ESCD. These ablations examine the effects of adding completion supervision and retaining the full set of valid completions. On both LiveCodeBench and TACO, full ESCD achieved higher average pass@1 than Root-only and Root + Single, while Root + Single scored below Root-only. This agrees with the local gradient analysis in the original paper and provides downstream evidence for supervising the total probability of the valid completion set.


2. Residual types, completion coverage, and computational cost (Reviewers ovvt and DMdp)

We analyzed full-length trajectories to examine how tokenization differences vary across teacher–student pairs. Newline residuals account for most of the completion supervision in the Qwen pair, while digit residuals also account for a substantial share in the GLM→Qwen pair. These results illustrate how the same byte-completion rule handles different tokenization mismatches across model pairs. The supervision weights describe how the training signal is distributed. Quantifying each residual type’s contribution to downstream scores would require a separate training ablation.

On the code trajectories analyzed, one-step completion covered approximately 99% of the probability mass of compatible teacher events at visited states in both settings. In paired measurements, enabling completion supervision increased the average time per full training step by approximately 8.4%, with the same peak allocated GPU memory per device at the reported precision. We also implemented exhaustive two-step completion. It requires computing continuation probabilities for many candidate first tokens, and the cost of the current implementation has so far precluded a full training run.


3. Extending the evaluation to instruction following (Reviewer BDUY)

We added a cross-family experiment with Gemma-2-9B-IT → Qwen2.5-1.5B-Instruct and evaluated ESCD on IFEval, MT-Bench, AlpacaEval 2, and Arena-Hard. In this setting, ESCD achieved a higher average score than BPM on each of the four benchmarks. Together with the original experiments, we have now observed improvements over BPM across code, mathematics, and instruction following.


4. Comparing SFT on teacher-generated responses with OPD (Reviewer BDUY)

We added an SFT baseline trained on teacher-generated responses and compared its instruction-following performance and computational cost with ESCD. In this experiment, ESCD achieved higher average scores than SFT on IFEval, MT-Bench, and AlpacaEval 2. Training ESCD to the reported checkpoint also used fewer GPU-hours than generating the teacher responses and training the SFT student combined.

SFT can serve both as a baseline for OPD and as its starting point. The Kimi → Qwen experiment in the main paper follows this sequence. The student first learns from complete teacher responses through SFT. During OPD, the teacher then provides distributional supervision on prefixes generated by the student itself. Continuing from the SFT checkpoint with ESCD further improved the student’s scores on both Olympiad-100 and PHYRD-40.


5. Choice and performance of FKL, RKL, and SRKL in OPD (Reviewer BDUY)

In OPD, the student generates the training prefixes. The KL direction determines how teacher and student predictions are compared at those prefixes. For example, GKD uses student-generated trajectories with FKL in its few-shot chain-of-thought experiment on GSM8K. We adopted BPM’s FKL setting for the root-alignment term to keep the baseline comparison consistent. Following the reviewer’s suggestion, we have labeled the smoothed reverse KL used in our earlier experiment as SRKL and added a training experiment using the RKL formula. All three configurations retain the same child-completion objective.

FKL weights outcomes by teacher probabilities, while RKL weights them by student probabilities. SRKL also uses student probabilities as weights and mixes 10% of the student distribution into the target, softening the penalty on outcomes to which the teacher assigns very low probability. In our code experiments, RKL achieved lower Avg than both FKL and SRKL on LiveCodeBench and TACO. Some RKL checkpoints also showed higher repetition and truncation rates. One possible explanation is that once the student has generated the same token several times in succession, that repetition becomes part of the context and may make another repetition more likely. Even when the teacher favors a different continuation, the student may assign it very low probability. FKL directly reinforces continuations in proportion to the teacher’s probabilities. Under RKL, the direct corrective signal for such alternatives may be weaker. This may help explain why some RKL checkpoints struggled to break out of repetition and reached the generation length limit more often.


6. Cross-tokenizer alignment and the OPD objective can be improved independently (Reviewer BDUY)

Cross-tokenizer alignment and the choice of OPD objective are orthogonal design choices. ESCD determines which teacher and student predictions can be compared across different vocabularies. The OPD objective determines how those aligned predictions are used to train the student. Following the reviewer’s suggestion, we retained ESCD’s byte alignment and child-completion supervision and replaced only the FKL root term with CSD(T,S), which matches the teacher’s and student’s relative preferences over candidates. In our code experiments, ESCD+CSD(T,S) further improved on ESCD-FKL on both LiveCodeBench and TACO. These results show that improving the distribution-matching objective can yield further gains with the same cross-tokenizer alignment mechanism.

7. Random seeds and per-problem analysis of the science evaluations (Reviewer DMdp)

We used a fixed training seed for each configuration. In the supplementary code and instruction-following evaluations, we used multiple decoding seeds for the same checkpoint and reported both the individual scores and their average. This allows readers to see how the comparisons between methods vary across generation samples.

We also added per-problem analyses for both science benchmarks. On Olympiad-100, we counted the problems on which ESCD retained a correct answer, gained a correct answer, or changed from correct to incorrect relative to BPM. On PHYRD-40, we examined the change in each problem’s rubric score and the distribution of those changes. For both teacher–student pairs, ESCD outperformed BPM overall on both benchmarks. The per-problem analyses show which problems contributed to the aggregate gains.


We again thank the reviewers and the Area Chair. This discussion helped us add important experiments and give a fuller account of how the method works, how it performs, and where it applies. We will incorporate these experiments, analyses, and clarifications into the main text of the revised paper.

Dear Reviewers,

Thank you again for your review. We have provided our responses and additional results about your concerns, and would appreciate your thoughts when you have a chance to review them. Please let us know if any points remain unclear or require further clarification.

Thanks for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.