- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Summary
Paper proposes new technique to do on-policy distillation from a teacher to a student, where the 2 models have different tokenization logics.
Process
Continuing with the paper’s example of generating “Bread”, where student generates a “Br” token, and teacher has multiple tokens in its vocab starting with “Br”, but not specifically “Br” itself:
- Let the generated seq before “Bread” be ‘x’. Get possible teacher tokens set that complete “Br”, and their probabilities
- aggregate common prefix teacher tokens and uses their sum to supervise the total probability of the byte-compatible one-step completion set. For example, both ead and eads satisfy the residual byte constraint ead without requiring a prescribed probability split.
- Collect all native full-vocabulary student content tokens whose byte realizations begin with the required residual string
- Apply a teacher-mass-weighted negative log-likelihood loss on the student probability of this completion set + root (‘x’) losses.
Related and Novelty
- Token Mappings and Alignment (e.g., FuseLLM, DSKD, MultiLevelOT, and GOLD): construct comparable prediction targets through token mappings, sequence alignment, or distribution matching across vocabularies.
- Text/Byte-Level Representation and Scoring (e.g., ALM, CTLS, BLD, SimCT, and BPM): align chunks or compute likelihood scores over text/byte segments.
- These methods focus primarily on building comparable prediction targets or aligning tokens/spans rather than explicitly supervising the remaining byte constraint after partial event entry without forcing tokenizer-dependent probability splits.
- ESCD doesn’t enforce forced token-level splits. It formalizes the event-completion gap as aggregate set-probability supervision objective.
Observations
- Good metrics compared to BPM, SimCT for OPD tasks.
- Outperforms number of baselines on a variety of tasks
Required Theory
- On-policy Post training: training on student model’s own generated trajectories with RL (as opposed to off-policy, which uses a teacher model’s logits, or final token sequence to do distillation on student - SFT)
- On-policy Distillation: grading student model’s trajectories with a teacher and training on that with distillation instead of RL to get over the sparse reward issue with RL.
- “Event Completion Gap” - (Fig 2 in the paper explains this well) in context of OPD with diff tokenizations, a single teacher token might split across several student tokens - so, even if the student starts with the right characters, it might cross the teacher-equivalnet token boundaries incorrectly. the teacher supervision only specifies what bytes must eventually be completed, not how probability should be divided or allocated among these individual student completion tokens.
Critical strengths
- Very detailed analysis of different aspects of the study, including reliance on one-step rewards, vocab overlap, example walk-through, etc
- Good way to address tokenizer boundary mismatches via byte-level prefix aggregation and completion sets instead of rigid token-to-token mappings.
- Validated across multiple model architectures (dense and MoE), scales, and domains (mathematics, programming, and advanced physics/science).
Critical weaknesses
- auxiliary child supervision loss is bounded to one-step completions of teacher vocab tokens. Residuals requiring multiple tokens to finish ("Deeper" completions) have no auxiliary child loss
Did not find any serious and obvious flaw/missed opportunity in this paper
Questions for authors
- How sensitive is ESCD's final performance to the exclusion criteria used when filtering out boundary-crossing tokens or handling very short residuals (such as single-byte newlines)?
- I think the one-step limitation of the child loss can be extended to multi-step paths. I know the one-step covers most of the trajectories, but I believe there’s scope to research that.
- Can you provide concrete metrics on the overhead introduced by prefix aggregation and byte-index lookups?
5 comments
2. One-step limitation and multi-step extension
Thank you for this suggestion. ESCD retains root supervision but applies no child loss when one-step completion is unavailable. In our full-length code analysis (16 trajectories; 2,048 valid states per teacher setting), one-step completion covers 98.93% and 98.64% of visited compatible teacher-event mass under Qwen3-32B and GLM-Z1-9B, respectively. Coverage is weighted by teacher probability mass over visited compatible representative events after filtering and prefix aggregation.
Two-step extension. We implemented exhaustive two-step supervision for residuals without one-step completions, retaining the original root and one-step objectives. We sum all valid two-token path probabilities and apply a negative log-likelihood weighted by the original teacher-event mass, without event sampling or path pruning. Unlike the one-step objective, which reuses trajectory logits, this requires predictions after alternative first-token branches.
For Qwen3-32B→Qwen3.5-2B, the first four code trajectories required 559,919 branches after sharing computations for identical position/first-token pairs. On 24 GPUs (eight each for teacher, rollouts, and student training), computation of the added objective exceeded 100 minutes without completing the first training step. This measures the cost of our current exhaustive implementation, not average step time or the minimum cost of optimized methods. We therefore have no complete two-step training results. High one-step coverage does not rule out downstream benefits from deeper completion; we will clarify this limitation and discuss efficient extensions as future work.
Supplementary training comparison. These experiments use code-only data, whereas the main experiments use a 1:1 mathematics/code mixture. All models are evaluated with decoding seeds 42 and 43. Evaluation sampling changes can affect Teacher/Student scores relative to the main tables; distilled scores may additionally reflect the training-data difference.
LiveCodeBench (%, ↑)
| Method | Seed 42: pass@1 | Seed 43: pass@1 | Avg pass@1 | pass@2 |
|---|---|---|---|---|
| Teacher | 60.44 | 60.99 | 60.71 | 66.48 |
| Original Student | 13.74 | 12.09 | 12.91 | 15.38 |
| BPM | 27.47 | 24.18 | 25.82 | 31.87 |
| Root-only | 29.12 | 25.82 | 27.47 | 34.07 |
| Root + Single | 23.63 | 24.73 | 24.18 | 32.42 |
| ESCD-RKL | 28.02 | 25.82 | 26.92 | 31.87 |
| ESCD (ours, FKL) | 30.77 | 28.57 | 29.67 | 34.62 |
TACO (%, ↑)
| Method | Seed 42: pass@1 | Seed 43: pass@1 | Avg pass@1 | pass@2 |
|---|---|---|---|---|
| Teacher | 53.36 | 53.71 | 53.53 | 57.60 |
| Original Student | 8.83 | 6.36 | 7.60 | 10.25 |
| BPM | 22.26 | 19.43 | 20.85 | 28.62 |
| Root-only | 21.91 | 17.31 | 19.61 | 29.68 |
| Root + Single | 16.61 | 18.73 | 17.67 | 26.86 |
| ESCD-RKL | 24.73 | 24.03 | 24.38 | 32.16 |
| ESCD (ours, FKL) | 26.86 | 25.44 | 26.15 | 33.57 |
Avg pass@1 uses the unrounded results of both seeds; pass@2 is the fraction of problems for which either solution passes. Bold indicates the best distillation result. ESCD-FKL is our final method; ESCD-RKL uses a smoothed reverse-KL root objective with the same child objective.
ESCD leads among distillation configurations across both benchmarks and both decoding seeds, supporting consistency under the tested sampling conditions. Root + Single falls below Root-only in all four pass@1 comparisons. Section 3.3 offers a possible explanation: Single increases completion-set probability while additionally favoring one token within the set, although the teacher byte event does not specify this allocation.
This agrees with Section 5's local gradient diagnostic: COUF for Root/Single/Event is 0.7383/0.4323/0.8495 under Qwen and 0.8848/0.7493/0.9341 under GLM. These observations support preserving the completion set. However, checkpoints come from different training steps, so the comparisons do not isolate the cause of performance differences.
3. Computational overhead of prefix aggregation and byte-index lookups
Thank you for requesting concrete overhead measurements. We conducted paired measurements of complete training steps with child supervision disabled and enabled in the same implementation. The measured increment covers the entire child-supervision path, including prefix aggregation, byte-index lookups, completion-set probability computation, and backpropagation.
We evaluated Qwen3-32B→Qwen3.5-2B on eight H100 GPUs with a batch size of four. The workload comprised 24 full-length trajectories, with 12 mathematics and 12 code trajectories, totaling 620,227 output tokens. The maximum output length was 27,648 tokens per trajectory. For each batch, both modes were warmed up separately before timing, yielding six paired measurements.
Both modes used the same trajectories, teacher predictions, and student weights. We set the learning rate to zero to keep the weights fixed while still executing the full forward pass, backward pass, gradient synchronization, and optimizer step. Both modes used identical CPU offloading settings, with transfer time included in the measurements.
| Configuration | Mean time per step (s) | Peak allocated memory per GPU (GiB) |
|---|---|---|
| Root-only: child supervision disabled | 15.89 | 52.80 |
| ESCD: child supervision enabled | 17.23 | 52.80 |
Enabling child supervision increases the mean time per complete training step by approximately 8.4%, while peak allocated GPU memory remains unchanged at the reported precision. This increment reflects the cost of the full child-supervision path. Timing excludes trajectory generation, teacher pre-scoring, and one-time initialization. For memory, we report the maximum peak allocation across all training GPUs.
We will include these measurement details in the revised manuscript and limit the overhead claim to the tested model pair, long-sequence workload, and memory configuration.
We thank the reviewer for the careful assessment and constructive suggestions. We address the three questions below with additional analyses, supplementary training results, and computational overhead measurements.
1. Sensitivity to boundary rules and short residuals
The completion rule in Section 3.2 requires a student token to complete all residual bytes, while allowing it to extend beyond the residual boundary. For example, both
MemoryandMemoryWarningsatisfy the residual constraintMemory. This means that they satisfy the same byte-prefix requirement, without implying that their full continuations are semantically equivalent. The current implementation does not discard residuals merely because they contain only one byte; single-byte newline residuals can receive child supervision.Appendix C.3 already examines boundary-crossing completions. In the original Qwen and GLM subsets, these tokens account for 2.10% and 0.63% of the completion probability, respectively. Removing them produces gradients with cosine similarities of 0.9832 and 0.9887 to the full-set gradients. The original analysis also includes short residuals: 45 of the 48 Qwen events have one-byte residuals, predominantly newlines.
To extend this analysis, we generated responses to 32 questions—16 mathematics and 16 code questions—using the original Qwen3.5-2B student with thinking enabled and a maximum output length of 27,648 tokens. We analyzed the full generated trajectories, including responses reaching the length limit. Under each teacher setting, we examined 4,096 valid positions without updating model parameters.
The removal fractions are calculated using teacher-event probability weights, relative to the total weight of valid child supervision. They do not represent fractions of the overall training loss. These results show that boundary-crossing tokens carry a small share of completion probability in the analyzed trajectories, whereas one-byte residuals account for most of the weighted child supervision.