- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SEAM (Shared-Event Alignment via Marginalization) for cross-tokenizer model distillation and probability alignment. The authors argue that direct comparison of token-level distributions is inappropriate because the teacher and student may differ in their native token boundaries, vocabularies, and the semantics of special tokens. To address this issue, SEAM maps both models into a lower-level shared event space, such as byte, control, and stop events. At student token boundaries, the method defines local observations of the next byte or control event and marginalizes over compatible tokens that cover the corresponding boundary. This produces comparable local conditional distributions that can subsequently be used as distillation objectives.
Critical strengths
The paper introduces a clearer alignment objective for cross-tokenizer distillation by defining mutually exclusive shared next-byte and control events at student token boundaries and marginalizing over compatible tokens that span these boundaries. By separating output representations from assumptions about token boundaries, the method directly addresses the issue that existing interfaces, such as BPM and SimCT, may compare different prediction objects due to differences in token boundaries, vocabulary coverage, and length normalization. This provides a more interpretable and fairer probabilistic objective for cross-tokenizer distillation.
Critical weaknesses
The empirical evidence remains insufficient. In Table 6, the benchmark comparisons primarily use training seed 101, while Full SEAM requires substantially more GPU hours (19.33) than the symmetric/common control (12.50) and the equal-weight control (14.14). Without more comprehensive evaluations across multiple training seeds, compute-matched comparisons, and statistical significance tests, it is difficult to attribute the reported improvements of 31.70 versus 31.04/30.69 and 39.00 versus 38.50/38.38 entirely to shared-event marginalization. These gains may also result from differences in the query set, loss weighting, sampling volume, or training costs.
Questions for authors
The main question concerns whether the reported benefits of Full SEAM have been adequately controlled for differences in computational cost and sampling. Specifically, in Table 6, Full SEAM requires substantially more GPU hours (19.33) than the symmetric/common control (12.50) and the equal-weight control (14.14). Furthermore, Figure 5 indicates that the endpoints of different methods may be affected by differences in accepted-sample counts. Could the authors provide additional results under matched accepted-sample budgets or matched computational budgets, along with results from 2 to 3 independent training seeds and corresponding confidence intervals?
5 comments
Response to Reviewer Tgzs - Part 2 of 5
The original seed-101 SEAM run shared GPU resources with other processes, so its recorded execution time was affected by contention. The additional seed-202/303 experiments had exclusive use of their allocated two H200 GPUs. We combine benchmark endpoint scores across seeds for the training-run statistics, while the new budget comparisons use the seed-202/303 cost measurements. The original Table 6 costs were 19.33/12.50/14.14 GPU-hours for Full SEAM/symmetric-common/equal-weight. Comparisons between the original and new cost figures therefore also reflect their execution conditions. The new experiments do not re-evaluate the original separate 800-example validation panel.
Matched computational budgets. We additionally report direct downstream measurements from the budget-matched experiments for training seeds 202 and 303, at 4.30 and 6.50 GPU-hours and at each seed's symmetric/common endpoint budget. The scores below are direct measurements from the budget-matched evaluations. Each variant uses its largest completed update count within the budget, shown in parentheses. Bold marks the highest Avg5 in each row.
| Training seed | GPU-hour budget | Full SEAM Avg5 (update) | Symmetric/common Avg5 (update) | Equal-weight Avg5 (update) | Gain over common | Gain over equal-weight |
|---|---|---|---|---|---|---|
| 202 | 4.30 | 30.441 (20) | 30.435 (38) | 30.407 (23) | +0.006 | +0.034 |
| 202 | 6.50 | 31.030 (32) | 30.336 (60) | 30.782 (33) | +0.694 | +0.248 |
| 202 | 8.38 | 31.298 (44) | 30.287 (78) | 30.948 (46) | +1.011 | +0.350 |
| 303 | 4.30 | 30.575 (20) | 30.161 (38) | 30.114 (21) | +0.414 | +0.461 |
| 303 | 6.50 | 31.812 (33) | 30.556 (58) | 30.795 (35) | +1.256 | +1.017 |
| 303 | 8.67 | 31.426 (44) | 30.146 (78) | 31.017 (45) | +1.280 | +0.409 |
Full SEAM is above both controls at all six displayed seed/budget combinations. The gains over symmetric/common range from +0.006 to +1.280 points, and those over equal-weight range from +0.034 to +1.017 points. At seed 202 and 4.30 GPU-hours, the margins are only +0.006 and +0.034 points, respectively; this budget therefore shows near-tied performance. Budget points from the same training run are correlated, so they are not counted as additional independent training seeds.
Together, the matched-response and matched-compute measurements strengthen the evidence for the full configuration relative to the two controls. The equal-weight comparison directly probes the weighting rule while retaining the same all-student query rule. The symmetric/common comparison tests the expanded query coverage as a package, with query counts and teacher computation intentionally different. These results reduce the concern that gains arise solely from more accepted responses or a larger overall compute budget. The endpoint comparison spans three training seeds; the new budget controls cover two seeds on the Gemma pair. The results support the full configuration without uniquely isolating every component of shared-event marginalization.
2. Additional accepted-sample controls against the main baselines
We checked the training records to match the accepted responses actually used for optimization. For the five main methods, each complete update uses 256 accepted training responses. The evaluated intermediate checkpoints therefore provide matched accepted-sample budgets:
| Checkpoint | Accepted responses used for training |
|---|---|
| Step 20 | 5,120 |
| Step 40 | 10,240 |
| Step 60 | 15,360 |
This holds for all five methods in the seed-202 curves across the three model pairs. Comparing the same steps therefore matches both the number of completed updates and the accepted-response count. The 29 completed main runs also consume the same 20,000 prompt-occurrence budget, while accepted-response counts can differ at their endpoints. Equal accepted counts do not imply identical trajectories, since each method samples from its own student policy.
The measured Avg5 scores at these matched accepted-sample budgets are shown below. All results use training seed 202; scores are pass@1 × 100, averaged equally over the five benchmarks. These are direct checkpoint evaluations, without interpolation. Bold indicates the highest score in each row.
Response to Reviewer Tgzs - Part 3 of 5
| Model pair | Accepted training responses | SEAM | BPM | SimCT | GOLD | DSKD |
|---|---|---|---|---|---|---|
| Qwen3-32B → Gemma-2-2B-it | 5,120 | 30.441 | 30.760 | 29.903 | 30.287 | 27.036 |
| Qwen3-32B → Gemma-2-2B-it | 10,240 | 31.321 | 30.303 | 28.960 | 19.416 | 28.728 |
| Qwen3-32B → Gemma-2-2B-it | 15,360 | 30.806 | 30.095 | 28.486 | 21.922 | 28.706 |
| phi-4 → Qwen3.5-2B | 5,120 | 44.553 | 44.552 | 40.897 | 44.076 | 39.933 |
| phi-4 → Qwen3.5-2B | 10,240 | 45.073 | 44.510 | 44.746 | 44.293 | 43.574 |
| phi-4 → Qwen3.5-2B | 15,360 | 45.204 | 44.428 | 45.263 | 44.426 | 44.441 |
| Qwen3-32B → Qwen3.5-2B | 5,120 | 45.662 | 45.269 | 42.385 | 45.210 | 42.296 |
| Qwen3-32B → Qwen3.5-2B | 10,240 | 46.600 | 46.006 | 44.716 | 46.022 | 45.024 |
| Qwen3-32B → Qwen3.5-2B | 15,360 | 46.257 | 45.833 | 45.480 | 45.951 | 44.958 |
Under matched accepted-sample budgets, SEAM has the highest displayed Avg5 in seven of the nine evaluated settings and at every evaluated budget on the Qwen pair. On Gemma and Phi, its relative performance varies with the sample budget; in particular, the lead over BPM at the smallest Phi budget is only 0.001 points. These rankings describe the measured scores and are not tests of statistical significance.
3. Additional compute-budget controls against the main baselines
For training seed 202, we evaluated actual checkpoints under a common GPU-hour budget for each model pair: 8.32 GPU-hours for Gemma, 6.40 GPU-hours for Phi, and 9.87 GPU-hours for Qwen. Each budget is the smallest endpoint cost among the five methods. Each method uses the largest completed update count within that budget. The scores below are direct downstream measurements from those checkpoints. The common budget is a compute cap; discrete updates need not use exactly identical elapsed GPU-hours.
| Model pair | GPU-hour budget | SEAM | BPM | SimCT | GOLD | DSKD |
|---|---|---|---|---|---|---|
| Qwen3-32B → Gemma-2-2B-it | 8.32 | 31.239 | 30.362 | 29.044 | 20.212 | 28.859 |
| phi-4 → Qwen3.5-2B | 6.40 | 45.102 | 44.532 | 45.033 | 44.579 | 43.971 |
| Qwen3-32B → Qwen3.5-2B | 9.87 | 46.646 | 45.937 | 45.283 | 46.208 | 45.197 |
For Gemma, the evaluated updates are 42/61/48/43/77 for SEAM/BPM/SimCT/GOLD/DSKD, respectively; for Phi they are 33/72/45/48/78, and for Qwen they are 49/74/52/51/77. At these budgets, SEAM has the highest Avg5 in all three model pairs. Its gains over the strongest baseline at each budget are +0.877 over BPM on Gemma, +0.069 over SimCT on Phi, and +0.438 over GOLD on Qwen. The Phi margin over SimCT is small (0.069 points). These direct measurements show gains under the same compute cap in the tested settings, while the preferred quality–cost trade-off can still depend on the model pair and budget. These main-baseline budget comparisons use training seed 202; the Table 6 controls in Section 1 cover seeds 202 and 303.
4. Training-seed uncertainty against the main baselines
We added independent training seeds 202 and 303 for SEAM and the main baselines across the three model pairs. Complete downstream results are available for 29 of the 30 planned additional runs. We did not pursue further Gemma–GOLD training with seed 303 after observing substantial performance degradation with seed 202. Together with the original training seed 101, these results provide three independent OPD training runs for all other configurations.
The table reports Avg5 mean ± sample standard deviation across training seeds 101, 202, and 303. These standard deviations describe training-run variability, whereas the task-level standard deviations in the experiment tables describe decoding seeds 42–46. We use the original paper's reported seed-101 endpoint averages and the updated seed-202/303 endpoint averages; derived statistics are approximate to the precision of the reported inputs.
| Model pair | BPM | SimCT | GOLD | DSKD | SEAM | SEAM − strongest baseline |
|---|---|---|---|---|---|---|
| Qwen3-32B → Gemma-2-2B-it | 30.375 ± 0.227 | 29.112 ± 1.038 | — | 28.798 ± 0.180 | 31.276 ± 0.393 | +0.900 |
| phi-4 → Qwen3.5-2B | 44.528 ± 0.353 | 44.999 ± 0.157 | 44.576 ± 0.348 | 43.744 ± 0.206 | 45.677 ± 0.223 | +0.677 |
| Qwen3-32B → Qwen3.5-2B | 46.265 ± 0.554 | 45.657 ± 0.421 | 46.345 ± 0.217 | 45.229 ± 0.071 | 47.151 ± 0.083 | +0.806 |
The strongest baselines by three-seed mean Avg5 are BPM for Gemma, SimCT for Phi, and GOLD for Qwen. Gemma–GOLD is excluded from the three-seed aggregate because seed 303 is unavailable; its available scores are 30.20 and 22.021 for seeds 101 and 202. SEAM has the highest observed Avg5 among available methods in all nine model-pair/training-seed settings.
Response to Reviewer Tgzs - Part 1 of 5
Thank you for identifying the need to control accepted-response counts, computation, and training randomness. We extended the Table 6 comparison to three training seeds and added direct evaluations under matched accepted-response budgets and common compute caps for seeds 202 and 303. At the matched 78-update endpoints, Full SEAM improves Avg5 over symmetric/common and equal-weight by 0.785 and 0.812 percentage points on average; both paired 95% confidence intervals are positive. It also has the highest measured Avg5 at all six tested seed/compute-budget combinations, with near-tied results at the smallest budget for seed 202. Below, we present these controls first, followed by matched-budget comparisons against the main baselines.
All benchmark scores below are reported as pass@1 × 100 (percent). Avg5 is the equally weighted mean of the scores on GSM8K, MATH-500, MBPP, LiveCodeBench-v6, and TACO. Each job in the additional experiments reported here was run on two NVIDIA H200 GPUs. GPU-hours are calculated as the recorded run time in hours multiplied by the number of GPUs; for these two-GPU jobs, GPU-hours equal twice the run time. Except for the training seed and changes required by the specific comparison, such as the divergence, mixture weight, lookahead threshold, or control variant in the ablation experiments, all other training and evaluation parameters were kept identical to those of the corresponding seed-101 configuration. Score differences are expressed in percentage points.
1. Table 6: matched budgets and three training seeds
We have now completed the Full SEAM, symmetric/common, and equal-weight comparisons on Qwen3-32B → Gemma-2-2B-it for training seeds 202 and 303. These directly extend the comparison in Table 6, which reported seed 101. As defined in Appendix F.2, symmetric/common applies covering to both models only at common boundaries; Full SEAM uses all eligible student boundaries. The equal-weight control keeps the all-student query rule but scales each response's loss sum by its common-to-all boundary-count ratio. The new configurations differ only in the variant, training seed, device assignment, and workspace; the remaining configuration is shared.
Matched updates and accepted-response budgets. All three variants finish at step 78. Using 256 accepted training responses per complete update, steps 20/40/60/78 correspond to 5,120/10,240/15,360/19,968 accepted responses. These budgets match training-response counts, not byte lengths, query counts, teacher calls, or identical sampled trajectories. The following are direct checkpoint evaluations:
Full SEAM has the highest Avg5 in all eight matched-update comparisons. At the endpoint, its gains over symmetric/common are +0.637/+1.057 points for seeds 202/303, and its gains over equal-weight are +0.599/+0.826 points. These improvements are measured by Avg5, the equally weighted five-benchmark average.
Endpoint compute and uncertainty.
Full SEAM costs 1.72×/1.64× as much as symmetric/common, but only 1.02×/1.01× as much as equal-weight at these endpoints. Thus, the gain over equal-weight is observed under nearly equal total compute as well as equal update counts. The lower cost of symmetric/common motivates the fixed-compute comparison below.
These endpoint comparisons pair training seeds 101, 202, and 303. The seed-101 Avg5 values are 31.70/31.04/30.69 for Full SEAM/symmetric-common/equal-weight, as reported in the original Table 6; all three original variants also completed 78 updates. The mean gains are +0.785 and +0.812 points, respectively. Both marginal 95% confidence intervals are positive, with two-sided paired t-test p-values of 0.0289 and 0.0208, respectively. The p-values are unadjusted, and the confidence intervals are marginal. With only three training seeds, normality cannot be reliably assessed, and the statistics are approximate to the precision of the reported scores.