- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Lookahead KL Routing (LKR) for on-policy distillation. LKR redirects part of a negative sampled-token update toward alternatives selected using current teacher probabilities and one-step student–teacher KL divergence. A constrained reverse-KL projection yields a water-filling allocation, which preserves the donor’s local logit gradient. Budgeted event sampling reduces lookahead queries. Experiments cover a held-out mathematical reasoning set and six public benchmarks, with three-seed results for the primary model pair, component ablations, and a single-seed larger-scale comparison.
Critical strengths
-
A coherent and interpretable routing formulation. The donor–recipient decomposition makes the destination of negative updates explicit. The constrained projection provides a well-defined allocation, and the manuscript carefully limits its guarantees to the auxiliary objective and local logit gradients.
-
Meaningful empirical improvements with useful controls. The primary three-seed comparison improves held-out Mean@8 and Cons@8 over OPD by 2.41 and 2.27 percentage points, respectively. Gains also persist against longer OPD training under an approximately matched training-time budget, providing evidence beyond a step-matched comparison.
Critical weaknesses
-
Theory–implementation mismatch. The rollout-dependent deterministic seeding leaves the applicability of Proposition 3’s conditional-unbiasedness guarantee to the released sampler unresolved.
-
Unvalidated KL approximation. The coarse successor KL can reverse candidate preferences, yet its effect on routing allocations is not empirically quantified.
-
Insufficient component-level evidence. Single-run ablations and limited equal-cost comparisons with cheaper routing variants do not firmly establish the incremental benefits of lookahead and water-filling.
Questions for authors
-
How is the rollout-derived sampling seed reconciled with the conditional inclusion probabilities in Proposition 3? Was this sampler used in the reported training runs?
-
On representative training events, how frequently does coarse successor KL change candidate rankings or routing allocations relative to full-vocabulary KL? What are the intersection-size and retained-probability-mass distributions?
-
Do the improvements from lookahead and water-filling persist across matched training seeds and equal measured training-time budgets, relative to Teacher-only and normalized-score routing?
-
Which implementation and configuration produced the results, given the restriction in the released projection function?
3 comments
Thank you for the additional experiments. Two concerns remain:
- Top-100 is still a coarsened reference. A comparison against full-vocabulary KL on a small representative subset would more directly assess the approximation’s effect on routing.
- The time-matched ablations provide useful support for both components, but one shared training seed does not establish robustness across seeds. Results over multiple matched training seeds are still needed.
Response to Reviewer QEUV
Dear Reviewer QEUV,
Thank you for following up. Because time was limited during the initial rebuttal, we first ran and reported the time-matched ablation with one training seed shared across methods. We have just obtained the results for two additional matched seeds. We report the three-seed results below, along with the requested full-vocabulary KL comparison.
1. Full-vocabulary KL comparison
We randomly sampled 32 questions, yielding 578 valid routing events and 2,227 successor prefixes. We computed successor reverse KL in FP32 over all 151,665 tokens in the effective vocabulary shared by our 1.5B student and 7B teacher. On the same events, we compared candidate rankings and water-filling allocations using Top-20, Top-100, and full-vocabulary KL.
| Metric | Top-20 → Top-100 | Top-20 → Full | Top-100 → Full |
|---|---|---|---|
| KL ranking reversal (%) | 7.61 | 9.11 | 1.90 |
| Routing score ranking reversal (%) | 4.95 | 5.89 | 1.17 |
| Highest-scoring candidate changed (%) | 4.67 | 5.71 | 1.21 |
| Water-filling active set changed (%) | 10.90 | 12.98 | 2.77 |
| Mean allocation TV | 0.0381 | 0.0445 | 0.0084 |
| Median allocation TV | 0.0139 | 0.0168 | 0.0029 |
| P90 allocation TV | 0.1048 | 0.1216 | 0.0231 |
| Maximum allocation TV | 0.396 | 0.442 | 0.118 |
Ranking reversal rates use pairs strictly ordered in both settings. Allocation TV is half the L1 distance between the two vectors of allocation shares. We summarize it over all 578 events.
The full-vocabulary comparison supports Top-20 as a practical routing approximation in our setting. Compared with full KL, Top-20 changed the highest-scoring candidate in 5.71% of events and shifted 4.45% of the allocation mass on average, with larger shifts in some events. For Top-100, the corresponding change rate and mean allocation TV were 1.21% and 0.0084, supporting its use as a finer reference in our earlier analysis.
2. Time-matched ablations across three seeds
Each method used the same three training seeds and an 8-hour target training budget per run. All runs were evaluated on the same 1,000 held-out questions. We report mean ± sample standard deviation across seeds.
| Method | Training steps | Measured training time (s) | Mean@8 (%) | Cons@8 (%) |
|---|---|---|---|---|
| Teacher-only | 333.3 ± 1.2 | 28,864 ± 14 | 25.03 ± 0.26 | 35.23 ± 0.25 |
| Normalized-score | 251.7 ± 0.6 | 28,876 ± 12 | 24.84 ± 0.22 | 34.97 ± 0.31 |
| LKR | 250.3 ± 0.6 | 28,890 ± 14 | 25.73 ± 0.16 | 36.47 ± 0.21 |
LKR outperformed both controls on both metrics in all three matched seeds. The paired gains in Mean@8 and Cons@8 were 0.70 ± 0.14 and 1.23 ± 0.06 percentage points over Teacher-only, and 0.89 ± 0.14 and 1.50 ± 0.10 percentage points over Normalized-score. These results support the contributions of lookahead and water-filling under the matched time budget in this setting.
We sincerely thank you for taking the time to read our rebuttal and share your follow-up comments. Your suggestions helped us test both points more directly. We would be grateful for any further thoughts you may have on the new results and would be happy to clarify any remaining questions.
Response to Reviewer QEUV
Thank you for recognizing the routing formulation and experimental controls. We address your questions below.
1. Sampling seed and Proposition 3
Thank you for highlighting this detail. The reported training runs used this sampler: it selects at most m donors uniformly without replacement from each rollout's negative-feedback positions, using a seed derived from rollout-related inputs.
Proposition 3 considers random donor sampling conditional on a fixed rollout and scores. With the current seed construction, fixing the relevant inputs also fixes the selected donors. To align the implementation with this condition, future experiments will use an independent donor-sampling seed to initialize a separate random generator once at training start, advancing its state throughout training. We will document the historical seed setup and this adjustment in the revision.
2. Successor KL resolution and its effect on routing
Considering the experimental cost, we added successor Top-20/50/100 resolution comparisons, using Top-100 as a finer reference. All three merge the vocabulary outside the teacher-student intersection into one tail bucket.
Using 128 rollouts at the primary pair's step-250 checkpoint, we evaluated 2,311 valid routing events and 8,910 successor prefixes. We fixed the current candidates (cap k=4), current teacher/student probabilities, and clipped dose, changing only successor KL resolution before recomputing water-filling shares.
The table reports successor intersection size and both models' probability mass on it as median [P10, P90].
\text{Top-K}&\text{Intersection size}&\text{Student mass}&\text{Teacher mass}\\\\ 20&11\\ [4,16]&0.827\\ [0.412,0.968]&0.862\\ [0.478,0.981]\\\\ 50&29\\ [13,42]&0.913\\ [0.609,0.989]&0.931\\ [0.682,0.994]\\\\ 100&63\\ [31,85]&0.952\\ [0.738,0.995]&0.968\\ [0.804,0.998] \end{array}$$ Top-20 retains median masses of 82.7% and 86.2% for the student and teacher; higher resolutions represent more probability mass token by token. We also measured changes in rankings and allocations: $$\begin{array}{lrrrr} \text{Comparison}&\text{KL reversal (\\%)}&\text{Top-score change (\\%)}&\text{Mean TV}&\text{P90 TV}\\\\ 20\to50&4.74&2.99&0.0259&0.0746\\\\ 20\to100&7.59&4.72&0.0379&0.1042\\\\ 50\to100&3.03&2.03&0.0151&0.0413 \end{array}$$ Reversal rates use candidate pairs strictly ordered under both resolutions; other metrics use the 2,311 valid events. TV is half the L1 distance between allocation shares, measuring the share requiring reallocation. From Top-20 to Top-100, 7.59% of candidate pairs reverse KL ordering, but only 4.72% of events change their highest-scoring candidate, with an average reallocation of 3.79%. From Top-50 to Top-100, this falls to 1.51%. These results support Top-20 as a practical routing approximation. Based on this resolution trend, we conjecture that using full-vocabulary KL may also have a limited effect on overall allocations. We will add these experiments and metric definitions to the revision. **3. Lookahead and water-filling under matched training-time budgets** Thank you for suggesting controls with matched seeds and measured training time. We added an ablation with one shared training seed, the same initialization and data on eight GPUs, with k=4. The target training-time budget matches the main experiments. Evaluation uses the same 1,000 Held-out questions. Teacher-only retains current teacher scores and water-filling but removes both models' successor queries. Normalized-score retains lookahead scores but replaces water-filling with score normalization. $$\begin{array}{lrrrrr} \text{Method}&\text{Steps}&\text{Time (s)}&\text{Teacher successor queries}&\text{Mean@8 (\\%)}&\text{Cons@8 (\\%)}\\\\ \text{Teacher-only}&334&28{,}857&0&24.96&35.20\\\\ \text{Normalized-score}&252&28{,}879&2{,}327{,}846&24.84&34.90\\\\ \text{LKR}&250&28{,}894&2{,}311{,}492&25.61&36.40 \end{array}$$ Despite Teacher-only completing 84 more steps, LKR gains 0.65/1.20 percentage points in Mean@8/Cons@8, supporting the benefit of successor information. Against Normalized-score, with similar runtime, steps, and query counts, LKR gains 0.78/1.50 points, supporting water-filling. Both components retain their benefits in this time-matched ablation. We will add the comparison and configurations. **4. Implementation and configuration for k=8** Thank you for noting this version discrepancy. The main and new supplementary experiments use k=4; k=8 is a hyperparameter experiment. It used a training version supporting eight candidates, with the cap set to 8. We omitted the updated candidate-count check when uploading the code, leaving the old restriction in the released version. We apologize for this inconsistency. After acceptance, we will open-source the correct code version consistent with the paper's experiments, together with the corresponding launch configurations.