- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provides a theoretical analysis of on-policy distillation focusing on finite-sample generalization and Pass@k performance.
First, the authors derive a generalization upper bound relating the student's KL divergence from the teacher to empirical distillation loss, sample size, student description length, and the chi-squared divergence between training and evaluation prefix distributions. A complementary minimax lower bound characterizes the error caused by unobserved prefixes, with matching dependence on sample size and distribution shift in a specific construction.
Second, the paper establishes upper and lower bounds on the student-teacher Pass@k gap, showing a worst-case dependence of order for small KL error .
Third, the authors construct an exact KL proximal update that reduces distillation error and improves Pass@1 while decreasing Pass@k, even when the teacher is more accurate on every prompt.
Critical strengths
- A coherent theoretical characterization of OPD generalization.
The paper connects finite on-policy supervision, prefix distribution shift, and student-teacher KL divergence through a generalization upper bound and a complementary lower-bound construction. In particular, the analysis of unobserved prefixes provides a useful explanation of why fitting teacher feedback at observed prefixes may be insufficient for accurate imitation. The matching dependence on sample size and distribution shift in the rare-prefix construction strengthens the theoretical contribution.
- An insightful characterization of the Pass@1 versus Pass@k trade-off.
The paper establishes order-matching worst-case bounds relating KL error to the Pass@k gap. More importantly, Theorem 5 demonstrates that an exact KL proximal update can improve Pass@1 while degrading Pass@k, even when the teacher is more accurate than the initial student on every prompt. This is a meaningful theoretical observation because it shows that better teacher imitation and higher single-sample accuracy do not necessarily preserve the benefits of repeated sampling. The proposed mechanism involving correct responses on harder prompts is potentially useful for understanding distillation behavior.
Critical weaknesses
- The empirical evidence for the proposed Pass@k degradation mechanism is not sufficiently convincing.
The main empirical evidence relies on selected checkpoint pairs whose Pass@1 and Pass@64 changes are relatively small. For example, on AMC23, Pass@1 increases by 0.74 percentage points while Pass@64 decreases by 2.50 percentage points. However, the entire Pass@64 decline is attributable to a single problem whose correct-response count changes from 2/64 to 0/64. Similarly, the AIME25 result is driven by changes in a small number of problems.
Furthermore, the mechanism is not directly verified. Theorem 5 relies on teacher preferences suppressing particular correct responses on difficult prompts, but the LLM experiments do not measure the relevant teacher probabilities or directly establish this suppression. The problem groups in Table 1 are also defined partly by their observed performance changes, making the analysis descriptive rather than an independent validation of the mechanism.
- There is a substantial gap between the theoretical assumptions and the practical OPD experiments.
The generalization analysis assumes independent trajectories collected from a fixed sampling policy, bounded token-level KL losses, and a finite description-length representation of the student. In contrast, the LLM experiments involve iterative policy updates and a top-16 approximation to the distillation objective. The paper does not demonstrate that the resulting generalization bound is non-vacuous for the actual model sizes and training configurations.
Questions for authors
- Statistical reliability of the Pass@k trade-off.
Could the authors provide repeated evaluations with different sampling seeds and uncertainty estimates for the selected checkpoint comparisons in Figure 4? Given that the AMC23 Pass@64 decrease is driven by a single problem changing from two correct responses to zero, it is unclear whether the observed difference reflects an actual change in the underlying success probabilities. Reporting the frequency of similar trade-offs across all evaluated checkpoint pairs would also help establish how representative these examples are.
- Evidence for the mechanism.
Theorem 5 identifies a mechanism in which distillation suppresses correct responses on harder prompts. Could the authors measure the teacher's probabilities or log-probabilities on the student's correct responses before and after the relevant updates? Such an analysis would help distinguish the proposed mechanism from ordinary sampling variation or other changes in the response distribution.
- Applicability and interpretation of the generalization bound.
Can the authors provide a concrete evaluation of the terms in Theorem 1 for an actual OPD training round, including the description-length term, the untruncated prefix chi-squared divergence, and the corresponding KL risk? It would be particularly informative to determine whether the resulting bound is non-vacuous in the studied configurations.
Relatedly, could the authors clarify whether Theorem 1 bounds the original or clipped KL risk? Appendix A.1, Eq. (13), appears to require an additional tail term when clipping is used. Clarifying this distinction would materially affect the interpretation and applicability of the main generalization result.
3 comments
Official Comment by Authors (2/2)
W2 & Q3: The generalization analysis assumes independent trajectories collected from a fixed sampling policy, bounded token-level KL losses, and a finite description-length representation of the student. In contrast, the LLM experiments involve iterative policy updates and a top-16 approximation to the distillation objective. ... Can the authors provide a concrete evaluation of the terms in Theorem 1 for an actual OPD training round, including the description-length term, the untruncated prefix chi-squared divergence, and the corresponding KL risk?
Thank you very much for raising this point. We agree. We have added an OPD experiment with Qwen3-0.6B-Base and a frozen GRPO teacher of the same size on AQuA, using TinyLoRA and original KL over the complete vocabulary (Section 5.2 and Appendix D in our revision). One frozen Base generates approximately 97,000 independent training responses, and fresh responses from each policy provide the risk and prefix measurements. Following [1-2], the update has a description length of approximately 2,600. The empirical training risk is 0.013, the independently measured sequence KL under the updated policy is 0.012, and the estimate of without truncation is 0.014. In our numerical evaluation, Appendix A.2 tightens the bound by treating the majority of prefixes with small conditional KL separately from rare events with larger KL and weighting the larger KL bound by the probability of those events. Substituting the measured quantities into the refinement of Theorem 1 in Appendix A.2 gives an empirical value of 0.049 for the upper bound. Appendix D.3 provides the calculation of each term. We hope this concrete evaluation helps bridge the gap between our theory and actual LLM learning to some extent.
[1] Lotfi et al. Non-Vacuous Generalization Bounds for Large Language Models. ICML 2024.
[2] Zhu, Alur, and Kang. Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards. arXiv:2607.14506
Q3 (continued): Could the authors clarify whether Theorem 1 bounds the original or clipped KL risk? Appendix A.1, Eq. (13), appears to require an additional tail term when clipping is used.
Thank you for identifying this issue. In our revision, Theorem 1 bounds original sequence KL under Definition 1's uniform ratio condition relative to the teacher, adapted from [3]. The logarithm of this ratio bound bounds the original conditional KL, and Appendix A.1 applies concentration directly to that loss. Appendix C.8 complements this analysis with local KL measurements in our original 0.6B experiments: over 99.9% of positions in independent responses from checkpoints C40 and C80 have KL below 0.50, and the observed maxima in these response sets are 4.4 and 4.3, respectively. The result therefore controls the same KL used in Theorem 3, whose assumption and formula remain unchanged.
[3] Rohatgi et al. Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification. COLT 2025.
Dear Reviewer zJBe,
Thank you again for your thoughtful comments on our paper.
As the author-reviewer discussion period is drawing to a close, we wanted to respectfully follow up and ask whether you have had a chance to review our response.
Should you have any additional comments, we would be more than happy to address them before the discussion period ends.
Sincerely,
Authors of Paper 13148
Official Comment by Authors (1/2)
Thank you very much for the constructive feedback. We address each concern below.
Thank you for suggesting repeated evaluations. We repeated the complete checkpoint evaluation with fresh sampling seeds. In the first epoch of the 4B run, the new batch again shows increasing Pass@1 and decreasing Pass@64 at AIME25 C20–C40 and C80–C100; the latter also appears in the original batch (Appendix C.6). Across all 15 checkpoints, three model sizes, and three benchmarks, the frequency is 10 of 126 adjacent comparisons (7.9%) in each complete batch. The following counts describe occurrence within these evaluated runs; adjacent comparisons share checkpoints.
Appendix C.6 also reports three further sampling batches for the two selected comparisons in Figure 4. The crossing recurs in zero of three batches for AMC23 and one of three for AIME25. Pooling the 192 new responses per question gives the Pass@64 changes and 95% intervals below, obtained by paired bootstrap resampling of questions. Thus, fresh evaluations reproduce opposing changes across checkpoints. These results support the possibility described by Theorem 5.
Thank you very much for your valuable insight. As suggested by the reviewer, Section 5.3, Table 1, and Appendix C.7 now report the probabilities assigned to the same correct responses by the frozen 4B GRPO teacher and the student before and after the update. On the five selected harder problems, the teacher assigns lower probability than the old student to all 13 original correct responses, and the updated student also assigns lower probability to each response. The same ordering holds for 32 of 34 correct responses from three independent sampling batches. These measurements support the suppression of particular correct trajectories on harder prompts described in Theorem 5.