acceptodds
Under review as a conference paper at ICLR 2027

Theoretical Analysis of On-Policy Distillation: Generalization and Pass@k

12 views

Abstract

On-policy distillation trains a student using teacher feedback on its own generated responses. We provide a theoretical analysis of its generalization and its Pass@k performance. First, we establish a uniform finite-sample upper bound relating distillation error to empirical training loss, sample size, student description length, and the shift in prefix distributions between training and evaluation. A complementary lower bound quantifies the unavoidable error from unobserved prefixes and recovers the same dependence on sample size and distribution shift in a worst-case construction. Second, we establish upper and lower bounds on the student-teacher Pass@k gap that match in order when KL error is small. Finally, we theoretically show that an exact KL proximal update can reduce distillation error and improve Pass@1 while lowering Pass@k. We identify a mechanism for this trade-off: teacher preferences can suppress correct responses on harder prompts, while gains on easier prompts raise Pass@1 but contribute little at larger sampling budgets. Synthetic experiments verify our theory, and experiments on large language models support the practical relevance of our analysis.

Primary area:
learning theory
Keywords:
learning theory, pass@k, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zJBeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a theoretical analysis of on-policy distillation focusing on finite-sample generalization and Pass@k performance.

First, the authors derive a generalization upper bound relating the student's KL divergence from the teacher to empirical distillation loss, sample size, student description length, and the chi-squared divergence between training and evaluation prefix distributions. A complementary minimax lower bound characterizes the error caused by unobserved prefixes, with matching dependence on sample size and distribution shift in a specific construction.

Second, the paper establishes upper and lower bounds on the student-teacher Pass@k gap, showing a worst-case dependence of order kϵ\sqrt{k\epsilon} for small KL error ϵ\epsilon.

Third, the authors construct an exact KL proximal update that reduces distillation error and improves Pass@1 while decreasing Pass@k, even when the teacher is more accurate on every prompt.

Critical strengths
  1. A coherent theoretical characterization of OPD generalization.

The paper connects finite on-policy supervision, prefix distribution shift, and student-teacher KL divergence through a generalization upper bound and a complementary lower-bound construction. In particular, the analysis of unobserved prefixes provides a useful explanation of why fitting teacher feedback at observed prefixes may be insufficient for accurate imitation. The matching dependence on sample size and distribution shift in the rare-prefix construction strengthens the theoretical contribution.

  1. An insightful characterization of the Pass@1 versus Pass@k trade-off.

The paper establishes order-matching worst-case bounds relating KL error to the Pass@k gap. More importantly, Theorem 5 demonstrates that an exact KL proximal update can improve Pass@1 while degrading Pass@k, even when the teacher is more accurate than the initial student on every prompt. This is a meaningful theoretical observation because it shows that better teacher imitation and higher single-sample accuracy do not necessarily preserve the benefits of repeated sampling. The proposed mechanism involving correct responses on harder prompts is potentially useful for understanding distillation behavior.

Critical weaknesses
  1. The empirical evidence for the proposed Pass@k degradation mechanism is not sufficiently convincing.

The main empirical evidence relies on selected checkpoint pairs whose Pass@1 and Pass@64 changes are relatively small. For example, on AMC23, Pass@1 increases by 0.74 percentage points while Pass@64 decreases by 2.50 percentage points. However, the entire Pass@64 decline is attributable to a single problem whose correct-response count changes from 2/64 to 0/64. Similarly, the AIME25 result is driven by changes in a small number of problems.

Furthermore, the mechanism is not directly verified. Theorem 5 relies on teacher preferences suppressing particular correct responses on difficult prompts, but the LLM experiments do not measure the relevant teacher probabilities or directly establish this suppression. The problem groups in Table 1 are also defined partly by their observed performance changes, making the analysis descriptive rather than an independent validation of the mechanism.

  1. There is a substantial gap between the theoretical assumptions and the practical OPD experiments.

The generalization analysis assumes independent trajectories collected from a fixed sampling policy, bounded token-level KL losses, and a finite description-length representation of the student. In contrast, the LLM experiments involve iterative policy updates and a top-16 approximation to the distillation objective. The paper does not demonstrate that the resulting generalization bound is non-vacuous for the actual model sizes and training configurations.

Questions for authors
  1. Statistical reliability of the Pass@k trade-off.

Could the authors provide repeated evaluations with different sampling seeds and uncertainty estimates for the selected checkpoint comparisons in Figure 4? Given that the AMC23 Pass@64 decrease is driven by a single problem changing from two correct responses to zero, it is unclear whether the observed difference reflects an actual change in the underlying success probabilities. Reporting the frequency of similar trade-offs across all evaluated checkpoint pairs would also help establish how representative these examples are.

  1. Evidence for the mechanism.

Theorem 5 identifies a mechanism in which distillation suppresses correct responses on harder prompts. Could the authors measure the teacher's probabilities or log-probabilities on the student's correct responses before and after the relevant updates? Such an analysis would help distinguish the proposed mechanism from ordinary sampling variation or other changes in the response distribution.

  1. Applicability and interpretation of the generalization bound.

Can the authors provide a concrete evaluation of the terms in Theorem 1 for an actual OPD training round, including the description-length term, the untruncated prefix chi-squared divergence, and the corresponding KL risk? It would be particularly informative to determine whether the resulting bound is non-vacuous in the studied configurations.

Relatedly, could the authors clarify whether Theorem 1 bounds the original or clipped KL risk? Appendix A.1, Eq. (13), appears to require an additional tail term when clipping is used. Clarifying this distinction would materially affect the interpretation and applicability of the main generalization result.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you very much for the constructive feedback. We address each concern below.

W1 & Q1: The main empirical evidence relies on selected checkpoint pairs whose Pass@1 and Pass@64 changes are relatively small. ... the entire Pass@64 decline is attributable to a single problem whose correct-response count changes from 2/64 to 0/64. ... Could the authors provide repeated evaluations with different sampling seeds and uncertainty estimates for the selected checkpoint comparisons in Figure 4? ... Reporting the frequency of similar trade-offs across all evaluated checkpoint pairs would also help establish how representative these examples are.

Thank you for suggesting repeated evaluations. We repeated the complete checkpoint evaluation with fresh sampling seeds. In the first epoch of the 4B run, the new batch again shows increasing Pass@1 and decreasing Pass@64 at AIME25 C20–C40 and C80–C100; the latter also appears in the original batch (Appendix C.6). Across all 15 checkpoints, three model sizes, and three benchmarks, the frequency is 10 of 126 adjacent comparisons (7.9%) in each complete batch. The following counts describe occurrence within these evaluated runs; adjacent comparisons share checkpoints.

StudentAdjacent comparisonsOriginal batchNew batchPresent in both batches
Qwen3-0.6B42110
Qwen3-1.7B42541
Qwen3-4B42452
Total12610103

Appendix C.6 also reports three further sampling batches for the two selected comparisons in Figure 4. The crossing recurs in zero of three batches for AMC23 and one of three for AIME25. Pooling the 192 new responses per question gives the Pass@64 changes and 95% intervals below, obtained by paired bootstrap resampling of questions. Thus, fresh evaluations reproduce opposing changes across checkpoints. These results support the possibility described by Theorem 5.

ComparisonPooled change in Pass@64 (percentage points)95% interval
AMC23, C60–C80+0.48[−0.19, +1.7]
AIME25, C80–C100+2.2[−7.2, +11]

W1 & Q2: Theorem 5 relies on teacher preferences suppressing particular correct responses on difficult prompts, but the LLM experiments do not measure the relevant teacher probabilities or directly establish this suppression. The problem groups in Table 1 are also defined partly by their observed performance changes, making the analysis descriptive rather than an independent validation of the mechanism. ... Could the authors measure the teacher's probabilities or log-probabilities on the student's correct responses before and after the relevant updates?

Thank you very much for your valuable insight. As suggested by the reviewer, Section 5.3, Table 1, and Appendix C.7 now report the probabilities assigned to the same correct responses by the frozen 4B GRPO teacher and the student before and after the update. On the five selected harder problems, the teacher assigns lower probability than the old student to all 13 original correct responses, and the updated student also assigns lower probability to each response. The same ordering holds for 32 of 34 correct responses from three independent sampling batches. These measurements support the suppression of particular correct trajectories on harder prompts described in Theorem 5.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W2 & Q3: The generalization analysis assumes independent trajectories collected from a fixed sampling policy, bounded token-level KL losses, and a finite description-length representation of the student. In contrast, the LLM experiments involve iterative policy updates and a top-16 approximation to the distillation objective. ... Can the authors provide a concrete evaluation of the terms in Theorem 1 for an actual OPD training round, including the description-length term, the untruncated prefix chi-squared divergence, and the corresponding KL risk?

Thank you very much for raising this point. We agree. We have added an OPD experiment with Qwen3-0.6B-Base and a frozen GRPO teacher of the same size on AQuA, using TinyLoRA and original KL over the complete vocabulary (Section 5.2 and Appendix D in our revision). One frozen Base generates approximately 97,000 independent training responses, and fresh responses from each policy provide the risk and prefix measurements. Following [1-2], the update has a description length K(π)K(\pi) of approximately 2,600. The empirical training risk is 0.013, the independently measured sequence KL under the updated policy is 0.012, and the estimate of χ2(Pπ∥Pπold)\chi^2(P_\pi\Vert P_{\pi_{\mathrm{old}}}) without truncation is 0.014. In our numerical evaluation, Appendix A.2 tightens the bound by treating the majority of prefixes with small conditional KL separately from rare events with larger KL and weighting the larger KL bound by the probability of those events. Substituting the measured quantities into the refinement of Theorem 1 in Appendix A.2 gives an empirical value of 0.049 for the upper bound. Appendix D.3 provides the calculation of each term. We hope this concrete evaluation helps bridge the gap between our theory and actual LLM learning to some extent.

[1] Lotfi et al. Non-Vacuous Generalization Bounds for Large Language Models. ICML 2024.

[2] Zhu, Alur, and Kang. Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards. arXiv:2607.14506

Q3 (continued): Could the authors clarify whether Theorem 1 bounds the original or clipped KL risk? Appendix A.1, Eq. (13), appears to require an additional tail term when clipping is used.

Thank you for identifying this issue. In our revision, Theorem 1 bounds original sequence KL under Definition 1's uniform ratio condition relative to the teacher, adapted from [3]. The logarithm of this ratio bound bounds the original conditional KL, and Appendix A.1 applies concentration directly to that loss. Appendix C.8 complements this analysis with local KL measurements in our original 0.6B experiments: over 99.9% of positions in independent responses from checkpoints C40 and C80 have KL below 0.50, and the observed maxima in these response sets are 4.4 and 4.3, respectively. The result therefore controls the same KL used in Theorem 3, whose assumption and formula remain unchanged.

[3] Rohatgi et al. Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification. COLT 2025.

Dear Reviewer zJBe,

Thank you again for your thoughtful comments on our paper.

As the author-reviewer discussion period is drawing to a close, we wanted to respectfully follow up and ask whether you have had a chance to review our response.

Should you have any additional comments, we would be more than happy to address them before the discussion period ends.

Sincerely,
Authors of Paper 13148

Reviewer_eGGvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a theoretical analysis of on-policy distillation (OPD). For data sampled from an old student and next-token feedback supplied by a fixed teacher, it proves a uniform finite-sample upper bound involving empirical distillation loss, sample size, description length, and chi-squared shift between old and new prefix distributions. A complementary lower bound and minimax construction show how unobserved prefixes create unavoidable error. The paper then bounds the student-teacher Pass@k gap in terms of KL distillation error and constructs an exact KL proximal update where KL error decreases and Pass@1 improves while Pass@k decreases. Synthetic experiments and Qwen3 checkpoint analyses on AIME24, AIME25, and AMC23 provide qualitative support.

Critical strengths
  1. The rare-prefix construction gives a concrete lower-bound example showing that prefix-distribution shift can create unavoidable distillation error even with exact teacher feedback on observed prefixes.
  2. The Pass@1-up/Pass@k-down construction formalizes a practically relevant failure mode in which closer teacher imitation does not preserve performance at larger sampling budgets.
Critical weaknesses
  1. The practical usefulness of Theorem 1 is limited by strong assumptions that may hide large constants or be difficult to verify in real OPD. In particular, the description length K(π)K(\pi) for a learned LoRA or full-parameter update may be very large; finite χ2(Pπ∥Pπold)\chi^2(P_\pi \| P_{\pi_{\mathrm{old}}}) implicitly requires absolute continuity, while even finite density ratios can make the shift term very large; and the independent-trajectory assumption does not directly cover adaptive sampling, replay, or multi-round OPD. Consequently, the resulting guarantee may be non-vacuous only in a substantially narrower setting than practical OPD.
Questions for authors
  1. The statement of Theorem 1 appears to conflate clipped token-level KL with the original sequence-level KL defined in Eq. (1). Its conclusion is written as a bound on the original KL, but the concentration and change-of-measure arguments directly apply only to the clipped loss unless the original per-prefix KL is uniformly bounded. Appendix A.1 introduces an additional tail term for recovering the original KL, yet this term is absent from the theorem statement. Without controlling this tail, Theorem 1 provides only a clipped-risk guarantee and therefore cannot be directly combined with Theorem 3, whose Pass@k bound assumes a small original sequence-level KL. Could the authors revise the theorem statement and assumptions to clarify whether the result is for clipped risk or original KL, and explain what additional condition is required for the claimed connection to Theorem 3?
  2. The practical recommendations currently stated in the paper—evaluating at the intended sampling budget and inspecting teacher probabilities on student-correct responses—appear largely intuitive and follow directly from the definitions of Pass@k and distillation. Can the authors provide more insightful, non-trivial practical guidance derived from the theory, or more clearly discuss the theory’s actual practical significance and role beyond formalizing these observations?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you very much for the constructive feedback. We address each concern below.

W1: The practical usefulness of Theorem 1 is limited by strong assumptions that may hide large constants or be difficult to verify in real OPD. In particular, the description length K(π) for a learned LoRA or full-parameter update may be very large; finite χ2(Pπ∥Pπold)\chi^2(P_\pi\Vert P_{\pi_{\mathrm{old}}}) implicitly requires absolute continuity, while even finite density ratios can make the shift term very large; and the independent-trajectory assumption does not directly cover adaptive sampling, replay, or multi-round OPD.

Thank you for raising these concrete concerns. To support the practical usefulness of Theorem 1, we add an OPD experiment with Qwen3-0.6B-Base on AQuA using TinyLoRA (Section 5.2 and Appendix D in our revision). Following [1-2], the update has a description length K(π)K(\pi) of approximately 2,600. The estimate of χ2(Pπ∥Pπold)\chi^2(P_\pi\Vert P_{\pi_{\mathrm{old}}}) without truncation is 0.014, and the refined upper bound evaluates empirically to 0.049. The analysis covers a single OPD round using independent responses from a fixed sampling policy, as in the added experiment. We hope this concrete evaluation helps bridge the gap between our theory and actual LLM learning to some extent.

[1] Lotfi et al. Non-Vacuous Generalization Bounds for Large Language Models. ICML 2024.

[2] Zhu, Alur, and Kang. Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards. arXiv:2607.14506.

Q1: The statement of Theorem 1 appears to conflate clipped token-level KL with the original sequence-level KL defined in Eq. (1). ... Appendix A.1 introduces an additional tail term for recovering the original KL, yet this term is absent from the theorem statement. ... Could the authors revise the theorem statement and assumptions to clarify whether the result is for clipped risk or original KL, and explain what additional condition is required for the claimed connection to Theorem 3?

Thank you for identifying this mismatch. We have revised Definition 1, Theorem 1, and Appendix A.1 to control the original sequence KL. Adapting [3], we assume that student token probabilities are uniformly bounded relative to the teacher at every relevant prefix. The logarithm of this ratio bound then bounds the original conditional KL, so the concentration argument and KL chain rule apply directly. Appendix A.2 further tightens the bound by treating the majority of prefixes with small conditional KL separately from rare events with larger KL and weighting the larger KL bound by the probability of those events. Appendix C.8 complements this analysis with local KL measurements in our original 0.6B experiments: over 99.9% of positions in independent responses from checkpoints C40 and C80 have KL below 0.5, and the observed maxima in these response sets are 4.4 and 4.3, respectively.

Consequently, Theorem 1 now controls the same original KL assumed small in Theorem 3; Theorem 3 requires no change. In our numerical evaluation, Appendix A.2 tightens the bound by treating the majority of prefixes with small conditional KL separately from rare events with larger KL and weighting the larger KL bound by the probability of those events.

[3] Rohatgi et al. Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification. COLT 2025.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q2: The practical recommendations currently stated in the paper—evaluating at the intended sampling budget and inspecting teacher probabilities on student-correct responses—appear largely intuitive ... Can the authors provide more insightful, non-trivial practical guidance derived from the theory, or more clearly discuss the theory's actual practical significance and role beyond formalizing these observations?

Thank you very much for this suggestion. The theory gives a quantitative relationship between imitation precision and the intended sampling budget: doubling the budget requires halving the allowable sequence KL to retain the same nontrivial Pass@k guarantee in the worst case. Theorem 1 further identifies how the update description length, number of independent responses, and prefix shift control the sequence KL. Section 5.2 now measures these quantities in a real OPD update.

Theorem 5 also shows that a teacher can be more accurate on every prompt while distillation reduces Pass@k. This motivates inspecting teacher probabilities on correct responses to harder prompts, where losses matter more at larger budgets. Appendix C.7 adds those measurements, finding the predicted probability directions for all 13 original and 32 of 34 fresh correct responses on the selected problems.

For training design, Section 3.1 suggests collecting responses from the latest student and limiting changes in prefix probabilities before refreshing the data, since a smaller shift gives a tighter bound at the same training loss. A complementary initialization strategy is SFT on teacher responses before OPD: [4] find that this improves accuracy over direct OPD and increases the initial overlap between the teacher's and student's top-kk token sets. Their overlap compares next-token predictions at the same prefix; our prefix divergence compares the updated student with the policy that collected the training responses. These measurements therefore inform initialization and data refresh, respectively.

[4] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016.

Thank you for the detailed response and revisions. The revised assumptions resolve my concern about Theorem 1 and its connection to Theorem 3. The TinyLoRA experiment also partially addresses my concerns about the assumptions and constants.

However, the additional experiments and discussion remain insufficient to establish the theory’s practical usefulness or provide substantially more insightful guidance. While I appreciate the improvements, they do not sufficiently change my overall assessment, and I maintain my original score.

Thank you very much for your constructive feedback and continued positive assessment of our work. Your suggestions have helped us clarify the theoretical scope and strengthen the empirical evidence.

Reviewer_HgmKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

On-policy distillation (OPD) is an increasingly popular algorithm for post-training LLMs. Despite its empirical success, its theoretical properties are still relatively poorly understood. This paper makes a step into that direction; its contributions are:

  1. A generalization bound for one-step of OPD, along with a lower bound that shows that some part of it is tight.
  2. A construction showing that one step of OPD can increase pass@1 while decreasing pass@k, even if the teacher is strictly better than the student everywhere.
Critical strengths

The generalization bounds combine existing tools (compression-based generalization bounds and change of sampling distribution from offline RL) and apply them to OPD. The lower bound usefully shows that a distribution-shift term cannot be removed in general.

The sqrt(k eps) characterization of the student–teacher Pass@k gap is clean and provides some insights into why Pass@1 rising while Pass@k falls has been widely observed (both in RL and OPD). The more specific claim, that this can happen under a proximal step toward a teacher that is more accurate on every prompt, is interesting.

Critical weaknesses

The main concern I have with the paper is that the experiments do not convincingly show that the theory is relevant for real-world LLMs.

The generalization bound is probably very loose in practice:

  • K(π) is likely large for any realistic architecture, and G can also be large.
  • The chi squared prefix compounds multiplicatively with response length, so the shift term is effectively exponential in H unless updates are tiny.
  • The analysis covers a single round with a fixed sampling policy, rather than the iterative procedure used in practice. The only empirical verification related to that section is to show that the theoretical constructions behave as claimed. From my understanding of the of the learning theory field, vacuous bounds are pretty. However, actual learning experiments, for example a small student trained by OPD on a synthetic teacher, are needed to confirm the predicted trends when varying the variables of interest for the theory to be valuable to LLM post-training.

The effect guaranteed by the pass@k construction is extremely small. The empirical support seems pretty weak; as far as I understand experiments do not show that the same behavior than the one described in the construction occurs in practice, only that pass@k decreases while pass@1 improves (which is now a pretty well established result for on-policy fine-tuning algorithms).

Questions for authors

See the weaknesses, I would appreciate any effort to show that the insights from the theory extend to LLMs, or at least to a less toyish OPD setup.

6 comments

Thank you very much for the constructive feedback. We address each concern below.

W1: The generalization bound is probably very loose in practice: K(π) is likely large for any realistic architecture, and G can also be large.

Thank you for this important point. We have added an OPD experiment with Qwen3-0.6B-Base and a frozen GRPO teacher of the same size on AQuA, using TinyLoRA and KL over the complete vocabulary (Section 5.2 and Appendix D in our revision). Following [1-2], the update has a description length K(π)K(\pi) of approximately 2,600 nats relative to the fixed Base and decoder. With approximately 97,000 training responses, the empirical training risk is 0.013 and the estimate of χ2(Pπ∥Pπold)\chi^2(P_\pi\Vert P_{\pi_{\mathrm{old}}}) is 0.014. The largest observed conditional KL from the student to the teacher is 0.51; the estimated probability that the conditional KL exceeds 0.025 under the updated policy is 0.0045. The uniform token probability ratio bound in Definition 1 enters Theorem 1 through its logarithm, moderating the effect of large ratios. Appendix A.2 further weights the larger conditional KL bound by the probability of the corresponding event, allowing a smaller threshold in the concentration and transfer terms. Substituting these measurements into the refinement of Theorem 1 in Appendix A.2 gives an empirical value of 0.049 for the upper bound. The compact update make this LLM setting favorable for a potentially non-vacuous bound in OPD.

[1] Lotfi et al. Non-Vacuous Generalization Bounds for Large Language Models. ICML 2024.

[2] Zhu, Alur, and Kang. Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards. arXiv:2607.14506.

W2: The chi squared prefix compounds multiplicatively with response length, so the shift term is effectively exponential in H unless updates are tiny.

Thank you for this professional comment. We agree with your concern from a theoretical perspective. To help bridge the gap between theory and practice, we report prefix shift measurements in our LLM experiments (Appendix C.2 in our revision). These estimates range from 1.6 to 6.5 across three model sizes and three benchmarks for C120–C139. Our added Qwen3-0.6B OPD experiment on AQuA, with a maximum response length of 4 tokens, gives an estimate without truncation of 0.014 (Section 5.2 and Appendix D.2 in our revision). We sincerely hope these measurements help convey our effort to connect our theory with practical OPD.

W3 & Q: The analysis covers a single round with a fixed sampling policy, rather than the iterative procedure used in practice. ... actual learning experiments, for example a small student trained by OPD on a synthetic teacher, are needed to confirm the predicted trends when varying the variables of interest ...

I would appreciate any effort to show that the insights from the theory extend to LLMs, or at least to a less toyish OPD setup.

Thank you for encouraging us to connect the analysis more closely to LLM learning. We have added an OPD experiment with Qwen3-0.6B-Base and a frozen GRPO teacher on AQuA, using TinyLoRA and KL over the complete vocabulary (Section 5.2 and Appendix D in our revision). We measure the update description length, training KL, prefix divergence, and risk under the updated policy, obtaining an empirical value of 0.049 for the refined upper bound. We hope this concrete evaluation helps bridge the gap between our theory and actual LLM learning to some extent.

W4: The effect guaranteed by the pass@k construction is extremely small. ... experiments do not show that the same behavior than the one described in the construction occurs in practice, only that pass@k decreases while pass@1 improves ...

Thank you for emphasizing the mechanism. First, Table 2 in the revision (Table 1 in the original submission) and Appendix C.5 show the allocation of gains and losses underlying the construction. For Qwen3-4B on AMC23 from C60 to C80, 20 easy problems gain 71 correct responses, contributing 2.8 percentage points to Pass@1 but no gain to Pass@64. Two hard problems lose three correct responses, accounting for the full decline of 2.5 percentage points in Pass@64. Thus, the original evidence identifies how gains on easier problems and losses on harder ones affect different sampling budgets.

As suggested by the reviewer, to examine this pattern more directly, we have added measurements of teacher probabilities. In particular, we score fixed correct responses under the frozen teacher and both student checkpoints (Section 5.3 and Appendix C.7 in our revision). On the five selected hard problems across AMC23 and AIME25, all 13 original correct responses and 32 of 34 fresh correct responses have lower probability under both the teacher and the updated student than under the old student. This provides direct evidence of the suppression of particular correct trajectories described in the construction.

Thank you for the response and the new experiments. I still have concerns, could you please clarify:

  • On the numerical bound (Sec. 5.2).
    • Definition 1 requires a uniform bound on the token ratio π/πT\pi/\pi_T. Using the largest observed local KL (0.514) for log⁡W2\log W_2 underestimates it. Table 19 shows sampled ratios up to ~3×10⁷ (log = 17), and a value of that size would dominate the bound.
    • W1W_1 is chosen post hoc, and α\alpha and χ2\chi^2 are point estimates without confidence bounds. So 0.049 is a plug-in estimate, not a bound that holds with probability 1−δ1-\delta. With thresholds set in advance (Table 23), the value seems to be 35 to 70× the measured risk.
    • The setting uses H = 4, a tiny update (χ2\chi^2 = 0.014) and gives no significant test gain. What is the base model’s KL to the teacher before training? This seems to be exactly the regime where the bound should be tight. It also doesn’t vary nn, KK or the shift, which was my original request.
  • Prefix shift. The realistic estimates (1.6-6.5, ratios truncated at 10, H = 15,872) are never plugged into the bound. Could you evaluate it, even roughly, in that setting?
  • Pass@k.
    • In C.7, negative r(y)r(y) and Δ(y)\Delta(y) are expected for almost any long sample from πold\pi_{old}. Do the same signs hold for correct responses on easy problems, or for incorrect responses? Without that control, the result doesn’t seem to isolate the mechanism in Theorem 5.
AuthorsOpenReview ↗

Official Comment by Authors: Follow-up Response (1/3)

Thank you very much for your thoughtful and constructive follow-up suggestions, which have helped us improve the clarity and precision of our paper; we have submitted a revised PDF incorporating the corresponding clarifications and improvements, and provide our responses below.

Definition 1 requires a uniform bound on the token ratio π/πT\pi / \pi_T. Using the largest observed local KL (0.514) for log⁡W2\log W_2 underestimates it. Table 19 shows sampled ratios up to ∼3×107\sim 3 \times 10^7 (log = 17), and a value of that size would dominate the bound.

Thank you for this careful observation. Definition 1 was motivated by bounded likelihood-ratio assumptions in prior theoretical work (Rohatgi et al., 2025). Its role in our proof is to provide a uniform upper bound on conditional KL at every relevant prefix. Indeed, our proof also applies when this uniform KL bound is assumed directly. In this case, the resulting bound can be tighter if a smaller valid KL upper bound is available. Therefore, we will state this weaker condition (i.e., uniform boundedness of conditional KL rather than of token probability ratios) explicitly in Appendix A.2. This clarifies how the refinement can avoid the conservatism of controlling KL through the largest token ratio.

Under this formulation, a large ratio on a token with very small student probability need not imply a comparably large conditional KL. In the H=4 experiment in Section 5.2, the largest observed conditional KL is approximately 0.51. Using this observed maximum as an empirical proxy for the uniform KL bound, together with the two-part refinement in Appendix A.2, gives an empirical evaluation of the bound of 0.049, approximately four times the measured risk. Thus, the numerical evaluation reflects the conditional KL and its tail contribution, rather than being determined directly by the largest token ratio.

W1W_1 is chosen post hoc, and α\alpha and χ2\chi^2 are point estimates without confidence bounds. So 0.049 is a plug-in estimate, not a bound that holds with probability 1−δ1-\delta. With thresholds set in advance (Table 23), the value seems to be 35 to 70× the measured risk.

Thank you for pointing this out. Appendix A.2 refines Theorem 1 to facilitate numerical evaluation and examine when the guarantee could be informative in practical OPD settings. Obtaining a confidence guarantee would involve fixing the threshold in advance, establishing valid bounds on conditional KL and prefix density ratios, and constructing upper confidence bounds for α\alpha and χ2\chi^2 from independent trajectories, with an appropriate allocation of the total failure probability. Our current numerical evaluation uses point estimates and an observed maximum KL, so 0.049 remains an empirical evaluation. As suggested by the reviewer, we clarify this status in Appendix D.2 in our revision and outline these additional steps for future work toward non-vacuous bounds with confidence guarantees.

Regarding Table 23, as explained above, a valid uniform conditional KL bound can be less conservative than a bound derived from the largest token ratio. Within this formulation, we apply the two-part refinement in Appendix A.2 using the same empirical measurements, a threshold of 0.025, and assumed uniform conditional KL upper bounds of 1 or 2. This gives empirical values of approximately 0.058 or 0.075, respectively—about 4.7 or 6.2 times the measured risk. This sensitivity comparison illustrates the numerical benefit of the refinement.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.