acceptodds
Under review as a conference paper at ICLR 2027

Reverse-KL Policy Distillation: Statistical Guarantees and Limits

Abstract

everse Kullback–Leibler (KL) distillation has shown empirical promise for train- ing language models, but its statistical guarantees and limits remain poorly under- stood. In particular, the effects of two design choices remain unclear: Does access to the teacher’s full action distribution provide an advantage over observing only the probability it assigns to the sampled action? Must training trajectories come from the student (online distillation), or can teacher-generated trajectories (offline distillation) suffice? Our work takes a first step toward understanding the theory underlying these questions. We establish matching upper and lower bounds for online reverse-KL distillation when the teacher reveals its full action distribution, and show that substantially more data may be required when the teacher reveals only the probability of the student-generated action. In contrast, offline reverse-KL distillation can attain the same optimal estimation rate whether the teacher reveals its full distribution or only its probability of the teacher-generated action. The distinction is that teacher- generated actions themselves reveal the teacher’s preferences, while queries at student-selected actions can leave those preferences unresolved. This highlights an additional statistical challenge for online distillation compared to offline distillation. These statistical guarantees do not follow simply from optimizing standard dis- tillation objectives: our lower bounds show that common online objectives can fail when candidates are evaluated only on past student-generated trajectories. We develop trajectory-loss and importance-ratio truncation techniques together with policy-space learning to achieve optimal online distillation guarantees.

Primary area:
learning theory
Keywords:
Learning Theory, Imitation Learning, Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Kt1QOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the statistical theory of reverse-KL distillation. It compares four settings: training on student-generated or teacher-generated data, with the teacher revealing either its full output distribution or only the probability of the sampled output. With full teacher distributions, the authors give matching upper and lower bounds in both online and offline settings.

Critical strengths

The paper sets up a clean framework with four settings and gives matching upper and lower bounds for three of them. The main messages are easy to follow.

Critical weaknesses

None

Questions for authors

Can the author provide some empirical validation?

2 comments

We thank the reviewer for the constructive feedback. You questions are elaborated below.

Question:

Can the author provide some empirical validation?

Answer.

We have added an offline full-feedback experiment using an MLP student with partial observations. In this experiment, trajectory clipping improves trajectory reverse KL over the unclipped and forward-KL baselines. Please see Section 4.3 and Appendix C.3 of the revised paper for the setup and results.

Thank you for the detail rebuttal. I would keep my positive score.

Reviewer_v2kXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the problem of reverse KL learning in imitation learning. In reverse KL learning, the student is penalized for assigning probability to actions that are unlikely under the teacher's policy, as opposed to forward KL learning, where the student is penalized for not assigning probability to actions that are very likely under the teacher's policy. The authors handle online (teachers' probabilities are revealed in an online manner) and offline configurations (teachers' probability assignments are available offline) with full feedback (the teacher's probability assignments for every action are available) and action feedback (feedback of only the action chosen by the teacher is available).

They handle the specific problem of visitation mismatch, which is, for example, a high-loss action under a state that is very rarely visited on the teacher's policy might still make the teacher's policy look worse. They handle this by clipping the probability ratio of the teacher's and student's policies.

Furthermore, they also provide a lower bound only if action feedback is available. Thus, they show that high sample complexity is required in this case.

Critical strengths

The problem statement is clearly stated, and development toward the solution is organic.

I have not checked the proof line by line, but the proofs seem correct.

Critical weaknesses

I think the related work and comparisons aren't discussed clearly. Empirical evidence is not provided. Although this paper is theoretical, I would like to see some empirical results. The contribution seems incremental, since the authors only provide clipping, which, to my knowledge, is very common in these areas.

Questions for authors

This question might be very naive. You mentioned that your algorithm tries to reduce the influence of prefixes that the evaluated policy would rarely reach. However, in some real-world applications, like vehicle driving, aren't these kinds of rare prefixes very important for safety?

What is the intuition behind the result where, in the offline setting, both full feedback and Action feedback are able to achieve the same complexity?

I might reconsider my score based on the answers by the authors.

2 comments

We thank the reviewer for the constructive feedback. You questions are elaborated below.

Question 1:

This question might be very naive. You mentioned that your algorithm tries to reduce the influence of prefixes that the evaluated policy would rarely reach. However, in some real-world applications, like vehicle driving, aren't these kinds of rare prefixes very important for safety?

Answer:

Our correction follows the importance-weighting principle used in off-policy evaluation: losses observed under the data-collection policy should be weighted according to how often the evaluated candidate would encounter the same prefixes. If the candidate rarely reaches a dangerous prefix that the data-collection policy frequently visits, evaluating the candidate without this correction would overrepresent that prefix in its expected loss. Reducing the influence means having smaller importance weight. The weight accounts for this difference in visitation frequency; it does not imply that the prefix is unimportant for safety.

Question 2:

What is the intuition behind the result where, in the offline setting, both full feedback and Action feedback are able to achieve the same complexity?

Answer.

In the offline setting, actions are sampled from the teacher, so the observed actions themselves carry information about teacher's distribution. Action feedback alone suffices to match the full-feedback reverse-KL guarantee under Assumption 2.2. Full feedback additionally admits guarantees under the weaker Assumption 2.1 (first row in Table 1), which is insufficient for action feedback, as shown by Proposition C.1.

Question 3:

The contribution seems incremental, since the authors only provide clipping, which, to my knowledge, is very common in these areas.

Answer:

Although clipping is widely used empirically, its statistical benefits for reverse-KL policy distillation remain poorly understood.

Our clipping constructions differ from existing approaches, and we develop new analytical techniques to study the resulting losses. Our new online loss compares the increments of the evaluated policy’s and rollout policy’s clipped cumulative KL losses, using clipped prefix likelihood ratios as weights. These two clipping operations control the loss. To analyze this loss and achieve the optimal guarantee, we develop new techniques including a stopping-time construction and change-of-measure bounds that control the candidate’s clipped risk through its prefix-weighted loss and the rollout policy’s clipped risk (Lemma B.1).

In the offline setting, Proposition 4.1 shows that the naive unclipped loss is suboptimal, while our clipped objective achieves the optimal guarantee (Theorem 4.2). To our knowledge, this objective is also new. We also use the new technique in Lemma B.1 to transfer guarantees for empirical minimization on teacher-generated data to guarantees on the candidate’s own risk.

Question 4:

I think the related work and comparisons aren't discussed clearly.

Answer:

We will try to improve the clarity of the related work section, and it would be great if the reviewer could point out more specifically where the comparison feels unclear. We elaborated the most important related work below, which is also stated in the end of page 2.

To our knowledge, our work provides the first systematic study of statistical guarantees for sequence-level reverse-KL policy distillation. The closest related studies are Rohatgi et al. (2025), which analyzes sequence-level Hellinger distance, and Xu, Yuan, and Zhan (2026), which analyzes sequence-level forward KL. Both focus on offline learning with action feedback. We study reverse KL, a widely used distillation objective, across both online and offline settings with either full or action feedback. We introduce new algorithms and techniques to handle these settings. Appendix A provides a detailed discussion of these connections and distinctions.

Question 5:

Empirical evidence is not provided. Although this paper is theoretical, I would like to see some empirical results.

Answer.

We have added an offline full-feedback experiment using an MLP student with partial observations. In this experiment, trajectory clipping improves trajectory reverse KL over the unclipped and forward-KL baselines. Please see Section 4.3 and Appendix C.3 of the revised paper for the setup and results.

Thank you for the clarifications and revisions to the paper. I maintain my positive score.

Reviewer_nHzYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies reverse-KL distillation when the student class may not contain the teacher, separating the source of training trajectories from the feedback available. It establishes optimal guarantees for full-feedback learning, shows that offline sampled-action feedback can match these guarantees under a likelihood-ratio bound, and proves a substantial disadvantage for online sampled-action feedback. The main arguments look sound; my reservations are mostly about the usefulness and scope of the guarantees.

Critical strengths
  • The feedback separation is interesting. Teacher-generated actions provide information that querying the teacher on student-generated actions may miss. The lower bound makes this distinction precise, even under realizability and at horizon one.
  • The clipping argument is a useful technical contribution. Combining clipped trajectory losses with truncated prefix ratios handles visitation mismatch without requiring ordinary importance weights to have controlled variance. The matching lower bounds strengthen the result.
  • The connection to existing theory is clear - the paper complements Rohatgi et al.’s Hellinger analysis and Xu et al.’s forward-KL guarantees. It also appropriately credits the existing Hellinger estimator underlying its offline action-feedback result.
Critical weaknesses
  • The misspecified guarantee can become vacuous fairly quickly. Under Assumption 2.2, every candidate already satisfies L(π)≤B∞L(\pi)\le B_\infty. Consequently, an approximation term O(B∞L⋆)O(B_\infty L^\star) stops improving on this trivial bound when L⋆L^\star is order one or larger. For long sequences and genuinely limited students, small total sequence-level KL is a demanding regime. The paper should explain more clearly when its guarantee is informative.
  • There is a substantial computational gap. Algorithm 1 maintains weights over the entire policy class and evaluates every candidate after each episode. This establishes statistical possibility, but provides little guidance for training a neural student. Experiments are not necessary for a theory paper, though an efficient special case or a concrete implementation of the proposed correction would make the practical implications much stronger.
  • The failure result has limited implications for practical distillation. Proposition 3.1 concerns exact refitting over a finite class, with a specified initialization and tie-retention rule. It does not establish failure of gradient updates or sequence-level policy gradients. MiniLLM [1], for example, already propagates future log-ratio rewards to earlier actions. The authors acknowledge this distinction, but the result should primarily be read as a limitation of the particular refitting surrogate.

Minor points

  • “We give a complete picture” is too strong: online action feedback receives a lower bound, without a matching upper bound characterizing its optimal rate.
  • The abstract’s offline full/action-feedback equivalence should explicitly mention Assumption 2.2. Under Assumption 2.1 alone, Proposition C.1 shows that this equivalence fails.

[1] Gu, Y., Dong, L., Wei, F. and Huang, M., 2024, May. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations (Vol. 2024, pp. 32694-32717).

Questions for authors
  • Can the authors give a simple regime where BB or B∞B_\infty and L⋆L^\star make the misspecified bound meaningfully nonvacuous?
  • How much can the approximation coefficient improve if one accepts a worse estimation rate? The current optimality result fixes that rate, rather than characterizing the full tradeoff.
2 comments

We thank the reviewer for the constructive feedback. You questions are elaborated below.

Question 1:

Can the authors give a simple regime where BB or B∞B_\infty and L⋆L^\star make the misspecified bound meaningfully nonvacuous?

Answer:

A concrete example is the two-policy, two-step instance in Proposition 3.1. Neither candidate exactly matches the teacher, but the better candidate has small trajectory-level reverse KL because it rarely visits the state with large local error. In this construction, B,B∞=Θ(λ)B,B_\infty=\Theta(\lambda) and 0<L⋆=O(λe−λ)0 < L^\star=O(\lambda e^{-\lambda}). With T=Θ(λ3)T=\Theta(\lambda^3) episodes and a fixed confidence level, Theorem 3.3 gives L(π^)=O!(λ2e−λ+1λ2),L(\widehat{\pi})=O\\!\left(\lambda^2e^{-\lambda}+\frac{1}{\lambda^2}\right), which tends to zero as λ\lambda grows. Thus, the guarantee yields a substantial improvement over the trivial O(λ)O(\lambda) bound.

Question 2:

How much can the approximation coefficient improve if one accepts a worse estimation rate? The current optimality result fixes that rate, rather than characterizing the full tradeoff.

Answer:

A constant approximation factor is achievable at the cost of a larger estimation term. In the online full-feedback setting under Assumption 2.1, a simple policy-evaluation procedure achieves approximation factor 22.

Let T≥2∣Π∣T \ge 2|\Pi|. Allocate n=⌊T/∣Π∣⌋n=\lfloor T/|\Pi|\rfloor independent rollouts to each candidate. For a trajectory generated by π\pi, compute Xπ(τ)=∑h=1HKL!(πh(⋅∣sh),∣,πh⋆(⋅∣sh)).X_\pi(\tau) =\sum_{h=1}^{H} \mathrm{KL}\\!\left( \pi_h(\cdot\mid s_h)\\,\\|\\,\pi_h^\star(\cdot\mid s_h) \right). Full feedback makes this loss observable. Estimate L(π)L(\pi) by its sample average L^(π)\widehat L(\pi), and return π^∈arg⁡min⁡π∈ΠL^(π)\widehat\pi\in\arg\min_{\pi\in\Pi}\widehat L(\pi).

Assumption 2.1 and the KL chain rule give 0≤Xπ≤B,E[Xπ]=L(π),Var⁡(Xπ)≤BL(π).0\le X_\pi\le B, \mathbb E[X_\pi]=L(\pi), \operatorname{Var}(X_\pi)\le BL(\pi). Bernstein’s inequality and a union bound imply that, with probability at least 1−δ1-\delta, simultaneously for all π∈Π\pi \in \Pi,

∣L^(π)−L(π)∣≤13L(π)+O!(Bnlog⁡∣Π∣δ).|\widehat L(\pi)-L(\pi)| \le \frac13L(\pi) +O\\!\left(\frac{B}{n}\log\frac{|\Pi|}{\delta}\right).

Comparing the empirical minimizer with the best candidate therefore yields

L(π^)≤2L⋆+O!(B∣Π∣Tlog⁡∣Π∣δ).L(\widehat\pi) \le 2L^\star +O\\!\left(\frac{B|\Pi|}{T}\log\frac{|\Pi|}{\delta}\right).

Thus, the approximation coefficient can be reduced to a constant but the estimation error scale linearly with ∣Π∣|\Pi|, which is usually vacuous.

This larger estimation cost can be unavoidable. Keeping ∣Π∣|\Pi| as a free parameter in Appendix B.4.2 gives the following lower bound. For ∣Π∣≥8|\Pi| \ge 8, T≥∣Π∣T\ge |\Pi|, and B≥32log⁡∣Π∣B\ge32\log |\Pi|, any online learner returning a candidate π^∈Π\widehat{\pi} \in \Pi has an instance on which, with probability at least 3/43/4,

L(π^)−2L⋆≥B∣Π∣128T.L(\widehat\pi)-2L^\star \ge \frac{B|\Pi|}{128T}.

Question 3:

The failure result has limited implications for practical distillation. Proposition 3.1 concerns exact refitting over a finite class, with a specified initialization and tie-retention rule. It does not establish failure of gradient updates or sequence-level policy gradients. MiniLLM [1], for example, already propagates future log-ratio rewards to earlier actions. The authors acknowledge this distinction, but the result should primarily be read as a limitation of the particular refitting surrogate.

Answer:

We have added Proposition 3.2 and its proof in Appendix B.1.1. For a smooth one-parameter policy family, projected gradient updates with sufficiently small step sizes can remain trapped at high reverse-KL risk with constant probability under uniform random initialization. This holds for both local KL gradients and full trajectory gradients with future-return terms. The result therefore addresses gradient updates directly.

Thank you for providing the clarifications and updating the paper. The new additions strengthen the paper, and I maintain my recommendation leaning towards acceptance.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We have revised the manuscript to strengthen our results and address their concerns. The main changes are:

  • Weaker boundedness assumption. We replace the original pathwise local KL bound in Assumption 2.1 with a relative second-moment condition while retaining the full-feedback guarantees. This weaker condition is implied by both the original Assumption 2.1 and Assumption 2.2. Our algorithms and analysis extend naturally to this condition, broadening the scope of the guarantees beyond uniformly bounded cumulative local KL losses.

  • Failure of gradient updates. We add Proposition 3.2 and Appendix B.1.1 to extend our failure result beyond exact minimization of the training objective. For a smooth one-parameter policy family, gradient updates can remain trapped at high risk with constant probability under random initialization. This holds for both local KL gradients and full trajectory gradients with future-return terms.

  • Offline neural-distillation experiment. We add an offline full-feedback experiment in a corridor MDP, with a optimal teacher and a student parameteried by neural networks receiving partial observations. The partial observations prevent the student from representing the teacher, allowing us to examine clipping under misspecification. Across 20 independent seeds, trajectory clipping reduces the original trajectory reverse KL by approximately 75% compared with both the unclipped reverse-KL and forward-KL baselines. Section 4.3 presents the experiment and results, and Appendix C.3 provides the implementation details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.