acceptodds
Under review as a conference paper at ICLR 2027

What Does On-Policy Distillation Optimize?

Abstract

On-policy distillation uses teacher feedback on student-generated prefixes, so an update changes both the student and the distribution of its later supervision. Yet practical methods often replace sequence gradients with token-local, bounded, support-restricted, or future-truncated updates. This raises a basic question: *what objective do these updates represent?* We answer this question at three levels: First, every continuous coordinate-local reward admits an explicit separable potential; we characterize its proper and Bregman subclasses. Next, universal autoregressive composition within the separable Bregman class forces the logarithmic generator, exposing a tradeoff between bounded rewards and sequence consistency. Finally, under student occupancy, an exact future-credit decomposition yields an intrinsic objective-fidelity coefficient, a finite-step descent guarantee, and circulation tests for integrability. To validate these results, exact finite trees verify these identities and separate composition from update fidelity. Complementing these exact checks, Qwen3.5 measurements on GSM8K, ARC-Challenge, and HellaSwag, using 0.8B-to-4B and 4B-to-9B model pairs, reveal horizon-dependent transitions from descent to near orthogonality or ascent. Matched training further shows that cached-prefix teacher fit and task utility can rank the same updates differently. Together, these results identify the objective each surrogate represents, the sequence terms it omits, and when its local update preserves progress.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Knowledge Distillation, Objective Fidelity, Bregman Divergences, Credit Assignmen

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Wx3xOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates fixed-prefix propriety, autoregressive composition, and the fidelity of practical OPD updates to an occupancy-weighted objective. It maps coordinate-local rewards to potentials, characterizes the logarithmic composition boundary within separable Bregman divergences, and derives omitted future credit, descent criteria, and circulation tests. Finite trees, projected Qwen3.5 measurements, and matched training examine these distinctions.

Critical strengths
  1. The paper makes a consequential distinction between a proper conditional objective, composition into the same path-space divergence, and descent under student occupancy. The logarithmic and power constructions demonstrate both nonimplications. Its value extends beyond restating nonconservative distillation updates: it connects bounded rewards and support restriction to a declared objective while acknowledging earlier results.
  2. The validation separates exact identities from neural observations and includes unfavorable outcomes. In particular, corrected log improves ARC accuracy relative to local log but hurts HellaSwag accuracy. Paired schedules, seed-level intervals, and explicit limits on parameter projections make this a useful investigation rather than an unconditional recommendation to restore future credit.
Critical weaknesses
  1. The neural fidelity statistic appears to differ from the population quantity that certifies descent. Equation (16) uses expected vectors, whereas Appendix H.2 defines Equation (84) for each prompt trajectory and H.4 then averages within seed. Averaging these ratios need not preserve the sign of the ratio formed from mean vectors. Likewise, averaging per-trajectory wedge norms cannot certify a nonzero expected wedge. Thus the reported negative intervals alone do not establish population ascent or nonintegrability, even within the measured subspace. This affects the central interpretation of Figure 2, beyond the projection limitation already acknowledged.
  2. The sampling measure is insufficiently specified for the claimed on-policy geometry. Sampling uses temperature 0.8 and top-p 0.95, but the score and divergences are written using qθ. Without identifying these distributions, the score-function correction need not target the stated objective. Furthermore, H.5 reuses initialization prompt trajectories at trained endpoints, while Observation 7 attributes changes to current occupancy. Reusing old response tokens requires an explicit distribution correction; holding samples fixed within a central difference is a separate, valid variance-reduction step.
Questions for authors
  1. Could the authors provide aggregation pseudocode and results that average the direct field, future-credit vector, and signed wedge matrix before computing fidelity or norms, with trajectory-count convergence and seed-level intervals? Do the reported ascent cases persist?
  2. Could the authors clarify whether qθ in rollout sampling, log probabilities, divergences, and directional scores includes temperature and nucleus truncation, and explain any correction needed to make these estimators target Equations (6) and (14)?
  3. Could the authors clarify whether endpoint responses are regenerated at each final adapter or reused from initialization, and provide the occupancy correction if reused?
3 comments

Thank you for the precise questions on population fidelity, the rollout distribution, and endpoint evaluation. We address them below.


Q1 (Q1 & Weakness 1): Could the authors provide aggregation pseudocode and results that average the direct field, future-credit vector, and signed wedge matrix before computing fidelity or norms, with trajectory-count convergence and seed-level intervals? Do the reported ascent cases persist?

We recomputed fidelity by aggregating the projected vectors before forming ratios. We report three estimates: the manuscript's per-trajectory ratio, the ratio formed from within-seed mean vectors, and a cross-fitted ratio of seed-averaged inner products over pairs of distinct trajectories. Cross-fitting removes the within-trajectory product bias from both the numerator and the denominator. All intervals use the seed as the bootstrap unit.

Table 3 (p. 24) reports the three estimates. After vector aggregation, the ARC and HellaSwag estimates remain negative. The cross-fitted intervals exclude zero for ARC at horizon 64 and at every reported HellaSwag horizon, while GSM8K remains at the zero boundary. Table 4 (p. 24) subsamples 2, 4, and 8 trajectories per seed and shows the same pattern. On HellaSwag, the mean-vector interval excludes zero at every count.

For circulation, we first aggregate the signed wedge matrices within each seed and its fixed direction basis, and only then compute norms. Cross-fitted trajectory pairs estimate the squared norm of the expected wedge. Its intervals exclude zero on HellaSwag with eight directions at horizons 8 and 16.

Manuscript revision: App. H.7 (pp. 23–24) now gives the step-by-step aggregation procedure, Table 3, and Table 4.


Q2 (Q2 & Weakness 2): Could the authors clarify whether qθ in rollout sampling, log probabilities, divergences, and directional scores includes temperature and nucleus truncation, and explain any correction needed to make these estimators target Equations (6) and (14)?

Rollouts follow the generation law q̃_θ, with temperature 0.8, top-k 50, and top-p 0.95. The conditional divergences use the untempered student distribution q_θ.

Under this sampling law, the occupancy-weighted objective is

L_B^q̃(θ) = E_{Y∼q̃_θ} Σ_t B(q_θ(·|H_t), p(·|H_t)).

The direct field is obtained by differentiating the conditional divergence at fixed prefixes (Eq. 79). The corresponding future-credit term uses the rollout score s̃_t = ∇log q̃_θ(Y_t|H_t). Where the retained token set N_t is fixed, this score is

s̃_t = (1/τ)(∇z_{Y_t} − Σ_{k∈N_t} q̃_θ(k|H_t)∇z_k).

This gives the matched-law decomposition for the generation distribution.

Manuscript revision: App. H.8 now states the rollout law, the conditional divergence, the objective, and the matched score (Eqs. 88–89, p. 24). App. I lists the full sampling settings (p. 25).


Q3 (Q3 & Weakness 2): Could the authors clarify whether endpoint responses are regenerated at each final adapter or reused from initialization, and provide the occupancy correction if reused?

The endpoint diagnostic in App. H.5 evaluates every trained adapter on the common initialization trajectories. Holding the reference responses and parameter directions fixed allows the local fields to be compared directly across trained states.

Current-occupancy objectives are evaluated separately, on fresh responses sampled from each final adapter (App. H.6).

For current-occupancy estimates from reused responses, the required sequence likelihood ratio is q̃_{θM}(y|x)/q̃_{θ0}(y|x), provided that the reference law covers the target law. Fresh rollouts sample from the trained policy directly.

Manuscript revision: App. H.5 now distinguishes the fixed-reference endpoint diagnostic from current-occupancy evaluation and states the likelihood ratio (p. 23). Observation 7 (p. 9) and App. K.19 (p. 58) describe the endpoint change on the common reference trajectories, and App. K.19 points to the fresh-sequence evaluation for current occupancy.

Thanks for the detailed rebuttal and the new experiments. The additional analysis clears up my concern regarding averaging per-trajectory fidelity ratios, and your clarification on the endpoint evaluation makes sense—using fixed trajectories for direct comparison while evaluating the objective on fresh rollouts for each final model is much clearer now.

Just one quick question remaining: were the fidelity estimates in Tables 3 and 4 computed using the score of the actual rollout distribution (including temperature and truncation), as defined in Appendix H.8?

Since the theoretical side of on-policy distillation isn't my main area of expertise, I'd suggest the AC weigh the assessments of the higher-confidence reviewers more heavily on this aspect.

Thank you for your constructive follow-up. We appreciate that the vector-aggregation results and endpoint clarification have addressed your first and third concerns.

To answer your remaining question directly, Tables 3 and 4 use the untempered student score defined in Eq. 78. The underlying directional measurements are held fixed across all three aggregation methods, allowing us to isolate the effect of aggregating vectors before computing fidelity ratios.

For the actual generation law, Appendix H.8 derives the corresponding future-credit score, ∇log q̃_θ (Eq. 89), accounting for temperature and truncation. The matched-law estimator retains the same conditional divergence but replaces the future-credit score with that of the actual rollout distribution.

Thus, Tables 3 and 4 establish the results of the vector-aggregation comparison, while Appendix H.8 specifies the estimator for the generation-law objective. Theorem 4.5 and Propositions 4.8–4.9 establish the theoretical separation and descent criteria independently of these empirical aggregation choices.

We appreciate your careful consideration.

Reviewer_xKu6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper seeks to provide timely understanding about the objective and implementation of on-policy distillation (OPD) algorithms. To my understanding, the authors asks three questions:

1) Is the per-token objective a well-defined objective minimized when student = teacher?

Yes for both KL and more general Bregman divergences.

2) Do the per-token objective compose into the correct sequence objective? (i.e. sequence between student and teacher)

Only for KL

3) Does the update ascend on the sequence objective?

Not necessarily, due to ignoring the effects of earlier tokens on future losses.

Critical strengths

The paper attempts to address an important research question about the correctness and understanding the objective of a popular class of algorithms (on policy distillation).

Critical weaknesses

I enjoyed the premise of the paper and what the authors set out to answer. However, upon attempting to read it in detail, I found the paper to be effectively incomprehensible. Non-standard terms and definitions are created without exposition (for e.g. “coordinate local”, “logarithmic boundary”, and L93: “We derive objective fidelity, finite-step descent, future-neutrality, support omission, circulation, and approximate-integrability bounds”), thus the interesting questions the author set out to answer is hidden behind walls of poorly written, dense technical language with insufficient intuition or exposition in my opinion.

Additionally, to my understanding, the main result of the paper is largely the same as previous works [1, 2] without proper citations. For instance [1, 2] already discussed and addressed how token-level KL updates can fail to estimate the gradient of sequence-level KL due to not accounting for dependencies on earlier tokens.

All in all, it is currently impossible for me to tell what it is about or follow the paper’s argumentation. As such, I unfortunately cannot provide a detailed review and recommend rejection based on poor intelligibility.


[1] Tang, Yunhao, and Rémi Munos. "On a few pitfalls in kl divergence gradient estimation for rl." arXiv preprint arXiv:2506.09477 (2025).

[2] Shah, Vedant, et al. "A comedy of estimators: On kl regularization in rl training of llms." arXiv preprint arXiv:2512.21852 (2025).

Questions for authors

How do your results in Section 4.3 differ from the analysis of token-level versus sequence-level KL gradients in [1] and [2]?

[1] Tang, Yunhao, and Rémi Munos. "On a few pitfalls in kl divergence gradient estimation for rl." arXiv preprint arXiv:2506.09477 (2025).

[2] Shah, Vedant, et al. "A comedy of estimators: On kl regularization in rl training of llms." arXiv preprint arXiv:2512.21852 (2025).

3 comments

Thank you for recognizing the importance of the research question. We address your concerns below.


Weakness 1: Non-standard terms and definitions are created without exposition (for e.g. "coordinate local", "logarithmic boundary", and L93: "We derive objective fidelity, finite-step descent, future-neutrality, support omission, circulation, and approximate-integrability bounds").

The central idea is simple. A token-level update can represent a valid objective at each prefix, yet fail to decrease the corresponding objective over complete responses.

The paper separates three questions.

  1. What objective does a token reward represent? Every continuous coordinate-local reward has an explicit conditional potential (Theorem 4.1). "Coordinate-local" means that the reward depends only on the teacher and student probabilities of the sampled token, for example log p − log q, or p − q.
  2. Does this objective compose over complete responses? Among separable Bregman divergences, only the logarithmic one adds up to the same divergence over complete responses (Proposition 4.3). A bounded reward can therefore stay strictly proper at each prefix without inheriting this chain rule (Corollary 4.4).
  3. Does the practical update decrease its objective? When the student generates the prefixes, the token-local update omits a future-credit term (Theorem 4.6). The objective-fidelity coefficient Φ_B decides whether the update still descends. A positive value means descent, zero means no first-order progress, and a negative value means ascent (Proposition 4.8). Proposition 4.9 gives the matching finite-step guarantee.

Manuscript revision: Sec. 1 now defines "coordinate-local" at its first use (p. 1) and states the contributions through these three questions (p. 2). Sec. 3 maps the three questions to Secs. 4.1–4.4 (p. 3).


Q1 (Q1 & Weakness 2): How do your results in Section 4.3 differ from the analysis of token-level versus sequence-level KL gradients in [1] and [2]?

[1] and [2] analyze KL-gradient estimation in RL, including the effect of sequential dependencies. Our work adds three results.

  • Objectives beyond KL. Theorem 4.1 identifies the conditional potential of general coordinate-local rewards, including rewards outside the Bregman class. Secs. 4.1–4.2 then characterize autoregressive composition within the separable Bregman class.
  • Descent, not only gradient bias. A biased update can still decrease its objective. Theorem 4.5 gives constructions that separate composition from fidelity in both directions. Propositions 4.8–4.9 give an exact descent criterion and a finite-step guarantee, and Fig. 2 shows how this criterion changes with horizon in Transformers.
  • Integrability, not bias against one chosen objective. Theorem 4.10 gives an explicit test for whether the local update is the gradient of any scalar objective. It uses an explicit score–gradient wedge and a circulation test.

Together, these results identify the objective that an OPD surrogate represents, its sequence-level properties, and the optimization behavior of its implemented update.

Manuscript revision: Sec. 2 now cites [1, 2] and states this difference (p. 3).

I thank the authors for the response. I am reassured that my understanding of the paper is correct, as my initial summary seems to match the response to W1 above.

That said, I remain unconvinced that my concerns have been addressed. Regarding novelty w.r.t. Tang et al.: the generalization to "descent, not only bias" and "integrability" is a minor result in my opinion. It is unclear to me what the implication should be beyond those discussed in previous work (even if the authors manages to characterize the symptom more precisely). This points to my biggest concern in the initial review: comprehensibility. The draft reads like a list of facts. Some of the facts may be slightly novel, but their context and implications are missing. This, to me, is not an effective piece of scientific communication that would benefit the community. Thus---with the willingness to be convinced by the other two reviewers / AC---I maintain my strong reject.

Thank you for the follow-up. We disagree that our results are minor extensions of prior KL-gradient analyses, or that they only describe a known symptom more precisely. The paper establishes results that do not follow from the cited work and that have direct implications for OPD.

First, gradient bias does not determine descent. Theorem 4.5 proves this with two explicit constructions. Logarithmic KL composes exactly over complete responses, yet its token-local update ascends the sequence objective. Conversely, a power divergence does not compose into the same path-space divergence, yet its local update exactly descends its occupancy-weighted objective. Composition and update fidelity are therefore independent. Identifying omitted gradient terms, as in [1, 2], establishes neither direction of this separation.

Second, we give criteria that decide optimization behavior. Proposition 4.8 decides whether the local field descends its objective, and Proposition 4.9 gives a finite-step guarantee. Theorem 4.10 tests whether the field is the gradient of any scalar objective, rather than whether it matches one particular KL gradient. These are different mathematical questions, not refinements of a bias calculation.

Third, the implications are concrete. Corollary 4.4 establishes the tradeoff between bounded, strictly proper transforms and universal sequence composition within the separable Bregman class. Eq. 22 identifies exactly what support restriction omits. Fig. 2 shows horizon-dependent changes in the measured fidelity, while Observations 5–6 show that future correction changes task outcomes and update scale. These results explain why local teacher fit, sequence composition, and optimization progress cannot be treated as interchangeable criteria.

The revised Introduction and Sections 2–3 connect these results through three explicit questions.

We therefore ask that the assessment reflect these results and their demonstrated consequences.

Reviewer_GCD7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what objective on-policy distillation updates actually optimize, distinguishing three often-conflated questions. It shows that token-local rewards always admit separable potentials, with the proper and Bregman cases fully characterized, but that sequence-level composition forces the logarithmic form, so no bounded transform is both strictly proper and sequence-consistent. The implemented update decomposes exactly into the negative gradient of the intended objective plus an omitted future-credit term; the sign of a resulting scale-free fidelity coefficient then determines whether the update descends that objective, and circulation of the update field provides a test for non-integrability.

Critical strengths
  • The three-level separation is a genuinely useful clarification for the OPD literature

  • The future-credit decomposition and the fidelity coefficient are exact, checkable tools rather than heuristics

Critical weaknesses
  • The headline composition result is, by the authors' own account, a specialization of classical theory rather than a new boundary

  • The Transformer evidence is far removed from the regimes where OPD is deployed. Fidelity is measured only within rank-eight LoRA parameter subspaces spanned by four (at most eight) parameter directions, over supervised horizons of at most 64 tokens, and training uses plain SGD for 128 updates.

Questions for authors
  • Which single result in this paper is not already implied by Czarnecki et al. (2019) together with the classical proper-scoring characterizations?

  • Can fidelity be estimated, even noisily, in the full parameter space of the 0.8B student (or a smaller fully fine-tuned model), rather than in a rank-eight LoRA subspace?

  • All horizons are at most 64 tokens, while deployed OPD typically supervises reasoning traces of hundreds to thousands of tokens. Does the fidelity transition sharpen, saturate, or change character at horizons of 512+ tokens, and is there any measurement or theoretical reason to expect the same sign pattern?

1 comment

Thank you for recognizing the value of the three-level separation and the exact fidelity analysis. We answer the novelty question first, then the experimental questions.


Q1 (Q1 & Weakness 1): Which single result in this paper is not already implied by Czarnecki et al. (2019) together with the classical proper-scoring characterizations?

Theorem 4.5. It proves that autoregressive composition and update fidelity are independent properties, and it gives explicit constructions for both directions:

  • Logarithmic KL composes exactly over complete responses, yet its token-local update ascends the sequence objective.
  • A power Bregman divergence does not compose into the same path-space divergence, yet its local update equals the exact descent gradient of its occupancy-weighted objective.

Czarnecki et al. show that a student-sampled local field need not be conservative. Classical scoring theory characterizes proper scores. Neither result separates composition from descent.

The paper then makes this separation measurable:

  • Definition 4.7 and Propositions 4.8–4.9 give an intrinsic fidelity coefficient, an exact descent criterion, and a finite-step descent guarantee.
  • Theorem 4.10 gives the score–gradient wedge in closed form. It tests integrability, and Eq. 21 gives an independent closed-loop check.
  • Theorem 4.1 covers every continuous coordinate-local reward, including rewards outside the Bregman class.

Proposition 4.3 connects the classical logarithmic characterization to OPD. Its OPD consequence is the bounded-reward tradeoff in Corollary 4.4.


Q2 (Q2 & Weakness 2): Can fidelity be estimated, even noisily, in the full parameter space of the 0.8B student (or a smaller fully fine-tuned model), rather than in a rank-eight LoRA subspace?

Our Rademacher directions are random directions in the complete trainable LoRA space, drawn independently for each seed. They are not selected coordinate planes. For a normalized random direction a in dimension d, E[(aᵀg)(aᵀu)] = ⟨g, u⟩/d. Averaging projected inner products over seeds therefore estimates the corresponding inner products in the complete trainable space.

We recomputed local-log fidelity this way from the stored measurements, with cross-fitted trajectory pairs. It is negative, with seed intervals that exclude zero, on HellaSwag at horizons 8, 16, 32, 48, and 64, in the eight-direction checks at horizons 8 and 16, and on ARC at horizon 64. The same directional estimator applies to full-model parameters without explicitly constructing the full gradient.

Plain SGD removes optimizer-state differences, so the matched 128-step training panel compares the update methods under the same optimization settings (Fig. 3).

Manuscript revision: App. H.7 now gives the cross-fitted estimator and reports these estimates in Table 3 (pp. 23–24).


Q3: All horizons are at most 64 tokens, while deployed OPD typically supervises reasoning traces of hundreds to thousands of tokens. Does the fidelity transition sharpen, saturate, or change character at horizons of 512+ tokens, and is there any measurement or theoretical reason to expect the same sign pattern?

Eq. 13 and its advantage form in Eq. 45 identify the mechanism at every horizon. An earlier action contributes future credit through its effect on later divergence. Longer responses give earlier actions more opportunities to affect future divergence. Whether fidelity keeps falling, saturates, or changes sign depends on how the resulting future credit aligns with the exact gradient.

Our measurements show relative future credit rising with horizon (Fig. 2c). The 0.8B student reaches or crosses the zero-fidelity boundary on every task. The 4B student stays aligned over longer horizons (Fig. 2).

At 512 tokens and beyond, the same decomposition and the same exact criterion apply. The sign of Φ_B decides whether the update descends or ascends its objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.