acceptodds
Under review as a conference paper at ICLR 2027

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

Abstract

On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragility remain incompletely understood. We study sequential distillation from multiple teachers, where the student minimizes its average divergence from the teachers. Forward Kullback–Leibler (KL) divergence yields a weighted arithmetic mixture, while reverse KL yields a normalized weighted geometric aggregate. We develop algorithms that learn these targets under off-policy and on-policy feedback, respectively, establishing logarithmic regret bounds in the tabular setting and extending the analysis to function approximation. By analyzing these aggregation targets, we identify mechanisms that help explain both the benefits and fragility of OPD. Relative to forward KL, reverse KL can better retain a confident expert's preferences under uninformative feedback, but is more sensitive to teachers that assign very low probabilities to correct responses. Its token-level conditionals also reveal a dependence on continuation distributions that can favor incorrect prefixes over long horizons.

Primary area:
learning theory
Keywords:
On policy distillation, off policy distillation, theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2B6LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares supervised finetuning (SFT) and on-policy distillation (OPD). A theoretical perspective is taken; the paper connects SFT with forward KL and OPD with reverse KL and study the mathematical consequences of the respective implied target policies. The authors assert that 1. reverse KL may be more robust than forward KL against uninformative teachers, 2. reverse KL may be more unstable with misleading teachers, and 3. OPD may suppress correct prefixes when the horizon is sufficiently long. With these assertions the authors aim to provide insights into why OPD may outperform SFT and why it is sometimes observed unstable. For independent interest, the authors provide regret bounds in approximating the respective target quantities.

Critical strengths

Firstly, the decision to study the difference between SFT and OPD is one which is highly relevant for practical RL and distillation. Moreover, the choice of connecting SFT with forward KL and OPD with reverse KL is also a natural and relevant one respecting the natural modelling choice of many existing works. Within this relevant setting, the authors provided quantitative results characterising scenarios where OPD may work better or worse than SFT, particularly in the long-horizon setting - this provides grounds for better insights into the two practical methods.

Secondly, the paper is well written. I was able to understand the intended contribution of the paper, the setup of the problem, the assumptions and results of the paper clearly.

Critical weaknesses

As is with any theoretical work, demonstrating the connections between the described theoretical scenario and actual empirical observations is difficult. The scenarios described by the authors - propositions 5.1, 5.3 and 5.5 - are mathematical statements and not clear if they actually explains the situations seen in practice. To establish a connection between the theoretical formulation and practical observations, the authors would either need to analyse a family of large empirical experiments and show that the situations outlined in the paper occurs in a significant number of cases.

Questions for authors
  • Assumption 4.3 asserts a one sided bound whereas the proof of theorem 4.5 uses a two sided bound. should assumption 4.3 be modified?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 2B6L

Q1. As with any theoretical work, demonstrating the connections between the described theoretical scenario and actual empirical observations is difficult. The scenarios described by the authors (Propositions 5.1, 5.3 and 5.5) are mathematical statements, and it is not clear whether they actually explain the situations seen in practice. To establish a connection between the theoretical formulation and practical observations, the authors would need to analyse a family of large empirical experiments and show that the situations outlined in the paper occur in a significant number of cases.

A1. We agree that this connection is the hardest part to establish, and we thank the reviewer for raising it. We respond in two parts.

New experiments. We have added training experiments with language models in the new Appendix "Experiments", which test two of the three scenarios.

Uninformative teacher. We distill a Qwen3-1.7B student from an expert and a much weaker teacher on DAPO-Math. As the weak teacher's weight grows from 0 to 0.5, SFT falls from 69.0 to 63.3 on MATH-500 and from 22.3 to 11.7 on held-out problems, below the initial student, while token-level OPD is unaffected (71.4 → 73.6 and 24.3 → 25.2). 

Long horizon. We train a Qwen3-4B student with the sequence-level reverse KL and vary how many future tokens are credited to each token. With a short window the student is preserved; with a long window it abandons derivations and outputs only the final answer, and accuracy falls from 84% to about 25% while the training objective decreases.
In both cases, the behavior of the trained student is consistent with what the corresponding target predicts.

Scope of the claims. We do not claim that these scenarios account for a significant fraction of the behavior seen in practice. Our claim is narrower: the two objectives have different optimal targets, and Propositions 5.1, 5.3, and 5.5 identify conditions under which this difference alone produces the retention and the fragility reported in the empirical literature. The new experiments show that two of these mechanisms can be reproduced in training under controlled conditions.


Q2. Assumption 4.3 asserts a one-sided bound whereas the proof of Theorem 4.5 uses a two-sided bound. Should Assumption 4.3 be modified?

A2. Assumption 4.3 already imposes a two-sided bound via absolute-value bars: ∣log⁡[pi(a∣x,u)/πref(a∣x,u)]∣≤B.\bigl|\log\bigl[p_i(a\mid x, u) / \pi_{\mathrm{ref}}(a\mid x, u)\bigr]\bigr| \le B. We have rewritten it in the more explicit two-sided form −B≤log⁡pi(a∣x,u)πref(a∣x,u)≤B-B \le \log \frac{p_i(a\mid x, u)}{\pi_{\mathrm{ref}}(a\mid x, u)} \le B to make this clearer.

Reviewer_YiwbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies off-policy and on-policy distillation through multi-teacher aggregation. It derives the optimal targets under forward and reverse KL, provides learning algorithms with regret guarantees, and uses several constructions to illustrate differences in capability retention, sensitivity to misleading teachers, and behavior over long sequences.

Critical strengths
  1. The paper clearly characterizes the two aggregation targets and their token-level conditionals, providing a useful perspective on autoregressive distillation.

  2. The constructions involving uninformative teachers, misleading teachers, and long sequences illustrate both the potential benefits and risks of reverse-KL aggregation.

Critical weaknesses
  1. The connection to practical OPD updates needs clarification. The sequence-level reverse-KL optimum accounts for the effects of future tokens. OPD implementations that use only immediate token feedback may not optimize the same objective. In particular, it is unclear whether the preference for an incorrect prefix over long horizons also arises under these updates.

  2. There are no actual training experiments to validate the proposed mechanisms. All figures evaluate closed-form expressions. They illustrate properties of the ideal targets, but do not show whether a trained student exhibits the corresponding behavior. Since the paper uses these results to explain capability retention and degradation in OPD, a small-scale training experiment testing at least one of these mechanisms would strengthen the contribution.

  3. The comparison does not separate sampling effects from the choice of KL direction. The paper pairs off-policy sampling with forward KL and on-policy sampling with reverse KL, changing both the data source and the objective. This makes it difficult to attribute the discussed differences to on-policy versus off-policy training itself. An on-policy forward-KL baseline with the same student and teachers would help clarify where the differences come from.

Typo: On page 1, lines 45–46, “off-policy and off-policy feedback” should read “off-policy and on-policy feedback.”

Questions for authors
  1. Which OPD implementations correspond to the sequence-level optimum analyzed here? Does the long-horizon preference reversal still occur when updates use only immediate token feedback?

  2. Could the authors add a small-scale training experiment to test whether a student exhibits the predicted capability retention, sensitivity to misleading teachers, or degradation over long sequences?

  3. Could the authors include an on-policy forward-KL baseline under the same model and teacher settings to separate the effects of sampling from those of KL direction?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Yiwb

Q1. Which OPD implementations correspond to the sequence-level optimum analyzed here? Does the long-horizon preference reversal still occur when updates use only immediate token feedback? (Weakness 1 + Question 1)

A1. Thank you for pointing this out. We agree that this distinction should be stated explicitly. For a context xx, let KLh(θ,y<h):=∑i∈Iwi(x) KL(πθ(⋅∣x,y<h) ∥ pi(⋅∣x,y<h))\mathrm{KL}_h(\theta, y_{<h}) := \sum_{i\in\mathcal{I}} w_i(x)\,\mathrm{KL}(\pi_\theta(\cdot\mid x,y_{<h})\,\big\|\,p_i(\cdot\mid x,y_{<h})) denote the per-token reverse KL. Writing the sequence-level objective (2.1) as Lx(θ)=Ey∼πθ(⋅∣x)[∑h=1HKLh(θ,y<h)]L_x(\theta)=\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\big[\sum_{h=1}^H \mathrm{KL}_h(\theta, y_{<h})\big], differentiating with respect to θ\theta yields two terms because θ\theta also governs the prefix distribution: ∇θLx(θ)=Ey∼πθ(⋅∣x)[∑h=1H∇θKLh(θ,y<h)+∑h=1H∇θlog⁡πθ(ah∣x,y<h)∑h′=h+1HKLh′(θ,y<h′)].\nabla_\theta L_x(\theta) = \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\bigg[\sum_{h=1}^H \nabla_\theta \mathrm{KL}_h(\theta, y_{<h}) + \sum_{h=1}^H \nabla_\theta\log\pi_\theta(a_h\mid x,y_{<h})\sum_{h'=h+1}^H \mathrm{KL}_{h'}(\theta, y_{<h'})\bigg]. Because our paper directly optimizes the sequence-level reverse KL (2.1), our analysis accounts for this full objective and yields the minimizer πR∗\pi_{\mathrm{R}}^* (which Algorithm 2 learns via the backward recursion for VhV_h). In practice, directly optimizing this objective corresponds to implementations that estimate the full gradient above, such as REINFORCE or PPO using the cumulative teacher log-ratio from step hh to HH as return-to-go.

By contrast, updates using only immediate token feedback drop the second sum and minimize KLh(θ,y<h)\mathrm{KL}_h(\theta, y_{<h}) locally at each prefix, yielding π(a∣x,y<h)∝gh(a∣x,y<h)\pi(a\mid x,y_{<h})\propto g_h(a\mid x,y_{<h}) without the continuation value Vh+1V_{h+1}. This token-level objective is the one used by many practical OPD implementations. Its target is still a reverse-KL aggregate, now of the teachers' next-token distributions at each prefix, so the comparisons for uninformative and misleading teachers in Section 5 (Propositions 5.1 and 5.3) apply to it prefix by prefix. The long-horizon preference reversal of Proposition 5.5 does not occur, since it is caused by the continuation value. We have added a remark after Theorem 4.1 to clarify this distinction.


Q2. There are no actual training experiments to validate the proposed mechanisms. All figures evaluate closed-form expressions. They illustrate properties of the ideal targets, but do not show whether a trained student exhibits the corresponding behavior. Since the paper uses these results to explain capability retention and degradation in OPD, a small-scale training experiment testing at least one of these mechanisms would strengthen the contribution. Could the authors add a small-scale training experiment to test whether a student exhibits the predicted capability retention, sensitivity to misleading teachers, or degradation over long sequences? (Weakness 2 + Question 2)

A2.

We thank the reviewer for this suggestion. We have added training experiments in the new Appendix "Experiments". They test mechanisms with language models; all teachers and students are Qwen3 models trained on DAPO-Math.

Uninformative teacher. We distill a Qwen3-1.7B student from an expert (Qwen3-30B-A3B, weight α) and a much weaker teacher (Qwen3-0.6B, weight 1−α). As the weak teacher's weight grows from 0 to 0.5, SFT degrades from 69.0 to 63.3 on MATH-500 and from 22.3 to 11.7 on held-out DAPO-Math, falling below the initial student (66.6 / 21.3). Token-level OPD is unaffected (71.4 → 73.6 and 24.3 → 25.2). An on-policy forward-KL baseline does not inherit this robustness (68.4 / 11.7 at α = 0.5), which is consistent with the robustness coming from the direction of the divergence. This agrees with the comparison for uninformative teachers in Section 5. Sequence-level OPD with a 128-token credit window lies in between: it matches token-level OPD at α = 1 and 0.9, and at α = 0.5 it drops to 65.6 / 17.4, still above SFT.

Long sequences. We train a Qwen3-4B student with the sequence-level reverse KL from two teachers and vary the number K of future tokens credited to each token. For K ≤ 128 the student is preserved (MATH-500 within 4 points of the initial 84.4). For K ≥ 512 it stops writing derivations and outputs only the final answer, and accuracy falls to about 25%, while the training objective decreases. This is consistent with the continuation effect of Proposition 5.5: the preferred first token changes with the horizon although the teachers' token-level distributions are fixed.

We also added a numerical evaluation of Algorithms 1 and 2 on tabular instances, which confirms the logarithmic regret of Theorems 3.3 and 4.6. These experiments are small in scale and are intended to check the qualitative predictions rather than to propose new practical methods.

AuthorsOpenReview ↗

Response to Reviewer Yiwb

Q3. The comparison does not separate sampling effects from the choice of KL direction. The paper pairs off-policy sampling with forward KL and on-policy sampling with reverse KL, changing both the data source and the objective. This makes it difficult to attribute the discussed differences to on-policy versus off-policy training itself. An on-policy forward-KL baseline with the same student and teachers would help clarify where the differences come from. Could the authors include an on-policy forward-KL baseline under the same model and teacher settings to separate the effects of sampling from those of KL direction? (Weakness 3 + Question 3)

A3. At the sequence level (Eq. 2.1), the pairing follows directly from the KL definition: KL(pi(⋅∣x)∣∣π(⋅∣x))\mathrm{KL}(p_i(\cdot\mid x)||\pi(\cdot\mid x)) takes an expectation over teacher rollouts y∼piy\sim p_i (off-policy), whereas KL(π(⋅∣x)∣∣pi(⋅∣x))\mathrm{KL}(\pi(\cdot\mid x)||p_i(\cdot\mid x)) takes an expectation over student rollouts y∼πy\sim\pi (on-policy).

An on-policy forward-KL baseline can instead be defined at the token level (given full teacher logits) by sampling prefixes y<h∼πt(⋅∣x)y_{<h}\sim\pi_t(\cdot\mid x) from the student and minimizing Ey∼πt(⋅∣x)[∑h=1H∑i∈Iwi(x) KL(pi(⋅∣x,y<h)∣∣π(⋅∣x,y<h))].\mathbb{E}_{y\sim\pi_t(\cdot\mid x)}\bigg[\sum_{h=1}^H \sum_{i\in\mathcal{I}} w_i(x)\,\mathrm{KL}\bigl(p_i(\cdot\mid x,y_{<h})||\pi(\cdot\mid x,y_{<h})\bigr)\bigg]. Unlike off-policy forward KL, where y<h∼pi(⋅∣x)y_{<h}\sim p_i(\cdot\mid x) weights each teacher by wi(x)pi(y<h∣x)w_i(x)p_i(y_{<h}\mid x) inside the sum over ii (yielding the prefix-posterior weights wi(x,y<h)w_i(x,y_{<h}) in Eq. 3.2), here πt(y<h∣x)\pi_t(y_{<h}\mid x) does not depend on ii and factors outside ∑i∈I\sum_{i\in\mathcal{I}}. Minimizing over π(⋅∣x,y<h)\pi(\cdot\mid x,y_{<h}) at each visited prefix therefore yields the static-weight arithmetic mixture πOnF∗(a∣x,y<h)=∑i∈Iwi(x)pi(a∣x,y<h).\pi_{\mathrm{OnF}}^\ast(a\mid x,y_{<h}) = \sum_{i\in\mathcal{I}} w_i(x)p_i(a\mid x,y_{<h}). Because πOnF∗\pi_{\mathrm{OnF}}^\ast is still an arithmetic mixture of the teachers (and coincides with πF∗\pi_{\mathrm{F}}^* at h=1h=1), all three comparisons in Section 5 behave the same under on-policy forward KL as under off-policy forward KL—confirming that the differences in Section 5 come from the KL direction (arithmetic vs. geometric aggregation) rather than from on-policy sampling. We have added a remark on this comparison at the end of Section 5.

Following your suggestion, we also ran this baseline (Appendix K). With the same student (Qwen3-1.7B), teachers, and student rollouts as token-level OPD, we only reverse the direction of the per-token divergence. When the weak teacher's weight is 0.5, on-policy forward KL reaches 68.4 on MATH-500 and 11.7 on held-out DAPO-Math, compared with 73.6 and 25.2 for OPD and 63.3 and 11.7 for SFT. It therefore does not inherit the robustness of OPD, which is consistent with the difference coming from the KL direction rather than from on-policy sampling.


Q4. On page 1, lines 45–46, "off-policy and off-policy feedback" should read "off-policy and on-policy feedback."

A4. Thank you for pointing these out. We have fixed all the typos.

Thank you for the response and the additional experiments. I share the concern raised by other reviewers about the link between the theory and practical OPD training. The paper still needs more verification, clearer explanations, and experiments. I will maintain my score.

Thank you for the reply.

We believe we have fully addressed all the concerns in your original review, which raised three specific questions that we answered with the exact analysis and experiments you asked for:

  • Q1: We provided the exact gradient decomposition relating sequence-level and immediate token-level OPD and characterized the token-level fixed point π(a∣x,y<h)∝gh(a∣x,y<h)\pi(a\mid x,y_{<h})\propto g_h(a\mid x,y_{<h}), clarifying when the long-horizon reversal arises.
  • Q2: We conducted actual LLM distillation experiments (training a Qwen3-1.7B student on DAPO-Math and evaluating on MATH-500 and held-out DAPO-Math, Appendix K). This directly fulfills the "small-scale training experiment testing at least one of these mechanisms" requested in your review.
  • Q3: We derived the explicit on-policy forward-KL target πOnF∗(a∣x,y<h)=∑iwi(x)pi(a∣x,y<h)\pi_{\mathrm{OnF}}^*(a\mid x,y_{<h})=\sum_i w_i(x)p_i(a\mid x,y_{<h}) (added as a remark at the end of Section 5) and ran the exact on-policy forward-KL baseline under the same Qwen3-1.7B student, teachers, and student rollouts (reaching 68.4 on MATH-500 and 11.7 on held-out DAPO-Math at weight 0.5, vs. 73.6 and 25.2 for OPD and 63.3 and 11.7 for SFT, Appendix K), directly isolating KL direction from on-policy sampling.

Your follow-up asks for "more verification, clearer explanations, and experiments" but does not identify any step in Q1 or Q3 that is unclear, or any aspect of the Q2/Q3 experiments in Appendix K that fails to verify the mechanisms. Without a specific point, it is unclear what further question or issue remains for us to address.

If a concrete technical concern remains, please state it and we will gladly respond within the discussion period. Otherwise, since all three questions in your original review have been resolved, we respectfully ask you to reconsider your score.

Reviewer_8v8WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a sequential distillation framework and proposes two gradient-free algorithms, with a theoretical analysis of their regret.

Critical strengths
  1. The analysis of uninformative and misleading teachers provides useful insights into when reverse KL can preserve expert preferences and when it may suppress correct responses.
  2. The paper establishes logarithmic regret bounds for the proposed tabular algorithms, showing that the target aggregates can be learned under the respective feedback settings.
Critical weaknesses
  1. The closed-form solution of the sequential distillation framework appears to be a simple variant of existing work.

  2. The theoretical analysis provides regret bounds for two gradient-free algorithms, but does not compare them with existing gradient-based optimization methods, either theoretically or empirically.

Questions for authors
  1. The paper proposes a sequential distillation framework and conducts its analysis within this framework. What are the main differences from prior work? For example, Heskes (1997) [1], Eqs. (1)–(2), already provides a result similar to Eq. (4.1), and the same derivation seems applicable here. Am I missing other implications of the sequential distillation framework?

  2. I understand that the two tabular algorithms facilitate theoretical analysis. Are they mainly theoretical constructions, or do they also have practical value? How do they compare with other gradient-based methods, either theoretically or empirically? Would a simple experiment be feasible to analyze this?

[1] Tom Heskes. Selecting Weighting Factors in Logarithmic Opinion Pools

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8v8W

Q1. The closed-form solution of the sequential distillation framework appears to be a simple variant of existing work. For example, Heskes (1997)

[1] Tom Heskes. Selecting Weighting Factors in Logarithmic Opinion Pools.

A1. Thank you for the reference. At the sequence level, Eq. (4.1) indeed matches the logarithmic opinion pool of Heskes (1997), which we now cite before Theorem 4.1. In our paper, however, Eq. (4.1) is only the starting point that identifies the population target; the main purpose of our framework is to study how this target behaves in multi-token generation and how it connects to distillation.

First, while Heskes (1997) considers one-shot distributions, language models generate responses token by token. We therefore derive the token-level conditionals of both targets (Eqs. 3.2 and 4.2), showing that forward KL updates teacher weights to the prefix posterior wi(x,y<h)w_i(x,y_{<h}), whereas reverse KL multiplies the local geometric pool ghg_h by the backward continuation value Vh+1V_{h+1} over future tokens.

Second, rather than static pooling of known distributions, our framework connects these two targets to off-policy (SFT) and on-policy (OPD) distillation, showing that both protocols provably learn their respective targets with logarithmic regret from sequential single-teacher feedback (Theorems 3.3 and 4.6).


Q2. The theoretical analysis provides regret bounds for two gradient-free algorithms, but does not compare them with existing gradient-based optimization methods, either theoretically or empirically. I understand that the two tabular algorithms facilitate theoretical analysis. Are they mainly theoretical constructions, or do they also have practical value? How do they compare with other gradient-based methods, either theoretically or empirically? Would a simple experiment be feasible to analyze this? (Weakness 2 + Question 2)

A2. Algorithms 1 and 2 are mainly theoretical constructions, while direct gradient-based updates do not admit finite-time regret guarantees in this sequential partial-feedback setting. Their purpose is to show that the two feedback protocols — teacher rollouts for off-policy distillation, and scalar teacher log-probabilities along student rollouts for on-policy distillation — are statistically sufficient to learn πF∗\pi_{\mathrm{F}}^\ast and πR∗\pi_{\mathrm{R}}^\ast, with O~(log⁡T)\widetilde{O}(\log T) regret. We do not propose them as replacements for gradient-based training, and the comparison with gradient methods is not central to our contribution, which is the characterization of the two aggregation targets and of how they differ (Section 5).

Gradient-based SFT and OPD optimize the same forward- and reverse-KL objectives over a parameterized policy class, so when the class can represent the target they aim at the same solutions πF∗\pi_{\mathrm{F}}^\ast and πR∗\pi_{\mathrm{R}}^\ast. We do not claim regret or global-convergence guarantees for them.

Following your suggestion, we ran a simple tabular experiment under the protocol of Section 2 (two teachers, m=8m=8 rollouts per round, T=20,000T=20{,}000 rounds, 5 seeds, three problem sizes). The gradient baseline uses a tabular softmax policy and, in every round, takes gradient steps on the empirical objective built from all data observed so far: the cross-entropy of the teacher rollouts (off-policy), or the sequence-level reverse KL to the empirical mean of the observed teacher log-probabilities (on-policy). Cumulative regret (2.2) at T=20,000T=20{,}000:

S=2,A=3,H=3S{=}2,A{=}3,H{=}3S=3,A=3,H=4S{=}3,A{=}3,H{=}4S=2,A=4,H=4S{=}2,A{=}4,H{=}4
Off-policy: Algorithm 131114195
Off-policy: gradient descent31107203
On-policy: Algorithm 22188198
On-policy: gradient descent1643134

The two approaches behave similarly: both reach the same targets, and their regret curves flatten at a comparable level; the remaining gap in the on-policy case comes from the early rounds. For Algorithm 2 we scale the confidence bonus to 10−310^{-3} of its theoretical value; the theoretical constant is set for the worst case and over-explores at this horizon. We have added this experiment to Appendix K.

Thank you for the response. However, the two algorithms are mainly theoretical constructions, and the new comparison is limited to small tabular experiments. My concerns are therefore only partially addressed, and I will maintain my score.

Thank you for reading our response. We would like to clarify two points:

First, this paper is a theoretical study of off- and on-policy distillation. Constructing theoretical algorithms (Algorithms 1 and 2, together with their function-approximation counterparts in Appendices H and I) to establish formal learnability and O~(log⁡T)\widetilde{O}(\log T) regret guarantees and connect the two feedback protocols to their respective targets πF∗\pi_{\mathrm{F}}^\ast and πR∗\pi_{\mathrm{R}}^\ast is a standard and intended theoretical contribution, rather than a limitation.

Second, beyond the tabular comparison, in order to validate our theoretical observations on real LLMs, we also conducted real LLM distillation experiments using standard gradient-based SFT and OPD. Please see our responses to Reviewer Yiwb and Reviewer 2B6L, which demonstrate the practical relevance of our theoretical analysis in actual language model training.

In light of these clarifications and the additional LLM experiments, we kindly ask if you would reconsider your evaluation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.