acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning, On-Policy Distillation, or Both? A Finite-Budget Theory

Abstract

Reinforcement learning (RL) learns from rewards, while on-policy distillation (OPD) trains a student to imitate a teacher at states the student visits. RL updates can be noisy; OPD can inherit teacher errors. We ask when combining them improves on either method alone under a finite budget. We compare expected final regularized reward regret in a known nonlinear two-stage model, charging for rollouts, rewards, and teacher queries. Under explicit local conditions and bounded learning rates, a fixed mixture can outperform both pure methods even after independent rate tuning and additional updates funded by saved query costs. For the specified estimators, this guarantee covers all permitted RL schedules chosen before training and adaptive OPD rates and stopping. An explicit sufficient window relates the advantage to teacher error, reward noise, learner-dependent visitation, and budget. Retaining sampling variation is essential: average update directions can predict the wrong winner. A separate two-update result quantifies the benefit of adapting RL rates to observed progress and gives sufficient conditions favoring mixing or adaptive RL. Controlled simulations check these criteria. A controlled Qwen study provides complementary evidence: after independent tuning under equal query allowances, mixing has lower mean regret with noisy rewards, while RL has lower mean regret with clean rewards. Together, the results identify a benefit that cannot be explained by rate tuning or cost reallocation alone. Teacher guidance can help when rewards are noisy and teacher mismatch is moderate; reliable rewards, cheaper pure updates, or effective adaptation can remove that benefit.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, On-policy distillation, Learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xDgXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a finite-budget theory that shows a combination of reinforcement learning (RL) and on-policy distillation (OPD) can beat either methods alone. On the one hand, RL learns from rewards but is sensitive to reward model accuracy and can be noisy. On the other hand, OPD trains a weaker student to imitate the stronger teacher but may inherit teacher errors. The paper proposes a local nonlinear two-stage RL and OPD, and show the condition that the algorithm has fewer error than two pure methods. Finally, the paper implements the method in a real experiment, and shows the comparison against pure RL and OPD.

Critical strengths

I find the high level idea of this paper interesting, though I have not checked the whole proofs.

Critical weaknesses
  1. The current form of paper is hard to read. Theorem 4.1 takes a total of 12 lines, but one can hardly understand the central message it conveys. What is Equation (7)? Why do you assume it is positive? What is the interpretation? The authors simply define the math without any explanations, and it is unclear whether the condition is justified.

I also spot a lot of weird patterns in the writeup. Many sentences follow the pattern "X is Y, not Z" (e.g., Line 61, 71, 141, 147, 168, 192). For example, in Line 339, the authors write "These are theoretical certificates, not new experiments". Many sentences seem to answer questions the text has never asked, which read very unnatural.

  1. Experiments do not support the theory. Figure 1 (right panel) does not show the mixture method can beat pure RL or pure OPD, which raises the question why the experiments do not support the theory. The “not CI” text inside Figure 1 also reads odd.
Questions for authors

The paper repeatedly uses the word "certificate". What does it mean? Is it a common terminology in this field? If not, you should explain the meanings.

Reviewer_3nQoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a finite-budget theoretical analysis to demonstrate when combining reinforcement learning and on-policy distillation can lead to benefits over either method alone in a two-stage decision problem. Controlled experiments using Qwen show that the numerical results agree in the settings where the theory provides a certificate.

Critical strengths
  • [S1]: The paper considers a novel finite-budget theoretical comparison between reinforcement learning, on-policy distillation, and a mixture of the two approaches.

  • [S2]: Understanding when to favor reinforcement learning vs. on-policy distillation vs. a mixture is an important topic for LLM post-training.

Critical weaknesses
  • [W1]: The scope of the theoretical results is limited to a two-stage decision problem, rather than the standard sequential structure in autoregressive LLM prediction.
  • [W2]: It can be difficult to extract practical guidance from the theoretical results, which require knowledge of reward noise and teacher error (in addition to the limited problem scope). In the experiments where these values are known, the theory does not provide a certified choice in the majority of settings.
  • [W3]: Experiments do not consider standard LLM benchmark tasks to demonstrate if the main theoretical conclusions hold empirically for realistic settings despite the restricted theoretical scope.
  • [W4]: The paper’s main theoretical contributions require a 70+ page appendix to verify. This is not practical for a conference review cycle where the main paper should be self-contained.
Questions for authors
  • [Q1]: Please provide theoretical and/or empirical evidence that the main conclusions also apply to the realistic sequential structure of autoregressive LLM prediction.
Reviewer_yTaiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper establishes a theoretical comparison between RL, OPD and a mixture of the two. The analysis is focused on a two-stage decision process with binary choices. Under stated assumptions, they give sufficient conditions under which the mixed method is better than the pure methods. They also show that adaptive RL can outperform the mixed method under certain conditions.

Critical strengths

The paper focuses on a relatively simple problem which can be rigorously study. The theoretical analysis and calculations are solid.

Critical weaknesses
  1. The authors does a poor job in clearly stating their simplified model in the main text. The theoretical claims in the main text is difficult to follow unless referring to the appendix. I suggest the authors to reorganize the paper to make the main text self-consistent.
  2. Although the two-stage task is already very complex for analysis, it may be over-simplified compared to LLMs. The experiments are also limited to the two-stage task. It is unclear how to verify the conditions and whether the results preserve for practical LLM training.
Questions for authors
  1. To prove Theorem 4.1, it is assumed that p∗<qA<pp^*<q_A<p. Is this a reasonable assumption?
  2. According to Proposition 4.2, the both the teacher's error and the reward's noise need to be "large enough and small enough" to make the mixed strategy the best one. In other words, there are cases when the other pure methods are better. Then how can we find this window in real-world scenarios? What is the implication of Prop. 4.2 to pratical LLM training?
  3. Can the adaptive RL algorithm in Propsition 4.3 be implemented in real-world scenarios? Or does it indicate some potential directions for developing practical algorithms?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.