acceptodds
Under review as a conference paper at ICLR 2027

A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching

Abstract

Reinforcement Learning (RL) from outcome rewards suffers from sparse supervision, particularly on difficult, long-horizon tasks where successful trajectories are rare and costly to generate. On-Policy Distillation (OPD) offers an attractive alternative by providing dense token-level supervision from a stronger teacher along the student’s own generations. Self-distillation methods further remove the need for a separate teacher model by conditioning the same policy on privileged information to serve as its own teacher. However, privileged conditioning alone does not guarantee that the resulting distillation update improves the student. Indeed, privileged information can lead the teacher to solve tasks through shortcuts unavailable to the student, producing supervision poorly matched to the student’s current behavior. Consequently, even a higher-performing teacher can provide guidance that degrades student performance. To address this, we analyze how the choice of privileged teacher affects the student’s update. We derive a necessary and sufficient condition for the teacher’s local distillation update to be a positive multiple of the student’s reward gradient. Our analysis suggests that the teacher should not only perform well on the task, but also provide guidance suited to the student’s current capabilities. This characterization motivates a practical teacher-training surrogate that combines outcome rewards with token-level Kullback–Leibler (KL) regularization toward the student. Based on this result, we propose Joint On-Policy Learning and Teaching (JOLT), which jointly trains a single policy in two roles: a privileged teacher using a KL-regularized objective, and an unprivileged student using dense on-policy distillation. Our experiments across mathematical reasoning, code generation, tool use, and agentic terminaluse tasks demonstrate improvements in both training efficiency and evaluation performance. On MATH, JOLT uses 13× fewer completion tokens and 6.5× fewer processed training tokens than GRPO at matched accuracy, without direct student reward. With student outcome rewards, JOLT+ achieves absolute gains over GRPO of +4.2% on MATH, +3.0% on LCBv6, +9.9% on AppWorld, and +4.5% in Terminal-Bench pass@32.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, on-policy distillation, self-distillation, learning with privileged information, teacher-student learning, KL regularization, language model post-training.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Rs6cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper suggests that a teacher in an OPSD setup may have access to shortcuts or extra information that a student cannot easily emulate. Hence they propose an objective that trains the teacher model to be more “student-like” by mixing in a KL penalty with reward for the teacher. This trained teacher proves to be better supervision in their OPSD experiments on MATH, LCBv6, and AppWorld with DeepSeek-V2-Lite, Qwen3-0.6B, and Qwen3-4B-Instruct. Additional results on Qwen3-30B-A3-Instruct on Terminal Bench 2.1 are also provided.

Critical strengths

As far as I can tell, the insight of making the teacher more on-policy for the student to learn is novel. Most other papers try to train the student to align with the teacher, and I think it is clever (and useful) to train the teacher toward this objective.

Critical weaknesses

My main criticisms are:

  • The evaluation section is quite confused in its current form. The main results are provided with DeepSeek-V2-Lite (a rather old but large-ish 16BA3B model), Qwen3-0.6B (an extremely tiny model?), Qwen3-4B-Instruct (a non-thinking model?) and Qwen3-30B-A3B (a large model, but on a different benchmark: TerminalBench-v2.0?) on different tasks (GSM8K which is very easy, to TerminalBench 2.1 which is frontier-ish). It would be much cleaner to fix a fixed set of three models in the 1-8B scale (e.g. Qwen3-8B, Olmo3-7B-Think, etc.) and fix the benchmarks to be a fixed set of 1-3. In the current form I find it hard to take anything away from the presented results.
  • The writing is generally good, but in my opinion, it should be more focused. Some points here are (a) Although I appreciate the math in the paper, it should spend more time stating the intuition “we need the teacher to be more like the student for the student to learn effectively” more plainly instead of having the reader decode it from the equations (b) I don’t think Figure 1 conveys the method very well (c) Some ablations like the bias-variance trade-off seem not to be a good fit to this paper (d) Similarly, e.g., Figure 5 doesn’t seem to add any information to the presented analysis. I suspect that the authors initially intended the paper to be a more theoretical analysis with a small amount of empirical evidence, but the current theme of the paper is clearly more empirical. I recommend that they lean into this direction and move the theory or more peripheral ablations to the background or appendix and answer the more pressing empirical questions in the main text (e.g., more thorough comparisons as mentioned above).

So I believe that there indeed are some interesting contributions here in this paper, but it appears unfinished/unpolished for publication in the given state.

Questions for authors

Since my concerns are more about the global framing or thoroughness of experiments, I do not think there are specific questions whose answers would materially affect my assessment.

2 comments

Thank you for recognizing the novelty and usefulness of our proposed method for adapting the teacher to the student. We appreciate the feedback on experimental organization and presentation. We have clarified the rationale for the evaluation settings and brought the most informative comparisons into the main text.

“It would be much cleaner to fix a fixed set of three models in the 1-8B scale … and fix the benchmarks to be a fixed set of 1-3.”

Within each setting, methods are compared using the same base model, task, and evaluation protocol. These experiments evaluate our method across different tasks and sources of privileged information. The model–task pairings accommodate differences in task difficulty and model capability, providing settings where learning progress can be meaningfully evaluated.

Our model–task choices also build on established experimental settings. We use Qwen3-8B on LiveCodeBench, as in Hübotter et al. (2026), and Qwen3-4B-Instruct on AppWorld, as in Wang et al. (2026), adopting the same model–task pairings used in these prior studies. Our MATH experiments follow the solution-conditioned teaching approach of Zhao et al. (2026), and our terminal experiments use the environments and agent infrastructure of Ivison et al. (2026). We have made these connections explicit at the start of Section 4, with detailed protocols in Appendix A.

“it should spend more time stating the intuition … more plainly”

We have developed the central intuition throughout the abstract, introduction, and methodology. Following your suggestion, we have strengthened the connections between these explanations and the mathematical objectives in the revision. We have also revised Figure 1 to show how teacher and student trajectories are generated, scored, and used to update the shared model, with privileged information available only to the teacher.

“The writing is generally good, but in my opinion, it should be more focused.”

We have revised the manuscript to address your concerns about the relevance of the bias–variance analysis and token-level visualization, and to give the empirical comparisons greater prominence.

On the bias–variance trade-off: This analysis addresses a central difficulty in self-distillation. Conventional self-distillation trains the student to imitate a privileged teacher. When the teacher relies on shortcuts unavailable to the student, that supervision can push the student toward behavior that does not improve its own performance. More samples make the resulting update less noisy, but do not correct this systematic mismatch. This is the bias we seek to address through teacher adaptation. We have clarified this motivation in the revised manuscript and connected it directly to Figure 6’s analysis of the student’s learning signal.

On Figure 5: The token-level visualization provides a concrete example of the feedback the student receives. The teacher–student log-probability differences shown in the figure supply the token-level learning signal used in distillation. The visualization therefore lets readers see how broadly and strongly the teacher’s feedback acts on a student response. In the example, OPSD produces differences throughout much of the response, whereas JOLT’s differences are smaller and more localized. We have connected this example more directly to the intuition of providing guidance suited to the student.

On the empirical results: We have strengthened the empirical presentation by bringing the component comparisons from Appendix C into Table 3 in Section 4. In ablations that vary the teacher and student learning signals, JOLT+ achieves 73.80% accuracy on MATH, versus 66.52% for OPSD with student rewards and 69.12% for JOLT+ without student distillation. These controls support contributions from both teacher optimization and student distillation. We have clarified the connection between the theoretical characterization and the practical teacher objective.

Reviewer_Rs6cOpenReview ↗

Thank you for the rebuttal

Thank you to the authors for their rebuttal. I have since read the revisions to the paper and I think the writing has been improved meaningfully. I still find issues with the experimental results and find them confusing, and would strongly urge the authors to pick the experimental settings from one clear prior work instead of mixing-and-matching several.

Nonetheless, the writing revision addresses my main concern with the paper, so I am raising my score to "3: weak accept".

Reviewer_FEX4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how a privileged self-teacher can provide supervision that is better matched to what its student can learn. JOLT trains a single model in two roles: a teacher that receives additional information and learns from rewards with KL regularization toward the student, and a student that learns through distillation on its own generations. JOLT+ also incorporates student outcome rewards. In my view, the main contribution is explicitly adapting the teacher alongside the student, motivated by an analysis of when distillation follows the student's reward gradient. Experiments cover mathematical reasoning, coding, and two agentic settings, with substantial token savings on MATH and the strongest overall results obtained by JOLT+

Critical strengths

I lean toward acceptance because the method is coherent and the MATH ablations provide convincing evidence that its components work well together. My main reservations concern whether the same explanation accounts for the agentic gains and how broadly the coding results generalize.

1/ The paper makes a useful distinction between a teacher that solves tasks well and one that provides helpful supervision to the current student. The analysis provides a clear motivation for combining teacher reward learning with regularization toward the student, and the resulting shared-model training procedure accommodates several forms of privileged information

2/ The MATH ablations provide substantive support for the proposed combination. JOLT+ reaches 73.80% accuracy, compared with 66.52% for OPSD with student rewards and 69.12% when student distillation is removed (Table 8). These controls support contributions from both teacher optimization and distillation. The token-efficiency results also strengthen the practical case, particularly because the accounting includes teacher generations and cross-context scoring

Critical weaknesses

1/ The most informative control is limited to MATH. On AppWorld and Terminal-Bench, JOLT falls below GRPO, while JOLT+ improves over it (Tables 1–2). However, the paper does not evaluate OPSD with student rewards on these benchmarks. This leaves uncertainty about how much of the agentic improvement comes specifically from optimizing the teacher, beyond combining distillation with student reinforcement learning.

2/ The theoretical justification does not extend to the joint update as implemented. The ideal teacher depends on student continuation advantages, whereas the practical objective uses teacher returns; teacher updates also directly change the student through shared parameters. The authors acknowledge this, but the gradient diagnostics still exclude the teacher gradient and do not isolate the effect of teacher optimization (Appendix E). The proposed mechanism is therefore less firmly established than the empirical gains.

3/ The evidence for generalization and robustness is uneven. LCBv6 evaluates the same 131 problems used during training, with additional held-out test inputs, so it does not demonstrate transfer to unseen coding problems (Appendix A.3). Terminal-Bench uses only one training run per method, leaving the stability of its reported improvement across training seeds unresolved. Both limitations are disclosed, but they reduce the weight I place on these results.

Questions for authors

1/ When the student objective, rollout budget, and privileged-information construction are matched, does JOLT+ still outperform OPSD with student outcome rewards on AppWorld or Terminal-Bench? This would help establish whether teacher optimization accounts for the agentic gains.

2/ Is there evidence that the complete shared-parameter update, including the teacher gradient, aligns better with the student's reward gradient? The current diagnostics evaluate only the student-side update.

3 comments

Thank you for recognizing the motivation for adapting the teacher to the student and the evidence provided by our MATH component controls. We now report an AppWorld comparison with OPSD + outcome rewards, alongside new experiments examining parameter sharing and teacher–student gradient geometry.

“When the student objective, rollout budget, and privileged-information construction are matched, does JOLT+ still outperform OPSD with student outcome rewards on AppWorld or Terminal-Bench?”

We have now evaluated OPSD + outcome rewards using the AppWorld training and evaluation protocol described in Appendix A.4. It achieves a development-set state-test pass rate of 64.19 ± 5.72%, compared with 72.09 ± 6.01% for JOLT+, a 7.90 percentage-point improvement in mean performance. Results are reported as means ± standard deviations across runs.

This extends the comparison with OPSD + outcome rewards from MATH to AppWorld. JOLT+ achieves higher mean performance in both settings. The MATH component controls, now presented in Table 3, further support contributions from both teacher optimization and student distillation.

“Is there evidence that the complete shared-parameter update, including the teacher gradient, aligns better with the student's reward gradient?”

We have extended Appendix E with a parameter-sharing comparison and a probe of teacher–student gradient geometry.

The parameter-sharing comparison evaluates training both roles within a single policy against separate policies that retain distillation coupling:

ConfigurationStudent train accuracyTeacher train accuracyTrain KLStudent held-out accuracy
Shared JOLT71.88%82.58%0.024867.28%
Separate JOLT31.80%97.50%0.554028.63%
Shared JOLT+73.20%82.58%0.020967.96%
Separate JOLT+61.33%97.27%0.754856.27%

This is a separate parameter-sharing experiment with one training run per configuration, using the MATH evaluation protocol in Appendix A.2.

For both methods, sharing parameters yields higher held-out student accuracy and substantially smaller teacher–student divergence despite lower privileged-teacher training accuracy. This supports parameter sharing in these MATH controls and the motivation that teacher success alone does not determine how successfully the student learns.

The gradient analysis finds nearly orthogonal teacher and student gradients on average, indicating distinct update directions. It does not measure the complete update’s alignment with the student reward gradient. Our theory motivates the teacher objective, which uses teacher returns in place of student-continuation advantages.

“how broadly the coding results generalize”

Our LCBv6 evaluation follows Hübotter et al.’s protocol: training and evaluation use the same 131 problems, with evaluation on the complete test suites, including 2,431 test cases withheld from training. These held-out cases test new inputs for the same coding problems, as clarified in Section 4 and Appendix A.3.

“the stability of its reported improvement across training seeds”

We have added 90% confidence intervals to Table 2 using 10,000 bootstrap resamples of benchmark tasks, retaining all observed evaluation trajectories within each sampled task. These are per-method task-bootstrap intervals, conditional on the observed trajectories, using one training run per method.

Thanks for your replies, while this addresses my question, I have some other questions:

(1) why for different tasks you are using difference backbone llm? also the models used are relatively old, why not use recent models (in 2026) like qwen3.5/gemma4...

(2) I want to check and confirm my understanding of this method, could you describe to me in simple words that compared to standard opsd, what extra stuff you added, like an extra regularization term? what else?

(3) Still want to confirm my understanding, regarding the sentence "a necessary and sufficient condition for the teacher’s local distillation update to be a positive multiple of the student’s reward gradient" in abstract, could you point me to what exactly does this condition look like (in the paper)? why and how the distillation has to be the positive multiple of the student reward? you can just give me the pointer or explain intuitively, I just want to make sure I understand the method accurately.

Thank you for the follow-up questions. We are happy to clarify.

“why for different tasks you are using different backbone llm?”

We choose model-task combinations with meaningful room for learning and follow established experimental settings where possible. This means avoiding settings where the model already saturates the task or obtains essentially no reward because the task is too difficult. For example, Qwen3-8B on LiveCodeBench follows Hübotter et al. (2026), and Qwen3-4B-Instruct on AppWorld follows Wang et al. (2026). Within each setting, all compared methods use the same backbone and evaluation protocol.

“compared to standard opsd, what extra stuff you added, like an extra regularization term? what else?”

In addition to OPSD’s student distillation objective, JOLT jointly trains the privileged teacher on its own generated responses using outcome rewards and a teacher-to-student KL penalty. The reward encourages successful responses, while the KL penalty keeps the teacher’s predictions close to the student’s predictions. The teacher and student share the same model weights.

“what exactly does this condition look like … why and how the distillation has to be the positive multiple of the student reward?”

The exact necessary-and-sufficient condition is Equation (5), Section 2.2, page 4. It requires teacher supervision and student advantages to produce the same expected update, up to positive scaling. Equation (6) gives a simpler sufficient condition: the two learning signals match at every state-action pair, rather than only after averaging, up to a state-dependent baseline that does not affect the expected gradient.

Reviewer_YH9yOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

JOLT jointly trains a shared language model as a privileged teacher and an unprivileged student. The teacher learns from outcome rewards with KL regularization toward the student, while the student receives on-policy distillation at its own generated prefixes. JOLT+ additionally uses student outcome rewards. A local gradient characterization motivates a teacher adapted to the student's continuation behavior. Experiments span mathematics, coding, AppWorld, and Terminal-Bench, with repeated runs in most settings, component controls on MATH, and token-budget scaling analyses.

Critical strengths
  1. The paper turns a useful distinction into a practical algorithm: a successful privileged policy need not provide supervision suited to an unprivileged student's current behavior. Joint teacher optimization and student-prefix distillation are simple enough to implement, while accommodating several forms of privileged information. JOLT+ improves the reported evaluation metrics over GRPO across the studied domains.

  2. The empirical controls are consequential. Appendix C compares against student reward plus OPSD without explicit teacher optimization and against teacher training without student distillation. These controls support the combination beyond either component alone on MATH. Repeated training runs, explicit evaluation protocols, and accounting for teacher and cross-context processed tokens further strengthen the evidence.

Critical weaknesses
  1. The strength of the broad empirical conclusion varies with the evaluation setting. LCBv6 evaluates the same 131 problems used in training against an expanded test suite, so its gain is not evidence of generalization to unseen coding problems. AppWorld reports development-set state-test fractions, and Terminal-Bench has only one training run per method. Furthermore, JOLT without student rewards remains below GRPO in the two agentic settings. These are legitimate experiments, but the strongest evidence is for the specific JOLT+ configurations and protocols, rather than a uniformly superior replacement for reward-based learning.

  2. The theoretical characterization remains a motivation rather than an explanation established for the actual combined update. The practical teacher uses teacher returns in place of student-continuation advantages, and teacher gradients also change the shared student parameters. The paper acknowledges both gaps. Appendix E includes useful normalized student-gradient comparisons, but excludes the teacher gradient and therefore does not test the complete update. Likewise, the 13×/6.5× efficiency numbers are fitted token-budget point estimates without ratio uncertainty, rather than measured runtime speedups. These limits prevent a stronger assessment of the claimed mechanism and practical efficiency.

Overall assessment: 3 — Weak acceptance. The method is useful, the multi-domain evidence is substantial, and the MATH controls support the combination of teacher optimization and student distillation. Evaluation-scope limitations, the gap between the local characterization and the actual joint update, and uncertain runtime savings keep this below clear acceptance.

Questions for authors
  1. Can you quantify how the complete shared-parameter update, including the teacher component, affects student reward at fixed MATH checkpoints? Existing component measurements or a small fixed-checkpoint analysis would help distinguish the proposed teacher mechanism from the behavior of the student estimator alone.

  2. For LCBv6, are withheld-test-only results available, and is there any evaluation on unseen problems? Please maintain the distinction between new test inputs and new coding tasks. For Terminal-Bench, can you report paired task-level uncertainty while clearly separating it from the unmeasured training-seed variation?

  3. How stable are the token-budget ratios under run resampling or direct comparison of nearby observed checkpoints? Is matched-accuracy wall-clock information available for the MATH setting? The processed-token accounting is valuable, but does not by itself establish a runtime ratio.

1 comment

Thank you for recognizing the usefulness of adapting the teacher to the student and the evidence provided by our component controls. We have added MATH experiments examining parameter sharing and teacher–student gradient geometry, and address your questions about evaluation scope and efficiency below.

“Can you quantify how the complete shared-parameter update, including the teacher component, affects student reward at fixed MATH checkpoints?”

We have extended Appendix E with a parameter-sharing comparison and a probe of teacher–student gradient geometry.

The parameter-sharing comparison evaluates the effect of training both roles within a single policy. Separate policies retain distillation coupling. The results are:

ConfigurationStudent train accuracyTeacher train accuracyTrain KLStudent held-out accuracy
Shared JOLT71.88%82.58%0.024867.28%
Separate JOLT31.80%97.50%0.554028.63%
Shared JOLT+73.20%82.58%0.020967.96%
Separate JOLT+61.33%97.27%0.754856.27%

This is a separate parameter-sharing experiment with one training run per configuration, using the MATH evaluation protocol in Appendix A.2.

For both methods, sharing parameters yields higher held-out student accuracy and substantially smaller teacher–student divergence despite lower privileged-teacher training accuracy. This supports parameter sharing in these MATH controls and the motivation that teacher success alone does not determine how successfully the student learns. Table 3’s component controls additionally support contributions from both teacher optimization and student distillation.

The additional gradient analysis finds that teacher and student gradients are nearly orthogonal on average. In this probe, teacher gradients contribute directions largely distinct from the student update.

The sharing comparison supports the joint training design. We have not measured the complete update’s one-step effect on student reward. Our theory motivates the teacher objective, which uses teacher returns in place of student-continuation advantages.

“the strongest evidence is for the specific JOLT+ configurations and protocols”

JOLT+ improves the reported evaluation metrics over GRPO across the studied settings. In AppWorld and Terminal-Bench, student outcome rewards are important: JOLT+ outperforms GRPO, while JOLT alone does not. The agentic results support JOLT+ as a way to combine distillation with student outcome rewards in the evaluated settings. AppWorld measures the mean fraction of state tests passed on its 57-task development split, as described in Section 4 and Appendix A.4.

“For LCBv6, are withheld-test-only results available, and is there any evaluation on unseen problems?”

Our LCBv6 evaluation follows Hübotter et al.’s protocol: training and evaluation use the same 131 problems, with evaluation including 2,431 test cases withheld from training. The reported scores cover the full test suites, rather than the withheld tests alone. These held-out cases test new inputs for the same coding problems, as clarified in Section 4 and Appendix A.3.

“For Terminal-Bench, can you report paired task-level uncertainty while clearly separating it from the unmeasured training-seed variation?”

We have added 90% confidence intervals to Table 2 using 10,000 bootstrap resamples of benchmark tasks, retaining all observed evaluation trajectories within each sampled task. These are per-method task-bootstrap intervals, conditional on the observed trajectories, using one training run per method. They quantify uncertainty in each method’s score, rather than in the difference between methods.

“How stable are the token-budget ratios under run resampling or direct comparison of nearby observed checkpoints? Is matched-accuracy wall-clock information available for the MATH setting?”

The approximately 13× and 6.5× figures describe completion-token and processed-token savings at matched accuracy for JOLT without student outcome rewards. They compare an observed GRPO checkpoint with JOLT’s fitted budget at the same accuracy. The revised introduction explicitly identifies these as approximate fitted estimates.

The existing Appendix B analysis examines sensitivity to the accuracy target. At 55%, 60%, and 65% MATH accuracy, the fitted comparisons yield approximately 9.5–14.4× fewer completion tokens and 4.3–7.4× fewer processed training tokens. The estimated token advantage therefore persists across these targets. These comparisons use fitted curves; run-resampled ratio intervals, a comparison based entirely on nearby observed checkpoints, and matched-accuracy wall-clock measurements are not reported. Our efficiency claim concerns training-token expenditure, including teacher generations and cross-context scoring, rather than measured runtime speedup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.