- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper suggests that a teacher in an OPSD setup may have access to shortcuts or extra information that a student cannot easily emulate. Hence they propose an objective that trains the teacher model to be more “student-like” by mixing in a KL penalty with reward for the teacher. This trained teacher proves to be better supervision in their OPSD experiments on MATH, LCBv6, and AppWorld with DeepSeek-V2-Lite, Qwen3-0.6B, and Qwen3-4B-Instruct. Additional results on Qwen3-30B-A3-Instruct on Terminal Bench 2.1 are also provided.
Critical strengths
As far as I can tell, the insight of making the teacher more on-policy for the student to learn is novel. Most other papers try to train the student to align with the teacher, and I think it is clever (and useful) to train the teacher toward this objective.
Critical weaknesses
My main criticisms are:
- The evaluation section is quite confused in its current form. The main results are provided with DeepSeek-V2-Lite (a rather old but large-ish 16BA3B model), Qwen3-0.6B (an extremely tiny model?), Qwen3-4B-Instruct (a non-thinking model?) and Qwen3-30B-A3B (a large model, but on a different benchmark: TerminalBench-v2.0?) on different tasks (GSM8K which is very easy, to TerminalBench 2.1 which is frontier-ish). It would be much cleaner to fix a fixed set of three models in the 1-8B scale (e.g. Qwen3-8B, Olmo3-7B-Think, etc.) and fix the benchmarks to be a fixed set of 1-3. In the current form I find it hard to take anything away from the presented results.
- The writing is generally good, but in my opinion, it should be more focused. Some points here are (a) Although I appreciate the math in the paper, it should spend more time stating the intuition “we need the teacher to be more like the student for the student to learn effectively” more plainly instead of having the reader decode it from the equations (b) I don’t think Figure 1 conveys the method very well (c) Some ablations like the bias-variance trade-off seem not to be a good fit to this paper (d) Similarly, e.g., Figure 5 doesn’t seem to add any information to the presented analysis. I suspect that the authors initially intended the paper to be a more theoretical analysis with a small amount of empirical evidence, but the current theme of the paper is clearly more empirical. I recommend that they lean into this direction and move the theory or more peripheral ablations to the background or appendix and answer the more pressing empirical questions in the main text (e.g., more thorough comparisons as mentioned above).
So I believe that there indeed are some interesting contributions here in this paper, but it appears unfinished/unpolished for publication in the given state.
Questions for authors
Since my concerns are more about the global framing or thoroughness of experiments, I do not think there are specific questions whose answers would materially affect my assessment.
2 comments
Thank you for the rebuttal
Thank you to the authors for their rebuttal. I have since read the revisions to the paper and I think the writing has been improved meaningfully. I still find issues with the experimental results and find them confusing, and would strongly urge the authors to pick the experimental settings from one clear prior work instead of mixing-and-matching several.
Nonetheless, the writing revision addresses my main concern with the paper, so I am raising my score to "3: weak accept".
Thank you for recognizing the novelty and usefulness of our proposed method for adapting the teacher to the student. We appreciate the feedback on experimental organization and presentation. We have clarified the rationale for the evaluation settings and brought the most informative comparisons into the main text.
Within each setting, methods are compared using the same base model, task, and evaluation protocol. These experiments evaluate our method across different tasks and sources of privileged information. The model–task pairings accommodate differences in task difficulty and model capability, providing settings where learning progress can be meaningfully evaluated.
Our model–task choices also build on established experimental settings. We use Qwen3-8B on LiveCodeBench, as in Hübotter et al. (2026), and Qwen3-4B-Instruct on AppWorld, as in Wang et al. (2026), adopting the same model–task pairings used in these prior studies. Our MATH experiments follow the solution-conditioned teaching approach of Zhao et al. (2026), and our terminal experiments use the environments and agent infrastructure of Ivison et al. (2026). We have made these connections explicit at the start of Section 4, with detailed protocols in Appendix A.
We have developed the central intuition throughout the abstract, introduction, and methodology. Following your suggestion, we have strengthened the connections between these explanations and the mathematical objectives in the revision. We have also revised Figure 1 to show how teacher and student trajectories are generated, scored, and used to update the shared model, with privileged information available only to the teacher.
We have revised the manuscript to address your concerns about the relevance of the bias–variance analysis and token-level visualization, and to give the empirical comparisons greater prominence.
On the bias–variance trade-off: This analysis addresses a central difficulty in self-distillation. Conventional self-distillation trains the student to imitate a privileged teacher. When the teacher relies on shortcuts unavailable to the student, that supervision can push the student toward behavior that does not improve its own performance. More samples make the resulting update less noisy, but do not correct this systematic mismatch. This is the bias we seek to address through teacher adaptation. We have clarified this motivation in the revised manuscript and connected it directly to Figure 6’s analysis of the student’s learning signal.
On Figure 5: The token-level visualization provides a concrete example of the feedback the student receives. The teacher–student log-probability differences shown in the figure supply the token-level learning signal used in distillation. The visualization therefore lets readers see how broadly and strongly the teacher’s feedback acts on a student response. In the example, OPSD produces differences throughout much of the response, whereas JOLT’s differences are smaller and more localized. We have connected this example more directly to the intuition of providing guidance suited to the student.
On the empirical results: We have strengthened the empirical presentation by bringing the component comparisons from Appendix C into Table 3 in Section 4. In ablations that vary the teacher and student learning signals, JOLT+ achieves 73.80% accuracy on MATH, versus 66.52% for OPSD with student rewards and 69.12% for JOLT+ without student distillation. These controls support contributions from both teacher optimization and student distillation. We have clarified the connection between the theoretical characterization and the practical teacher objective.