acceptodds
Under review as a conference paper at ICLR 2027

Online Symbolic Distillation: Learning from Teacher Continuations at Student States

Abstract

We present Online Symbolic Distillation (OSD). At each iteration, the evolving student policy generates a new prefix, the teacher continues it autoregressively, and the student is updated using cross-entropy computed on the teacher-generated tokens. Each design choice targets a corresponding limitation of existing distillation methods: (1) sequential covariate shift in offline supervised fine-tuning (SFT) on fixed teacher trajectories, (2) fragmented supervision under prefix failure in token-level on-policy distillation (OPD), and (3) the need for access to teacher token probabilities in distribution-matching distillation. OSD achieves higher reasoning performance than OPD (with a top-16 KL approximation) at comparable GPU-hour cost. Our asynchronous implementation further reduces OSD's total training time by 23.8%. We evaluate OSD on both hard reasoning tasks and agentic tasks which reflects modern post-training scenarios, and it consistently outperforms existing distillation methods under the same training budget. By regenerating prefixes from the evolving student, OSD continues improving after offline distillation plateaus while better preserving the general capabilities and plasticity of the student. Using only text from GPT-5.4-mini, continuously training with OSD outperforms offline SFT from the same teacher by 13% on ScienceWorld. These results support OSD as an effective and efficient approach to online language-model distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, post-training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_J5nHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed OSD, training students on teacher continuations from prefixes generated by the evolving student. It uses cross-entropy on teacher tokens and supports text-only teachers. Experiments on reasoning and interactive tasks report improved performance, reduced training time with asynchronous execution, and better retention and sequential learning.

Critical strengths
  • The agent experiments show meaningful gains over the tested OPD variants, especially on ScienceWorld and TextCraft.
Critical weaknesses
  1. Novelty appears incremental. The cited Revisiting DAgger work already uses online student prefixes followed by teacher continuations. The benefit of OSD's changes is not clearly established.

  2. The experiments do not isolate the claimed benefit of evolving student states. In Section 5.2, offline SFT reuses teacher trajectories while OSD obtains fresh continuations. State adaptation is therefore mixed with data freshness and potentially different teacher budgets. Fixed-prefix sampling with fresh teacher continuations, at matched budgets, would clarify the mechanism.

  3. The manuscript contains repeated reporting errors. Figure 6 labels avg@8 results as pass@8, and Figure 11 still calls the method ROSE. Its 12.5% pass@4 also cannot arise from a single evaluation of 50 equally weighted tasks with four attempts each without an unstated aggregation scheme. Table 2 contradicts the claim of consistently outperforming existing distillation methods (BabyAI: 67.50 vs GuidedOPD 67.78). These errors affect interpretation of the results and indicate insufficient checking of the manuscript.

Questions for authors
  • What is the main contribution beyond the cited Revisiting DAgger work?

  • How do the existing results separate the benefit of evolving student states from fresh teacher continuations and different teacher budgets in Figure 7? Please clarify teacher calls, token counts and error bars.

  • How were k and the OPD baseline settings selected? Was OPD independently tuned, given the different learning rates and top-16 KL approximation?

  • Please clarify the metric labels in Figure 6 and the evaluation and averaging in Figure 11. How was 12.5% pass@4 obtained from 50 tasks with four attempts each?

3 comments

[Part 1/2]

TLDR:

  • Contribution beyond existing works (W1/Q1): OSD distills through continuations at the evolving student's states, using only the teacher's text. Unlike DAgger-style per-state labeling, it needs no label for every visited state and no verifier, and keeping this supervision on the evolving student helps beyond more offline data.
  • Evolving states vs fresh continuations and teacher budget (W2/Q2): With fresh continuations and the teacher budget matched, the gain comes from supervising the evolving student's states. Intervening on the frozen initial student's states stays at the offline-SFT level, while OSD clearly improves.
  • Quick clarification on typo (W3/Q4): The reporting issues were presentation errors that leave our results and conclusions unchanged, and OSD's advantage extends to long-horizon agents. The labels in Fig. 6 and Fig. 11 and the Table 2 sentence have been corrected in the revised version, and on long-horizon terminal tasks OSD also leads Teacher-SFT and OPD.
  • OPD settings (Q3): Both methods follow the standard learning rate for their own objective. OSD's objective, cross-entropy on teacher-written text, is closer to SFT, so it follows the standard SFT learning rate; OPD follows the default recipe of Li et al. (2026c), standard in recent on-policy distillation work, and the revision includes a comparison across OPD settings.

W1 / Q1. "What is the main contribution beyond the cited Revisiting DAgger work?"

Revisiting DAgger interpolates student and teacher turn by turn and supervises the student with teacher labels at the visited states. OSD rests on two different choices. First, it is continuation-only: the student writes a contiguous prefix, the teacher writes one continuation from that state, and the loss is applied only to the teacher's tokens. Second, the prefixes are regenerated from the evolving student throughout training, so the supervision always follows the states the student currently reaches, which helps beyond more offline data. Starting from an SFT-initialized Qwen3-1.7B (DeepSeek-V4.1-Flash teacher) on ScienceWorld, avg success rate (%):

epoch+ continued offline SFT+ OSD
1 (shared start)25.1225.12
220.5034.88
321.3839.12
427.2539.12

We have included this experiment in the revised manuscript (Appendix F).

W2 / Q2. "Separate the benefit of evolving student states from fresh teacher continuations and different teacher budgets"

Thank you for raising this. To separate these factors, we designed a frozen-prefix control on ScienceWorld (Qwen3-1.7B student, GPT-5.4-mini teacher) that holds the teacher budget fixed and changes only one thing: whether the teacher intervenes on the states of the student being trained or on those of the initial student. Both methods receive fresh teacher continuations, in the same number, so data freshness and teacher budget are matched by construction. At the same budget, supervising the current student's states outperforms supervising the initial student's, which stays at the level of offline SFT: OSD aligns its supervision with the states of the evolving student.

Whose states to supervise. Frozen-prefix control: the teacher writes fresh continuations, as in OSD, but always from prefixes of the frozen initial student, with the same total number of continuations as 4 OSD epochs. Avg success rate (%) at the data volume of 4 epochs (± = std over 4 evaluation passes):

Avg success rate (%)
offline SFT15.12 ± 2.72
frozen-prefix + fresh continuations15.75 ± 0.87
OSD20.25 ± 2.25

Fresh continuations alone perform on par with offline SFT, which suggests that the gain comes mainly from supervising the current student's states. We have included this control in the revised manuscript (Appendix E).

[Part 2/2]

Takeaway for a fixed teacher budget: our results so far suggest spending it on the current student's states (OSD 20.25 vs 15.75 for the same number of continuations on the initial student's states). The same holds in continual learning: starting from a student already trained with offline distillation, further training with OSD keeps improving it (39.1%), while more offline SFT does not (at most 27.3%). As for where to hand off within a trajectory, the prefix length k was not specifically tuned for OSD. The paper already contains a handoff ablation (App., Fig. 9), where varying how many turns the student plays before the teacher takes over shows that an intermediate handoff works best. To further address the reviewer's concern, we additionally ran a prefix-length ablation on RLVE with the teacher budget fixed: every prefix length we tried clearly improves over the original student, and pass@8 varies by less than four points across them, which suggests that OSD is not overly sensitive to where the teacher takes over. Please refer to Appendix C.1 of the revised manuscript for details.

W3 / Q4. "Clarify the metric labels in Figure 6 and the evaluation and averaging in Figure 11"

We thank the reviewer for the careful reading. These are typos, and we have corrected them in the revision. In Fig. 6, the caption should read avg@8 rather than pass@8; the bars are the same avg@8 values as in Fig. 5. In Fig. 11, "ROSE" is an earlier internal name of OSD, and the reported numbers are avg@4 rather than pass@4: 12.5% corresponds to 25 successes out of 200 episodes (50 tasks × 4 attempts). For Table 2, we have rewritten the sentence on consistent superiority so that it describes the table precisely in the revised manuscript.

Since Fig. 11 concerns terminal agents, we have added a larger study on TMax, a suite of long-horizon terminal tasks in which the agent works inside a real sandboxed shell over up to 60 turns. We distill Qwen3.5-27B into Qwen3.5-9B. We compare OSD with the same alternatives as in the paper, all trained on the same tasks from the same teacher: supervised fine-tuning on the teacher's own trajectories, and OPD. In OSD the student starts every episode on its own and the teacher takes over partway through, so the teacher's actions are learned in the situations the student actually reaches. We report results on the 80 TMax tasks closest to the training distribution, three runs per model (success rate, %; 95% CI from a task-level bootstrap):

success rate95% CI
Qwen3.5-9B (original)64.17[54.58, 73.75]
Teacher-SFT67.08[57.50, 76.67]
OPD68.75[58.75, 78.33]
OSD72.08[62.08, 81.67]

OSD reaches the highest success rate, suggesting that learning the teacher's actions in the situations the student itself reaches carries over to long-horizon terminal agents at a larger scale. We have included this study in the revised manuscript (Appendix C.3).

Q3. "Was OPD independently tuned?"

Its configuration follows the standard recipe of Li et al. (2026c), as stated in the original manuscript. Its configuration, including the learning rate, is also the standard choice in recent on-policy distillation work (Ye et al., 2025; Zhong et al., 2026; Xu et al., 2026; Xin et al., 2026; Hao et al., 2026; He et al., 2026). We have included a detailed comparison across OPD settings in the manuscript revision (Appendix D).

References

  • Hao, Dongze, et al. "Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization." arXiv preprint arXiv:2606.09091 (2026).
  • Ye, Tianzhu, et al. "Black-box on-policy distillation of large language models." arXiv preprint arXiv:2511.10643 (2025).
  • Li, Yaxuan, et al. "Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe." arXiv preprint arXiv:2604.13016 (2026).
  • Xin, Haoran, et al. "Escaping the KL Agreement Trap in On-Policy Distillation." arXiv preprint arXiv:2606.09471 (2026).
  • Zhong, Qiyong, et al. "SOD: Step-wise on-policy distillation for small language model agents." arXiv preprint arXiv:2605.07725 (2026).
  • Xu, Yuanda, et al. "Beyond GRPO and on-policy distillation: An empirical sparse-to-dense reward principle for language-model post-training." arXiv preprint arXiv:2605.12483 (2026).
  • He, Qiangqiang, Jin Li, and MingCai Chen. "Calibrating Teacher--Student Discrepancy for On-Policy Distillation." arXiv preprint arXiv:2609.21619 (2026).

Dear Reviewer J5nH,

Thank you again for your careful review. As the discussion period is drawing to a close, we wanted to kindly follow up on our response. Following your comments, we have clarified OSD’s contribution relative to existing literature and added substantial new experiments: a frozen-prefix control with fresh continuations and a matched teacher budget, continued training from an SFT-initialized student, an ablation of the OPD baseline’s settings, and a larger study on long-horizon terminal tasks (TMax), where OSD outperforms Teacher-SFT and OPD. We have also corrected the labels in Fig. 6 and Fig. 11 and the Table 2 sentence. All changes are marked in blue in the revised manuscript.

We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.

Best Regards,

Authors

Reviewer_Ji25OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Online Symbolic Distillation (OSD), in which the current student generates a prefix, the teacher continues from that prefix, and the student is trained with cross-entropy only on the teacher-generated continuation. Student prefixes are refreshed throughout training. The method extends naturally to agent tasks, where the teacher takes over interaction after several student turns, and supports teachers accessible only through generated text. An asynchronous implementation overlaps student prefix generation with teacher continuation. Experiments cover reasoning and multi-turn agent tasks, with additional analyses of efficiency, forgetting, and continued learning.

Critical strengths
  1. A simple, intuitive approach to providing actionable teacher supervision. Instead of only scoring a completed student trajectory, OSD lets the teacher demonstrate how to continue from a state reached by the student. This is particularly appealing when a weak student cannot independently produce useful continuations. The masked cross-entropy objective is straightforward and requires only teacher-generated text, making the method applicable to black-box teachers.

  2. Encouraging results across reasoning and interactive tasks. The evaluation includes multiple agent environments and comparisons with both offline distillation and online distillation methods. The teacher-takeover mechanism has a clear practical interpretation in these settings, and the asynchronous implementation provides an additional efficiency benefit. I find the combination of simplicity, applicability, and empirical improvements sufficient to support acceptance.

Critical weaknesses
  1. The sensitivity to the student–teacher handoff point is not fully characterized. The prefix length determines both the states receiving supervision and the amount of student and teacher generation. Appendix C.1 provides a useful turn-count ablation on ALFWorld, but the choice of 4,096 student tokens for single-turn reasoning remains insufficiently justified. It is unclear whether the preferred handoff point depends strongly on student capability or task difficulty. This is an important practical detail for applying the method beyond the reported settings.

  2. The continued-learning analysis does not fully isolate the effect of tracking the evolving student. In Section 5.2, offline distillation repeatedly uses a fixed dataset, whereas OSD obtains new teacher continuations throughout training. Consequently, the comparison combines adaptation to current student states with access to fresh supervision. The OEC comparison in Section 5.1 provides useful supporting evidence, but a control that repeatedly collects new continuations from a frozen initial student would more directly identify the contribution of updating the prefix-generating policy. The broader claims about preserving plasticity would also benefit from this distinction.

Questions for authors
  1. What motivates the choice of 4,096 student tokens in the reasoning experiments? Could you provide a prefix-length ablation with the teacher continuation budget held fixed, reporting both performance and generation cost? Does the preferred handoff point change with student capability or task difficulty?

  2. In the continued-learning experiments, how does OSD compare with repeatedly sampling fresh prefixes from a frozen initial student and obtaining new teacher continuations under a matched teacher-generation budget? This would help distinguish the benefits of tracking the evolving student from those of receiving fresh supervision.

2 comments

TLDR:

  • Prefix length and student capability (W1/Q1): OSD does not hinge on a finely tuned handoff point, and it still helps stronger students. At a fixed teacher budget, the prefix lengths we tested give similar gains on RLVE; from an SFT-initialized student, OSD keeps improving while continued offline SFT does not.
  • Tracking the student vs fresh supervision (W2/Q2): We separate the two factors with a frozen-prefix control, and the gain comes from tracking the evolving student. Fresh continuations from the frozen initial student's states stay at the offline-SFT level, while OSD clearly improves on both.

W1 / Q1. "What motivates the choice of 4,096 student tokens?"

Prior work suggests that supervision is most useful early to midway through a student's solution (Li et al., 2026c; Zhang et al., 2026), so we set K = 4,096 without tuning, within the 7,168-token per-rollout budget of the baselines. As suggested, we ran a prefix-length ablation (RLVE, Qwen3-4B student). The student writes the first K tokens of a solution and the teacher continues for a fixed budget; only K varies, everything else is the paper's OSD run. Evaluated on the enlarged RLVE test set (the original problems plus new problems from the same environments and difficulty), %:

K (student tokens)2,0483,0724,096
pass@847.551.450.8
student tokens / rollout2,0483,0724,096
teacher tokens / rollout≤1,024≤1,024≤1,024

Every prefix length we tried clearly improves over the original student (41.7 pass@8 on the same test set), and pass@8 varies by less than four points across K, which suggests that OSD is not overly sensitive to where the teacher takes over; the paper's 4,096 is within a point of the best setting. We have included this ablation in the revised manuscript (Appendix C.1).

To clarify the role of student capability, we further raise the student's starting capability on ScienceWorld: Qwen3-1.7B is first fine-tuned for one epoch on teacher (DeepSeek-V4.1-Flash) trajectories, then trained further two ways with the same teacher: OSD (the student plays the first turns, the teacher continues) vs further SFT on offline trajectories. Avg success rate (%) of the checkpoints trained from Qwen3-1.7B:

epoch+ continued offline SFT+ OSD
1 (shared start)25.1225.12
220.5034.88
321.3839.12
427.2539.12

Starting from the same stronger checkpoint, OSD reaches 39.1% after two further epochs, while more epochs of offline SFT stay between 20.5% and 27.3%. This suggests that a more capable student still benefits from supervision on its own states beyond more offline data. We have included this experiment in the revised manuscript (Appendix F).

References

  • Li, Yaxuan, et al. "Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe." arXiv preprint arXiv:2604.13016 (2026).
  • Zhang, Yaocheng, et al. "Are Full Rollouts Necessary for On-Policy Distillation?." arXiv preprint arXiv:2605.31490 (2026).

W2 / Q2. "Distinguish the benefits of tracking the evolving student from those of receiving fresh supervision"

We ran the control the reviewer suggested on ScienceWorld, in the paper's continued-learning setting (Fig. 7): it keeps the fresh teacher outputs but removes the tracking of the student. The question is whether the headroom OSD gains over offline SFT can be pinned down to on-policy prefixes, rather than to receiving new teacher text every epoch. The three methods differ as follows:

teacher text written during trainingwritten at the current student's states
offline SFTnono
frozen-prefix controlyesno
OSDyesyes

The frozen-prefix control (Qwen3-1.7B student, GPT-5.4-mini teacher) keeps everything that fresh supervision offers (the teacher writes fresh continuations, nothing is filtered, and it receives as many continuations as OSD), but the states it starts from always come from the initial student, so they stop reflecting what the student does as it learns. Avg success rate (%) at the same amount of teacher data, i.e., after 4 epochs for offline SFT and OSD (± = std over evaluation passes):

Avg success rate (%)
offline SFT15.12 ± 2.72
frozen-prefix + fresh continuations15.75 ± 0.87
OSD20.25 ± 2.25

Fresh teacher text alone does not account for the gain: continuing from the initial student's states does about as well as offline SFT (15.75 vs 15.12), whereas supervising the states the student actually reaches as it trains gives 20.25. This suggests that what keeps OSD learning is not the novelty of the teacher's text but where it is placed. We have included this frozen-prefix control in the revised manuscript (Appendix E).

Dear Reviewer Ji25,

Thank you again for your thoughtful review. As the discussion period is drawing to a close, we wanted to kindly follow up on our response. To address your questions, we have added a prefix-length ablation with the teacher budget fixed, an experiment starting from a stronger, SFT-initialized student, where OSD keeps improving while continued offline SFT does not, and a frozen-prefix control that separates tracking the evolving student from receiving fresh supervision. All additions are marked in blue in the revised manuscript.

We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.

Best Regards,

Authors

Comments on the paper

9 comments

[Part 1/3]

TLDR:

  • Contribution beyond roll-in + takeover (W1/Q1): In the budget- and verifier-matched comparison the reviewer suggested, refreshing prefixes from the evolving student brings a clear gain. On RLVE in the Fig. 5 setting, OSD reaches 33.3 pass@8 against 28.3 for OEC with the same verifier; on ScienceWorld, fresh continuations from frozen prefixes stay at the offline-SFT level (15.8% vs 15.1%), while OSD reaches 20.3%.
  • Cost and learning rate (W2/Q2): The comparison with OPD is fair in both cost and learning rate. The two methods have comparable end-to-end cost. For OPD we follow standard learning-rate practice in recent on-policy distillation work; OSD uses a different learning rate because its objective, cross-entropy on teacher-written text, is closer to SFT, and hence follows the standard SFT learning rate.
  • Uncertainty and test-set size (W3/Q3): On the enlarged RLVE test set the reviewer asked for, OSD's advantage holds with confidence intervals. OSD reaches 23.0 avg@8 against at most 18.5 for the baselines, and 50.8 pass@8 against at most 47.5; the gain spans environments rather than coming from one.
  • Effective intervention (W4/W5): OSD keeps helping as the student improves. Teacher interventions at the student's own states provide supervision whether or not a continuation succeeds, and from an SFT-initialized student OSD keeps improving beyond continued offline SFT; the revised Sec. 4 now states the main results precisely.

W1 / Q1. "How much gain remains from refreshing prefixes?"

Refreshing the prefixes brings a clear gain. OSD learns the teacher's intervention at the states the evolving student reaches and, in our main experiments, needs no verifier; OEC draws prefixes once from the initial student and keeps only verifier-approved completions, which ties its data to extra teacher inference compute. With student rollouts matched, OSD outperforms OEC both with and without the filter on RLVE. We further provide a frozen-prefix control in a multi-turn agentic setting (ScienceWorld).

In the paper's Fig. 5, OEC samples one prefix per training prompt from the initial student and lets the teacher continue it four times, while OSD samples four prefixes per prompt online and continues each once. To match OEC's budget and filter, OSD here also keeps only verifier-approved continuations, so that the comparison tests whether the evolving student's prefixes elicit better supervision. We re-ran OEC on RLVE to mirror OSD exactly: four prefixes per training prompt from the initial student, each continued once by the same teacher, so both methods use 36k different student prefixes; the OSD row is the run from Fig. 5.

Methodprefixes fromstudent rolloutsfilterpass@8
OECinitial student (once)36knone15.0
OECinitial student (once)36kverifier28.3
OSDcurrent student (online)36kverifier33.3

Without a filter, OEC stays clearly below OSD. With the same verifier filter as OSD, the remaining gap comes from the prefix states: prefixes drawn from the evolving student elicit supervision that stays aligned with the learner as it changes, which prefixes collected once from the initial student cannot provide.

To test whether OSD's gain comes only from learner roll-in followed by expert takeover, we further run a frozen-prefix control on ScienceWorld (Qwen3-1.7B student, GPT-5.4-mini teacher), with no filtering for any method, including OSD. If roll-in and takeover were enough, rolling in with a learner that never changes while still asking the teacher for fresh continuations should recover most of OSD's gain. The control matches OSD in fresh teacher continuations, no filtering and the number of continuations, and all methods are compared after 4 epochs; the only difference is that the teacher always starts from the initial student's states.

offline SFTfrozen-prefixOSD
avg success rate (%)15.12 ± 2.7215.75 ± 0.8720.25 ± 2.25

Rolling in with a frozen learner and refreshing the teacher's continuations does about as well as offline SFT on full teacher trajectories (15.75 vs 15.12), while OSD reaches 20.25. Roll-in and takeover alone therefore do not explain the gain. The difference is that OSD's teacher intervenes on the mistakes the student makes now, not on those it made at the start. This is the principle we establish: a student can be taught by learning the teacher's intervention at the states it actually reaches, and the supervision it receives should stay up to date as it evolves.

[Part 2/3]

W2. "The resource comparison is incomplete"

Thank you for raising this. Per training run (RLVE, Qwen3-1.7B, same prompts and hardware; GPU-hours count both the student and the teacher):

OPDOSD
student tokens sampled≤ 257M147M
teacher tokens scored / prefilled≤ 257M (scored)158M (prefilled)
teacher tokens generated037M
teacher calls36k36k
supervised tokens≤ 257M37M
wall clock (h)~5.1~4.8
GPU-h~40~39

OSD and OPD thus have comparable end-to-end cost, while OSD supervises far fewer tokens. Teacher-SFT has no student rollouts: its teacher generates up to 258M tokens over 36k calls before training, all of which are supervised.

Beyond accuracy and cost, there is also a practical difference in what each method asks of the teacher. OPD needs logit-level information: the teacher has to score every token the student writes and return its probability distribution, which requires white-box access, a shared vocabulary, and serving top-K log-probabilities for every student token. OSD only needs the text the teacher writes. This makes it applicable to API-only black-box teachers and to teachers whose tokenizer differs from the student's.

Q2. "The rationale for the tenfold learning-rate difference between OPD and OSD"

We would like to clarify that for OPD we use the default recipe of Li et al. (2026c) and keep its learning rate as published, so that the baseline reflects the configuration its authors recommend rather than one we picked. This learning rate is the standard choice in recent on-policy distillation work (Ye et al., 2025; Zhong et al., 2026; Xu et al., 2026; Xin et al., 2026; Hao et al., 2026; He et al., 2026). OSD's learning rate follows a simple rule: its objective is ordinary token-level cross-entropy on teacher-written text, the same objective as the Teacher-SFT baseline, so it shares that baseline's learning rate. The two methods optimise quite different losses, a divergence to the teacher's distribution on the student's own samples versus cross-entropy on text the teacher wrote, so their learning rates are not directly comparable, and small step sizes are common for on-policy objectives of the former kind. We have included a detailed comparison across OPD learning rates in the manuscript revision (Appendix D).

References

  • Hao, Dongze, et al. "Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization." arXiv preprint arXiv:2606.09091 (2026).
  • Ye, Tianzhu, et al. "Black-box on-policy distillation of large language models." arXiv preprint arXiv:2511.10643 (2025).
  • Li, Yaxuan, et al. "Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe." arXiv preprint arXiv:2604.13016 (2026).
  • Xin, Haoran, et al. "Escaping the KL Agreement Trap in On-Policy Distillation." arXiv preprint arXiv:2606.09471 (2026).
  • Zhong, Qiyong, et al. "SOD: Step-wise on-policy distillation for small language model agents." arXiv preprint arXiv:2605.07725 (2026).
  • Xu, Yuanda, et al. "Beyond GRPO and on-policy distillation: An empirical sparse-to-dense reward principle for language-model post-training." arXiv preprint arXiv:2605.12483 (2026).
  • He, Qiangqiang, Jin Li, and MingCai Chen. "Calibrating Teacher--Student Discrepancy for On-Policy Distillation." arXiv preprint arXiv:2609.21619 (2026).

[Part 3/3]

W3 / Q3. "Enlarge the RLVE test set and report multiple seeds, uncertainty intervals"

Following the suggestion, we re-evaluated every Table 1 model on an enlarged RLVE test set (the original problems plus new problems from the same environments and difficulty). Each model is sampled 8 times per problem; the 95% CIs come from a problem-level bootstrap, so they reflect both problem and sampling variation. We have included these results in the revised manuscript (Appendix C.4). Qwen3-4B, %:

avg@895% CI (avg@8)pass@8
Original16.77[14.8, 18.7]41.7
Teacher-Gen.18.09[16.0, 20.2]47.5
OPD18.54[16.4, 20.7]41.9
OSD22.96[20.8, 25.1]50.8

Following the reviewer's suggestion, we also present a detailed breakdown across representative environments (avg@8, %), which shows that the gain is not driven by a single environment; we have also included this comparison in the revised paper (Appendix C.4):

EnvironmentOriginalTeacher-Gen.OPDOSD
CoinSquareGame7.518.14.322.5
Patrol34.427.518.153.1
SubarraySumXor48.129.423.155.0
DoubleCrossCounting1.97.55.010.6

W4 / W5. "The claim of consistent superiority exceeds the reported results" and "the conditions for effective intervention remain unclear"

Thank you for these points. By "consistently outperforms", we meant that OSD performs best overall across task families and student sizes and never falls behind a baseline beyond noise, not that it wins every single comparison; we have clarified this in the revision, as detailed at the end of this response.

The problem we study is learning from teacher continuations placed at the student's own states. OSD does not assume that a failed prefix can always be rescued: the teacher shows what it would do from the state the student reached, and the student learns from it whether or not the continuation succeeds. In a case study on a subset of RLVE problems, the fraction of failed rollouts that the teacher recovers grows with its budget K:

teacher budget K7,1688,1929,21610,24011,264
recovered4.2%5.6%8.7%11.7%14.7%

What makes the intervention effective is that it keeps teaching the student as it changes, which our analysis section examines: OSD learns more while forgetting less than its offline counterparts (Sec. 5.1, Fig. 6), keeps improving where offline distillation plateaus (Sec. 5.2, Fig. 7), and preserves the student's plasticity (Sec. 5.2, Fig. 8). The same holds from a stronger student: starting from Qwen3-1.7B fine-tuned for one epoch on DeepSeek-V4.1-Flash trajectories, OSD keeps improving while continued SFT does not (ScienceWorld avg success rate, %):

epoch1 (shared start)234
+ continued SFT25.1220.5021.3827.25
+ OSD25.1234.8839.1239.12

The main tables therefore show that OSD is a strong distillation method rather than one that wins every single comparison: OSD has the best average SR and ranks first on ALFWorld, ScienceWorld, TextCraft and SearchQA; on BabyAI, 67.50 vs 67.78 corresponds to 243 vs 244 of 360 episodes, a single-episode difference within the noise of one evaluation run; on 4B RLVE, OSD leads every baseline on both avg@8 and pass@8 on the enlarged test set (W3). We have revised the corresponding sentence in the manuscript (Sec. 4) accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.