acceptodds
Under review as a conference paper at ICLR 2027

On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

Abstract

Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) to consolidate capabilities from multiple teachers, but near-perfect in-distribution (IND) distillation can mask incomplete capability consolidation beyond the supervised distribution. A preliminary synthetic reasoning study shows incomplete transfer of teacher generalization. We then experiment on real-world extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers’ mean IND scores, yet their mean out-of-distribution (OOD) scores remain 2.45-16.25% below the base. Across three compositional settings, MOPD improves mean IND performance by 0.48-22.46% over the base but remains 18.19-24.34% below the best teachers on composed tasks. We analyse teacher capability and data coverage as two potential bottlenecks: jointly easing them eliminates OOD forgetting in our experiments and shows potential for generalization beyond the teacher. Moreover, relative to their RL baselines, OPD and MOPD reduce OOD forgetting on average. Geometry analysis finds more compact OPD and MOPD updates on average than those of their RL baselines, along with greater shared update structure under MOPD. Based on the analysis, we propose geometry-based patching and base-preserving replay to improve OOD performance while retaining most IND performance. Our findings identify a blind spot in capability consolidation via MOPD and motivate distillation algorithms that explicitly support OOD retention and generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Multi-teacher distillation, LLM post-training, Capability consolidation, Out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_krbJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates the in-distribution (IND) performance and out-of-distribution (OOD) performance gaps of (Multi-teacher) On-Policy Distillation ((M)OPD) across various tasks. The paper further compares these results under different teacher configurations, including base models and RL-trained models. Additionally, the authors propose two methods, "Geometry-Based Patching" and "Base-Preserving Replay," to mitigate OOD performance degradation.

Critical strengths

This paper designs experiments to investigate OOD generalization by evaluating IND and OOD performance within the same domain and across combinations of different domains. The study is particularly interesting because it examines these phenomena across diverse domains, as well as under same-size and strong-to-weak distillation settings. Furthermore, the authors propose two methods to mitigate OOD performance degradation.

Critical weaknesses
  • Weak Results. Figure 2 generally shows high performance on IND tasks but lower performance on OOD tasks. However, Figure 4 shows that models trained on MATH achieve higher performance on GSM8K than those trained on GSM8K itself, while models trained on GSM8K perform better on MATH500 than those trained on MATH. Furthermore, additional analysis is needed to explain why performance on OOD tasks drops significantly below that of the student base model, as shown in Figures 2–4. It would have been more convincing if the authors had provided evidence that this degradation is primarily attributable to the OOD nature of the tasks, rather than to other factors in the training process. Regarding Figures 5 and 6, performance sometimes falls below that of the student base model even when MOPD is applied, which appears to weaken the authors' central claims.
  • Weak Discussion. In Section 6.1, the authors state that data coverage is expanded using the OpenR1 dataset. However, the definition of "data coverage" and its role, particularly whether it effectively turns previously OOD tasks into IND tasks, remain unclear. Furthermore, Figure 9 reports AIME24 and AIME25 results using "pass" (pass@8), whereas Figure 4 uses "avg." This inconsistency makes it difficult to assess the actual impact of incorporating OpenR1 on OOD performance degradation. In particular, reporting pass@8 may obscure performance degradation (Figure 9 reports OPD has better accuracy than student-base in both cases) that would otherwise be visible under average accuracy, making it appear as though OOD performance does not degrade at all.
Questions for authors
  • OOD performance degradation. In Figures 2–4, why does student performance on OOD tasks drop significantly below that of the student base model? Could the authors provide further analysis demonstrating that this degradation is primarily caused by OOD generalization issues rather than other factors in the training process?

  • Data coverage in Section 6.1. What exactly does increasing data coverage using OpenR1 mean in this experimental setting? Does incorporating this additional data effectively turn previously OOD tasks into IND tasks? How is the IND/OOD distinction defined after incorporating OpenR1?

  • Inconsistent AIME reporting. Why are AIME24 and AIME25 results reported using "avg" in Figure 4 but "pass" in Figure 9? Since pass@8 may obscure performance degradation observed under average accuracy, could the authors report the same metric in both figures to clarify whether OpenR1 actually mitigates OOD performance degradation?

  • Section 6, lines 399–403. The authors state that the "largest singular direction ... decreases from 19.0 for RL to 6.90 for OPD." Where are these values reported in the table, and how are they computed?

  • Section 6, lines 409–410. The authors state that the corresponding quantity accounts for "11.0–18.6% of MOPD's squared update norm and less than 0.240% of mixed RL's (Table 1)." Could the authors clarify how these percentages are calculated from the values reported in Table 1?

  • Section 3, citation formatting. In the Settings paragraph of Section 3, "Following" appears to be incorrectly included as part of a citation. Could the authors correct this formatting issue?

5 comments
AuthorsOpenReview ↗

Response to Reviewer krbJ [1/2]

We thank all reviewers for their valuable feedback and recognition of our “timely, practical” (M7V5) and “interesting and important” research question (dbDM), “clear framework” (M7V5), broad experimental coverage (M7V5, dbDM, krbJ), “well-designed and particularly interesting” compositional experiments (dbDM), and “interesting” geometry analysis (dbDM). We address their questions and concerns below.


Re-W1: Consistent evidence of generalization limitation across seeds and settings\color{purple}{\textbf{Re-W1: Consistent evidence of generalization limitation across seeds and settings}}

Our central finding is that near-perfect in-distribution distillation does not reliably ensure broader generalization in (M)OPD. To assess robustness, we repeated experiments with three independent training seeds. We report the results below and will update the corresponding figures and numerical summaries in the paper.

  • The synthetic transfer gap persists with longer training. In the preliminary synthetic reasoning experiment, the IND–OOD gain-transfer gap persists across seeds when the training budget is extended to 2T and 4T, where T approximately marks the plateau in IND performance. (Update in Fig. 3 and Sec. 1.)
OPD training stepsIND gain transferOOD gain transfer
Original budget T95.2% ± 0.4%82.8% ± 0.6%
2T94.8% ± 0.3%84.4% ± 0.5%
4T94.9% ± 0.5%84.5% ± 0.7%
  • OOD forgetting persists with self-RL specialists. Across the three training datasets, OPD reaches 97.25% ± 0.78% of its teachers’ IND scores, while its OOD performance remains 10.78% ± 3.52% below the base. (Update in Fig. 2 and Sec. 4.)

  • A stronger teacher does not reliably ensure OOD improvement. With an external generalist teacher, OPD reaches 99.12% ± 0.36% of the teacher’s IND score, while its OOD performance is 2.53% ± 5.48% below the base on average. (Update in Fig. 4 and Sec. 4.)

  • Strong component distillation coexists with a composed-task gap. In long-context coding, MOPD reaches 96.22% ± 2.84% of the teachers’ IND performance, but only 80.95% ± 7.17% of the best teacher’s composed-task performance. (Update in Fig. 5 and Sec. 5.)

Overall, the multi-seed results support our central finding across the tested settings: near-perfect IND distillation does not reliably ensure broader generalization.


Re-W1: Consistent evidence of improvement from patching methods\color{purple}{\textbf{Re-W1: Consistent evidence of improvement from patching methods}}

We extend geometry-based patching and base-preserving replay to additional generalization settings.

  • Geometry-based patching improves compositional generalization. On long-context coding, patching improves MOPD's composed-task (OOD) performance from 44% to 56%, exceeding the base and code-only OPD models by 12 and 14 percentage points, respectively. Component-task (IND) performance also improves slightly:HumanEval, HumanEval+, and LongBench increase by 1.22, 0.61, and 0.83 percentage points, respectively.

  • Geometry-based patching also improves extrapolative generalization. With MATH as the training dataset, patching improves OOD performance over plain OPD by 1.25 percentage points. IND performance also slightly exceeds that of both the base and the best teacher.

  • Geometry-based patching outperforms random perturbations. Compared with random perturbations at the same perturbation strength, patching achieves average gains of 1.07 percentage points on IND and 6.32 percentage points on OOD evaluations.

  • Base-preserving replay improves compositional generalization. On long-context coding, we test two replay datasets: math (GSM8K) and instruction-following (Nemotron-RL IF subset). GSM8K replay improves MOPD's composed-task performance from 44% to 58%, with a 0.62-percentage-point decrease in mean IND performance. Nemotron-RL instruction-following replay also improves composed-task performance to 56%, with a 1.64-percentage-point decrease in mean IND performance.

We will include these results and their evaluation protocols in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer krbJ [2/2]

Re-W1/Q1: OOD Gaps Despite Successful IND Distillation\color{purple}{\textbf{Re-W1/Q1: OOD Gaps Despite Successful IND Distillation}}

  • Cross-benchmark rankings do not contradict our finding. Training on MATH can yield better GSM8K performance than training on GSM8K, and vice versa. Our conclusion does not require each training source to produce the best score on its corresponding benchmark. Successful source-task fitting can still coexist with incomplete cross-benchmark retention. The multi-seed results in our response [1/2] further support this observation.

  • Below-base composed-task results illustrate the limitation we study. Our central finding is that learning individual capabilities does not reliably improve their combination. Thus, the below-base results in Figs. 5–6 are consistent with this finding. We also retain the positive transfer results.

  • We distinguish observed degradation from its causes. We do not claim that OOD status itself causes degradation. Teacher-side forgetting, supervision coverage, and training dynamics may contribute. The external-teacher experiments address teacher-side forgetting, while the coverage experiment examines broader supervision. These comparisons suggest these are valid factors, but we agree it does not fully isolate a causal mechanism.


Re-W2/Q2/Q3: Broader Supervision and Consistent AIME Reporting\color{purple}{\textbf{Re-W2/Q2/Q3: Broader Supervision and Consistent AIME Reporting}}

  • The coverage experiment examines the role of supervision breadth. We replace GSM8K-only distillation with a mixture of 90% GSM8K and 10% OpenR1, keeping Qwen3-32B as the teacher. The purpose is to examine supervision coverage as a contributing factor to the generalization gap.

  • Expanded supervision may cover previously OOD behaviors. The purpose of this experiment is to examine data coverage as a contributing factor to the observed gap, rather than present broader supervision as a general solution to the problem. Although evaluation examples remain held out, some previously OOD problem types may fall within the support of the expanded training distribution. The IND/OOD labels in this comparison refer to the original GSM8K-only setting. We do not establish that these benchmarks remain OOD after adding OpenR1. We will clarify that in the paper.

  • AIME reporting is now consistent. We have updated Fig. 9 to report AIME avg@8 consistently with Fig. 4. Broader supervision raises AIME24/25 scores from 70.83%/64.17% to 78.33%/73.33%, exceeding the base on both and supporting the revised claim that broader supervision improves held-out performance.


Re-Q4/Q5: Stable Rank and Explained Update Energy\color{purple}{\textbf{Re-Q4/Q5: Stable Rank and Explained Update Energy}}

  • The values 19.0 and 6.90 are stable ranks. They do not denote the largest singular value (Table 1) itself. Stable rank is computed as
=\frac{\|\Delta W\|_F^2}{\|\Delta W\|_2^2} =\frac{\sum_i \sigma_i^2}{\sigma_1^2}.$$ These values are reported in the text. Table 1’s left panel instead reports update norms and top-64 energy fractions. Update norms measure magnitude, while stable rank and top-64 energy capture spectral concentration from complementary perspectives. We will make these distinctions explicit. - **Table 1 reports group means, while the text reports per-setting ranges.** Multiplying Table 1’s $R^2$ values by 100 gives MOPD means of 15.7% and 12.1% for the extrapolative and compositional groups, versus 0.173% and 0.0245% for mixed RL. The text reports the minimum–maximum range across individual settings for MOPD (11.0–18.6%) and an upper bound for mixed RL (<0.240%). We will clarify these aggregation levels in the revised paper. --- $$\color{purple}{\textbf{Re-Q6: Citation Formatting Corrected}}$$ We have corrected the citation formatting in Section 3. Thank you for pointing this out!

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

However, after revisiting the manuscript and the authors' responses, I still have concerns about how IND and OOD evaluations are defined within the same domain. In particular, the observation that MATH-trained models outperform GSM8K-trained models on GSM8K, and vice versa on MATH500, raises questions about the appropriateness of the current IND/OOD categorization. I believe the authors should provide further analysis to clarify and justify how these benchmarks are categorized as IND or OOD.

Additionally, although the authors state that Figure 9 has been updated to report avg@8, this change does not currently appear to be reflected in the revised manuscript.

While I appreciate the additional results and clarifications, my main concerns remain only partially addressed. Therefore, I decide to maintain my original rating.

Reviewer_dbDMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that (M)OPD can perfectly teach the student the in-distribution capabilities that it's distilling on without transferring any of the generalization capabilities of the teacher. In particular, they look at two types of generalization, (1) extrapolative (solving harder tasks) and (2) compositional (solving tasks that require more than one taught skill). They show that even when the students achieve near-perfect in-distribution accuracy relative to their teachers, they can still lose OOD performance, or fail to generalize to tasks that require multiple tasks. Their experiments cover a range of tasks, including math, coding, agentic, and long-context. They also analyze the geometry of model updates and explore two approaches, geometry-based patching and base-preserving replay, to mitigate OOD degradation.

Critical strengths
  • They study an interesting and important question about whether on-policy distillation actually transfers the generalization capabilities of the teacher or merely its in-domain performance.

  • The compositional generalization experiments (Section 5) are well-designed and particularly interesting to me. By testing across different combinations of important tasks, they show that while (M)OPD can yield high in-distribution gains, it falls short on composed tasks.

  • The analysis of model update geometry is interesting. The proposed potential solutions through patching and replay are well-motivated and show promising gains.

Critical weaknesses
  • It's unclear how much of the observed generalization gap is specific to OPD/MOPD rather than a general limitation of training on narrow distributions. In fact, mixed RL also suffers from similar compositional generalization issues, and Fig. 9 shows that adding more diverse training data can eliminate OOD forgetting. Better isolation of the factors responsible for these gaps would strengthen the paper.

  • The definition of IND and OOD in the main mathematical experiments is somewhat concerning. In Appendix D.1, they classify benchmarks based on the teachers' performance after training, rather than their actual distribution relative to the training data. Therefore, I'm not sure how much of the reported OOD degradation is influenced by the evaluation setup. Also, the main experiments are limited to Qwen3-8B students, with one seed per training run.

Questions for authors
  • Can you provide more evidence for the connection between model update geometry and OOD generalization? For example, have you considered comparing against simply scaling down the parameter updates or using random-direction controls to understand whether the gains from patching come from the proposed geometric structure?

  • Have you tried evaluating the patching and replay methods across more tasks and model sizes? The current results look promising, but it's unclear how general the improvements are.

  • Please see the points mentioned above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer dbDM [1/2]

We thank all reviewers for their valuable feedback and recognition of our “timely, practical” (M7V5) and “interesting and important” research question (dbDM), “clear framework” (M7V5), broad experimental coverage (M7V5, dbDM, krbJ), “well-designed and particularly interesting” compositional experiments (dbDM), and “interesting” geometry analysis (dbDM). We address their questions and concerns below.


Re-Q2: Consistent evidence of generalization limitation across seeds and settings\color{purple}{\textbf{Re-Q2: Consistent evidence of generalization limitation across seeds and settings}}

Our central finding is that near-perfect in-distribution distillation does not reliably ensure broader generalization in (M)OPD. To assess robustness, we repeated experiments with three independent training seeds. We report the results below and will update the corresponding figures and numerical summaries in the paper.

  • The synthetic transfer gap persists with longer training. In the preliminary synthetic reasoning experiment, the IND–OOD gain-transfer gap persists across seeds when the training budget is extended to 2T and 4T, where T approximately marks the plateau in IND performance. (Update in Fig. 3 and Sec. 1.)
OPD training stepsIND gain transferOOD gain transfer
Original budget T95.2% ± 0.4%82.8% ± 0.6%
2T94.8% ± 0.3%84.4% ± 0.5%
4T94.9% ± 0.5%84.5% ± 0.7%
  • OOD forgetting persists with self-RL specialists. Across the three training datasets, OPD reaches 97.25% ± 0.78% of its teachers’ IND scores, while its OOD performance remains 10.78% ± 3.52% below the base. (Update in Fig. 2 and Sec. 4.)

  • A stronger teacher does not reliably ensure OOD improvement. With an external generalist teacher, OPD reaches 99.12% ± 0.36% of the teacher’s IND score, while its OOD performance is 2.53% ± 5.48% below the base on average. (Update in Fig. 4 and Sec. 4.)

  • Strong component distillation coexists with a composed-task gap. In long-context coding, MOPD reaches 96.22% ± 2.84% of the teachers’ IND performance, but only 80.95% ± 7.17% of the best teacher’s composed-task performance. (Update in Fig. 5 and Sec. 5.)

Overall, the multi-seed results support our central finding across the tested settings: near-perfect IND distillation does not reliably ensure broader generalization.


Re-W2: Consistent evidence of improvement from patching methods\color{purple}{\textbf{Re-W2: Consistent evidence of improvement from patching methods}}

We extend geometry-based patching and base-preserving replay to additional generalization settings.

  • Geometry-based patching improves compositional generalization. On long-context coding, patching improves MOPD's composed-task (OOD) performance from 44% to 56%, exceeding the base and code-only OPD models by 12 and 14 percentage points, respectively. Component-task (IND) performance also improves slightly:HumanEval, HumanEval+, and LongBench increase by 1.22, 0.61, and 0.83 percentage points, respectively.

  • Geometry-based patching also improves extrapolative generalization. With MATH as the training dataset, patching improves OOD performance over plain OPD by 1.25 percentage points. IND performance also slightly exceeds that of both the base and the best teacher.

  • Geometry-based patching outperforms random perturbations. Compared with random perturbations at the same perturbation strength, patching achieves average gains of 1.07 percentage points on IND and 6.32 percentage points on OOD evaluations.

  • Base-preserving replay improves compositional generalization. On long-context coding, we test two replay datasets: math (GSM8K) and instruction-following (Nemotron-RL IF subset). GSM8K replay improves MOPD's composed-task performance from 44% to 58%, with a 0.62-percentage-point decrease in mean IND performance. Nemotron-RL instruction-following replay also improves composed-task performance to 56%, with a 1.64-percentage-point decrease in mean IND performance.

We will include these results and their evaluation protocols in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer dbDM [2/2]

Re-W2: Generalization limitation is Robustness to an Alternative IND/OOD Definition\color{purple}{\textbf{Re-W2: Generalization limitation is Robustness to an Alternative IND/OOD Definition}}

Our paper examines generalization across different types, including extrapolative across complexity (Section 3), extrapolative across difficulty (Section 4), and compositional across capabilities (Section 5). The performance-based IND/OOD partition concerns the aggregates on difficulties in Section 4.

  • Difficulty extrapolation has clear precedent. Prior work studies GSM/MATH-to-AIME transfer as easy-to-hard or OOD generalization [1,2,3]. These precedents motivate our evaluations, while the analysis below tests whether our conclusions depend on the original grouping.

  • The key finding survives in an alternative, source-based partition. We additionally treat held-out evaluations corresponding to the training sources as IND and other benchmarks as cross-benchmark evaluations. For example, GSM8K test is IND for GSM8K training, and MATH500 is IND for MATH training. All self-RL OPD configurations and all mathematical MOPD mixtures remain below the base on their cross-benchmark means, with MOPD losses of 7.49-12.91 percentage points. In the external-teacher setting, the student is within 0.84 points of the teacher on IND, yet remains 4.17-1.66 points below the base on OOD.

These results show that successful source-task fitting can coexist with incomplete cross-benchmark retention, independently of the original teacher-performance-based grouping. We will distinguish this source-based analysis from difficulty extrapolation and clarify the evaluation definitions throughout.

[1] Nishad Singhi et al. When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning. COLM, 2025.

[2] Ziyu Wan et al. ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning. NeurIPS, 2025.

[3] Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, and Jinwoo Shin. Beyond Correctness: Learning Robust Reasoning via Transfer. arXiv preprint arXiv:2602.08489, 2026.


Re-Q1: Empirical findings and the scope of the geometry analysis\color{purple}{\textbf{Re-Q1: Empirical findings and the scope of the geometry analysis}}

Our main contribution is the empirical finding that near-perfect IND distillation does not reliably ensure broader generalization. The geometry analysis offers a potential explanation, but we have not established a causal connection.

Patching outperforms the selected weight-interpolation baseline on both IND and OOD performance (93.48%/65.00% versus 93.10%/62.92%). We also report the random-direction control results in Re-W2, which further examine whether the gains depend on the proposed geometric structure.


Re-W1: Generalization Gaps Beyond Successful IND Distillation\color{purple}{\textbf{Re-W1: Generalization Gaps Beyond Successful IND Distillation}}

We agree that lack of generalization is not a unique problem to OPD/MOPD. Our central finding is that strong performance on directly supervised tasks does not reliably certify broader capability consolidation, which we believe is an overlooked blind spot.

  1. Incomplete transfer of a teacher's generalization. Section 3 demonstrates this in a controlled setting where the teacher and student share the same base and post-training data. Section 4 further shows that OOD gaps persist even with a stronger external teacher on practical tasks.
  2. Component gains without consistent composed-task gains. Section 5 shows that MOPD can improve individual capabilities while leaving composed-task performance unchanged or degraded relative to the base. Mixed RL also exhibits this gap, so we do not claim it is unique to MOPD.

Broader data, as explored in Fig. 9, changes the support of supervision and may cover previously OOD behaviors. It therefore helps characterize the role of coverage, but does not establish transfer beyond the expanded training distribution. We will clarify that in our paper.

I thank the authors for the additional experiments, in particular, multiple seeds/extended runs, and consistent improvement from patching methods. However, I still have concerns regarding the contributions of the paper. Based on the answer to Q1, it seems that the contribution claim of the paper is stated to be on the IND-OOD gap in (M)OPD and not the causality of the geometric discussion. Moreover, I believe the lack of generalization in their setting is somewhat expected. Basically, when the questions covered in the training set have a certain type and cap of difficulty, there's no reason to believe the model needs to learn the capabilities of the teacher on harder or composed problems. Essentially, I think this is more of a data-coverage problem rather than a fundamental limitation of (M)OPD as stated in the paper. As a result, I keep my score.

AuthorsOpenReview ↗

Respectful Disagreement to Reviewer dbDM

Thank you for acknowledging the additional experiments and patching results. We respectfully disagree that limited data coverage makes our findings expected or fully explains them.

  • No generalization guarantee does not make the observed failures self-evident. We study whether (M)OPD transfers capabilities its teachers already possess (Sec 3, 4.2). Even with a stronger external teacher, near-perfect IND matching does not reliably yield OOD improvement. Likewise, component-task gains can coexist with unchanged or degraded composed-task performance. Limited coverage permits these outcomes, but does not establish that they should be expected.

  • Data coverage alone does not account for the intervention results. Without expanding the distillation data, geometry-based patching raises composed-task performance from 44% to 56%, with a marginal decrease in IND performance. This does not isolate the original cause, but shows that capability retention also depends on how model updates are constructed. Broader coverage is a contributing factor, not an established complete explanation.

  • Establishing a causal geometric mechanism is not a prerequisite for our empirical contribution. Our experiments establish that near-perfect IND distillation does not reliably ensure broader capability consolidation across the tested settings. The geometric analysis offers a potential explanation, and patching demonstrates its practical utility. The remaining uncertainty about the causal mechanism does not weaken the evidence establishing the generalization gaps.

We invite the reviewer to reconsider their assessment in light of this evidence and the distinction between establishing an empirical finding and fully explaining its mechanism.

Reviewer_M7V5OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether on-policy distillation (OPD) and multi-teacher OPD (MOPD) preserve capabilities beyond the training distribution. Students nearly match their teachers in-distribution but fall below the base model on OOD benchmarks and fall short of the teachers on composed tasks. The authors also propose patching and replay as mitigations.

Critical strengths

The paper asks a timely, practical question and answers it with a clear framework. It shows across several domains that near-perfect in-distribution distillation can hide incomplete consolidation.

Critical weaknesses

The evidence for OOD forgetting is weak. The IND/OOD split depends on how the teachers performed, so it is partly circular, and the criterion is stated inconsistently. Every result comes from a single run, and many of the gaps are small.

Questions for authors
  1. Could you state a single, consistent IND/OOD criterion? Do the conclusions still hold under a partition that does not depend on the teachers, for example, one based on difficulty or problem source?

  2. Could you report results averaged over several training seeds, with variance and confidence intervals for the key comparisons? How was the patching strength chosen, and was any test data used to choose it?

  3. Do patching and replay help beyond the single math setting shown?

  4. How do the mitigations compare with simple baselines such as weight interpolation toward the base or an explicit KL penalty to the base model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer M7V5 [1/2]

We thank all reviewers for their valuable feedback and recognition of our “timely, practical” (M7V5) and “interesting and important” research question (dbDM), “clear framework” (M7V5), broad experimental coverage (M7V5, dbDM, krbJ), “well-designed and particularly interesting” compositional experiments (dbDM), and “interesting” geometry analysis (dbDM). We address their questions and concerns below.


Re-Q2: Consistent evidence of generalization limitation across seeds and settings\color{purple}{\textbf{Re-Q2: Consistent evidence of generalization limitation across seeds and settings}}

Our central finding is that near-perfect in-distribution distillation does not reliably ensure broader generalization in (M)OPD. To assess robustness, we repeated experiments with three independent training seeds. We report the results below and will update the corresponding figures and numerical summaries in the paper.

  • The synthetic transfer gap persists with longer training. In the preliminary synthetic reasoning experiment, the IND–OOD gain-transfer gap persists across seeds when the training budget is extended to 2T and 4T, where T approximately marks the plateau in IND performance. (Update in Fig. 3 and Sec. 1.)
OPD training stepsIND gain transferOOD gain transfer
Original budget T95.2% ± 0.4%82.8% ± 0.6%
2T94.8% ± 0.3%84.4% ± 0.5%
4T94.9% ± 0.5%84.5% ± 0.7%
  • OOD forgetting persists with self-RL specialists. Across the three training datasets, OPD reaches 97.25% ± 0.78% of its teachers’ IND scores, while its OOD performance remains 10.78% ± 3.52% below the base. (Update in Fig. 2 and Sec. 4.)

  • A stronger teacher does not reliably ensure OOD improvement. With an external generalist teacher, OPD reaches 99.12% ± 0.36% of the teacher’s IND score, while its OOD performance is 2.53% ± 5.48% below the base on average. (Update in Fig. 4 and Sec. 4.)

  • Strong component distillation coexists with a composed-task gap. In long-context coding, MOPD reaches 96.22% ± 2.84% of the teachers’ IND performance, but only 80.95% ± 7.17% of the best teacher’s composed-task performance. (Update in Fig. 5 and Sec. 5.)

Overall, the multi-seed results support our central finding across the tested settings: near-perfect IND distillation does not reliably ensure broader generalization.

For patching methods, in practice, we randomly sample a subset of the training dataset and check the accuracy to select the best α\alpha. We don’t use a test set.


Re-Q3: Consistent evidence of improvement from patching methods\color{purple}{\textbf{Re-Q3: Consistent evidence of improvement from patching methods}}

We extend geometry-based patching and base-preserving replay to additional generalization settings.

  • Geometry-based patching improves compositional generalization. On long-context coding, patching improves MOPD's composed-task (OOD) performance from 44% to 56%, exceeding the base and code-only OPD models by 12 and 14 percentage points, respectively. Component-task (IND) performance also improves slightly:HumanEval, HumanEval+, and LongBench increase by 1.22, 0.61, and 0.83 percentage points, respectively.

  • Geometry-based patching also improves extrapolative generalization. With MATH as the training dataset, patching improves OOD performance over plain OPD by 1.25 percentage points. IND performance also slightly exceeds that of both the base and the best teacher.

  • Geometry-based patching outperforms random perturbations. Compared with random perturbations at the same perturbation strength, patching achieves average gains of 1.07 percentage points on IND and 6.32 percentage points on OOD evaluations.

  • Base-preserving replay improves compositional generalization. On long-context coding, we test two replay datasets: math (GSM8K) and instruction-following (Nemotron-RL IF subset). GSM8K replay improves MOPD's composed-task performance from 44% to 58%, with a 0.62-percentage-point decrease in mean IND performance. Nemotron-RL instruction-following replay also improves composed-task performance to 56%, with a 1.64-percentage-point decrease in mean IND performance.

We will include these results and their evaluation protocols in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer M7V5 [2/2]

Re-W1,Q1: Generalization limitation is Robustness to an Alternative IND/OOD Definition\color{purple}{\textbf{Re-W1,Q1: Generalization limitation is Robustness to an Alternative IND/OOD Definition}}

Our paper examines generalization across different types, including extrapolative across complexity (Section 3), extrapolative across difficulty (Section 4), and compositional across capabilities (Section 5). The performance-based IND/OOD partition concerns the aggregates on difficulties in Section 4.

  • Difficulty extrapolation has clear precedent. Prior work studies GSM/MATH-to-AIME transfer as easy-to-hard or OOD generalization [1,2,3]. These precedents motivate our evaluations, while the analysis below tests whether our conclusions depend on the original grouping.

  • The key finding survives in an alternative, source-based partition. We additionally treat held-out evaluations corresponding to the training sources as IND and other benchmarks as cross-benchmark evaluations. For example, GSM8K test is IND for GSM8K training, and MATH500 is IND for MATH training. All self-RL OPD configurations and all mathematical MOPD mixtures remain below the base on their cross-benchmark means, with MOPD losses of 7.49-12.91 percentage points. In the external-teacher setting, the student is within 0.84 points of the teacher on IND, yet remains 4.17-1.66 points below the base on OOD.

These results show that successful source-task fitting can coexist with incomplete cross-benchmark retention, independently of the original teacher-performance-based grouping. We will distinguish this source-based analysis from difficulty extrapolation and clarify the evaluation definitions throughout.

[1] Nishad Singhi et al. When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning. COLM, 2025.

[2] Ziyu Wan et al. ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning. NeurIPS, 2025.

[3] Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, and Jinwoo Shin. Beyond Correctness: Learning Robust Reasoning via Transfer. arXiv preprint arXiv:2602.08489, 2026.


Re-Q4: Our mitigations are better on IND performance and OOD retention\color{purple}{\textbf{Re-Q4: Our mitigations are better on IND performance and OOD retention}}

  • Patching outperforms the selected weight-interpolation baseline on both IND and OOD. In the setting of Fig. 10, the best interpolation strength, selected by the mean IND and OOD score, achieves 93.10% IND and 62.92% OOD accuracy. Our patching method achieves 93.48% and 65.00%, respectively, providing a better trade-off at these operating points.

  • Explicit base-KL regularization reduced IND performance. We interpret this baseline as adding a reverse-KL penalty toward the base model on the same student-generated histories used for distillation. In our preliminary experiments, this regularization reduced IND performance relative to plain OPD. At a fixed history hh, the combined objective satisfies DKL(pθ∥qT)+λDKL(pθ∥qB)=(1+λ)DKL(pθ∥qmix)+C(h),D_{\mathrm{KL}}(p_\theta\|q_T) +\lambda D_{\mathrm{KL}}(p_\theta\|q_B) =(1+\lambda)D_{\mathrm{KL}}(p_\theta\|q_{\mathrm{mix}})+C(h), where qmix(a∣h)=qT(a∣h)1/(1+λ)qB(a∣h)λ/(1+λ)Z(h),q_{\mathrm{mix}}(a\mid h)=\frac{q_T(a\mid h)^{1/(1+\lambda)}q_B(a\mid h)^{\lambda/(1+\lambda)}}{Z(h)}, and C(h)=−(1+λ)log⁡Z(h)C(h)=-(1+\lambda)\log Z(h) is independent of the student distribution at that history. Thus, the base penalty modifies the effective distillation target on teacher-training prompts. This motivates our base-preserving replay approach, which uses teacher supervision on target-task prompts and base-model supervision on a separate replay distribution.

My major concerns have been addressed; I decided to raise my score to weak accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and a Warm Invitation to Read Our Responses

Hi Reviewers,

First of all, we sincerely appreciate the effort you have devoted to reviewing our work.

As a gentle reminder, the ICLR discussion period closes in approximately 30 hours. We would be grateful if you could review our responses and share any further thoughts. We would be more than happy to discuss. We've also updated the manuscript accordingly, with revisions marked in blue.

Thank you again.

Respectfully,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.