- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates the in-distribution (IND) performance and out-of-distribution (OOD) performance gaps of (Multi-teacher) On-Policy Distillation ((M)OPD) across various tasks. The paper further compares these results under different teacher configurations, including base models and RL-trained models. Additionally, the authors propose two methods, "Geometry-Based Patching" and "Base-Preserving Replay," to mitigate OOD performance degradation.
Critical strengths
This paper designs experiments to investigate OOD generalization by evaluating IND and OOD performance within the same domain and across combinations of different domains. The study is particularly interesting because it examines these phenomena across diverse domains, as well as under same-size and strong-to-weak distillation settings. Furthermore, the authors propose two methods to mitigate OOD performance degradation.
Critical weaknesses
- Weak Results. Figure 2 generally shows high performance on IND tasks but lower performance on OOD tasks. However, Figure 4 shows that models trained on MATH achieve higher performance on GSM8K than those trained on GSM8K itself, while models trained on GSM8K perform better on MATH500 than those trained on MATH. Furthermore, additional analysis is needed to explain why performance on OOD tasks drops significantly below that of the student base model, as shown in Figures 2–4. It would have been more convincing if the authors had provided evidence that this degradation is primarily attributable to the OOD nature of the tasks, rather than to other factors in the training process. Regarding Figures 5 and 6, performance sometimes falls below that of the student base model even when MOPD is applied, which appears to weaken the authors' central claims.
- Weak Discussion. In Section 6.1, the authors state that data coverage is expanded using the OpenR1 dataset. However, the definition of "data coverage" and its role, particularly whether it effectively turns previously OOD tasks into IND tasks, remain unclear. Furthermore, Figure 9 reports AIME24 and AIME25 results using "pass" (pass@8), whereas Figure 4 uses "avg." This inconsistency makes it difficult to assess the actual impact of incorporating OpenR1 on OOD performance degradation. In particular, reporting pass@8 may obscure performance degradation (Figure 9 reports OPD has better accuracy than student-base in both cases) that would otherwise be visible under average accuracy, making it appear as though OOD performance does not degrade at all.
Questions for authors
-
OOD performance degradation. In Figures 2–4, why does student performance on OOD tasks drop significantly below that of the student base model? Could the authors provide further analysis demonstrating that this degradation is primarily caused by OOD generalization issues rather than other factors in the training process?
-
Data coverage in Section 6.1. What exactly does increasing data coverage using OpenR1 mean in this experimental setting? Does incorporating this additional data effectively turn previously OOD tasks into IND tasks? How is the IND/OOD distinction defined after incorporating OpenR1?
-
Inconsistent AIME reporting. Why are AIME24 and AIME25 results reported using "avg" in Figure 4 but "pass" in Figure 9? Since pass@8 may obscure performance degradation observed under average accuracy, could the authors report the same metric in both figures to clarify whether OpenR1 actually mitigates OOD performance degradation?
-
Section 6, lines 399–403. The authors state that the "largest singular direction ... decreases from 19.0 for RL to 6.90 for OPD." Where are these values reported in the table, and how are they computed?
-
Section 6, lines 409–410. The authors state that the corresponding quantity accounts for "11.0–18.6% of MOPD's squared update norm and less than 0.240% of mixed RL's (Table 1)." Could the authors clarify how these percentages are calculated from the values reported in Table 1?
-
Section 3, citation formatting. In the Settings paragraph of Section 3, "Following" appears to be incorrectly included as part of a citation. Could the authors correct this formatting issue?
5 comments
Response to Reviewer krbJ [2/2]
-
Cross-benchmark rankings do not contradict our finding. Training on MATH can yield better GSM8K performance than training on GSM8K, and vice versa. Our conclusion does not require each training source to produce the best score on its corresponding benchmark. Successful source-task fitting can still coexist with incomplete cross-benchmark retention. The multi-seed results in our response [1/2] further support this observation.
-
Below-base composed-task results illustrate the limitation we study. Our central finding is that learning individual capabilities does not reliably improve their combination. Thus, the below-base results in Figs. 5–6 are consistent with this finding. We also retain the positive transfer results.
-
We distinguish observed degradation from its causes. We do not claim that OOD status itself causes degradation. Teacher-side forgetting, supervision coverage, and training dynamics may contribute. The external-teacher experiments address teacher-side forgetting, while the coverage experiment examines broader supervision. These comparisons suggest these are valid factors, but we agree it does not fully isolate a causal mechanism.
-
The coverage experiment examines the role of supervision breadth. We replace GSM8K-only distillation with a mixture of 90% GSM8K and 10% OpenR1, keeping Qwen3-32B as the teacher. The purpose is to examine supervision coverage as a contributing factor to the generalization gap.
-
Expanded supervision may cover previously OOD behaviors. The purpose of this experiment is to examine data coverage as a contributing factor to the observed gap, rather than present broader supervision as a general solution to the problem. Although evaluation examples remain held out, some previously OOD problem types may fall within the support of the expanded training distribution. The IND/OOD labels in this comparison refer to the original GSM8K-only setting. We do not establish that these benchmarks remain OOD after adding OpenR1. We will clarify that in the paper.
-
AIME reporting is now consistent. We have updated Fig. 9 to report AIME avg@8 consistently with Fig. 4. Broader supervision raises AIME24/25 scores from 70.83%/64.17% to 78.33%/73.33%, exceeding the base on both and supporting the revised claim that broader supervision improves held-out performance.
- The values 19.0 and 6.90 are stable ranks. They do not denote the largest singular value (Table 1) itself. Stable rank is computed as
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.
However, after revisiting the manuscript and the authors' responses, I still have concerns about how IND and OOD evaluations are defined within the same domain. In particular, the observation that MATH-trained models outperform GSM8K-trained models on GSM8K, and vice versa on MATH500, raises questions about the appropriateness of the current IND/OOD categorization. I believe the authors should provide further analysis to clarify and justify how these benchmarks are categorized as IND or OOD.
Additionally, although the authors state that Figure 9 has been updated to report avg@8, this change does not currently appear to be reflected in the revised manuscript.
While I appreciate the additional results and clarifications, my main concerns remain only partially addressed. Therefore, I decide to maintain my original rating.
Response to Reviewer krbJ [1/2]
We thank all reviewers for their valuable feedback and recognition of our “timely, practical” (M7V5) and “interesting and important” research question (dbDM), “clear framework” (M7V5), broad experimental coverage (M7V5, dbDM, krbJ), “well-designed and particularly interesting” compositional experiments (dbDM), and “interesting” geometry analysis (dbDM). We address their questions and concerns below.
Re-W1: Consistent evidence of generalization limitation across seeds and settings
Our central finding is that near-perfect in-distribution distillation does not reliably ensure broader generalization in (M)OPD. To assess robustness, we repeated experiments with three independent training seeds. We report the results below and will update the corresponding figures and numerical summaries in the paper.
OOD forgetting persists with self-RL specialists. Across the three training datasets, OPD reaches 97.25% ± 0.78% of its teachers’ IND scores, while its OOD performance remains 10.78% ± 3.52% below the base. (Update in Fig. 2 and Sec. 4.)
A stronger teacher does not reliably ensure OOD improvement. With an external generalist teacher, OPD reaches 99.12% ± 0.36% of the teacher’s IND score, while its OOD performance is 2.53% ± 5.48% below the base on average. (Update in Fig. 4 and Sec. 4.)
Strong component distillation coexists with a composed-task gap. In long-context coding, MOPD reaches 96.22% ± 2.84% of the teachers’ IND performance, but only 80.95% ± 7.17% of the best teacher’s composed-task performance. (Update in Fig. 5 and Sec. 5.)
Overall, the multi-seed results support our central finding across the tested settings: near-perfect IND distillation does not reliably ensure broader generalization.
Re-W1: Consistent evidence of improvement from patching methods
We extend geometry-based patching and base-preserving replay to additional generalization settings.
Geometry-based patching improves compositional generalization. On long-context coding, patching improves MOPD's composed-task (OOD) performance from 44% to 56%, exceeding the base and code-only OPD models by 12 and 14 percentage points, respectively. Component-task (IND) performance also improves slightly:HumanEval, HumanEval+, and LongBench increase by 1.22, 0.61, and 0.83 percentage points, respectively.
Geometry-based patching also improves extrapolative generalization. With MATH as the training dataset, patching improves OOD performance over plain OPD by 1.25 percentage points. IND performance also slightly exceeds that of both the base and the best teacher.
Geometry-based patching outperforms random perturbations. Compared with random perturbations at the same perturbation strength, patching achieves average gains of 1.07 percentage points on IND and 6.32 percentage points on OOD evaluations.
Base-preserving replay improves compositional generalization. On long-context coding, we test two replay datasets: math (GSM8K) and instruction-following (Nemotron-RL IF subset). GSM8K replay improves MOPD's composed-task performance from 44% to 58%, with a 0.62-percentage-point decrease in mean IND performance. Nemotron-RL instruction-following replay also improves composed-task performance to 56%, with a 1.64-percentage-point decrease in mean IND performance.
We will include these results and their evaluation protocols in the revised paper.