acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training

Abstract

Modern LLM post-training composes supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) into multi-stage pipelines, yet these stages are typically designed and evaluated in isolation. We show that this composition is consequential: a stage that improves the current model can make the next stage less effective. Through controlled experiments with Qwen3 models on math and science reasoning, we first characterize OPD across nine student–teacher pairs spanning 2x to 53x parameter ratios and show that OPD effectiveness depends on student–teacher compatibility rather than teacher scale alone. That surrounding stages reshape this compatibility in three ways: (1) A brief SFT warm-up improves subsequent OPD, while an RLVR-strengthened student regresses under distillation from the same teacher. (2) Adapting the teacher with RLVR raises downstream OPD accuracy in proportion to the capability it adds. Following these two interventions, we find that combining teacher adaptation and student warm-up alone raise average OPD accuracy from 29.2% to 43.8% (50% relative improvement) after the same number of distillation steps, with additional preparatory training. (3) At comparable accuracy, OPD leaves a stronger initialization for downstream RLVR than SFT, with a gap that widens as RL compute scales. Our results suggest each post-training stage should be chosen not only for the capability it adds, but for the learning interface it creates for the next stage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-Training, SFT, RLVR, OPD, Interplay, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SSG5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper empirically studies how supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) interact when composed in an LLM post-training pipeline. It first evaluates OPD across nine Qwen3 student-teacher size pairs and proposes the initial reinforceable fraction as a diagnostic of student-teacher compatibility. It then varies the state entering each stage and reports that a short SFT warm-up improves subsequent OPD, whereas an RLVR-strengthened student can regress under the same teacher; adapting the teacher with RLVR improves downstream OPD; and OPD provides a better initialization than SFT for subsequent RLVR at comparable starting accuracy. Combining teacher RLVR adaptation and student SFT warm-up raises the final OPD accuracy from 29.2% to 43.8% in the main Qwen3-4B/14B setting.

Critical strengths
  • The paper studies an important practical question that is usually hidden by end-to-end recipe comparisons: how one post-training stage affects the usefulness of the next stage. Several observations are informative, especially that a stronger RLVR-prepared student can regress under OPD while a briefly SFT-warmed student improves, and that OPD and SFT checkpoints with similar initial accuracy behave differently under subsequent RLVR.
  • The study covers more stage combinations than most existing work. The teacher-size grid, student and teacher adaptation experiments, checkpoint sweeps, and OPD-versus-SFT initialization comparison provide a useful empirical map of the main Qwen3/mathematics setting, even where the results are negative or non-monotonic.
Critical weaknesses
  • The central “learning interface” explanation is not independently defined or tested. If a stage helps downstream training, the paper calls the resulting interface productive; if it hurts, the interface is unfavorable. This is descriptive unless a quantity measured before downstream training can predict the outcome. The proposed reinforceable fraction does not yet provide that evidence: it selects the best teacher for only two of three students, is strongly confounded with teacher size in the main grid, and is not prospectively validated on a held-out model family or domain. The experiments therefore show that current accuracy and scalar KL are insufficient, but do not identify what property actually determines distillability.
  • The claimed synergy and composition recipe are not established by the final experiment. Student SFT alone adds 6.1 points and teacher RLVR alone adds 13.0, while combining them adds 14.6 rather than the 19.1 points expected under an additive effect; the interaction is therefore sub-additive. Moreover, the 50% relative improvement matches only the final OPD steps while adding teacher RLVR and student SFT compute. Without a fixed-total-compute factorial comparison, it is unclear whether the ordering is preferable to spending the same compute on student RLVR, longer OPD with checkpoint selection, or another teacher. The teacher-adaptation experiment also does not separate an RLVR-specific transfer effect from teacher accuracy because the SFT teacher is not matched in accuracy.
Questions for authors
  1. Can the authors provide a fixed-total-compute factorial comparison of student warm-up and teacher adaptation, including the interaction term? In the current numbers the combined improvement is sub-additive, so it is unclear in what sense the result demonstrates synergy.
  2. Can the reinforceable fraction select the teacher or stopping point prospectively on a held-out model family or domain, without using downstream OPD outcomes to choose the rule? Otherwise, “learning interface” remains a retrospective description rather than a predictive explanation.
5 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2: W1-Q2

Thank you for your thoughtful and detailed feedback! We appreciate your recognition of the practical importance of stage interactions and the breadth of our study. Below, we report mixture of clarifications, additional analyses of our experiments, and describe the revisions we will make in response to your concerns. Collectivelly, we believe these significantly improve the paper!


1. Testing the diagnostics beyond the original scaling grid (W1, Q2)

“The central ‘learning interface’ explanation is not independently defined or tested.”

We use learning interface to describe the policy, its generated trajectories, and the feedback available when the next stage begins. These properties exist before the downstream outcome is observed. We will define the term explicitly as a framework for studying stage interactions, while distinguishing it from a complete causal explanation of distillability.

“Can the reinforceable fraction select the teacher or stopping point prospectively on a held-out model family or domain, without using downstream OPD outcomes to choose the rule?”

We addressed this through an additional evaluation of the submission’s unchanged selection rule: choose the teacher with the largest initial reinforceable fraction. We evaluated it across 19 comparison settings, including Qwen2.5 models, OpenThoughts training data, different student states, and fixed-size teacher checkpoint sweeps. Selection uses the initial diagnostic; downstream accuracy is used only to score the choice. No thresholds or weights were fitted in this analysis.

Table R1: Mean teacher-selection regret at OPD step 200, in percentage points. Lower is better.

Selection ruleTeacher size varies: 9 settingsTeacher size fixed: 10 settingsAll 19 settings
Largest reinforceable fraction0.351.310.86
Highest teacher accuracy1.500.651.06
Smallest teacher0.361.621.03
Smallest initial reverse KL1.232.631.97

Each setting receives equal weight. For the size baseline, tied candidates receive equal probability, and we report expected regret. One candidate without a step-200 evaluation is excluded. Some settings share configurations, so these are comparison settings rather than independent replications.

The reinforceable fraction has the lowest observed pooled regret and selects the best candidate in 9 settings, compared with 7 for teacher accuracy and 5 for initial KL. The fixed-size comparisons clarify its scope: teacher accuracy performs best, while reinforceable fraction improves substantially over initial KL and the size-only baseline. Across varying teacher sizes, its performance remains close to choosing the smallest teacher.

These results extend the evidence beyond the original grid and support complementary diagnostic value. They do not establish a universal selector or identify a complete mechanism governing transfer.

We also evaluated the unchanged stopping rule on additional runs. Applying the original κ≤0.6\kappa\leq0.6 threshold to 56 complete trajectories outside the original nine-run stopping analysis gives:

Table R2: Stopping comparisons on 56 additional runs.

Stopping ruleMean stopping stepMean accuracy
Fixed step 4040.027.94%
Fixed step 8080.028.19%
κ≤0.6\kappa\leq0.685.428.40%
Fixed step 100100.028.53%
Fixed step 200200.028.76%

Without retuning, the threshold retains accuracy within 0.37 percentage points of step 200 while using 57.3% fewer optimizer steps. The nearby fixed-step baselines show the accuracy–budget trade-off directly. This provides additional evidence for the heuristic’s usefulness beyond its original evaluation setting, through retrospective testing on recorded trajectories.

We will add both analyses, including their candidate sets, evaluation horizons, and eligibility criteria.

AuthorsOpenReview ↗

Rebuttal Part 2/2: W2-Q1

2. Composition, interaction, and total compute (W2, Q1)

“The interaction is therefore sub-additive.”

For the reported Figure 8 endpoints, the numerical contrast is

43.8−35.3−42.2+29.2=−4.543.8-35.3-42.2+29.2=-4.5

percentage points. These values do not demonstrate a positive interaction. Section 5 notes that the gains are not additive, but “synergy” can suggest a stronger conclusion.

We will describe the reported combination as beneficial composition with diminishing returns and state this contrast explicitly. The SFT budgets also differ between the reported warm-up-only and combined configurations, so this comparison should not be presented as an isolated factorial estimate. Its practical contribution is evidence about how preparation changes subsequent OPD performance.

“Without a fixed-total-compute factorial comparison, it is unclear whether the ordering is preferable to spending the same compute…”

Our experiment fixes downstream OPD steps, while teacher adaptation, warm-up data generation, and student SFT add preparation costs. It therefore does not establish the optimal allocation of a fixed total budget.

We checked the direct student-RLVR alternative as well. Qwen3-4B-Base reaches 43.60% after 260 direct-RLVR steps, making it a strong reference that should be visible alongside the recipe. For workload accounting, 200 OPD steps generate 51,200 student responses, whereas 260 direct-RLVR steps generate 133,120. OPD additionally requires teacher scoring, and the full recipe includes preparatory training; these response counts do not establish a FLOP advantage.

Early checkpoint selection is also relevant, and Appendix B.9 already quantifies its benefit for vanilla OPD. We will make this comparison more visible and distinguish the observed effects of preparation from claims about compute-optimal training. Teacher reuse can amortize adaptation costs, but we do not establish a break-even point in this study.


3. Teacher accuracy versus an RLVR-specific transfer effect (W2)

“The teacher-adaptation experiment also does not separate an RLVR-specific transfer effect from teacher accuracy because the SFT teacher is not matched in accuracy.”

The self-training control does not reach the RLVR teacher’s accuracy, so it cannot isolate an RLVR-specific transfer advantage. Section 4.2 acknowledges this limitation. Additional existing runs nevertheless provide useful comparisons at the same teacher size and similar accuracy:

Table R3: Teacher accuracy and downstream transfer under different preparation histories.

StudentTeacher preparationTeacher accuracyStudent accuracy at OPD step 200
Qwen3-4B-BaseOfficial Qwen3-14B42.11%29.23%
Qwen3-4B-BaseQwen3-14B-Base + RLVR44.14%38.40%
Qwen3-1.7B-BaseOfficial Qwen3-8B40.09%23.00%
Qwen3-1.7B-BaseQwen3-8B-Base + RLVR40.56–40.93%25.61–25.63%

The official 14B-teacher result averages three runs. The final row summarizes two teacher checkpoints, not seed replications; the remaining results use one run per configuration.

Transfer differs substantially even where teacher size matches and aggregate accuracy is similar. These comparisons motivate examining teacher preparation beyond its final accuracy. Their training histories differ, however, so they do not isolate the adaptation objective. We will include them with this qualification and describe RLVR as an effective teacher-adaptation method in the studied settings.


Revisions in response to your review

We will:

  1. Add the selection and stopping analyses with explicit candidate sets, horizons, and baseline rules.
  2. Define the learning interface clearly and distinguish diagnostic evidence from causal explanation.
  3. Replace the synergy framing, report the numerical interaction contrast, and clarify the differing preparation budgets.
  4. Add compact main-text tables for the teacher–student grid and four recipe settings, with clear evaluation horizons, seed counts, and uncertainty reporting.
  5. Make the direct-RLVR reference and additional training costs visible, and add the teacher comparisons above.
  6. Correct the recipe sentence suggesting an additional, untested transfer into a fresh student. The combined experiment adapts the teacher, warms the student, and distills into that same student; subsequent RLVR is examined separately.

Thank you again for your helpful feedback. We believe these additional analyses and planned revisions strengthen the paper. Do they address your concerns and support reconsidering your score? If anything remains unresolved, what further clarification or changes would you find helpful?

Thank you for the detailed response and additional analyses. I appreciate the clarification of the synergy claim and the more careful interpretation of the diagnostics. However, the fixed-total-compute comparison remains missing, and the additional diagnostic results do not yet convincingly establish the predictive value of the proposed learning-interface explanation beyond simpler alternatives.

These remain central concerns in my assessment, so I will maintain my original rating.

Reviewer_yGnuOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The main contribution of the paper is to study how the three most common post-training methods, SFT, RLVR and OPD, interact when composed, rather than evaluated in isolation. The paper supports reproducibility by including detailed hyperparameters, which would be helpful for further study. It also finds that OPD is a better initialisation than SFT for subsequent RLVR, and this finding is consistent across three students.

Critical strengths

The paper main contribution of the paper is to study the most common three post-training methods: how SFT, RLVR and OPD interact when composed, rather than evaluated in isolation. The paper shows reproducibility which would be helpful for further study, with detailed hyperparameter included. The finding shows that OPD is a better initialisation than SFT for subsequent RLVR and is consistent across three students.

Critical weaknesses
  1. The presentation is hard to follow. The main paper has no results tables, and most figure captions do not state the evaluation benchmarks. I would suggest moving the main results table (e.g., Table 5) into the main text.

  2. While The idea is interesting, several descriptions are unclear and hard to follow. For example, Figure 6 and other related tables do not state which benchmark its accuracy refers to.

  3. Most results lack standard deviations or significance tests. For those that comparisons are very close, e.g., Table 7, the differences are often below one point and mostly from single runs.

adapt the teacher with RLVR, briefly warm the student with SFT, distill with OPD, and finally distill to the cold-start student.

  1. The proposed recipe is not fully tested. Figure 8 omits its final step (“distill to the cold-start student”, which is also unclear) and a student-only RLVR baseline.

  2. Few baselines beyond vanilla OPD. Comparisons with on-policy self-distillation or compute-matched student RLVR would strengthen the study.

  3. The setting is narrow: mainly one Qwen3-4B/14B pair, in non-thinking mode, with responses of at most 8,192 tokens. It is unclear whether the findings transfer to long chain-of-thought distillation, OPD’s main use case.

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1/4: W1-W2-W3

Thank you for your thoughtful and detailed feedback! Below, we have spent considerable time to provide a mixture of clarifications, additional results from our existing experiments, and edits to the paper to address your concerns. We believe these changes have substantially improved the paper.


1. Results tables and clearer evaluation descriptions (W1, W2)

"The main paper has no results tables, and most figure captions do not state the evaluation benchmarks. I would suggest moving the main results table (e.g., Table 5) into the main text."

"Figure 6 and other related tables do not state which benchmark its accuracy refers to."

Thank you, we agree. We will move the teacher–student grid from Table 5 into Section 3 and add a compact four-setting recipe table to Section 5:

Teacher0.6B student1.7B student4B student
8B14.2923.0032.70 ± 0.66
14B15.0020.9529.23 ± 0.63
32B13.6620.3429.25 ± 0.34

Seven-benchmark mean@4 accuracy (%) at OPD step 200. The 4B results report mean ± sample standard deviation across three training runs; smaller students use one run per teacher.

Here and in Figure 6, accuracy is the unweighted average of seven benchmark-level mean@4 scores: AIME 2024, AIME 2025, AMC 2023, MATH-500, Minerva Math, OlympiadBench, and GPQA-Diamond. Mean@4 averages correctness over four sampled responses per question. The paper defines this only in Section 2 and Appendix B.2, which is not enough.

Every caption will now state the benchmarks, student, teacher, training step, number of runs, and the meaning of the error bars. Captions will also distinguish best observed checkpoints from endpoints: Figure 6(b) reports the best accuracy observed during OPD, whereas Figure 8 compares checkpoints after 200 OPD steps. Any exception to the seven-benchmark aggregate will be labeled explicitly. For example, Figure 6's caption will read: "Qwen3-4B-Base distilled from Qwen3-14B checkpoints adapted with RLVR. Accuracy is mean@4 over the seven benchmarks; one run per teacher checkpoint."


2. Standard deviations and significance tests (W3)

"Most results lack standard deviations or significance tests. For those that comparisons are very close, e.g., Table 7, the differences are often below one point and mostly from single runs."

We agree that close single-run differences should not carry claims, and we will label every single-run comparison. Our replication is concentrated on the main setting. The largest student, Qwen3-4B-Base, carries the central scaling claim and has three seeds per teacher, and its nine runs agree (Section 1). The smaller students and the second model family are single runs. They test whether the 4B findings generalize across scale, and we read them as trends. The study already comprises about 190 training runs and 13,000 H200 GPU-hours, so we placed seeds where the main claims are made. Our existing runs also let us measure the noise directly:

  • Run-to-run noise. Seven configurations in our study were run twice. Across 45 matched evaluations, the two runs differ by a median of 0.56 points. The 4B seeds have a standard deviation of 0.3 to 0.7 points.
  • Teacher ranking. For the 4B student, each of the three 8B-teacher runs outperforms each of the six larger-teacher runs at both steps 200 and 260 (exact permutation p = 0.012). The 0.6B ranking is not stable: 14B leads by 0.71 points at step 200, while 8B leads at step 260. We will describe this as horizon-dependent behavior. The 8B teacher is also the cheapest: 200 OPD steps cost 43 GPU-hours with it and 51 with the 32B teacher (Section 4 explains how we measure compute).
  • Table 7. Its finding concerns the student's starting state. Qwen2.5-1.5B base students gain 18.54–19.14 points, compared with 2.48–2.66 points for instruct students. The much smaller differences between teachers within a row do not establish a reliable winner.
  • Teacher adaptation. All 17 RLVR-adapted teacher checkpoints with a step-200 evaluation outperform their unadapted teacher, across four student–teacher pairs (Section 3 below).

Stage ordering. Section 4.3's claim, that OPD is a better first stage than SFT before RLVR, is also significant. For the cleanest test, we use the two students where RLVR runs from both starting points with identical settings (eight responses per prompt, 8,192 tokens):

StudentRLVR from SFT startRLVR from OPD startOPD run ends higherPermutation p
Qwen3-1.7B-Base (step 260)23.13–25.7026.66–27.539 of 9 pairs0.05
Qwen2.5-1.5B-Base (step 200)18.69–19.6119.21–21.5433 of 36 pairs0.008

Seven-benchmark accuracy (%) at the end of RLVR. Each method was started from several checkpoints, with one run each, and the ranges cover them. Pairs compare every OPD-started run with every SFT-started run (3 × 3 for 1.7B, 6 × 6 for 1.5B).

AuthorsOpenReview ↗

Rebuttal Part 2/4: W4

Every OPD-started 1.7B run ends above every SFT-started run, which gives p = 0.05, the smallest value possible with three runs per side. Together, the two students give p = 0.003 (Fisher's method). OPD also hands RLVR a stronger starting point, which is part of why it is the better first stage. In the 4B comparison of Figure 7, the OPD-started runs used a larger RLVR budget (eight responses and 8,192 tokens, versus four and 4,096). For the revision, we will rerun it with matched settings and lead with the matched comparisons above.


3. The tested recipe and a student-RLVR baseline (W4)

"adapt the teacher with RLVR, briefly warm the student with SFT, distill with OPD, and finally distill to the cold-start student."

"Figure 8 omits its final step ('distill to the cold-start student', which is also unclear) and a student-only RLVR baseline."

Thank you for catching this. The recipe is a protocol of four steps, and each step is tested. The last clause of that sentence was worded badly. It does not mean another distillation into a fresh student. As the opening of the sentence indicates ("Together with §4.3"), it refers to the stage-ordering study, where RLVR continues from the distilled student. We will rewrite it as: "adapt the teacher with RLVR, briefly warm the student with SFT, distill with OPD, and then scale the distilled student with RLVR."

  • Steps 1–3 (teacher RLVR adaptation → student SFT warm-up → OPD into that same student) are tested together in Figure 8.
  • Step 4 (RLVR from the distilled student) is tested as a study of its own in Section 4.3, on three students. There, OPD checkpoints give better RLVR endpoints than SFT checkpoints (Section 2 above).

The introduction lists the first three steps, and we will state the recipe identically in both places. We study RLVR on both sides of distillation because RL is now the main scaling stage in frontier post-training, as in the recent MiMo-V2.6 report (LLM-Core Xiaomi, 2026; arXiv:2610.11959), which we will cite. RLVR before OPD is covered on the teacher (Section 4.2) and on the student (Section 4.1). RLVR after OPD is covered in Section 4.3.

We also have a direct student-RLVR baseline that should be visible. Starting from Qwen3-4B-Base, it reaches 41.69% at step 200 and 43.60% at step 260. The recipe reaches 43.8% after 200 OPD steps. Under identical RLVR settings, RLVR started from the OPD peak ends level with direct RLVR (43.11% vs. 43.60%). We will add this reference to Figure 8.

The recipe's components also extend beyond the main pair. Teacher adaptation improves transfer across four teacher–student pairs:

Student ← teacherUnadapted teacherRLVR-adapted teachersAdapted checkpoints
4B ← Qwen3-14B29.2333.32–42.905
1.7B ← Qwen3-14B20.9527.41–29.775
4B ← Qwen3-14B-Base30.8138.40–41.934
1.7B ← Qwen3-8B-Base19.6624.83–25.633

Seven-benchmark accuracy (%) at OPD step 200. Ranges span adapted teacher checkpoints, with one downstream run per checkpoint. The unadapted 4B ← 14B reference is the three-run mean.

The full recipe also gives the same ordering on a second student:

Qwen3-1.7B-Base settingAccuracy at OPD step 60
Vanilla OPD21.87
+ SFT warm-up23.56
+ Teacher RLVR24.88 / 25.22
+ Both27.24 / 27.94

Seven-benchmark accuracy (%). Two values denote two recorded runs. These runs come from the first round of our recursive study, which uses a shorter OPD stage than Section 5.

AuthorsOpenReview ↗

Rebuttal Part 3/4: W5

4. Additional baselines and a compute-matched comparison with student RLVR (W5)

"Few baselines beyond vanilla OPD. Comparisons with on-policy self-distillation or compute-matched student RLVR would strengthen the study."

The study compares more than vanilla OPD, though the paper does not make this visible. Each composition question has its own control:

  • Student preparation: SFT-warmed vs. RLVR-warmed student (Section 4.1).
  • Teacher preparation: RLVR-adapted vs. self-trained (STaR) teacher, and base vs. post-trained teachers (Section 4.2, Appendix B).
  • Stage order: SFT vs. OPD as the initialization for RLVR (Section 4.3).
  • Objective: sampled-token reverse KL vs. top-k forward KL (Appendix B.4).
  • Checkpoint selection: OPD stopped early vs. trained longer (Appendix B.9).
  • New: direct student RLVR, also at matched compute (below).

Compute-matched student RLVR. Our logs allow this comparison directly. All runs use the same 8×H200 node and the same training set, so we compare measured GPU-hours. We also estimate FLOPs from the logged token counts.

StudentCompute per studentOPD from the adapted teacherDirect RLVR, same GPU-hoursDirect RLVR, same FLOPs
Qwen3-4B-Base15 GPU-h (3.5 EFLOPs)41.936.937.0
Qwen3-4B-Base23 GPU-h (5.4 EFLOPs)41.637.637.8
Qwen3-4B-Base30 GPU-h (7.1 EFLOPs)43.738.438.5
Qwen3-4B-Base38 GPU-h (8.9 EFLOPs)42.939.439.9
Qwen3-1.7B-Base12 GPU-h (2.3 EFLOPs)27.322.725.1
Qwen3-1.7B-Base18 GPU-h (3.7 EFLOPs)28.523.826.7
Qwen3-1.7B-Base25 GPU-h (5.1 EFLOPs)29.024.828.2

Seven-benchmark accuracy (%). OPD uses the RLVR-adapted Qwen3-14B teacher; rows are OPD steps 80 to 200 (4B, one run) and 80 to 160 (1.7B, mean of two runs). Direct-RLVR accuracy is interpolated between evaluations. FLOPs count 2N per token for a forward pass and 4N for a backward pass, with N the parameter count, and include teacher scoring. 1 EFLOP = 10^18 FLOPs.

Per student, OPD from the adapted teacher is ahead of direct RLVR at every matched budget, under both measures. The 4B student reaches 41.9% in 15 GPU-hours. Direct RLVR needs 63 GPU-hours to reach that level, and 93 GPU-hours for its final 43.6%. The early advantage also does not depend on one teacher checkpoint: at 15 GPU-hours, the students of all nine adapted teacher checkpoints are ahead of direct RLVR.

More compute alone does not help OPD without preparation. Over 260 steps (58 to 69 GPU-hours), the best checkpoint of vanilla OPD is 32.7–33.9% with any of the three released teachers.

Adapting the teacher is a one-time cost of 148 GPU-hours (33 EFLOPs). If it is charged to a single student, direct RLVR is the cheaper route to the same accuracy, and we will say so in Section 5. Shared across students, the adaptation pays for itself from about the fourth student of this size, and the adapted teacher above already serves both students in the table. This is the regime in which distillation is used in practice. In Qwen3, for example, the largest models are trained with RL and the smaller models are distilled from them.

A second model family shows the same per-student pattern. On Qwen2.5-1.5B-Instruct, OPD reaches 20.6–20.7% in about 20 GPU-hours with any of the three teachers, while direct RLVR reaches 20.0% in 46 GPU-hours.

We have not evaluated on-policy self-distillation. It conditions the model on privileged information, such as reference solutions, which answers a different question from ours. We will discuss it as related work.

AuthorsOpenReview ↗

Rebuttal Part 4/4: W6 and Summary of Overall Changes

5. Generalization beyond the main pair and to long reasoning (W6)

"The setting is narrow: mainly one Qwen3-4B/14B pair, in non-thinking mode, with responses of at most 8,192 tokens. It is unclear whether the findings transfer to long chain-of-thought distillation, OPD's main use case."

The combined-preparation experiment is concentrated on Qwen3-4B/14B, while the individual stage comparisons cover more models and settings:

AxisCovered
StudentsQwen3 0.6B, 1.7B, 4B, 8B (base); Qwen3 1.7B, 4B (post-trained); Qwen2.5 1.5B, 3B (base and instruct)
TeachersQwen3 4B, 8B, 14B, 32B; Qwen3 8B-Base, 14B-Base; Qwen2.5 7B, 14B, 32B; RLVR-adapted 8B-Base, 14B-Base, 14B
Training dataDAPO-Math-17K, OpenThoughts3-Math
ObjectivesSampled-token reverse KL, top-k forward KL

Coverage of the existing experiments.

Several findings hold beyond the main pair:

  • Teacher adaptation helps in all four teacher–student pairs we tested.
  • The recipe ordering holds on Qwen3-1.7B.
  • OPD gives better RLVR endpoints than SFT on Qwen3-1.7B and Qwen2.5-1.5B.
  • The teacher ranking holds on both training distributions.

We agree that long chain-of-thought distillation is the most important next setting. We used non-thinking mode so that the full grid of students, teachers, and stages fit a controlled budget. The questions we study concern how the teacher is prepared, how the student is prepared, and in which order the stages run, and they do not depend on response length. These results do not yet establish transfer to long chain-of-thought distillation. We will state that boundary explicitly and report the response limit of every experiment.


Changes in response to your review

We thank you again for taking the time to review our paper. We will:

  1. Bring the teacher–student grid and a compact recipe table into the main text.
  2. State benchmarks, aggregation, training horizons, run counts, error bars, and checkpoint type in every caption, including Figure 6.
  3. Add the noise statistics and significance tests, and label single-run comparisons explicitly.
  4. Support the stage-ordering claim in Section 4.3 with the matched-setting significance tests, and rerun the 4B comparison with matched RLVR settings.
  5. Reword the recipe sentence so that its final step clearly refers to the RLVR stage of Section 4.3, state the recipe identically in the introduction, and add direct student RLVR to Figure 8.
  6. Add the compute-matched comparison with student RLVR (GPU-hours and FLOPs), with the teacher's one-time cost and the break-even point.
  7. Add the baseline list, the cross-pair teacher-adaptation and 1.7B recipe results, and a clear statement of scope for long chain-of-thought distillation.

We believe these changes have improved the paper, considerably! Do the above actions address your concerns and warrant reconsidering your score? If not, what further clarification or modifications could we make?

Reviewer_Em8KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work observes and studies the interactions between different post-training techniques, such as SFT, RLVR, and OPD, in order to better combine them and achieve improved final model performance.

Critical strengths
  • I think this paper motivates the problem they study quite well: tunnel vision on any specific part of post-training will give you a weaker overall model at the end. Thus, we must think of post-training holistically.
  • The work creates important metrics to study the effectiveness of distillation, such as distillation effectiveness and reinforceable fraction, that help us better understand this problem and measure better approaches.
Critical weaknesses
  • Related to my first question in "Questions For Authors": I think you should have run a more robust set of interactions between all the students and teachers. Your claim that 8B teaches 4 and 1.7B best while 8B teaches 0.6B best is puzzling, especially since those Qwen3 models are trained on the same corpus. I think you need a more statistically robust experiment (more seeds) since, as you say, "0.6B and 1.7B use one run per teacher." This is very important, especially since some of your other results suggest that learning from models closer in capability/scale to the student is better (Reinforceable signal and teacher size in Figure 4). Also, I have concerns about how well the simple recipe you propose in Section 5 generalizes beyond a 14B teacher and a 4B student.

  • Some deeper analysis into why the most powerful teacher is not always the best teacher would greatly improve this work. I have some intuition for why this is the case, rigorous analysis would be able to tell us what's going on here?

  • I'm not sure why you're not using tables in the main paper. This makes it harder to follow your specific results. Perhaps something like students as columns and teachers as rows.

  • A few concerns I have about training data and evals. You use the DAPO dataset, which is not very large. I have concerns about how generalizable this makes your work. For example, do your findings hold when I use more or different data mixes? Also, I have concerns about you evaluating only on math datasets. While I understand this instinct, a concern during post-training is getting better at some skills while degrading on others. How do we know that your pipeline didn't degrade other model skills?

Questions for authors
  • From line 192: "The preferred teacher also varies across students: Qwen3-8B performs best for both the 4B and 1.7B students, while Qwen3-14B leads for the 0.6B student." This makes no sense to me. Can you explain why?
  • How well does your simple recipe work for other model pairs?
3 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2: W1-W2-W3-Q1

We thank reviewer for the thoughtful and detailed feedback! Below, we provide a mixture of clarifications, additional results from our experiments, and edits to the paper to address the mentioned concerns. In doing so, we believe the quality of the paper has improved substantially!


1. Teacher rankings, statistical support, and clearer presentation (W1, W3, Q1)

“I think you need a more statistically robust experiment (more seeds).”

“Qwen3-8B performs best for both the 4B and 1.7B students, while Qwen3-14B leads for the 0.6B student.”

“Perhaps something like students as columns and teachers as rows.”

Thank you for raising this. Our main finding is that increasing teacher size does not reliably improve transfer. The replicated 4B results support this directly:

Teacher0.6B student1.7B student4B student
8B14.2923.0032.70 ± 0.66
14B15.0020.9529.23 ± 0.63
32B13.6620.3429.25 ± 0.34

Seven-benchmark mean@4 accuracy (%) at step 200. The 4B results report mean ± sample standard deviation across three training runs; smaller students use one run per teacher.

For the 4B student, each of the three 8B-teacher runs outperforms each of the six larger-teacher runs, at both steps 200 and 260. At step 200, even the lowest 8B result, 32.21%, exceeds the highest larger-teacher result, 29.82%.

The 0.6B comparison deserves a more specific explanation. Almost all of the 14B teacher’s 0.71-point lead comes from GPQA-Diamond. Across the six math benchmarks, the students distilled from 8B and 14B score almost identically: 15.00% and 15.05%. Their GPQA scores are 10.03% and 14.72%, respectively. The ranking also reverses at step 260, when 8B performs better. We will clarify that this small, single-run difference does not establish a reliable preference for 14B.

Shared pretraining data does not imply identical probabilities on student-generated continuations. Different teachers can therefore provide different supervision to the same student, even when their standalone accuracies are similar.

For presentation suggestion, we will move this grid into Section 3 and add a compact four-setting recipe table to Section 5. Captions will state the benchmarks, evaluation step, seed counts, and meaning of the error bars. Four responses per evaluation question are used for mean@4 and are separate from training-seed replication.


2. Why the strongest teacher need not transfer best (W2)

“Some deeper analysis into why the most powerful teacher is not always the best teacher would greatly improve this work.”

We agree that this distinction needs a clearer explanation. Teacher accuracy measures performance on its own generations, while OPD applies teacher feedback to trajectories generated by the student.

For the 4B student, initial reverse KL is nearly unchanged across the three teachers: 0.370, 0.376, and 0.382. Yet the fraction of tokens receiving positive distillation advantage falls substantially: 54.79%, 47.83%, and 33.11% for 8B, 14B, and 32B. Thus, similar average divergence can conceal markedly different supervision on the student’s behavior.

A positive advantage encourages the sampled token; a negative advantage discourages it. The reinforceable fraction makes this difference visible, although positive advantage should not be confused with token correctness.

There is also relatively little additional teacher capability in this scaling comparison: teacher accuracies range from 40.1% to 42.8%. When we instead improve capability at fixed model size through RLVR, downstream transfer increases substantially, as shown below. Together, these results motivate examining both teacher capability and the supervision it provides on student trajectories. We will present the reinforceable fraction consistently as a complementary diagnostic.

AuthorsOpenReview ↗

Rebuttal Part 2/2: W1-W4-Q2 and Summary of Overall Changes

3. Does the recipe extend to other model pairs? (W1, Q2)

“How well does your simple recipe work for other model pairs?”

Our existing experiments include several smaller-model comparisons that we should have made more visible.

Teacher adaptation improves transfer across four teacher–student pairs. The table below uses the common step-200 OPD evaluation:

Student ← teacherUnadapted teacherRLVR-adapted teachersAdapted checkpoints
4B ← Qwen3-14B29.2333.32–42.905
1.7B ← Qwen3-14B20.9527.41–29.775
4B ← Qwen3-14B-Base30.8138.40–41.935
1.7B ← Qwen3-8B-Base19.6624.83–25.633

Seven-benchmark accuracy (%). Ranges span adapted teacher checkpoints, with one downstream run per checkpoint. The unadapted 4B ← 14B reference is the three-run mean.

All 18 adapted checkpoints in these comparisons outperform their unadapted references. This extends the teacher-adaptation result to a smaller student, a smaller teacher, and base-model teachers.

The combined preparation also appears in the recursive study: an RLVR-adapted 14B teacher, followed by SFT preparation of a 1.7B student and OPD, reaches 27.24% and 27.94% after 60 OPD steps in two recorded first-round executions. These are smaller-student applications of the recipe, although they use a shorter OPD stage and are not training-seed replications of the Section 5 ablation.

Finally, the advantage of OPD over SFT as preparation for subsequent RLVR is also observed with Qwen3-1.7B and Qwen2.5-1.5B in Appendix E.5. We will bring these cross-model results into the discussion of the recipe.


4. Training distributions, non-math evaluation, and model families (W4)

“Do your findings hold when I use more or different data mixes?”

Thank you for pointing this out. Figure 3(d) and Appendix B.5, Table 9 already compare DAPO-Math and OpenThoughts-Math:

TeacherDAPO-MathOpenThoughts-Math
8B32.70 ± 0.6631.29
14B29.23 ± 0.6328.80
32B29.25 ± 0.3428.00

Qwen3-4B-Base accuracy (%) at step 200; three runs per teacher for DAPO and one for OpenThoughts.

The 8B teacher performs best on both distributions. This addresses whether the result is specific to DAPO, while larger training budgets and broader mixtures remain separate questions.

“How do we know that your pipeline didn’t degrade other model skills?”

This is an important check. Our primary evaluation includes GPQA-Diamond, and Appendix B.5, Table 10 additionally reports SciKnowEval biology, chemistry, and physics.

We also compared each replicated 4B run against its own initial score. All 36 run–benchmark comparisons improve at step 200: nine training runs evaluated on four science benchmarks.

Non-math benchmarkGain with 8BGain with 14BGain with 32B
GPQA-Diamond+14.21 ± 1.92+19.59 ± 0.70+17.94 ± 1.03
SciKnowEval-Biology+7.00 ± 1.80+10.00 ± 4.09+12.67 ± 3.33
SciKnowEval-Chemistry+9.17 ± 4.96+13.42 ± 1.13+11.33 ± 1.38
SciKnowEval-Physics+13.96 ± 6.51+28.33 ± 3.34+23.75 ± 0.62

Accuracy gains in percentage points, reported as mean ± sample standard deviation across three runs.

Thus, the math gains are accompanied by science gains in these experiments. We will make clear that this analysis covers the OPD scaling runs, rather than every stage of the complete recipe.

We also test Qwen2.5 students with 7B, 14B, and 32B teachers in Appendix B.5:

Qwen2.5 studentAccuracy range (%)Across-teacher spread (pp)
1.5B-Base19.85–20.440.59
1.5B-Instruct20.56–20.730.17
3B-Base25.28–26.100.82
3B-Instruct24.40–25.561.16

Step-200 accuracy; ranges span teacher sizes, with one run per configuration.

Despite the increase from 7B to 32B, all teachers finish within 0.6 points for the 1.5B students and within 1.2 points across all four settings. The largest teacher sometimes has the highest numerical score, but the advantage is small and teacher scaling does not improve transfer consistently.


Changes in response to your review

We thank you again for taking the time to review our paper. We will:

  1. Bring the teacher–student grid and a compact recipe table into the main text, with clear evaluation and replication details.
  2. Explain the 0.6B ranking through its benchmark breakdown and training-horizon dependence.
  3. Add the cross-pair teacher-adaptation results and highlight the smaller-student composition experiments.
  4. Make the alternative training distribution and non-math evaluations more visible.

We believe these changes have improved the paper, considerably! Do the above actions address your concerns and warrant reconsidering your score? If not, what further clarification or modifications could we make?

AuthorsOpenReview ↗

Kind Follow-up to Review Rebuttal for Submission 36180

Dear Reviewer Em8K,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. In response to your concern regarding statistical robustness, we re-examined our replicated runs. For the Qwen3-4B student, each of the three 8B-teacher runs outperforms each of the six larger-teacher runs, at both steps 200 and 260. We also explained the single 0.6B exception: the 14B teacher's 0.71-point lead comes almost entirely from one benchmark and reverses at step 260. On why the strongest teacher need not transfer best, we showed that the initial reverse KL is nearly identical across the three teachers (0.370 to 0.382), while the fraction of tokens with a positive distillation advantage falls from 54.8% to 33.1%. On generality, teacher adaptation improves transfer in all four teacher–student pairs we tested, the 8B teacher remains best on a second training distribution, and all 36 non-math run–benchmark comparisons improve. These results directly address the core questions raised in your review: whether the teacher ranking is reliable, why it arises, and whether it extends beyond the main setting.

We believe these additions help address your concerns, and we would be happy to discuss any remaining questions or provide further clarification. If you find our responses satisfactory, we would greatly appreciate it if you could reflect this in an updated overall rating.

Thank you very much for your time and helpful suggestions.

Sincerely,

Authors of Paper 36180

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.