- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper empirically studies how supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) interact when composed in an LLM post-training pipeline. It first evaluates OPD across nine Qwen3 student-teacher size pairs and proposes the initial reinforceable fraction as a diagnostic of student-teacher compatibility. It then varies the state entering each stage and reports that a short SFT warm-up improves subsequent OPD, whereas an RLVR-strengthened student can regress under the same teacher; adapting the teacher with RLVR improves downstream OPD; and OPD provides a better initialization than SFT for subsequent RLVR at comparable starting accuracy. Combining teacher RLVR adaptation and student SFT warm-up raises the final OPD accuracy from 29.2% to 43.8% in the main Qwen3-4B/14B setting.
Critical strengths
- The paper studies an important practical question that is usually hidden by end-to-end recipe comparisons: how one post-training stage affects the usefulness of the next stage. Several observations are informative, especially that a stronger RLVR-prepared student can regress under OPD while a briefly SFT-warmed student improves, and that OPD and SFT checkpoints with similar initial accuracy behave differently under subsequent RLVR.
- The study covers more stage combinations than most existing work. The teacher-size grid, student and teacher adaptation experiments, checkpoint sweeps, and OPD-versus-SFT initialization comparison provide a useful empirical map of the main Qwen3/mathematics setting, even where the results are negative or non-monotonic.
Critical weaknesses
- The central “learning interface” explanation is not independently defined or tested. If a stage helps downstream training, the paper calls the resulting interface productive; if it hurts, the interface is unfavorable. This is descriptive unless a quantity measured before downstream training can predict the outcome. The proposed reinforceable fraction does not yet provide that evidence: it selects the best teacher for only two of three students, is strongly confounded with teacher size in the main grid, and is not prospectively validated on a held-out model family or domain. The experiments therefore show that current accuracy and scalar KL are insufficient, but do not identify what property actually determines distillability.
- The claimed synergy and composition recipe are not established by the final experiment. Student SFT alone adds 6.1 points and teacher RLVR alone adds 13.0, while combining them adds 14.6 rather than the 19.1 points expected under an additive effect; the interaction is therefore sub-additive. Moreover, the 50% relative improvement matches only the final OPD steps while adding teacher RLVR and student SFT compute. Without a fixed-total-compute factorial comparison, it is unclear whether the ordering is preferable to spending the same compute on student RLVR, longer OPD with checkpoint selection, or another teacher. The teacher-adaptation experiment also does not separate an RLVR-specific transfer effect from teacher accuracy because the SFT teacher is not matched in accuracy.
Questions for authors
- Can the authors provide a fixed-total-compute factorial comparison of student warm-up and teacher adaptation, including the interaction term? In the current numbers the combined improvement is sub-additive, so it is unclear in what sense the result demonstrates synergy.
- Can the reinforceable fraction select the teacher or stopping point prospectively on a held-out model family or domain, without using downstream OPD outcomes to choose the rule? Otherwise, “learning interface” remains a retrospective description rather than a predictive explanation.
5 comments
Rebuttal Part 2/2: W2-Q1
2. Composition, interaction, and total compute (W2, Q1)
“The interaction is therefore sub-additive.”
For the reported Figure 8 endpoints, the numerical contrast is
percentage points. These values do not demonstrate a positive interaction. Section 5 notes that the gains are not additive, but “synergy” can suggest a stronger conclusion.
We will describe the reported combination as beneficial composition with diminishing returns and state this contrast explicitly. The SFT budgets also differ between the reported warm-up-only and combined configurations, so this comparison should not be presented as an isolated factorial estimate. Its practical contribution is evidence about how preparation changes subsequent OPD performance.
“Without a fixed-total-compute factorial comparison, it is unclear whether the ordering is preferable to spending the same compute…”
Our experiment fixes downstream OPD steps, while teacher adaptation, warm-up data generation, and student SFT add preparation costs. It therefore does not establish the optimal allocation of a fixed total budget.
We checked the direct student-RLVR alternative as well. Qwen3-4B-Base reaches 43.60% after 260 direct-RLVR steps, making it a strong reference that should be visible alongside the recipe. For workload accounting, 200 OPD steps generate 51,200 student responses, whereas 260 direct-RLVR steps generate 133,120. OPD additionally requires teacher scoring, and the full recipe includes preparatory training; these response counts do not establish a FLOP advantage.
Early checkpoint selection is also relevant, and Appendix B.9 already quantifies its benefit for vanilla OPD. We will make this comparison more visible and distinguish the observed effects of preparation from claims about compute-optimal training. Teacher reuse can amortize adaptation costs, but we do not establish a break-even point in this study.
3. Teacher accuracy versus an RLVR-specific transfer effect (W2)
“The teacher-adaptation experiment also does not separate an RLVR-specific transfer effect from teacher accuracy because the SFT teacher is not matched in accuracy.”
The self-training control does not reach the RLVR teacher’s accuracy, so it cannot isolate an RLVR-specific transfer advantage. Section 4.2 acknowledges this limitation. Additional existing runs nevertheless provide useful comparisons at the same teacher size and similar accuracy:
Table R3: Teacher accuracy and downstream transfer under different preparation histories.
| Student | Teacher preparation | Teacher accuracy | Student accuracy at OPD step 200 |
|---|---|---|---|
| Qwen3-4B-Base | Official Qwen3-14B | 42.11% | 29.23% |
| Qwen3-4B-Base | Qwen3-14B-Base + RLVR | 44.14% | 38.40% |
| Qwen3-1.7B-Base | Official Qwen3-8B | 40.09% | 23.00% |
| Qwen3-1.7B-Base | Qwen3-8B-Base + RLVR | 40.56–40.93% | 25.61–25.63% |
The official 14B-teacher result averages three runs. The final row summarizes two teacher checkpoints, not seed replications; the remaining results use one run per configuration.
Transfer differs substantially even where teacher size matches and aggregate accuracy is similar. These comparisons motivate examining teacher preparation beyond its final accuracy. Their training histories differ, however, so they do not isolate the adaptation objective. We will include them with this qualification and describe RLVR as an effective teacher-adaptation method in the studied settings.
Revisions in response to your review
We will:
- Add the selection and stopping analyses with explicit candidate sets, horizons, and baseline rules.
- Define the learning interface clearly and distinguish diagnostic evidence from causal explanation.
- Replace the synergy framing, report the numerical interaction contrast, and clarify the differing preparation budgets.
- Add compact main-text tables for the teacher–student grid and four recipe settings, with clear evaluation horizons, seed counts, and uncertainty reporting.
- Make the direct-RLVR reference and additional training costs visible, and add the teacher comparisons above.
- Correct the recipe sentence suggesting an additional, untested transfer into a fresh student. The combined experiment adapts the teacher, warms the student, and distills into that same student; subsequent RLVR is examined separately.
Thank you again for your helpful feedback. We believe these additional analyses and planned revisions strengthen the paper. Do they address your concerns and support reconsidering your score? If anything remains unresolved, what further clarification or changes would you find helpful?
Thank you for the detailed response and additional analyses. I appreciate the clarification of the synergy claim and the more careful interpretation of the diagnostics. However, the fixed-total-compute comparison remains missing, and the additional diagnostic results do not yet convincingly establish the predictive value of the proposed learning-interface explanation beyond simpler alternatives.
These remain central concerns in my assessment, so I will maintain my original rating.
Rebuttal Part 1/2: W1-Q2
Thank you for your thoughtful and detailed feedback! We appreciate your recognition of the practical importance of stage interactions and the breadth of our study. Below, we report mixture of clarifications, additional analyses of our experiments, and describe the revisions we will make in response to your concerns. Collectivelly, we believe these significantly improve the paper!
1. Testing the diagnostics beyond the original scaling grid (W1, Q2)
We use learning interface to describe the policy, its generated trajectories, and the feedback available when the next stage begins. These properties exist before the downstream outcome is observed. We will define the term explicitly as a framework for studying stage interactions, while distinguishing it from a complete causal explanation of distillability.
We addressed this through an additional evaluation of the submission’s unchanged selection rule: choose the teacher with the largest initial reinforceable fraction. We evaluated it across 19 comparison settings, including Qwen2.5 models, OpenThoughts training data, different student states, and fixed-size teacher checkpoint sweeps. Selection uses the initial diagnostic; downstream accuracy is used only to score the choice. No thresholds or weights were fitted in this analysis.
Table R1: Mean teacher-selection regret at OPD step 200, in percentage points. Lower is better.
Each setting receives equal weight. For the size baseline, tied candidates receive equal probability, and we report expected regret. One candidate without a step-200 evaluation is excluded. Some settings share configurations, so these are comparison settings rather than independent replications.
The reinforceable fraction has the lowest observed pooled regret and selects the best candidate in 9 settings, compared with 7 for teacher accuracy and 5 for initial KL. The fixed-size comparisons clarify its scope: teacher accuracy performs best, while reinforceable fraction improves substantially over initial KL and the size-only baseline. Across varying teacher sizes, its performance remains close to choosing the smallest teacher.
These results extend the evidence beyond the original grid and support complementary diagnostic value. They do not establish a universal selector or identify a complete mechanism governing transfer.
We also evaluated the unchanged stopping rule on additional runs. Applying the original κ≤0.6 threshold to 56 complete trajectories outside the original nine-run stopping analysis gives:
Table R2: Stopping comparisons on 56 additional runs.
Without retuning, the threshold retains accuracy within 0.37 percentage points of step 200 while using 57.3% fewer optimizer steps. The nearby fixed-step baselines show the accuracy–budget trade-off directly. This provides additional evidence for the heuristic’s usefulness beyond its original evaluation setting, through retrospective testing on recorded trajectories.
We will add both analyses, including their candidate sets, evaluation horizons, and eligibility criteria.