acceptodds
Under review as a conference paper at ICLR 2027

Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO

Abstract

On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: in our math setting, OPD recovers the accuracy of the student's own GRPO checkpoint with 50× fewer distillation FLOPs than direct GRPO needs to reach that accuracy. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, though, separate GRPO checkpoints hold different capabilities, and consolidating them into a single model has a distinct advantage. Multi-teacher OPD consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the same data mix at 42.0% fewer total FLOPs.

Primary area:
reinforcement learning
Keywords:
on-policy distillation, reinforcement learning, post-training, analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZNe5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares OPD and GRPO in terms of training FLOPs. The authors find that OPD with external teachers generally performs worse than GRPO under the same FLOPs budget, while using the student's own GRPO checkpoint as the teacher can recover its performance with much fewer FLOPs. They further show that training separate GRPO models for different domains and merging them with OPD can save 42% FLOPs compared with joint GRPO.

Critical strengths

The paper conducts extensive experiments across different model sizes, datasets, teachers, and training settings.

Critical weaknesses

First, I doubt whether FLOPs are a meaningful metric for measuring practical training efficiency. Prefilling has much higher arithmetic intensity and GPU utilization than decoding, so fewer FLOPs do not necessarily mean shorter wall-clock time. Second, the OPD performance reported in this paper is unusually poor compared with Tinker's official examples and other related work [1,2] on similar models and datasets. This makes me question whether the results reflect general limitations of OPD or simply unsuccessful training setups.

[1] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation. 2026. https://arxiv.org/abs/2604.13010

[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. 2026. https://arxiv.org/abs/2604.13016

Questions for authors
  1. FLOPs is not 100% equals to efficiency in practice, for example, a small LLM decoding may consume less flops than a larger LLM prefilling, but the wall-clock time and the latency will be very different. Prefilling has much higher arithmetic density and GPU (tensor core) utility than decoding. Can the authors explain the set-up and authors provide wall-clock time or GPU-hours for their main experiments?

  2. Why does OPD perform so poorly in this paper compared with the successful training results reported in [1, 2]? In particular, [1] identifies teacher consistency between SFT and OPD as an important factor for successful training, while [2] emphasizes thinking-pattern alignment and teacher-generated cold starts. Although this paper includes SFT ablations, it does not seem to systematically test these recipes with its external teachers. Have the authors tried reproducing these settings?

  3. In my experience, without this SFT initialization, OPD can produce many negative token-level advantages, making the model to repeat certain high-probability words and inflate its response length. I suspect something similar may be happening in this paper. Can the author present some case study of the failure of OPD?

3 comments
AuthorsOpenReview ↗

Rebuttal 1/n

W1 First, I doubt whether FLOPs are a meaningful metric for measuring practical training efficiency. Prefilling has much higher arithmetic intensity and GPU utilization than decoding, so fewer FLOPs do not necessarily mean shorter wall-clock time.

Q1 FLOPs is not 100% equals to efficiency in practice, for example, a small LLM decoding may consume less flops than a larger LLM prefilling, but the wall-clock time and the latency will be very different. Prefilling has much higher arithmetic density and GPU (tensor core) utility than decoding. Can the authors explain the set-up and authors provide wall-clock time or GPU-hours for their main experiments?

FLOP is a well-accepted and conventional metric for comparing training algorithms [1, 2]. Compared with wall-clock time, FLOP is agnostic to system choices such as GPU layout (e.g., using 1 GPU vs 8 GPU for rollout).

We agree that FLOPs and wall-clock time differ. Decoding is memory-bandwidth-bound, while prefill-style passes run near peak throughput. We report additional measures for runs in Figure 1(a):

MethodWall-clock (h)GPU-hoursFLOPs in Figure 1(a), use max length (101810^{18})FLOPs, use generated tokens (101810^{18})
GRPO6.2850.2145.9
OPD, Qwen3-4B teacher11.0788.62416.9
OPD, Qwen3-8B teacher11.4491.52721.9
OPD, Qwen3-32B teacher15.2121.76855.8

From the table, the ordering among runs is the same under all four measures. Therefore, our conclusion does not change.

[1] Scaling Laws for Neural Language Models. 2020. https://arxiv.org/abs/2001.08361

[2] Measuring the Algorithmic Efficiency of Neural Networks. 2020. https://arxiv.org/abs/2005.04305

AuthorsOpenReview ↗

Rebuttal 2/n

W2 the OPD performance reported in this paper is unusually poor compared with Tinker's official examples and other related work [1,2] on similar models and datasets.

Q2 Why does OPD perform so poorly in this paper compared with the successful training results reported in [1, 2]? … Have the authors tried reproducing these settings?

As mentioned in the appendix (Line 1448), we follow the official setting in Tinker's official examples as much as they specified. Because of the limited budget, we use 64 prompts per batch and 40 steps instead of 512 and 200, and our batch-size ablation (Appendix D.3) shows that this choice does not change the ordering. We disclose our reproduction code to show our practice is faithful.

Is our OPD under-tuned? No.

Comparison with [2]: Our main result’s (Figure 1a) setting closely follows Figure 2 in [2].

SettingLi et al. (2026) [2]Ours (Figure 1a)
TrainingStudentQwen3-1.7B-BaseQwen3-1.7B-Base
TeacherQwen3-4B (non-thinking)Qwen3-4B (non-thinking)
DataDAPO-Math-17KDAPO-Math-17K
OPD lossclosed-form reverse KL, student top-16closed-form reverse KL, student top-16
Batch size (prompts)6464
Training length limit7,1687,168
Rollouts per prompt48
Learning rate1e-62e-6
EvaluationBenchmarksAIME24, AIME25, AMC23AIME24, AIME25, AMC23
Metricavg@16, macro over 3 benchmarksavg@16, macro over 3 benchmarks
Temperature / top-p0.7 / 0.950.7 / 0.95
Evaluation length limit31,7447,168

Bold marks where the two setups differ. Our tighter evaluation length limit can only lower our scores.

We use the Lightning OPD repository to reproduce OPD run in [1]:

AIME24 (avg@32)Lightning OPD [1], reportedOurs, Lightning OPD codebase
SFT student (Qwen3-4B-Base SFT'd on Qwen3-8B traces)56.759.7
Standard OPD (Qwen3-8B teacher, 150 steps), best checkpoint65.464.7

The table below reports the OOD macro avg@16; values for [2] are read from their Figure 2. We see that our OPD(Qwen3-4B) stays close to or outperform [2]; and the main results (Figure 1a) is reproducible using Lightning OPD codebase.

Li et al. (2026) [2]Our codebase (verl)Lightning OPD codebase (slime)
StepOPDOPDGRPOOPDGRPO
05.05.05.64.95.4
205.115.012.614.913.5
407.214.514.015.015.7
607.915.915.214.616.8
8010.315.814.815.216.9
10013.417.015.316.816.9
12014.016.415.917.118.9
14014.416.316.115.918.8
16015.016.818.118.018.8
18014.316.717.515.419.7
20015.116.418.416.320.1

In our SFT ablation study (Appendix D.6), we do not observe SFT init helps OPD, for example, after training Qwen3-4B-Base on QwQ-32B rollout, running OPD with QwQ-32B teacher doesn’t lead to better performance than using GRPO (Figure 16).

[1] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation. 2026. https://arxiv.org/abs/2604.13010

[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. 2026. https://arxiv.org/abs/2604.13016

AuthorsOpenReview ↗

Rebuttal 3/n

Q3 Can the author present some case study of the failure of OPD?

We observe many repetition in OPD with external teachers and we discuss them in Appendix E. We highlight two analyses that demonstrate it below:

a) response length dynamics during training (Figure 22). With a Qwen3-4B-Base student, OPD with Qwen3-4B teacher will make student rollout hit max response length within the first 20 steps, while GRPO and OPD with self-GRPO stays low.

StepGRPO lengthGRPO truncated (%)OPD (Qwen3-4B) lengthOPD (Qwen3-4B) truncated (%)OPD (self-GRPO) lengthOPD (self-GRPO) truncated (%)
17230.87100.87810.4
51,0281.92,73619.11,6293.9
91,0660.67,06795.83,96330.2
191,0170.67,167100.04,31628.9
501,1331.07,168100.03,63521.3
1002,1719.47,168100.02,85013.1
1502,60117.27,168100.02,2837.6
2003,81324.27,168100.03,74123.2

b) wasteful repetition (Table 8). We naively detect repetition by “when the 100-char window response[-200:-100] appears ≥ 3 times elsewhere in the response”. OPD with external teachers has substantial repetition, whereas OPD from the student's own GRPO checkpoint does not inflate.

MetricGRPOOPD, self-GRPO teacherOPD, external teacher
Repetition loop (naive detection)16.6%15.2%63.6%
Median response length (tokens)6,5046,54230,720 (= maximum)
  ↳ if detected loops removed4,5734,93721,031
Mean response length (tokens)11,23311,60530,229
  ↳ if detected loops removed7,7638,35319,627
≥ 95% of eval maximum19.0%21.0%97.8%
CJK language collapse0.0%0.0%0.2%
Missing \boxed{·}21.2%23.2%49.0%
Reviewer_xbyVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the training efficiency of on-policy distillation against GRPO in terms of accuracy and training FLOP. The authors conduct several experiments across different model size (e.g,. teacher size 4B/8B/14B student size 1.7b/4b), domains (math/code/Language understanding), and optimization. They found that distillating from external post-trained teacher is generally less compute-efficient than GRPO and can face response-length inflation. In contrast, using student model with GPRO post-training as teacher makes OPD to close to teacher's performance with substantially lower distillation-stage compute. (Although this way did not provide overall compute advantage, since we also need computation for each teacher training). Then, the authors conduct multi-teacher opd experiments, where they tried to distill multiple expert teachers into a single student models. The authors found, this way can exceed or match joint GRPO while need less total computation cost. In a two-domain settings, this pipeline reduce 42% total FLOPs.

Critical strengths
  • The experimental setup is solid. The paper compares opd, grpo in terms of accuracy and efficiency (FLOP) and controlled many conditions. The results are very clean to me.

  • The experiments are also reasonably comprehensive, spanning different model sizes, domains, training setup.

  • The presentation itself is also very clear.

Critical weaknesses
  • The multi-teacher OPD recipe has already been adopted in recent frontier-model post-training pipelines (e.g., deepseek), as acknowledged by the authors. Several other observations are also consistent with existing literature. While the systematic FLOPs-based comparison is valuable, the contribution is primarily empirical rather than introducing substantially new insights.

  • The analysis of token overlap and weight-spectrum similarity is mostly correlational and does not fully explain why self-GRPO teachers enable more efficient distillation.

  • The multi-teacher experiments are mainly conducted with a single 4B student model.

Questions for authors
  • Given that multi-teacher OPD has already been adopted in recent frontier-model post-training pipelines, what do the authors consider the main new conceptual or methodological insight of this work beyond the empirical results?
3 comments

W1 The multi-teacher OPD recipe has already been adopted in recent frontier-model post-training pipelines (e.g., deepseek), as acknowledged by the authors. Several other observations are also consistent with existing literature. While the systematic FLOPs-based comparison is valuable, the contribution is primarily empirical rather than introducing substantially new insights.

Q1 Given that multi-teacher OPD has already been adopted in recent frontier-model post-training pipelines, what do the authors consider the main new conceptual or methodological insight of this work beyond the empirical results?

Our new insight is what a matched-FLOPs comparison reveals, which no prior OPD work has characterized systematically: dense supervision alone does not make OPD cheaper than RL even in the setting that favors it the most (Section 4, Figure 2), and the teacher's training history (also called “provenance” in our paper) decides whether it does (Section 5). Our findings start from single-teacher OPD and extend to a widely adopted multi-teacher OPD recipe (Section 6). The conclusion (Lines 473–485) explains how the pieces fit together.

W2 The analysis of token overlap and weight-spectrum similarity is mostly correlational and does not fully explain why self-GRPO teachers enable more efficient distillation.

Our claim about which teachers work rests on interventions in Section 5, not on the correlational analyses. With the student, data and recipe fixed, changing only the teacher's training history (external thinking model, external model further trained with GRPO, or GRPO-trained base) changes OPD's compute efficiency. Changing only the teacher's size, from 4B to 14B, does not.

We agree that the overlap and spectrum analyses are correlational. They address a separate question, why training history matters, and we present them as a hypothesis.

To support the hypothesis, we measure weight-spectrum distance with the metrics of [1] (Appendix H):

  • The self-GRPO teacher is close to the student base, while the external teacher is far from it. In κspec\kappa_{\text{spec}}, the two differ by about 103×10^3 \times: 2.81 vs 3571 at 4B, and 1.19 vs 2077 at 8B.
  • OPD keeps the student within the RLVR range, even with an external teacher. The 4B OPD student reaches only κspec=3.42\kappa_{\text{spec}} = 3.42, far from its teacher's 3571.

Hypothesis: like RLVR updates [1], OPD updates change the base spectrum only slightly. OPD can therefore reach a teacher that is equally close to the base, such as the self-GRPO teacher, but not a teacher that is an SFT-scale change away.

Student base weight WBW_BW1W_1κspec\kappa_{\text{spec}}ρΣ\rho_{\Sigma}δΣ\delta_{\Sigma}∥ΔW∥\|\Delta W\|
Qwen3-4B-BaseQwen3-4B35710.1551.6×10−21.6 \times 10^{-2}162
Qwen3-4B-BaseGRPO-trained Qwen3-4B-Base2.810.0635.3×10−55.3 \times 10^{-5}1.33
Qwen3-4B-Basemid checkpoint of OPD (Qwen3-4B)3.160.0335.7×10−55.7 \times 10^{-5}2.69
Qwen3-4B-Baselast checkpoint of OPD (Qwen3-4B)3.420.0256.1×10−56.1 \times 10^{-5}3.76
Qwen3-8B-BaseQwen3-8B20770.1551.3×10−21.3 \times 10^{-2}202
Qwen3-8B-BaseGRPO-trained Qwen3-8B-Base1.190.1257.7×10−57.7 \times 10^{-5}1.48
Qwen3-8B-Basemid checkpoint of OPD (Qwen3-8B)1.500.0518.14×10−58.14 \times 10^{-5}3.83
Qwen3-8B-Baselast checkpoint of OPD (Qwen3-8B)1.610.0368.06×10−58.06 \times 10^{-5}5.38

[1] ISO: An RLVR-Native Optimization Stack. 2026. https://arxiv.org/abs/2607.19331

W3 The multi-teacher experiments are mainly conducted with a single 4B student model.

The multi-teacher experiments use one student, Qwen3-4B-Base, but the property that makes the recipe efficient is replicated across students. Multi-teacher OPD saves compute because each domain is distilled from the student's own GRPO checkpoint. We show that this single-teacher property holds along three independent axes (Section 5.1, Appendix G):

  • Student size: Qwen3-8B-Base.
  • Domain: code.
  • Model family and initialization: Olmo-3-7B-Instruct-SFT, which starts from an SFT checkpoint instead of a base model.

In all three, OPD recovers the self-GRPO teacher at a small fraction of GRPO's FLOPs. With the 4B student, the multi-teacher result holds on both domain mixes.

Thank you for the detailed response. I find the intervention results convincing in establishing that teacher provenance materially affects OPD efficiency, although the proposed explanation for why provenance matters remains largely correlational. My scale concern is also partially alleviated by the additional results with Qwen3-8B-Base and OLMo-3-7B-Instruct-SFT, although the full multi-teacher consolidation setting is still primarily evaluated with the 4B student.

My main remaining concern is therefore about the novelty of the contribution. I now better understand that the main contribution is the systematic matched-FLOPs characterization, rather than a new multi-teacher OPD method or a complete mechanistic explanation. While I find these empirical results useful, I am still somewhat uncertain whether they constitute a sufficiently substantial new insight beyond existing multi-teacher OPD practice.

I raised my soundness score to 2 to reflect the additional evidence and clarifications provided in the rebuttal.

We thank the reviewer for the positive feedback and for raising the soundness score.

My main remaining concern is therefore about the novelty of the contribution. [...] I am still somewhat uncertain whether they constitute a sufficiently substantial new insight beyond existing multi-teacher OPD practice.

Existing practice shows what frontier pipelines do, but it does not compare the recipe with its alternatives at matched compute. Section 6 makes that comparison. Counting specialist training, multi-teacher OPD matches joint GRPO with 42.0% fewer FLOPs on two domains and 31.6% fewer on four (Figure 5). The two obvious alternatives fail for different reasons.

Alternative 1: distill from one strong external teacher. Training with the Qwen3-8B teacher is unstable. On the 4-domain mix it collapses the student to 0% on math and code, even though this teacher is the strongest model overall (Figure 29). This is the multi-domain form of the external-teacher failure in Section 4.

Alternative 2: joint GRPO on the mixed data. Under joint GRPO, the domains interfere with each other, which shows up in response length (Appendix E.3). The single-domain specialists produce about 7.7K tokens on math and 1.0K on code (Table 9).

  • Two domains: math pulls code responses to 3.6–5.3× the code specialist's length (Figure 24). Accuracy does not improve: code stays near 41%, below the specialist's 46.4% (Figure 30).
  • Four domains: adding the short-response domains (IFBench, MMLU-Pro) pulls math down to 0.87× its specialist's length, and math accuracy becomes unstable (Figure 29).

Multi-teacher OPD avoids this interference. Each domain is trained by GRPO on its own, and distillation then reproduces each specialist's behavior: response lengths stay within 0.93–1.19× of the specialists on math, code and MMLU-Pro. Distillation costs only 5.2–8.0% of the specialist training FLOPs.

These results are what existing practice leaves unexplained: why the recipe separates RL by domain, and why it distills from the student's own checkpoints rather than from a stronger model.

Reviewer_WwF6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the training compute efficiency of strong-to-weak on-policy distillation (OPD) relative to GRPO. It compares external teachers with teachers obtained by applying GRPO to the student’s initialization, and studies whether separately trained domain specialists can be consolidated through multi-teacher OPD. The reported results show substantial sensitivity to teacher choice and training configuration. Distillation from a student-derived GRPO teacher can recover its performance with low additional compute, while the multi-domain pipeline reportedly outperforms joint GRPO even after including specialist training costs. The experimental scope is valuable, but several conclusions are broader than the evidence supports, and the quantitative efficiency claims require stronger validation.

Critical strengths
  • The paper addresses a practically important question: whether the dense supervision provided by OPD translates into lower compute requirements than outcome-based reinforcement learning. Comparing accuracy against cumulative FLOPs is more informative for this question than comparing optimizer steps alone.
  • The empirical investigation covers multiple relevant factors, including task difficulty, teacher and student sizes, initialization, domains, rollout group sizes, and implementation choices. This breadth helps reveal that OPD performance depends substantially on the experimental setting.
  • The distinction between distillation-stage efficiency and complete pipeline efficiency is useful and explicitly acknowledged. In particular, the paper recognizes that training a single-domain teacher and then recovering its performance through distillation does not itself save total compute. Figure 5 includes specialist training costs in the multi-domain comparison.
  • The multi-teacher experiments provide an interesting demonstration of capability consolidation. They compare against both instance-mixed and batch-mixed joint GRPO, and provide per-domain results in addition to aggregate scores. This makes the proposed use case more concrete than a distillation-only comparison.
  • The paper documents informative failure modes and boundary cases, including response-length inflation, unstable training, and domain-dependent effects of reducing the rollout group size. The observation that teacher strength alone does not predict successful transfer is useful for designing subsequent experiments.
Critical weaknesses
  • The broad negative characterization of external-teacher OPD is not fully consistent with the paper’s own results. In Figure 26, external-teacher OPD with G=1 exceeds direct GRPO at some overlapping compute budgets and achieves higher peak math accuracy for both student sizes. These results do not establish robust superiority across domains or full recovery of the external teacher, but they limit statements such as “no external teacher transfers compute-efficiently” in the Section 5 summary. The paper should distinguish recovery of each teacher’s own accuracy from performance at a common absolute target, and revise its conclusions to reflect the positive external-teacher results.
  • The multi-domain comparison does not sufficiently isolate the benefit of the proposed pipeline from differences in training configuration. Table 11 gives specialists domain-specific response caps, whereas Appendix J specifies a shared 16K cap for instance-mixed joint GRPO; the corresponding batch-mixed configuration is insufficiently documented. Data exposure is also described inconsistently: Section 6 says that domain counts match specialist training, while Appendix J states equal per-domain exposure. These ambiguities do not prove that the baseline is unfair, but they prevent a clear assessment. Joint GRPO should be evaluated with comparable domain-specific caps and reasonable tuning of domain sampling and loss weights, with actual exposure counts reported.
  • The headline efficiency ratios are not sufficiently reproducible from the reported information. Appendix B.1 counts generated tokens but excludes prompt processing, and its parameter-based approximation does not explicitly capture context-length-dependent attention costs. These omissions need not affect methods equally when teacher inference, response lengths, and rollout group sizes differ. The paper should disclose the target accuracy, checkpoint, and interpolation rule behind each ratio, and test sensitivity to more complete accounting. Numerical inconsistencies also require correction: the Olmo experiment is described as approximately 40× in Appendix G but 56× in Figure 27, while its teacher accuracy is reported as 42% in Section 5.1 and 40% in Appendix G.
  • Section 6 does not report variability across independent training seeds or confidence intervals, and mean@16 measures repeated generation from a trained model rather than independent training replication. Peak checkpoint comparisons also require a clear selection protocol, particularly when the same evaluation benchmarks are inspected repeatedly. The statement that the Olmo improvement is too large to attribute to randomness is therefore insufficiently supported. Repeated training runs for the key comparisons, paired evaluation uncertainty, and results from explicitly identified checkpoints would substantially strengthen the claims.
  • Response-length inflation and initial token overlap correlate with transfer outcomes, but the experiments do not establish which factors cause failure or explain the multi-domain advantage. Appendix H also interprets a scale-invariant spectral alignment statistic as evidence of distance, although that statistic alone does not measure displacement magnitude. Controlled interventions on length handling and domain weighting, together with comparisons separating specialist training from the subsequent distillation step, are needed to support the mechanistic interpretation. Without these controls, the paper establishes useful empirical associations rather than a general explanation of when OPD succeeds.
Questions for authors

Please refer to weaknesses.

9 comments
AuthorsOpenReview ↗

Rebuttal 1/3

W1 First, I doubt whether FLOPs are a meaningful metric for measuring practical training efficiency. Prefilling has much higher arithmetic intensity and GPU utilization than decoding, so fewer FLOPs do not necessarily mean shorter wall-clock time.

Q1 FLOPs is not 100% equals to efficiency in practice, for example, a small LLM decoding may consume less flops than a larger LLM prefilling, but the wall-clock time and the latency will be very different. Prefilling has much higher arithmetic density and GPU (tensor core) utility than decoding. Can the authors explain the set-up and authors provide wall-clock time or GPU-hours for their main experiments?

FLOP is a well-accepted and conventional metric for comparing training algorithms [1, 2]. Compared with wall-clock time, FLOP is agnostic to system choices such as GPU layout (e.g., using 1 GPU vs 8 GPU for rollout).

We agree that FLOPs and wall-clock time differ. Decoding is memory-bandwidth-bound, while prefill-style passes run near peak throughput. We report additional measures for runs in Figure 1(a):

MethodWall-clock (h)GPU-hoursFLOPs in Figure 1(a), use max length (101810^{18})FLOPs, use generated tokens (101810^{18})
GRPO6.2850.2145.9
OPD, Qwen3-4B teacher11.0788.62416.9
OPD, Qwen3-8B teacher11.4491.52721.9
OPD, Qwen3-32B teacher15.2121.76855.8

From the table, the ordering among runs is the same under all four measures. Therefore, our conclusion does not change.

[1] Scaling Laws for Neural Language Models. 2020. https://arxiv.org/abs/2001.08361

[2] Measuring the Algorithmic Efficiency of Neural Networks. 2020. https://arxiv.org/abs/2005.04305

AuthorsOpenReview ↗

Rebuttal 2/3

W2 the OPD performance reported in this paper is unusually poor compared with Tinker's official examples and other related work [1,2] on similar models and datasets.

Q2 Why does OPD perform so poorly in this paper compared with the successful training results reported in [1, 2]? … Have the authors tried reproducing these settings?

As mentioned in the appendix (Line 1448), we follow the official setting in Tinker's official examples as much as they specified. Because of the limited budget, we use 64 prompts per batch and 40 steps instead of 512 and 200, and our batch-size ablation (Appendix D.3) shows that this choice does not change the ordering. We disclose our reproduction code to show our practice is faithful.

Is our OPD under-tuned? No.

Comparison with [2]: Our main result’s (Figure 1a) setting closely follows Figure 2 in [2].

SettingLi et al. (2026) [2]Ours (Figure 1a)
TrainingStudentQwen3-1.7B-BaseQwen3-1.7B-Base
TeacherQwen3-4B (non-thinking)Qwen3-4B (non-thinking)
DataDAPO-Math-17KDAPO-Math-17K
OPD lossclosed-form reverse KL, student top-16closed-form reverse KL, student top-16
Batch size (prompts)6464
Training length limit7,1687,168
Rollouts per prompt48
Learning rate1e-62e-6
EvaluationBenchmarksAIME24, AIME25, AMC23AIME24, AIME25, AMC23
Metricavg@16, macro over 3 benchmarksavg@16, macro over 3 benchmarks
Temperature / top-p0.7 / 0.950.7 / 0.95
Evaluation length limit31,7447,168

Bold marks where the two setups differ. Our tighter evaluation length limit can only lower our scores.

We use the Lightning OPD repository to reproduce OPD run in [1]:

AIME24 (avg@32)Lightning OPD [1], reportedOurs, Lightning OPD codebase
SFT student (Qwen3-4B-Base SFT'd on Qwen3-8B traces)56.759.7
Standard OPD (Qwen3-8B teacher, 150 steps), best checkpoint65.464.7

The table below reports the OOD macro avg@16; values for [2] are read from their Figure 2. We see that our OPD(Qwen3-4B) stays close to or outperform [2]; and the main results (Figure 1a) is reproducible using Lightning OPD codebase.

Li et al. (2026) [2]Our codebase (verl)Lightning OPD codebase (slime)
StepOPDOPDGRPOOPDGRPO
05.05.05.64.95.4
205.115.012.614.913.5
407.214.514.015.015.7
607.915.915.214.616.8
8010.315.814.815.216.9
10013.417.015.316.816.9
12014.016.415.917.118.9
14014.416.316.115.918.8
16015.016.818.118.018.8
18014.316.717.515.419.7
20015.116.418.416.320.1

In our SFT ablation study (Appendix D.6), we do not observe SFT init helps OPD, for example, after training Qwen3-4B-Base on QwQ-32B rollout, running OPD with QwQ-32B teacher doesn’t lead to better performance than using GRPO (Figure 16).

[1] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation. 2026. https://arxiv.org/abs/2604.13010

[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. 2026. https://arxiv.org/abs/2604.13016

AuthorsOpenReview ↗

Rebuttal 3/3

Q3 Can the author present some case study of the failure of OPD?

We observe many repetition in OPD with external teachers and we discuss them in Appendix E. We highlight two analyses that demonstrate it below:

a) response length dynamics during training (Figure 22). With a Qwen3-4B-Base student, OPD with Qwen3-4B teacher will make student rollout hit max response length within the first 20 steps, while GRPO and OPD with self-GRPO stays low.

StepGRPO lengthGRPO truncated (%)OPD (Qwen3-4B) lengthOPD (Qwen3-4B) truncated (%)OPD (self-GRPO) lengthOPD (self-GRPO) truncated (%)
17230.87100.87810.4
51,0281.92,73619.11,6293.9
91,0660.67,06795.83,96330.2
191,0170.67,167100.04,31628.9
501,1331.07,168100.03,63521.3
1002,1719.47,168100.02,85013.1
1502,60117.27,168100.02,2837.6
2003,81324.27,168100.03,74123.2

b) wasteful repetition (Table 8). We naively detect repetition by “when the 100-char window response[-200:-100] appears ≥ 3 times elsewhere in the response”. OPD with external teachers has substantial repetition, whereas OPD from the student's own GRPO checkpoint does not inflate.

MetricGRPOOPD, self-GRPO teacherOPD, external teacher
Repetition loop (naive detection)16.6%15.2%63.6%
Median response length (tokens)6,5046,54230,720 (= maximum)
  ↳ if detected loops removed4,5734,93721,031
Mean response length (tokens)11,23311,60530,229
  ↳ if detected loops removed7,7638,35319,627
≥ 95% of eval maximum19.0%21.0%97.8%
CJK language collapse0.0%0.0%0.2%
Missing \boxed{·}21.2%23.2%49.0%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.