- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper compares OPD and GRPO in terms of training FLOPs. The authors find that OPD with external teachers generally performs worse than GRPO under the same FLOPs budget, while using the student's own GRPO checkpoint as the teacher can recover its performance with much fewer FLOPs. They further show that training separate GRPO models for different domains and merging them with OPD can save 42% FLOPs compared with joint GRPO.
Critical strengths
The paper conducts extensive experiments across different model sizes, datasets, teachers, and training settings.
Critical weaknesses
First, I doubt whether FLOPs are a meaningful metric for measuring practical training efficiency. Prefilling has much higher arithmetic intensity and GPU utilization than decoding, so fewer FLOPs do not necessarily mean shorter wall-clock time. Second, the OPD performance reported in this paper is unusually poor compared with Tinker's official examples and other related work [1,2] on similar models and datasets. This makes me question whether the results reflect general limitations of OPD or simply unsuccessful training setups.
[1] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation. 2026. https://arxiv.org/abs/2604.13010
[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. 2026. https://arxiv.org/abs/2604.13016
Questions for authors
-
FLOPs is not 100% equals to efficiency in practice, for example, a small LLM decoding may consume less flops than a larger LLM prefilling, but the wall-clock time and the latency will be very different. Prefilling has much higher arithmetic density and GPU (tensor core) utility than decoding. Can the authors explain the set-up and authors provide wall-clock time or GPU-hours for their main experiments?
-
Why does OPD perform so poorly in this paper compared with the successful training results reported in [1, 2]? In particular, [1] identifies teacher consistency between SFT and OPD as an important factor for successful training, while [2] emphasizes thinking-pattern alignment and teacher-generated cold starts. Although this paper includes SFT ablations, it does not seem to systematically test these recipes with its external teachers. Have the authors tried reproducing these settings?
-
In my experience, without this SFT initialization, OPD can produce many negative token-level advantages, making the model to repeat certain high-probability words and inflate its response length. I suspect something similar may be happening in this paper. Can the author present some case study of the failure of OPD?
3 comments
Rebuttal 2/n
W2 the OPD performance reported in this paper is unusually poor compared with Tinker's official examples and other related work [1,2] on similar models and datasets.
Q2 Why does OPD perform so poorly in this paper compared with the successful training results reported in [1, 2]? … Have the authors tried reproducing these settings?
As mentioned in the appendix (Line 1448), we follow the official setting in Tinker's official examples as much as they specified. Because of the limited budget, we use 64 prompts per batch and 40 steps instead of 512 and 200, and our batch-size ablation (Appendix D.3) shows that this choice does not change the ordering. We disclose our reproduction code to show our practice is faithful.
Is our OPD under-tuned? No.
Comparison with [2]: Our main result’s (Figure 1a) setting closely follows Figure 2 in [2].
| Setting | Li et al. (2026) [2] | Ours (Figure 1a) | |
|---|---|---|---|
| Training | Student | Qwen3-1.7B-Base | Qwen3-1.7B-Base |
| Teacher | Qwen3-4B (non-thinking) | Qwen3-4B (non-thinking) | |
| Data | DAPO-Math-17K | DAPO-Math-17K | |
| OPD loss | closed-form reverse KL, student top-16 | closed-form reverse KL, student top-16 | |
| Batch size (prompts) | 64 | 64 | |
| Training length limit | 7,168 | 7,168 | |
| Rollouts per prompt | 4 | 8 | |
| Learning rate | 1e-6 | 2e-6 | |
| Evaluation | Benchmarks | AIME24, AIME25, AMC23 | AIME24, AIME25, AMC23 |
| Metric | avg@16, macro over 3 benchmarks | avg@16, macro over 3 benchmarks | |
| Temperature / top-p | 0.7 / 0.95 | 0.7 / 0.95 | |
| Evaluation length limit | 31,744 | 7,168 |
Bold marks where the two setups differ. Our tighter evaluation length limit can only lower our scores.
We use the Lightning OPD repository to reproduce OPD run in [1]:
| AIME24 (avg@32) | Lightning OPD [1], reported | Ours, Lightning OPD codebase |
|---|---|---|
SFT student (Qwen3-4B-Base SFT'd on Qwen3-8B traces) | 56.7 | 59.7 |
Standard OPD (Qwen3-8B teacher, 150 steps), best checkpoint | 65.4 | 64.7 |
The table below reports the OOD macro avg@16; values for [2] are read from their Figure 2. We see that our OPD(Qwen3-4B) stays close to or outperform [2]; and the main results (Figure 1a) is reproducible using Lightning OPD codebase.
| Li et al. (2026) [2] | Our codebase (verl) | Lightning OPD codebase (slime) | |||
|---|---|---|---|---|---|
| Step | OPD | OPD | GRPO | OPD | GRPO |
| 0 | 5.0 | 5.0 | 5.6 | 4.9 | 5.4 |
| 20 | 5.1 | 15.0 | 12.6 | 14.9 | 13.5 |
| 40 | 7.2 | 14.5 | 14.0 | 15.0 | 15.7 |
| 60 | 7.9 | 15.9 | 15.2 | 14.6 | 16.8 |
| 80 | 10.3 | 15.8 | 14.8 | 15.2 | 16.9 |
| 100 | 13.4 | 17.0 | 15.3 | 16.8 | 16.9 |
| 120 | 14.0 | 16.4 | 15.9 | 17.1 | 18.9 |
| 140 | 14.4 | 16.3 | 16.1 | 15.9 | 18.8 |
| 160 | 15.0 | 16.8 | 18.1 | 18.0 | 18.8 |
| 180 | 14.3 | 16.7 | 17.5 | 15.4 | 19.7 |
| 200 | 15.1 | 16.4 | 18.4 | 16.3 | 20.1 |
In our SFT ablation study (Appendix D.6), we do not observe SFT init helps OPD, for example, after training Qwen3-4B-Base on QwQ-32B rollout, running OPD with QwQ-32B teacher doesn’t lead to better performance than using GRPO (Figure 16).
[1] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation. 2026. https://arxiv.org/abs/2604.13010
[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. 2026. https://arxiv.org/abs/2604.13016
Rebuttal 3/n
Q3 Can the author present some case study of the failure of OPD?
We observe many repetition in OPD with external teachers and we discuss them in Appendix E. We highlight two analyses that demonstrate it below:
a) response length dynamics during training (Figure 22). With a Qwen3-4B-Base student, OPD with Qwen3-4B teacher will make student rollout hit max response length within the first 20 steps, while GRPO and OPD with self-GRPO stays low.
| Step | GRPO length | GRPO truncated (%) | OPD (Qwen3-4B) length | OPD (Qwen3-4B) truncated (%) | OPD (self-GRPO) length | OPD (self-GRPO) truncated (%) |
|---|---|---|---|---|---|---|
| 1 | 723 | 0.8 | 710 | 0.8 | 781 | 0.4 |
| 5 | 1,028 | 1.9 | 2,736 | 19.1 | 1,629 | 3.9 |
| 9 | 1,066 | 0.6 | 7,067 | 95.8 | 3,963 | 30.2 |
| 19 | 1,017 | 0.6 | 7,167 | 100.0 | 4,316 | 28.9 |
| 50 | 1,133 | 1.0 | 7,168 | 100.0 | 3,635 | 21.3 |
| 100 | 2,171 | 9.4 | 7,168 | 100.0 | 2,850 | 13.1 |
| 150 | 2,601 | 17.2 | 7,168 | 100.0 | 2,283 | 7.6 |
| 200 | 3,813 | 24.2 | 7,168 | 100.0 | 3,741 | 23.2 |
b) wasteful repetition (Table 8). We naively detect repetition by “when the 100-char window response[-200:-100] appears ≥ 3 times elsewhere in the response”. OPD with external teachers has substantial repetition, whereas OPD from the student's own GRPO checkpoint does not inflate.
| Metric | GRPO | OPD, self-GRPO teacher | OPD, external teacher |
|---|---|---|---|
| Repetition loop (naive detection) | 16.6% | 15.2% | 63.6% |
| Median response length (tokens) | 6,504 | 6,542 | 30,720 (= maximum) |
| ↳ if detected loops removed | 4,573 | 4,937 | 21,031 |
| Mean response length (tokens) | 11,233 | 11,605 | 30,229 |
| ↳ if detected loops removed | 7,763 | 8,353 | 19,627 |
| ≥ 95% of eval maximum | 19.0% | 21.0% | 97.8% |
| CJK language collapse | 0.0% | 0.0% | 0.2% |
Missing \boxed{·} | 21.2% | 23.2% | 49.0% |
Rebuttal 1/n
FLOP is a well-accepted and conventional metric for comparing training algorithms [1, 2]. Compared with wall-clock time, FLOP is agnostic to system choices such as GPU layout (e.g., using 1 GPU vs 8 GPU for rollout).
We agree that FLOPs and wall-clock time differ. Decoding is memory-bandwidth-bound, while prefill-style passes run near peak throughput. We report additional measures for runs in Figure 1(a):
From the table, the ordering among runs is the same under all four measures. Therefore, our conclusion does not change.
[1] Scaling Laws for Neural Language Models. 2020. https://arxiv.org/abs/2001.08361
[2] Measuring the Algorithmic Efficiency of Neural Networks. 2020. https://arxiv.org/abs/2005.04305