- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes Weak to Strong OPD method, with the goal to reduce computational cost of doing OPD at frontier scale models, they show in this work using Qwen3 Family models that Larget student model (8B) can get good learning signals from smaller teacher models from the same Family of models ( 4B, 0.6B). They aim to use directional update instead of imitating teacher and propose three ways to create proxy teacher with smaller model contrast pairs, namely : using expert RL, stronger base and Hints. They showed that their method improves performance on math/coding (non-agentic) benchmarks compared to baseline of plain OPD with smaller teacher and SFT.
Critical strengths
- The proposed method is simple and promising to improve student model with less computational resources, specifically extracting capability directions using smaller teacher model instead of training the same student with larger computational resources (expert RL).
- Performance Gains: From the Tables 2/3/4 , it's clear that the proposed method using all 3 different proxy teacher initialization works good compared to the baseline in the paper.
Critical weaknesses
-
One of my strongest concern regarding this method is that the paper only considers Qwen3 Family models, and lacks the evidence for cross family generalization of the method, because I believe its not practical to assume that everyone will have different sizes model from the same family. I suspect that the cross family experiments may not work this well, because "exact" directional capability extraction may not hold with models having different pertaining and SFT pipeline distributions, these differences can lead to contrast methods to encode capabilities that are not transferrable.
-
The work doesn't have any stronger baseline, like the exact OPD/MOPD setup, take student model do expert RL and then distill using the stronger Expert RL model to student. This brings out the question of training compute vs maximum performance gains, which to choose when sufficient compute is available?
Questions for authors
Following from my previous text:
- I would strongly encourage authors to provide any experiments with cross family models using their method.
- A stronger baseline with expert RL on same student not the weaker model of the same family.
I am happy to increase my scores if authors address these concerns.
3 comments
Thank you for providing the experiments to establish cross-family generalization and OPD baseline, I have increased my score to reflect the positive assessment of this work. Few things still missing, like end-to-end GPU-hours or training costs for the two pipelines, compute-efficiency gap still remains and for multi-domain setup an actual MOPD baseline.
Response to Reviewer bcfY
Thank you so much for your encouraging and positive assessment of our work! We further address your comments as follows:
Q1: End-to-end GPU-hours or training costs for the two pipelines, and compute-efficiency gap.
A1: Thank you for your helpful suggestions. We report the total GPU-hours for both training pipelines on math reasoning, including expert RL and subsequent distillation. The GPU-hours are measured on NVIDIA A100 GPUs.
| Training pipeline | Expert RL (GPU-hours) | Distillation (GPU-hours) | Total (GPU-hours) | Math Avg. |
|---|---|---|---|---|
| OPD with an 8B-RL expert | 1816.4 | 66.5 | 1882.9 | 49.0 |
| W2S-OPD with a 4B pre-/post-RL contrast | 1288.8 | 56.0 | 1344.8 | 51.8 |
W2S-OPD reduces end-to-end training GPU-hours by 28.6% while improving average math accuracy from 49.0% to 51.8%. These results demonstrate that W2S-OPD is able to achieve higher performance with lower training cost, when sufficient compute is available to train a student-sized RL expert. We added these results to Appendix E.7.
Q2: An actual MOPD baseline for the multi-domain setting.
A2: Thank you for your suggestion. We added a multi-teacher OPD (MOPD) baseline that jointly distills two Qwen3-8B-RL experts, trained separately on math and coding, into a single Qwen3-8B student. We compare this baseline with W2S-OPD using the corresponding Qwen3-4B pre-/post-RL contrasts. The results are shown below:
| Method | AIME24 | AIME25 | HMMT25 (Feb.) | HMMT25 (Nov.) | Math Avg. | HumanEval+ | MBPP+ | LCB | Code Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Neg. Model (4B) | 21.6 | 19.9 | 10.0 | 8.2 | 14.9 | 79.9 | 63.6 | 20.9 | 54.8 |
| Pos. Experts (4B-RL) | 62.0 | 57.9 | 34.8 | 40.4 | 48.8 | 86.3 | 71.0 | 27.1 | 61.5 |
| Student Base (8B) | 26.1 | 21.1 | 11.6 | 9.0 | 17.0 | 81.2 | 70.0 | 21.7 | 57.6 |
| Experts (8B-RL) | 70.2 | 59.2 | 35.0 | 47.5 | 53.0 | 87.0 | 72.0 | 28.6 | 62.5 |
| MOPD w/ 4B-RL experts | 64.3 | 54.9 | 29.6 | 37.4 | 46.5 | 83.4 | 69.5 | 25.6 | 59.5 |
| MOPD w/ 8B-RL experts | 66.2 | 58.0 | 32.5 | 42.9 | 49.9 | 84.3 | 70.1 | 26.4 | 60.3 |
| W2S-OPD | 67.4 | 61.0 | 34.7 | 45.2 | 52.1 | 85.2 | 71.0 | 27.1 | 61.1 |
W2S-OPD outperforms MOPD with student-sized 8B-RL experts on all seven benchmarks, improving average accuracy by 2.2 percentage points on math benchmarks and 0.8 percentage points on coding benchmarks. These results show that the performance advantage of W2S-OPD also extends to jointly transferring multiple domain capabilities into a single student. We added these results and discussions to Appendix E.6.
Thank you again for your helpful feedback and positive assessment of our work! We hope these additional results address your remaining concerns.
Response to Reviewer bcfY
We sincerely thank you for your valuable feedback and helpful suggestions. We appreciate your recognition of our method as simple and promising with good performance gains. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have.
Q1: The paper only considers Qwen3 models and lacks evidence of cross-family generalization. Differences in pretraining and SFT distributions may cause the contrast to encode capabilities that do not transfer to the student. Cross-family experiments are needed to assess whether the method remains effective beyond different sizes of the same model family.
A1: Thank you for your insightful comments and suggestion. We extend our evaluation to a cross-family setting using OLMo-3-7B-Instruct as the student, which differs from Qwen3 in both training background and vocabulary. We reuse the same contrast pairs from our main experiments: the math- and code-specific Qwen3-4B-RL experts serve as the positive models, and their pre-RL Qwen3-4B initialization serves as the negative model. To accommodate the tokenizer mismatch, we follow the cross-tokenizer alignment procedure in [1], applying supervision at strictly aligned token positions over the shared vocabulary. The results are presented below:
The results show that W2S-OPD consistently outperforms OPD on both math and coding benchmarks, providing evidence that the contrast extracted from Qwen3 models remains useful for an OLMo student despite differences in training background and vocabulary. These findings support the applicability of W2S-OPD beyond same-family teacher–student settings. We added these results and discussions to Section 4.4 in the main text.
Q2: The paper lacks a stronger OPD/MOPD baseline that first performs expert RL on the student-sized model and then distills the resulting stronger expert into the student. How does W2S-OPD compare with this baseline in terms of training compute and achievable performance, particularly when sufficient compute is available?
A2: Thank you for your helpful suggestion. We compare two pipelines: (i) performing expert RL on Qwen3-8B and distilling the resulting expert into the original Qwen3-8B model via OPD; and (ii) using the pre-/post-RL contrast from Qwen3-4B and Qwen3-4B-RL used in our main experiments to train Qwen3-8B via W2S-OPD. Both students start from the same Qwen3-8B initialization. We report math reasoning performance below.
The results show that W2S-OPD outperforms OPD using the student-sized 8B-RL expert on all four math benchmarks, achieving a 2.8 percentage-point gain in average accuracy. This suggests that training a student-sized RL expert does not necessarily yield a better distilled student. We hypothesize that a teacher’s standalone performance does not fully determine the effectiveness of its supervision. Although the 4B-RL expert is weaker than the 8B-RL expert, its pre-/post-RL contrast, when combined with the student’s initial distribution, may provide more effective guidance than directly distilling from the 8B-RL expert even though sufficient compute is available. We added the above results and discussion to the Appendix E.6.
[1] Hou, Bingxi, et al. "Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability." arXiv preprint arXiv:2610.08448 (2026).