- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on a weakness of OPD where the reverse-KL teacher signal is applied on every prompt, even when the teacher is confidently wrong on that prompt, and since reverse KL is mode-seeking these errors get amplified. The authors first show that teacher confidence is a poor predictor of whether the teacher actually solves a prompt. They propose a method where the teacher generates 3 probe rollouts per prompt, a verifier scores them, and if at least 2 pass, the prompt is trained with dense OPD, otherwise it gates to GRPO on the student’s own rollouts (essentially a gating mechanism between two routes as shown in Figure 3). The probes run during the teacher node’s idle time in asynchronous training, raising teacher GPU utilization from a low 10% to a high 79%. Experiments with 4B and 35B Qwen students show gains over standard OPD in all single domain settings and in multi-domain training, with ablations on the gate threshold and on what to do after the gate closes.
Critical strengths
-
The main idea is well motivated and the mechanism is verified with experiments. The idea resembles techniques used in classic KD papers which is here leveraged for OPD. Figure 2 shows that teacher confidence does not identify unreliable prompts, which justifies gating on verified outcomes instead of distributional proxies, and the closed-gate ablation shows that both gated variants beat standard OPD.
-
the efficiency part of the method is a genuine contribution and is honestly reported. In asynchronous OPD the teacher’s GPUs sit idle most of the time while waiting for student rollouts, the probes are scheduled into exactly those gaps, and the authors measure the remaining slowdown on matched runs and report it. The threshold ablation showing that a simple majority vote over the probes works about as well as any tuned threshold is useful.
Critical weaknesses
-
All the results seem to be a single training run per configuration (no seeds and variance) and the margins that matter are very small. At 4B the reported gap from the TrOPD baseline on the average of all architectures is only 0.16 points (very marginal), and the baseline RLSD-style performs worse than standard OPD, which raises concerns about fairness and how the baselines were tuned. Also, several benchmarks are evaluated with a single run (last 3 rows in Table 5), and the math benchmarks have only about 30 problems each, so consistently solving a single additional problem shifts accuracy by about 3 points which is larger than many of the reported margins between methods. The multi-domain experiments also compare only against vanilla OPD, with no other baselines. This basically reduces the claim to beating standard OPD and beating the baselines can't be claimed.
-
The authors never test simpler alternatives that the pipeline already has, so it is unclear what the probe-based gate contributes. The mask-only ablation shows that simply withholding the teacher signal on rejected prompts captures most of the gain, and even beats the GRPO method at MOPD, so the mechanism is actually prompt-level filtering of unreliable supervision, while the novel part (the routing) is the inconsistent part. Moreover, the verifier already scores the student’s own rollouts on every prompt (as shown in the algorithm in the appendix), so a student pass rate is available without probe cost, and since each teacher is a GRPO trained copy of the same base model, it should correlate strongly with the teacher’s pass rate. A baseline that gates on this signal is never run. Since the teacher is just the same model trained with GRPO, the obvious baseline is to skip distillation and run GRPO on the student directly and the teacher row in Table 1 shows this ends up higher than TGOPD at 4B. There is no comparison of these two at equal compute and no training curves.
Questions for authors
-
How many training runs are performed in Tables 1,2, and 3? Can you provide seeds and variances?
-
Can you report the correlation between the teacher probe pass rate and the student group pass rate (already computed for every prompt), and the result of gating on the student pass rate instead of probing the teacher?
-
Can you show TGOPD vs GRPO on the student at matched steps, including training curves?
-
The passing probes are verified teacher solutions, yet they are only used for the routing decision and then discarded. Can you compare against using them as SFT targets (with GRPO on the rest), i.e. the same probe compute used for training rather than only for gating? Also, please report the value of beta from equation 1?
13 comments
Author Response to Reviewer W242 (2/5): Baseline Fairness and MOPD
1. Training variability, baseline fairness, and additional MOPD comparisons (continued; W1, Q1)
Baseline fairness and multi-domain comparisons. Our distillation comparisons share the student initialization, frozen domain teachers, training prompts, and asynchronous training infrastructure. The RLSD-style baseline is an adaptation that uses verifier-derived update directions with teacher-scaled token magnitudes. The baseline adaptation and shared optimization settings are summarized in Appendix B.1 and Table 4. Our evaluation also includes competitive OPD baselines such as TrOPD and RG-OPD.
The original seven-benchmark improvement over TrOPD is +0.16 points at 4B and +0.87 points at 35B, while TGOPD exceeds Vanilla OPD by +2.10 and +1.09 points, respectively (Table 1).
To strengthen the multi-domain evaluation, we additionally implemented TrOPD in the MOPD setting, extending the original comparison beyond Vanilla MOPD. The complete results at 200 rollout steps are included in the revised Table 2.
Qwen3.5-4B MOPD
| Method | AIME25 | AIME26 | HMMT-Feb | LCB | OJBench | IFBench | IFEval | Avg. |
|---|---|---|---|---|---|---|---|---|
| Vanilla MOPD | 58.1 | 63.8 | 50.7 | 50.5 | 18.1 | 48.8 | 83.9 | 53.40 |
| MOPD + TrOPD (new) | 57.4 | 67.8 | 50.1 | 49.0 | 18.3 | 50.1 | 83.2 | 53.69 |
| MOPD + TGOPD | 58.1 | 67.6 | 51.3 | 50.0 | 19.0 | 51.7 | 84.1 | 54.54 |
Qwen3.6-35B-A3B MOPD
| Method | AIME25 | AIME26 | HMMT-Feb | LCB | OJBench | IFBench | IFEval | Avg. |
|---|---|---|---|---|---|---|---|---|
| Vanilla MOPD | 73.1 | 77.3 | 60.7 | 57.4 | 23.3 | 44.4 | 90.7 | 60.99 |
| MOPD + TrOPD (new) | 71.7 | 76.0 | 61.6 | 54.8 | 26.7 | 45.8 | 88.4 | 60.72 |
| MOPD + TGOPD | 73.0 | 78.0 | 62.4 | 57.8 | 27.6 | 44.0 | 90.8 | 61.94 |
TGOPD improves over TrOPD by approximately +0.85 points at 4B and +1.23 points at 35B, outperforming it on six of seven benchmarks at both scales. It also exceeds Vanilla MOPD by +1.14 and +0.95 points, respectively.
These additional results extend the multi-domain comparison beyond Vanilla MOPD and provide further evidence for the effectiveness of verifier-grounded teacher admission against competitive distillation baselines.
Author Response to Reviewer W242 (3/5): Teacher vs. Student Gating
2. Why teacher probing instead of student-only gating? (W2, Q2)
We thank the reviewer for suggesting this important alternative. Teacher and student pass rates measure related but distinct quantities: the student's success rate reflects its current ability to solve a prompt, whereas teacher probing evaluates the reliability of the supervision source that OPD would follow. We therefore investigate their correlation, agreement in admission decisions, and the resulting training performance.
Teacher–student pass-rate correlation. We analyzed matched teacher–student prompt occurrences from the first 100 rollout steps of the 35B Math and Code runs. For this offline diagnostic, we use ten teacher responses and four recorded student responses per prompt, comparing their pass rates under a matched three-response evaluation protocol.
The results show moderate positive correlations:
| Domain | Matched groups | Pearson | Spearman |
|---|---|---|---|
| Code | 6,400 | 0.521 | 0.432 |
| Math | 6,400 | 0.548 | 0.493 |
We further examined the resulting admission decisions under the same two-out-of-three threshold.
Although the teacher and student admitted sets exhibit substantial Jaccard overlap (92.1% on Code and 81.2% on Math), the overlap between their rejected sets is considerably lower (28.9% and 38.0%, respectively).
More importantly, student-based admission would accept 50.2% of the groups rejected by the teacher-based gate on Code and 43.3% on Math.
Thus, despite their positive correlation, the two signals can produce substantially different admission decisions, particularly on prompts where teacher verification indicates that dense supervision should be withheld.
The complete correlation analysis, admission-agreement statistics, and pass-count heatmaps are provided in Appendix G.3, Tables 13–14 and Figure 9.
Student-only gating baseline. We also implemented the alternative suggested by the reviewer. Specifically, we randomly select three responses from each four-response student rollout group and apply the same two-out-of-three admission rule to select between OPD and GRPO, keeping the student initialization, teachers, training prompts, remaining hyperparameters, and 100-rollout-step training horizon unchanged.
Math — Qwen3.6-35B-A3B
| Method | AIME25 | AIME26 | HMMT-Feb | Avg. |
|---|---|---|---|---|
| Student-only gate | 73.3 | 77.5 | 62.6 | 71.14 |
| TGOPD | 74.0 | 78.7 | 64.1 | 72.27 |
Code — Qwen3.6-35B-A3B
| Method | LCB Gen | OJBench | Avg. |
|---|---|---|---|
| Student-only gate | 59.9 | 26.2 | 43.05 |
| TGOPD | 64.0 | 28.7 | 46.35 |
TGOPD outperforms student-only gating by +1.12 points on Math and +3.30 points on Code, with improvements across all five benchmarks.
Together, the correlation analysis, differing admission decisions, and controlled training comparison demonstrate that independent teacher probing provides useful information beyond the student's existing verifier rewards. Directly evaluating teacher reliability is valuable because the gate determines whether the student's optimization should follow that teacher's dense supervision.
This also clarifies the contribution identified by the mask-only ablation: verifier-grounded prompt-level filtering is the consistent mechanism, while the additional benefit of GRPO fallback depends on the training setting. The student-only comparison further demonstrates the value of grounding this filtering decision in verified teacher outcomes rather than solely in student success.
Author Response to Reviewer W242 (1/5): Training Variability
We sincerely thank the reviewer for the thoughtful and constructive assessment of our work, particularly for recognizing the motivation behind verifier-grounded teacher gating and the systems-level contribution of reclaiming idle teacher GPU capacity. We appreciate the detailed suggestions regarding experimental robustness, baseline fairness, the necessity of teacher probing, and alternative uses of teacher supervision. To address these questions, we conducted additional experiments on training-seed variability, teacher–student reliability agreement, student-only gating, compute-matched GRPO, and verified-probe SFT. These results are now incorporated into the revised manuscript.
1. Training variability, baseline fairness, and additional MOPD comparisons (W1, Q1)
Training-seed variability and evaluation protocol. The original Tables 1–3 report one training run per configuration. To evaluate training stability, we conducted two additional independent TGOPD runs on both Math and Code using Qwen3.6-35B-A3B, resulting in three training seeds (1234, 1235, and 1236) per domain. These seeds control training randomness, including student rollout sampling.
Math — Qwen3.6-35B-A3B
Code — Qwen3.6-35B-A3B
The additional runs demonstrate consistent TGOPD performance across training seeds in both domains. In particular, all three Code runs outperform the original Vanilla OPD and TrOPD baselines on LiveCodeBench, providing further evidence for the repeatability of the Code improvements.
Given the computational demands of large-scale post-training and the available rebuttal training window, we prioritized additional independent TGOPD runs at the 35B scale across Math and Code. These experiments characterize the method's run-to-run variability across two distinct domains.
We also clarify the evaluation protocol. As detailed in Appendices C and J.2, each checkpoint is evaluated using 64 independent generations on AIME 2025/2026, 32 on HMMT-Feb, and 6 on LiveCodeBench, with scores averaged across generations. OJBench, IFBench, and IFEval use one evaluation run per checkpoint. Repeated generations reduce sensitivity to decoding randomness on fixed benchmark problems, while the additional training seeds separately assess variability across training runs.