acceptodds
Under review as a conference paper at ICLR 2027

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Abstract

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher–student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD improves the within-domain average over Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Teacher Reliability, Reinforcement Learning with Verifiable Rewards, LLM Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W242OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on a weakness of OPD where the reverse-KL teacher signal is applied on every prompt, even when the teacher is confidently wrong on that prompt, and since reverse KL is mode-seeking these errors get amplified. The authors first show that teacher confidence is a poor predictor of whether the teacher actually solves a prompt. They propose a method where the teacher generates 3 probe rollouts per prompt, a verifier scores them, and if at least 2 pass, the prompt is trained with dense OPD, otherwise it gates to GRPO on the student’s own rollouts (essentially a gating mechanism between two routes as shown in Figure 3). The probes run during the teacher node’s idle time in asynchronous training, raising teacher GPU utilization from a low 10% to a high 79%. Experiments with 4B and 35B Qwen students show gains over standard OPD in all single domain settings and in multi-domain training, with ablations on the gate threshold and on what to do after the gate closes.

Critical strengths
  1. The main idea is well motivated and the mechanism is verified with experiments. The idea resembles techniques used in classic KD papers which is here leveraged for OPD. Figure 2 shows that teacher confidence does not identify unreliable prompts, which justifies gating on verified outcomes instead of distributional proxies, and the closed-gate ablation shows that both gated variants beat standard OPD.

  2. the efficiency part of the method is a genuine contribution and is honestly reported. In asynchronous OPD the teacher’s GPUs sit idle most of the time while waiting for student rollouts, the probes are scheduled into exactly those gaps, and the authors measure the remaining slowdown on matched runs and report it. The threshold ablation showing that a simple majority vote over the probes works about as well as any tuned threshold is useful.

Critical weaknesses
  1. All the results seem to be a single training run per configuration (no seeds and variance) and the margins that matter are very small. At 4B the reported gap from the TrOPD baseline on the average of all architectures is only 0.16 points (very marginal), and the baseline RLSD-style performs worse than standard OPD, which raises concerns about fairness and how the baselines were tuned. Also, several benchmarks are evaluated with a single run (last 3 rows in Table 5), and the math benchmarks have only about 30 problems each, so consistently solving a single additional problem shifts accuracy by about 3 points which is larger than many of the reported margins between methods. The multi-domain experiments also compare only against vanilla OPD, with no other baselines. This basically reduces the claim to beating standard OPD and beating the baselines can't be claimed.

  2. The authors never test simpler alternatives that the pipeline already has, so it is unclear what the probe-based gate contributes. The mask-only ablation shows that simply withholding the teacher signal on rejected prompts captures most of the gain, and even beats the GRPO method at MOPD, so the mechanism is actually prompt-level filtering of unreliable supervision, while the novel part (the routing) is the inconsistent part. Moreover, the verifier already scores the student’s own rollouts on every prompt (as shown in the algorithm in the appendix), so a student pass rate is available without probe cost, and since each teacher is a GRPO trained copy of the same base model, it should correlate strongly with the teacher’s pass rate. A baseline that gates on this signal is never run. Since the teacher is just the same model trained with GRPO, the obvious baseline is to skip distillation and run GRPO on the student directly and the teacher row in Table 1 shows this ends up higher than TGOPD at 4B. There is no comparison of these two at equal compute and no training curves.

Questions for authors
  1. How many training runs are performed in Tables 1,2, and 3? Can you provide seeds and variances?

  2. Can you report the correlation between the teacher probe pass rate and the student group pass rate (already computed for every prompt), and the result of gating on the student pass rate instead of probing the teacher?

  3. Can you show TGOPD vs GRPO on the student at matched steps, including training curves?

  4. The passing probes are verified teacher solutions, yet they are only used for the routing decision and then discarded. Can you compare against using them as SFT targets (with GRPO on the rest), i.e. the same probe compute used for training rather than only for gating? Also, please report the value of beta from equation 1?

13 comments
AuthorsOpenReview ↗

Author Response to Reviewer W242 (1/5): Training Variability

We sincerely thank the reviewer for the thoughtful and constructive assessment of our work, particularly for recognizing the motivation behind verifier-grounded teacher gating and the systems-level contribution of reclaiming idle teacher GPU capacity. We appreciate the detailed suggestions regarding experimental robustness, baseline fairness, the necessity of teacher probing, and alternative uses of teacher supervision. To address these questions, we conducted additional experiments on training-seed variability, teacher–student reliability agreement, student-only gating, compute-matched GRPO, and verified-probe SFT. These results are now incorporated into the revised manuscript.

1. Training variability, baseline fairness, and additional MOPD comparisons (W1, Q1)

Training-seed variability and evaluation protocol. The original Tables 1–3 report one training run per configuration. To evaluate training stability, we conducted two additional independent TGOPD runs on both Math and Code using Qwen3.6-35B-A3B, resulting in three training seeds (1234, 1235, and 1236) per domain. These seeds control training randomness, including student rollout sampling.

Math — Qwen3.6-35B-A3B

Training seedAIME25AIME26HMMT-FebAvg.
1234 (original)74.078.764.172.27
123576.580.566.474.44
123676.178.364.773.04
Mean ± SD75.5 ± 1.379.2 ± 1.265.0 ± 1.273.25 ± 1.10

Code — Qwen3.6-35B-A3B

Training seedLCB GenOJBenchAvg.
1234 (original)64.028.746.35
123562.330.446.34
123662.731.046.85
Mean ± SD63.0 ± 0.930.0 ± 1.246.51 ± 0.29

The additional runs demonstrate consistent TGOPD performance across training seeds in both domains. In particular, all three Code runs outperform the original Vanilla OPD and TrOPD baselines on LiveCodeBench, providing further evidence for the repeatability of the Code improvements.

Given the computational demands of large-scale post-training and the available rebuttal training window, we prioritized additional independent TGOPD runs at the 35B scale across Math and Code. These experiments characterize the method's run-to-run variability across two distinct domains.

We also clarify the evaluation protocol. As detailed in Appendices C and J.2, each checkpoint is evaluated using 64 independent generations on AIME 2025/2026, 32 on HMMT-Feb, and 6 on LiveCodeBench, with scores averaged across generations. OJBench, IFBench, and IFEval use one evaluation run per checkpoint. Repeated generations reduce sensitivity to decoding randomness on fixed benchmark problems, while the additional training seeds separately assess variability across training runs.

AuthorsOpenReview ↗

Author Response to Reviewer W242 (2/5): Baseline Fairness and MOPD

1. Training variability, baseline fairness, and additional MOPD comparisons (continued; W1, Q1)

Baseline fairness and multi-domain comparisons. Our distillation comparisons share the student initialization, frozen domain teachers, training prompts, and asynchronous training infrastructure. The RLSD-style baseline is an adaptation that uses verifier-derived update directions with teacher-scaled token magnitudes. The baseline adaptation and shared optimization settings are summarized in Appendix B.1 and Table 4. Our evaluation also includes competitive OPD baselines such as TrOPD and RG-OPD.

The original seven-benchmark improvement over TrOPD is +0.16 points at 4B and +0.87 points at 35B, while TGOPD exceeds Vanilla OPD by +2.10 and +1.09 points, respectively (Table 1).

To strengthen the multi-domain evaluation, we additionally implemented TrOPD in the MOPD setting, extending the original comparison beyond Vanilla MOPD. The complete results at 200 rollout steps are included in the revised Table 2.

Qwen3.5-4B MOPD

MethodAIME25AIME26HMMT-FebLCBOJBenchIFBenchIFEvalAvg.
Vanilla MOPD58.163.850.750.518.148.883.953.40
MOPD + TrOPD (new)57.467.850.149.018.350.183.253.69
MOPD + TGOPD58.167.651.350.019.051.784.154.54

Qwen3.6-35B-A3B MOPD

MethodAIME25AIME26HMMT-FebLCBOJBenchIFBenchIFEvalAvg.
Vanilla MOPD73.177.360.757.423.344.490.760.99
MOPD + TrOPD (new)71.776.061.654.826.745.888.460.72
MOPD + TGOPD73.078.062.457.827.644.090.861.94

TGOPD improves over TrOPD by approximately +0.85 points at 4B and +1.23 points at 35B, outperforming it on six of seven benchmarks at both scales. It also exceeds Vanilla MOPD by +1.14 and +0.95 points, respectively.

These additional results extend the multi-domain comparison beyond Vanilla MOPD and provide further evidence for the effectiveness of verifier-grounded teacher admission against competitive distillation baselines.

AuthorsOpenReview ↗

Author Response to Reviewer W242 (3/5): Teacher vs. Student Gating

2. Why teacher probing instead of student-only gating? (W2, Q2)

We thank the reviewer for suggesting this important alternative. Teacher and student pass rates measure related but distinct quantities: the student's success rate reflects its current ability to solve a prompt, whereas teacher probing evaluates the reliability of the supervision source that OPD would follow. We therefore investigate their correlation, agreement in admission decisions, and the resulting training performance.

Teacher–student pass-rate correlation. We analyzed matched teacher–student prompt occurrences from the first 100 rollout steps of the 35B Math and Code runs. For this offline diagnostic, we use ten teacher responses and four recorded student responses per prompt, comparing their pass rates under a matched three-response evaluation protocol.

The results show moderate positive correlations:

DomainMatched groupsPearson rrSpearman ρ\rho
Code6,4000.5210.432
Math6,4000.5480.493

We further examined the resulting admission decisions under the same two-out-of-three threshold.

Although the teacher and student admitted sets exhibit substantial Jaccard overlap (92.1% on Code and 81.2% on Math), the overlap between their rejected sets is considerably lower (28.9% and 38.0%, respectively).

More importantly, student-based admission would accept 50.2% of the groups rejected by the teacher-based gate on Code and 43.3% on Math.

Thus, despite their positive correlation, the two signals can produce substantially different admission decisions, particularly on prompts where teacher verification indicates that dense supervision should be withheld.

The complete correlation analysis, admission-agreement statistics, and pass-count heatmaps are provided in Appendix G.3, Tables 13–14 and Figure 9.

Student-only gating baseline. We also implemented the alternative suggested by the reviewer. Specifically, we randomly select three responses from each four-response student rollout group and apply the same two-out-of-three admission rule to select between OPD and GRPO, keeping the student initialization, teachers, training prompts, remaining hyperparameters, and 100-rollout-step training horizon unchanged.

Math — Qwen3.6-35B-A3B

MethodAIME25AIME26HMMT-FebAvg.
Student-only gate73.377.562.671.14
TGOPD74.078.764.172.27

Code — Qwen3.6-35B-A3B

MethodLCB GenOJBenchAvg.
Student-only gate59.926.243.05
TGOPD64.028.746.35

TGOPD outperforms student-only gating by +1.12 points on Math and +3.30 points on Code, with improvements across all five benchmarks.

Together, the correlation analysis, differing admission decisions, and controlled training comparison demonstrate that independent teacher probing provides useful information beyond the student's existing verifier rewards. Directly evaluating teacher reliability is valuable because the gate determines whether the student's optimization should follow that teacher's dense supervision.

This also clarifies the contribution identified by the mask-only ablation: verifier-grounded prompt-level filtering is the consistent mechanism, while the additional benefit of GRPO fallback depends on the training setting. The student-only comparison further demonstrates the value of grounding this filtering decision in verified teacher outcomes rather than solely in student success.

Reviewer_Gx8POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Teacher-Gated On-Policy Distillation (TGOPD), which uses verified teacher performance on each prompt to decide whether the student should learn from dense teacher supervision or sparse-reward reinforcement learning. One motivation of the paper is that a teacher can be confidently wrong, so entropy or teacher–student agreement alone may not reveal harmful supervision. For each prompt, TGOPD generates three teacher probe responses and checks them with a task verifier: if at least two pass, it applies dense reverse-KL distillation to the student’s own rollouts; otherwise, it withholds teacher supervision and uses verifier-grounded GRPO, which produces an update when the student’s rollout group contains reward variation. The two supervision modes are mutually exclusive, making the central contribution a prompt-level admission decision for teacher guidance. The paper also schedules teacher probes concurrently with student generation to use otherwise-idle teacher capacity, keeping the measured runtime overhead modest. Empirical results show that TGOPD improves on OPD on several benchmarks.

Critical strengths

One critical strength of the paper is the practicality of the TGOPD method it proposes. The method setup is very straightforward and intuitive. The empirical results are promising.

Critical weaknesses

One critical weakness of this paper is the generalization of the TGOPD method. The paper does not sufficiently validate its teacher-reliability gate as a predictor of useful distillation supervision. The gate measures whether the teacher can solve a prompt through its own generated responses, success on the teacher’s trajectories does not necessarily indicate whether OPD will outperform verifier-based GRPO. I hope the authors can clarify that during the rebuttal.

Questions for authors

For the task verifier of the teacher's response, how do you correlate the verifier's judgement with the token-level supervision signals?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Gx8P (1/2): Teacher Reliability and OPD Scores

We sincerely thank the reviewer for recognizing the practicality of TGOPD and for raising this insightful and constructive question. We agree that a teacher's ability to solve a prompt does not necessarily imply that its token-level supervision is beneficial, nor that OPD is preferable to verifier-based GRPO. This observation motivated two complementary analyses: (1) how verified teacher reliability relates to the correctness preference of OPD scores on student-generated trajectories, and (2) whether teacher reliability is informative about the relative effectiveness of OPD versus GRPO.

In TGOPD, the verifier does not assign correctness labels to individual tokens or directly modify their OPD advantages. Instead, verified teacher outcomes determine whether the original dense OPD supervision is admitted for a prompt. When the gate opens, the token-level OPD signal is preserved; otherwise, teacher supervision is withheld in favor of verifier-grounded GRPO.

1. Connecting teacher verification to OPD scores on student trajectories

We analyzed the first 100 rollout steps of the Qwen3.6-35B-A3B single-domain Math and Code runs. Prompt instances are grouped by successful teacher probes (0/3, 1/3, 2/3, or 3/3). Within the same prompt and rollout step, we pair verified correct and incorrect student-generated responses, using mixed-outcome groups with valid, aligned token-level log-probabilities.

We use teacher–rollout-policy log-probability differences recorded before gating, allowing us to examine the original dense OPD signal on both admitted and rejected prompts. For each student response, we compute its mean per-token OPD score by averaging the token-level advantages in Eq. (1), as formalized in Eq. (8) of Appendix G.1:

s(x,y)=β∣y∣∑t=1∣y∣[log⁡πT(yt∣x,y<t)−log⁡πold(yt∣x,y<t)].s(x,y)=\frac{\beta}{|y|}\sum_{t=1}^{|y|}\left[\log\pi_T(y_t\mid x,y_{<t})-\log\pi_{\mathrm{old}}(y_t\mid x,y_{<t})\right].

Here, πold\pi_{\mathrm{old}} is the frozen rollout-policy snapshot and β=1.0\beta=1.0. We calculate Δ=s(x,y+)−s(x,y−)\Delta=s(x,y^+)-s(x,y^-), averaging valid correct/incorrect pairs within each prompt-step group before aggregating by teacher reliability. For Code, we additionally match responses of similar lengths.

Teacher probe passesCodeMath
0/3−1.13−1.20
1/3−0.33−0.99
2/3+0.80+0.92
3/3+2.32+3.58

Correct-minus-incorrect mean OPD score gap (10−310^{-3} nat/token), computed on mixed-outcome student groups.

The mean score gap increases monotonically with teacher verification success in both domains. With zero or one successful probe, incorrect responses receive higher mean OPD scores; with two or three successes, correct responses receive higher scores. This sign transition between 1/3 and 2/3 is consistent with TGOPD's two-out-of-three admission rule.

These results establish an empirical connection between verified teacher outcomes and the relative correctness preference of dense OPD scores on the student's own trajectories, providing a signal-level explanation for prompt-level teacher admission.

AuthorsOpenReview ↗

Author Response to Reviewer Gx8P (2/2): OPD vs. GRPO and Gating Implications

2. Does teacher reliability predict when OPD is preferable to GRPO?

The preceding analysis concerns the supervision signal itself. We further examine whether teacher reliability is informative about the actual training benefits of OPD relative to GRPO.

We conduct a controlled high-/low-reliability experiment on Code using Qwen3.6-35B-A3B. For each prompt, ten independently sampled teacher responses are scored by the verifier. We form a high-reliability subset (7–10/10 successful probes) and a low-reliability subset (0–5/10); prompts with exactly six successes are excluded to separate the subsets.

Each subset contains 512 training prompts and 181 held-out prompts. From the same student initialization, we train pure OPD and pure GRPO on identical training prompts within each subset for 100 rollout steps, yielding four controlled runs. The ten teacher samples are used only for this offline diagnostic; TGOPD itself uses three online teacher probes.

Held-out evaluation. We evaluate each resulting checkpoint on the corresponding unseen held-out prompts, sampling four responses per prompt (181 prompts, 724 responses per subset). Base accuracy characterizes initial solvability. As the subsets differ in difficulty, the comparison of interest is OPD versus GRPO within each subset, where initialization and training prompts are shared.

Teacher reliabilityBaseOPDGRPOOPD − GRPO
High (7–10/10)60.8%93.9%87.0%+6.9 pp
Low (0–5/10)22.8%38.0%52.9%−14.9 pp

The relative advantage of OPD versus GRPO reverses across the two reliability subsets. OPD leads by 6.9 percentage points on high-reliability prompts, whereas GRPO leads by 14.9 points on low-reliability prompts. The OPD–GRPO margin thus differs by 21.8 percentage points between subsets. These within-subset comparisons show that teacher-reliability stratification is informative about which objective is more effective under the evaluated conditions.

Independent benchmark evaluation. We also evaluate the same 100-step checkpoints on LiveCodeBench and OJBench.

Training subsetMethodLCB GenOJBench
High reliabilityOPD57.329.1
High reliabilityGRPO53.125.2
High reliabilityOPD − GRPO+4.3+3.9
Low reliabilityOPD56.125.0
Low reliabilityGRPO59.130.2
Low reliabilityOPD − GRPO−3.0−5.2

The same reversal appears on both independent benchmarks. High-reliability training favors OPD by +4.3 points on LiveCodeBench and +3.9 on OJBench, while low-reliability training favors GRPO by 3.0 and 5.2 points, respectively.

These comparisons are summarized in Figure 6b; full protocols appear in Appendix G.2.

3. Implications for prompt-level teacher gating

Together, these analyses provide complementary empirical support for TGOPD's teacher-admission mechanism: verified teacher reliability is informative about both the correctness preference of dense OPD scores and the relative effectiveness of OPD versus GRPO.

The signal-level analysis shows that, on mixed-outcome student groups, higher teacher verification success is associated with OPD scores that more strongly favor correct student responses. The controlled Code experiments further show that reliability-stratified subsets exhibit different relative benefits from OPD and GRPO, with consistent trends on held-out prompts and independent benchmarks.

These findings reinforce TGOPD's central principle: verified teacher reliability provides an outcome-grounded admission signal for dense OPD supervision, rather than assuming that a globally strong teacher provides equally useful token-level guidance on every prompt. The three-probe analysis relates teacher reliability to supervision-score preference on student trajectories, while the offline stratified experiments connect reliability to the relative value of the two training objectives.

We thank the reviewer for emphasizing the distinction between teacher solution correctness and supervision utility. This has helped us extend the paper beyond benchmark comparisons toward a more direct empirical account of why prompt-level verification is useful for selecting distillation supervision.

Both analyses and their experimental protocols are included in the revised manuscript (Section 4.7, Figure 6, and Appendix G.1–G.2, Tables 10–12).

AuthorsOpenReview ↗

Author Follow-up to Reviewer Gx8P: Teacher Reliability and OPD Supervision

Dear Reviewer Gx8P,

We sincerely appreciate your insightful feedback, which has helped us strengthen the empirical validation of our work. As the discussion period draws to a close, we would like to briefly highlight how our two-part rebuttal addresses your main concern.

We have provided two complementary analyses: (1) a signal-level study connecting verified teacher reliability to the correctness preference of token-level OPD scores on student-generated trajectories, and (2) controlled experiments examining the relative effectiveness of OPD and GRPO across teacher-reliability subsets. Both analyses have been incorporated into the revised manuscript (§4.7, Figure 6, and Appendix G.1–G.2).

We would be very grateful to know whether these additional results address your main concern. If any questions remain, we would be happy to provide further clarification before the discussion period ends.

Thank you again for your time and thoughtful consideration.

Best regards,
The Authors

I really appreciate the authors' rebuttal and the additional experiments further strengthen the empirical support on some of the claims. My biggest concern is the "reliability check" that determines which prompts would be routed to dense OPD. The reliability check is estimated through verifier-scored teacher probes, however, large teacher models' behavior is quite complex, and the estimation might not yield reliable prompt routing. I would maintain my score.

Reviewer_YJQGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Teacher-Gated On-Policy Distillation (TGOPD), a framework that addresses the risk of unreliable teacher supervision in on-policy distillation (OPD). The key idea is to estimate teacher reliability at the prompt level using a small number of verifier-scored teacher rollouts. Specifically, the empirical pass rate of these rollouts is used to determine whether a prompt receives dense OPD supervision or is instead trained using verifier-grounded GRPO. This selective routing mitigates the risk of applying misleading gradients from confidently incorrect teacher outputs. The method also leverages otherwise idle teacher GPU capacity during student rollout generation to perform reliability estimation. Experiments on Qwen3.5-4B and Qwen3.6-35B across mathematics, coding, and instruction-following tasks show improvements over standard OPD and multi-teacher distillation baselines.

Critical strengths
  1. The authors address an important limitation of OPD: confidently incorrect teacher predictions can negatively affect student learning. The motivation is clear and supported by the confidence-reliability analysis.

  2. The proposed approach is simple and practical, requiring relatively minor modifications to existing OPD pipelines. The mutually exclusive routing between OPD and GRPO is also intuitive.

  3. The experimental evaluation covers multiple domains and model scales, with comparisons against relevant baselines. The GPU utilization analysis provides an additional systems-level contribution. The authors evaluate TGOPD comprehensively across distinct domains (Math, Code, Instruction Following), two architectural scales (4B and 35B), and multi-domain (MOPD) settings.

Critical weaknesses
  1. The core approach combines verifier-based filtering with existing OPD and GRPO objectives. While the prompt-level routing is useful, the conceptual contribution appears somewhat incremental relative to existing reliability-aware distillation methods.

  2. The method uses only three teacher samples with a fixed majority-vote threshold. This introduces substantial estimation variance, particularly for prompts with intermediate teacher reliability. The paper would benefit from a more systematic analysis of probe count, routing errors, and sensitivity across domains.

  3. Although TGOPD consistently improves over vanilla OPD in domain averages, the improvement over TrOPD is only 0.16 points at 4B. Additionally, the mask-only ablation outperforms the proposed GRPO fallback in the 4B multi-domain setting, making the benefit of the fallback less convincing. However, the ablation study comparing the GRPO fallback to a "Mask only" approach (which simply drops the rejected prompt) reveals mixed results. While GRPO helps in some settings, masking actually outperforms the GRPO fallback in the 4B MOPD setting (54.97 vs. 54.54). This suggests that the primary driver of performance is simply filtering out bad teacher signals, making the necessity of the GRPO fallback setting-dependent.

Questions for authors

Few following questions:

  1. How sensitive is TGOPD to the number of teacher probes? Would increasing the probe count improve reliability estimation enough to justify the additional computation?

  2. Can the authors provide compute-matched comparisons against vanilla OPD and TrOPD, particularly in terms of final accuracy versus total GPU-hours?

  3. Why does mask-only gating outperform the GRPO fallback in the 4B multi-domain experiment? Are there particular prompt characteristics that determine when GRPO is beneficial?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer YJQG (1/4): Contribution and MOPD

We sincerely thank the reviewer for recognizing the motivation, practicality, comprehensive evaluation, and systems-level contributions of TGOPD. We appreciate the constructive suggestions regarding conceptual novelty, probe-count sensitivity, compute efficiency, and the role of GRPO fallback. The additional experiments and analyses are now included in the revised manuscript. We respond to each concern below.

1. Contribution beyond existing reliability-aware distillation (W1, W3)

We agree that TGOPD builds upon established OPD and GRPO objectives. Our central contribution is verifier-grounded, prompt-level teacher admission: determining whether dense teacher supervision should be trusted before it is applied.

Unlike approaches that regulate distillation primarily through distributional proxies or trajectory-level filtering, TGOPD directly audits the selected teacher using independently sampled, verifier-scored responses to the current prompt. When the audit passes, the original dense OPD supervision is preserved without attenuation; otherwise, potentially unreliable teacher supervision is withheld. This separates whether to trust the teacher from how to optimize the student, while leveraging otherwise-idle teacher capacity for efficient reliability estimation.

Our additional signal-level analysis further supports this design. Using matched correct/incorrect student-generated responses from the same prompt and rollout step, we find that the correct-minus-incorrect mean OPD score gap increases with verified teacher success. From 0/3 to 3/3 successful probes, this gap rises from −1.13 to +2.32 on Code and from −1.20 to +3.58 on Math (in units of 10⁻³ nat/token). The gap changes sign between one and two successful teacher probes, consistent with our two-of-three admission rule. These findings connect teacher reliability to the correctness preference of dense OPD scores on student-generated trajectories (Section 4.7, Figure 6a, and Appendix G.1, Table 10).

Regarding the modest 4B improvement over TrOPD, we acknowledge that the +0.16-point difference is small. However, the corresponding improvement reaches +0.87 points at 35B, while TGOPD exceeds Vanilla OPD by +2.10 and +1.09 points at 4B and 35B, respectively.

We also evaluated TrOPD in multi-domain training, extending the original comparison beyond Vanilla MOPD. These complete results after 200 rollout steps are now reported in Table 2 of the revised manuscript.

Qwen3.5-4B MOPD

MethodAIME25AIME26HMMT-FebLCBOJBenchIFBenchIFEvalAvg.
Vanilla MOPD58.163.850.750.518.148.883.953.40
MOPD + TrOPD (new)57.467.850.149.018.350.183.253.69
MOPD + TGOPD58.167.651.350.019.051.784.154.54

Qwen3.6-35B-A3B MOPD

MethodAIME25AIME26HMMT-FebLCBOJBenchIFBenchIFEvalAvg.
Vanilla MOPD73.177.360.757.423.344.490.760.99
MOPD + TrOPD (new)71.776.061.654.826.745.888.460.72
MOPD + TGOPD73.078.062.457.827.644.090.861.94

TGOPD improves over TrOPD by approximately +0.85 points at 4B and +1.23 points at 35B, outperforming it on six of seven benchmarks at both scales. TGOPD also exceeds Vanilla MOPD by +1.14 and +0.95 points, respectively. These additional comparisons strengthen the evidence for verifier-grounded teacher admission relative to competitive reliability-aware distillation baselines.

AuthorsOpenReview ↗

Author Response to Reviewer YJQG (2/4): Teacher Probe Count

2. Sensitivity to the number of teacher probes (W2, Q1)

We appreciate the concern regarding estimation variance with only three teacher probes, particularly for prompts with intermediate teacher reliability. Our objective is not to precisely estimate the teacher's success probability, but to obtain a lightweight and practically useful admission decision.

To systematically investigate probe-count sensitivity, we conducted additional experiments on Qwen3.6-35B-A3B with 3, 5, and 7 teacher probes, using the corresponding majority-vote rules of 2/3, 3/5, and 4/7. All configurations use the same training seed (1235) and matched training settings, and are evaluated after 100 completed rollout steps (checkpoint index 99). The three-probe configuration in this ablation therefore uses a different training seed from the original main-results run (seed 1234).

Math — Qwen3.6-35B-A3B

Probes / thresholdAIME25AIME26HMMT-FebAvg.
3 / 276.580.566.474.44
5 / 376.979.066.874.23
7 / 475.679.365.673.51

Code — Qwen3.6-35B-A3B

Probes / thresholdLCB GenOJBenchAvg.
3 / 262.330.446.34
5 / 361.128.544.75
7 / 462.928.245.55

Averages are computed before rounding; complete results appear in Appendix H.1, Table 15.

The three-probe configuration achieves the highest in-domain average on both Math and Code. On Math, five probes yield slightly higher scores on AIME25 and HMMT-Feb, but three probes achieve the highest domain average (74.44), outperforming five and seven probes by +0.21 and +0.93 points, respectively. On Code, three probes outperform five and seven probes by +1.59 and approximately +0.80 points.

As characterized by the finite-sample analysis in Appendix A.1, increasing the number of probes reduces estimator variance, but the additional estimation precision does not consistently translate into improved downstream accuracy. Meanwhile, five and seven probes require 66.7% and 133.3% more teacher-generated responses per prompt than three probes.

These findings complement the gate-threshold ablation in Section 4.5 (Figure 5) and show that a small number of probes provides an effective coarse teacher-admission signal in our evaluated settings. We therefore retain the three-probe majority-vote configuration as a practical balance between supervision quality and teacher sampling cost.

AuthorsOpenReview ↗

Author Response to Reviewer YJQG (3/4): Compute-Matched OPD

3. Compute-matched comparisons against Vanilla OPD and TrOPD (Q2)

We agree that comparing final accuracy under matched computational budgets is important, especially because TGOPD introduces additional teacher probing.

Our original systems analysis shows that most probing overlaps with otherwise-idle teacher capacity, with a measured 5.9% increase in mean time per rollout step relative to Vanilla OPD in the 35B Code experiment (Section 4.4 and Appendix D).

To further assess the accuracy–compute trade-off, we conducted additional GPU-hour-matched comparisons on Qwen3.6-35B-A3B. At the TGOPD budgets of approximately 303.0 GPU-hours on Math and 182.8 GPU-hours on Code (100 rollout steps), TrOPD and Vanilla OPD reach approximately 104 and 106 rollout steps, respectively. We evaluate the corresponding checkpoints below.

Math — Qwen3.6-35B-A3B

MethodRollout stepsGPU-hoursAIME25AIME26HMMT-FebAvg.
Vanilla OPD106≈303.073.877.163.471.44
TrOPD104≈304.774.378.263.872.08
TGOPD100303.074.078.764.172.27

Code — Qwen3.6-35B-A3B

MethodRollout stepsGPU-hoursLCB GenOJBenchAvg.
Vanilla OPD106≈182.859.828.744.24
TrOPD104≈184.058.428.943.63
TGOPD100182.864.028.746.35

GPU-hours are measured for TGOPD; TrOPD values are estimated from its measured runtime over 100 rollout steps, and Vanilla OPD values denote the approximately matched budget for the selected 106-rollout-step checkpoints. Averages and average improvements are computed before rounding. Full results appear in Appendix I.1, Tables 17–18.

Under approximately matched GPU-hour budgets, TGOPD improves the Math average by +0.82 points over Vanilla OPD and +0.18 points over TrOPD. The gains are more pronounced on Code, reaching +2.12 and +2.72 points, respectively.

The largest improvement appears on LiveCodeBench, where TGOPD exceeds Vanilla OPD by +4.2 points and TrOPD by +5.6 points. This is particularly relevant because our original analysis showed that teacher confidence is least informative about prompt-level reliability on Code.

These results complement the equal-rollout-step comparisons and provide further evidence that TGOPD's performance benefits persist when accounting for the additional computation introduced by teacher probing, particularly on Code.

AuthorsOpenReview ↗

Author Response to Reviewer YJQG (4/4): Gating vs. GRPO Fallback

4. Teacher admission versus GRPO fallback (W3, Q3)

We agree that the ablation identifies teacher admission as the more consistent source of improvement, while the additional benefit of GRPO fallback is setting-dependent.

Verifier-grounded prompt-level teacher admission is the central mechanism of TGOPD. The mask-only variant retains this decision while removing the replacement update. Its effectiveness therefore supports the admission mechanism itself. Both gated variants outperform ungated MOPD at both model scales (Section 4.6, Table 3).

GRPO fallback also provides additional benefits in single-domain mathematics. Compared with mask-only, it improves the three-benchmark average by +1.15 points at 4B and +0.60 points at 35B. Among teacher-rejected Math prompts, 79.0% and 84.7% of student rollout groups at the two scales contain mixed correct/incorrect outcomes, yielding non-zero verifier-grounded advantages. These observations support the additional benefit of fallback in single-domain mathematics (Appendix F, Figure 8).

Our additional 4B MOPD diagnostics shed further light on this setting dependence. Non-zero fallback signals from multiple domains co-occur in 40.7% of rollout batches. Math accounts for 52.9% of fallback groups with non-zero advantages and 53.2% of the summed squared fallback advantages (Appendix E.2, Table 9).

In multi-domain training, GRPO fallback introduces additional verifier-grounded RL updates from multiple domains into joint optimization. The interaction between these updates may create cross-domain optimization competition, consistent with the interference and see-saw effects discussed for Mix-RL in the MOPD study [1]. This provides a possible explanation for why fallback improves single-domain mathematics but does not offer the same additional benefit in 4B MOPD.

The ablation results reflect this setting dependence: mask-only performs better at 4B MOPD (54.97 vs. 54.54), whereas GRPO fallback is slightly stronger at 35B MOPD (61.94 vs. 61.78).

These results suggest setting-specific choices for the closed-gate update. In single-domain Math training, GRPO fallback is beneficial because rejected prompts frequently retain informative verifier-grounded learning signals.

For multi-domain capability integration, mask-only gating provides a simpler and competitive alternative, particularly when additional cross-domain RL interactions are undesirable. It retains verifier-grounded teacher admission while avoiding an additional RL objective on top of multi-domain OPD.

These findings further clarify the modularity of TGOPD: the core principle is to verify teacher reliability before admitting dense supervision, while the update applied to rejected prompts can be selected according to the training setting.

We thank the reviewer again for these valuable suggestions, which have helped us strengthen the experimental evidence and clarify the practical design choices of TGOPD.


Reference

[1] Ma, W., Wei, J., Zhao, L., et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv preprint arXiv:2606.30406. https://arxiv.org/abs/2606.30406

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Response and Manuscript Revisions

Dear Area Chairs and Reviewers,

We sincerely thank you for your thoughtful and constructive feedback. We also appreciate the positive comments on the practicality and intuitive design of TGOPD.

Our central contribution is a prompt-level teacher-reliability criterion for admitting dense OPD supervision, supported by empirical analyses connecting teacher reliability to supervision quality and training effectiveness. TGOPD combines this criterion with verifier-grounded GRPO fallback while reusing otherwise-idle teacher capacity for asynchronous probing.

We briefly summarize the key evidence and clarifications incorporated into the revised manuscript:

  • Reproducibility and baseline comparisons. We completed five-seed TGOPD–TrOPD comparisons at both model scales, with separate single-domain (SOPD) training on Math, Code, and instruction following. The seven-benchmark averages (mean ± sample SD) are 56.78 ± 0.28 vs. 56.18 ± 0.30 at 4B and 64.33 ± 0.46 vs. 62.78 ± 0.33 at 35B. TGOPD improves the overall average in all five seed-matched comparisons at both scales (Table 1, §4.2, Appendix J). The remaining baselines retain their original single-run evaluations; as noted in our earlier response, we plan to extend their multi-seed evaluation in a subsequent revision.

  • Empirical support for teacher-reliability gating. We added a signal-level analysis relating teacher verification success to OPD scores' preference for correct student responses. Controlled Code experiments further show that OPD outperforms GRPO on high-reliability prompts (+6.9 pp) but underperforms on low-reliability prompts (−14.9 pp), with consistent reversals on independent benchmarks (§4.7, Figure 6, Appendix G).

  • Teacher probe-count and gate sensitivity. We characterized finite-sample estimation uncertainty and evaluated gate thresholds and probe budgets. A five-probe threshold sweep peaks at the 3/5 majority rule on all three 4B Math benchmarks. In a controlled 35B comparison using the same training seed, three probes achieve the highest Math and Code domain averages among the tested budgets of 3, 5, and 7, while requiring fewer teacher samples (Appendix A.1, p. 13; §4.5, Figure 5, p. 8; Appendix H.1, Table 15, p. 21).

  • Supervision routing and efficiency. Both mask-only admission and GRPO fallback outperform ungated OPD in all four evaluated settings, supporting the value of teacher admission itself. GRPO fallback further outperforms masking in three settings, while mask-only performs better in 4B MOPD (§4.6, Table 3). We also clarified the teacher-training setup and the scope of approximately GPU-hour-matched GRPO comparisons with frozen teachers already available (§4.1, Appendix I). Reusing idle teacher capacity raises measured teacher-node GPU utilization from 9.8% to 78.9% at 4B and from 8.8% to 82.8% at 35B under SOPD (§4.4, Appendix D).

We hope this summary helps clarify how the revised manuscript addresses the main concerns raised during review. Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.