acceptodds
Under review as a conference paper at ICLR 2027

DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation

158 views

Abstract

On-policy distillation (OPD) has emerged as a widely used paradigm for post-training large language models, reducing the train–test mismatch of conventional distillation by supervising the student on its own generated trajectories. However, existing OPD objectives remain largely token-local and outcome-agnostic, optimizing teacher–student agreement at each prefix despite reasoning quality being determined at the trajectory level. Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), provides complementary outcome-level supervision but suffers from sparse rewards and coarse credit assignment. We show that OPD and RLVR exhibit complementary blind spots: teacher signals provide dense local guidance but are weakly aligned with rollout correctness, whereas group-relative rewards capture task success but provide coarse token-level credit and vanish on all-failure groups. We introduce DiffGate, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance. Teacher supervision is applied only to failed trajectories, scaled by group difficulty, and smoothly bounded to prevent extreme teacher–student discrepancies from dominating optimization. The verifier therefore determines which trajectories receive teacher guidance, while the teacher provides dense token-level update directions within those trajectories. Across Qwen3-0.6B and Qwen3-1.7B students, DiffGate improves code avg@8 over matched GRPO by and points and pass@8 by and points, respectively. On mathematics, avg@8 remains within points of GRPO while pass@8 improves by and points. Overall, DiffGate improves pass@8 across all four model–domain settings, demonstrating improved solution coverage under our evaluation protocol.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Reinforcement Learning with Verifiable Rewards, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bMgyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DiffGate, which combines GRPO with on-policy distillation. It adds teacher guidance only to failed responses, adjusts its strength based on the group solve rate, and uses tanh to limit the size of the teacher signal. The authors evaluate it on mathematics and code tasks with Qwen3 models, report improvements in average pass@8, and analyze difficulty weighting, signal bounding, and training behavior.

Critical strengths

S1: The approach is clear and easy to understand. Adding teacher guidance when the student cannot solve a problem is an understandable approach. It can also provide a learning signal for all-failure groups without changing the basic process of generating student responses.

S2: Some metrics improve. Average pass@8 is higher than GRPO on both mathematics and code, and average code accuracy also improves. The authors include training analyses and ablations, so the evaluation goes beyond final scores.

Critical weaknesses

W1: Directly related methods are missing from the comparison, and the new contribution is not very clear.

Several prior works already combine GRPO and OPD, but the paper does not directly compare with methods such as KDRL and RLAD. In particular, KDRL already studies distillation on incorrect responses and all-failure groups. DiffGate mainly adds difficulty weighting and tanh bounding, which are fairly straightforward changes. Although the authors report some ablations, removing difficulty weighting also increases the total amount of teacher guidance. This leaves it unclear whether the proposed design is better than simply adjusting the teacher weight. I do not see a clear advance over prior methods from the current results.

W2: The results are not convincing enough, and the model coverage is narrow.

The method does not consistently outperform GRPO. Average mathematics accuracy is slightly lower at both student sizes, and both avg@8 and pass@8 on LiveCodeBench are lower at both sizes. Average pass@8 does improve, but this alone does not make the overall results particularly strong. Also, all students and teachers come from Qwen3. Testing several model sizes does not show that the method works across model families, and transfer to science tasks does not answer that question either. I therefore still have doubts about how broadly the method applies.

Questions for authors

Q1: Why are directly related methods such as KDRL, KDRL-Masking, and RLAD not included? Are there results under the same settings that show where DiffGate improves over these methods? This would affect my assessment of the new contribution.

Q2: Was the teacher weight tuned separately for the baseline without difficulty weighting? If the overall strength of teacher guidance is made comparable, does DiffGate still have a clear advantage? I would like to understand whether the gains come from difficulty-based allocation or simply from reducing the teacher's influence.

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for their constructive feedback and for noting that our approach is “clear and easy to understand” and provides “a learning signal for all-failure groups.” We address each concern point by point below.

W1 and Q1: Related Work and Contribution.

Response: We have added direct comparisons with plain KDRL and KDRL-Masking and clarified what DiffGate contributes. KDRL already explores joint GRPO–OPD training. Our contribution is how teacher guidance is weighted and bounded within this setting.

Additional baseline results. DiffGate reports mean ± standard deviation over three seeds; the KDRL variants report single-run results.

MethodMath 0.6B avg@8Math 0.6B pass@8Math 1.7B avg@8Math 1.7B pass@8Code 0.6B avg@8Code 0.6B pass@8Code 1.7B avg@8Code 1.7B pass@8
Plain KDRL12.225.621.033.731.348.343.656.7
KDRL-Masking12.426.119.331.331.247.243.855.8
DiffGate (3 seeds)13.1±0.729.4±1.221.2±0.637.5±1.432.7±0.749.9±0.444.9±0.461.4±0.4

DiffGate’s mean scores are higher on both metrics across all four settings. Its pass@8 gains are 1.6–4.7 points over KDRL and 2.7–6.2 points over KDRL-Masking. These are observed improvements; we do not claim statistical significance because the KDRL variants lack multi-seed uncertainty estimates.

What DiffGate adds. A binary mask determines whether a response receives teacher guidance. DiffGate additionally determines how much guidance it receives: the weight (1−Rˉ)γ(1-\bar R)^\gamma assigns stronger guidance to groups with lower solve rates and reduces it as the student becomes more successful. Its bounded teacher advantage also limits extreme token-level contributions.

The manuscript evaluates these components separately:

  • Difficulty weighting: Sec. 5.2 and Appendix E.5, Table 11 report 1.3–2.4-point avg@8 gains over the tested constant-weight baseline with bounded, failure-only guidance.
  • Bounding: Appendix E.4, Table 10 reports 1.0–1.5-point avg@8 gains, alongside reduced gradient spikes.
  • Response selection: Appendix E.6, Table 12 provides additional all-response and failed-only controls.

Removing difficulty weighting changes both where guidance is allocated and its overall strength. Table 11 therefore evaluates their combined effect. We address the reviewer’s question about lower constant teacher weights separately in Q2, including the additional controls.

RLAD. We have not included numerical comparisons with RLAD because we could not find an official publicly released implementation. Reimplementing it would require additional validation to ensure a fair comparison.

W2. Empirical Strength and Model Coverage.

Response: We have strengthened the evaluation in three ways: multiseed run, additional reasoning tasks, and experiments beyond the Qwen family. The new results show improvements in both average correctness and solution coverage across the additional task and model settings.

1. Updated three-seed results. Our evaluation across three random seeds shows that DiffGate achieves higher mean avg@8 and pass@8 than GRPO, with gains of 0.9–1.5 points in avg@8 and 2.1–5.1 points in pass@8 across the aggregate mathematics and code results at both student sizes. These results strengthen the original findings: DiffGate improves both aggregate accuracy and solution coverage.

We also repeated the LiveCodeBench experiments three times. As shown below, DiffGate achieves higher mean scores on both metrics at both model sizes, although these gains are modest relative to the variability across runs.

Model sizeMethodavg@8pass@8
0.6BDiffGate13.9 ± 0.4424.7 ± 1.72
0.6BGRPO13.1 ± 0.5023.0 ± 0.47
1.7BDiffGate18.6 ± 0.3630.3 ± 1.68
1.7BGRPO18.2 ± 0.8329.7 ± 1.05

2. Broader task coverage. We added multimodal reasoning with a Qwen3-VL-2B student and 8B teacher, and multi-turn text-to-SQL with a Qwen3-1.7B student and 8B teacher. See General Comment response.

DiffGate improves both avg@8 and pass@8 over GRPO in all six evaluations. It also outperforms OPD on both metrics except MathVista pass@8, where OPD leads by 0.2 points. The Spider scores remain unchanged when evaluated only on questions shared across all models—1,032 development and 2,147 test questions—confirming that the gains do not arise from different evaluation subsets.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

3. Coverage beyond Qwen. To directly address the model-family concern, we added code-generation experiments with Gemma 2 (2B student, 9B teacher) and Gemma 3 (4B student, 12B teacher). See General Comment response.

DiffGate outperforms GRPO on both metrics in every reported Gemma comparison. For Gemma 2, gains are 2.2–3.8 points in avg@8 and 5.2–10.5 points in pass@8. For Gemma 3, gains hold across all three benchmarks, including LiveCodeBench, where avg@8 improves by 1.7 points and pass@8 by 5.8 points.

These experiments directly address the coverage concern: DiffGate’s benefits extend beyond Qwen and beyond pass@8 alone, with improvements in both metrics across the reported Gemma, visual reasoning, and multi-turn SQL evaluations.

Q2. Teacher Weight and Difficulty based Allocation.

Response: We have added lower constant-weight controls to test whether simply reducing teacher strength can reproduce DiffGate’s gains. These experiments complement Appendix E.5, Table 11.

All variants use the same bounded teacher advantage on failed responses. They differ in how that guidance is weighted (setting λ\lambda):

  • Constant-weight controls: Every failed response receives the same coefficient, with γ=0\gamma=0.
  • DiffGate: The coefficient is 1−Rˉ1-\bar R, where Rˉ\bar R is the group solve rate. Harder groups therefore receive stronger guidance.

We test a constant coefficient of 1.0 and reduced coefficients of 0.80, 0.70, 0.87, and 0.78 for Math 0.6B, Math 1.7B, Code 0.6B, and Code 1.7B, respectively.

MethodMath 0.6B avg@8Math 0.6B pass@8Math 1.7B avg@8Math 1.7B pass@8Code 0.6B avg@8Code 0.6B pass@8Code 1.7B avg@8Code 1.7B pass@8
Constant weight, λ=1\lambda=112.427.419.435.632.650.344.560.7
Constant weight, reduced λ\lambda12.027.520.636.331.848.244.959.6
DiffGate13.729.521.139.133.049.845.261.4

DiffGate outperforms the reduced-weight control on both metrics in all four settings, improving avg@8 by 0.3–1.7 points and pass@8 by 1.6–2.8 points. It also exceeds both constant-weight controls on avg@8 in all four settings and on pass@8 in three of four.

The key distinction is that DiffGate adjusts guidance separately for each group, even within the same update. A lower constant weight reduces guidance equally for every failed response; DiffGate concentrates stronger guidance on groups the student finds harder. The results show that the tested constant-weight reductions do not recover DiffGate’s overall performance, supporting the benefit of its adaptive weighting design.

We would be happy to answer any remaining questions and respectfully ask you to consider raising your score if these revisions satisfactorily address your concerns.

AuthorsOpenReview ↗

Follow up on Rebuttal

Dear Reviewer bMgy,

We look forward to your feedback on our rebuttal. To address your concerns, we added direct comparisons with KDRL and KDRL-Masking, lower constant-weight controls to assess difficulty-based allocation, and three-seed evaluations, including LiveCodeBench. We also clarified DiffGate’s contribution and expanded evaluation to Gemma 2 and Gemma 3, multimodal reasoning, and multi-turn text-to-SQL.

We hope these additions address your concerns about novelty, teacher weighting, and generalization. We would be happy to discuss any remaining questions.

Thank you,
Authors

Reviewer_zSrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DiffGate, which combines on-policy distillation (OPD) with GRPO. The method applies bounded teacher guidance only to failed trajectories and scales its strength through difficulty gating. The authors compare DiffGate with existing methods and conduct ablation studies to evaluate its design choices.

Critical strengths

The experimental evaluation covers multiple student model sizes, mathematical reasoning and code generation, with comparisons against several baselines and extensive ablation studies.

Critical weaknesses
  1. The method mainly combines existing OPD and GRPO signals through teacher guidance on failed trajectories, difficulty gating, and bounded teacher guidance. Its technical contribution is relatively limited, and the necessity of these designs compared with simpler combination strategies has not been sufficiently established.

  2. Performance gains depend on the task and metric. On mathematics, avg@8 is slightly below GRPO, with improvements mainly in pass@8. On code, average performance improves, but DiffGate underperforms GRPO on LiveCodeBench, and some gains over the strongest existing baseline are small. In the mathematical ablations, the complete method clearly outperforms variants without difficulty weighting or with unbounded teacher guidance, but still does not exceed GRPO on avg@8. These results show that the proposed components improve the adopted combination scheme, but do not sufficiently establish whether they offer additional benefits over simpler training strategies.

  3. The ablation studies are important for validating the effectiveness of the proposed design choices and would be better presented in the main text rather than in the appendix.

Questions for authors
  1. Why combine OPD and GRPO within each update rather than use a two-stage approach, such as OPD followed by GRPO? Under a matched total compute budget and a reasonable allocation between stages, does DiffGate provide additional benefits?
  2. After introducing teacher scoring, what are the per-stage runtimes and end-to-end Model FLOPs Utilization (MFU)? Are there additional model-switching, communication, or waiting costs? Under the same GPU-hours budget, do the performance gains over GRPO and two-stage OPD→GRPO training still hold?
8 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for their constructive feedback and for recognizing that our evaluation “covers multiple student model sizes, mathematical reasoning and code generation”. We address each concern point by point below.

W1. Necessity of the proposed design choices.

Response: DiffGate’s contribution is how it allocates and bounds teacher guidance: it assigns stronger guidance to harder groups and limits extreme token-level teacher contributions. We evaluate both choices against simpler alternatives in Sec. 5.2 and App. E.4–E.5.

Controlled comparisons. We test two alternatives (setting γ=1\gamma = 1):

  • Constant-weight guidance: Uses the same bounded teacher signal on failed responses but removes difficulty weighting (Appendix E.5, Table 11). Data, teacher, rollout count, and training-step budget are matched.
  • Unbounded guidance: Replaces the bounded teacher signal with the raw teacher–student log-probability gap, keeping the remaining design unchanged (Appendix E.4, Table 10).

DiffGate outperforms both alternatives on both metrics across all four settings. Difficulty weighting improves avg@8 by 1.3–2.4 points over the tested constant-weight control, while bounding improves it by 1.0–1.5 points.

Bounding also improves the observed optimization stability. On 0.6B code, removing the bound increases the maximum observed gradient norm from 4.3 to 342.3 (Appendix E.4, Table 10 and Figure 6). This provides direct evidence that bounding controls extreme updates, alongside its performance benefits.

Together, these comparisons support the value of adaptive weighting and bounding beyond simply adding OPD to GRPO. Table 11 evaluates the combined effect of changing guidance allocation and strength. Our additional lower-weight controls, discussed in Reviewer bMgy Q2, further examine whether reducing a constant teacher coefficient can reproduce these gains.

W2. Task- and metric-dependent gains and benefits over simpler training strategies.

Response: DiffGate’s gains vary across tasks and metrics, but the expanded evaluation shows benefits beyond aggregate pass@8. We have added comparisons with simpler training strategies and experiments on Gemma models, multimodal reasoning, and multi-turn text-to-SQL.

1. Updated three-seed results. Our evaluation across three random seeds shows that DiffGate achieves higher mean avg@8 and pass@8 than GRPO, with gains of 0.9–1.5 points in avg@8 and 2.1–5.1 points in pass@8 across the aggregate mathematics and code results at both student sizes. These results strengthen the original findings: DiffGate improves both aggregate accuracy and solution coverage.

We also repeated the LiveCodeBench experiments three times. See Reviewer bMgy W2 response. DiffGate achieves higher mean scores on both metrics at both model sizes, although these gains are modest relative to the variability across runs.

2. Benefits over simpler training strategies. The controlled comparisons provide several complementary findings:

  • Additive GRPO+OPD: Appendix E.5, Table 11 uses the same bounded teacher signal on failed responses, with matched data, teacher, rollout count, and training steps. DiffGate’s difficulty weighting improves avg@8 by 1.3–2.4 points across the four settings.
  • Two-stage OPD→GRPO: Our new GPU-hour-matched comparison shows higher DiffGate scores on both metrics in all four settings, with gains of 0.5–1.8 points in avg@8 and 1.8–3.2 points in pass@8. These results support keeping adaptive teacher guidance active throughout RL under the tested stage allocation.
  • Smaller rollout groups: Appendix E.3, Table 9 shows that reducing group size from 16 to 4 lowers avg@8 by 1.9 points for GRPO versus 1.2 for DiffGate, averaged across settings. On 0.6B mathematics, DiffGate moves from 0.5 points below GRPO to 1.2 points above it. This supports its usefulness when group-relative reward supervision becomes less informative.

3. New results across model families and tasks. The additional experiments show improvements in both average correctness and solution coverage. See General Comment response.

W3. Presentation of ablation studies in the main text.

Response: We agree that readers should be able to assess the main design choices directly from the main text. Sec. 5.2 currently summarizes the ablation findings; we have added the compact table below, while retaining the detailed analyses and optimization diagnostics in App. E.4–E.5, Tables 10–11.

Full DiffGate achieves higher avg@8 and pass@8 than either ablated variant in all four settings. Difficulty weighting improves avg@8 by 1.3–2.4 points over the tested constant-weight control, while bounding improves it by 1.0–1.5 points. Including this table in Sec. 5.2 will make the contribution of each component easier to evaluate without referring to the appendix.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

Q1. Joint Updates vs Two Stage Training.

Response: We added OPD→GRPO comparisons under matched GPU-hours and training steps to test the benefit of continued teacher guidance.

Why combine signals within each update? OPD→GRPO removes teacher guidance when RL begins. DiffGate scores current student responses, guides failed attempts, and emphasizes harder groups. Guidance adapts to the student’s changing errors, including all-failure groups where GRPO provides no reward advantage.

(a) Same total GPU-hours. We include both stages’ costs and stop GRPO at DiffGate’s budget, allowing 150 GRPO steps for mathematics and 50 for code after OPD. This accounts for OPD’s lower cost with one rollout per prompt.

SettingOPD→GRPO avg@8OPD→GRPO pass@8DiffGate avg@8DiffGate pass@8
Math 0.6B13.127.113.729.5
Math 1.7B19.337.321.139.1
Code 0.6B32.547.633.049.8
Code 1.7B43.958.245.261.4

DiffGate improves both metrics in all four settings, gaining 0.5–1.8 points in avg@8 and 1.8–3.2 points in pass@8.

(b) Same total training steps. We allocate 75 OPD + 96 GRPO steps for mathematics and 30 OPD + 32 GRPO steps for code, matching DiffGate’s 171 and 62 steps, respectively.

SettingOPD→GRPO avg@8OPD→GRPO pass@8DiffGate avg@8DiffGate pass@8
Math 0.6B12.226.413.729.5
Math 1.7B18.836.821.139.1
Code 0.6B31.551.333.049.8
Code 1.7B42.657.945.261.4

DiffGate improves avg@8 in all four settings and pass@8 in three of four.

The original two-stage baseline completed OPD followed by a full GRPO schedule, exceeding DiffGate’s budget. These controlled comparisons, particularly at equal GPU-hours, support maintaining adaptive teacher guidance throughout RL under the tested stage allocations.

Q2. Runtime, MFU and Equal Compute Comparison.

Response: We measured training costs, optimized teacher scoring, and compared against teacher-free GRPO at equal GPU-hours. On code, DiffGate achieves higher pass@8 with comparable avg@8.

1. Training time and overhead. For the 1.7B student on four A100 GPUs:

Training settingTeacher scoringOther stagesTotal wall timeGPU-hours
Mathematics20.5 h (54%)17.6 h38.1 h≈152
Code≈7 h (37%)≈12 h≈19 h72–79
OPD, mathematics; one rollout per prompt1.3 h2.3 h3.6 h≈14

The mathematics and code runs comprise 171 and 62 steps. Teacher scoring dominates mathematics; unit tests account for ≈40% of code runtime. Explicit model switching costs ≈1% of a mathematics step. Waiting for the longest rollout costs ≈8% on mathematics and 3–5% on code, affecting both methods.

Our original GRPO implementation unnecessarily ran teacher scoring with a zero teacher coefficient. The budget-matched experiments remove this cost and use teacher-free GRPO, whose estimated runtime for the original schedules is 17.6 h on mathematics and 11.3 h on code.

2. Faster teacher scoring. Profiling identified repeated CPU-to-GPU teacher-weight transfers as a major overhead. Keeping the teacher on GPU and increasing the scoring microbatch from 2 to 16 sequences yielded:

Teacher scoring per step, 1.7B studentMathematicsCode
Original implementation436 s392 s
Optimized implementation69 s38 s
Speedup6.3×10.4×

The objective is unchanged. On mathematics, this reduces DiffGate’s total runtime from ≈2.2× to ≈1.2× teacher-free GRPO, or ≈20% overhead. A similar ratio is projected for code.

3. Model FLOPs utilization (MFU). Estimated end-to-end utilization after optimization:

SettingOriginal MFUEstimated optimized MFU
Mathematics, 1.7B14.6%26.0%
Mathematics, 0.6B9–12%18–22%
Code, 1.7B4.5–5.0%7.1–7.9%
Code, 0.6B≈5.0%≈7.9%

These estimates use teacher-stage speedups measured at 1.7B. MFU measures hardware utilization; GPU-hours measure training cost. Since MFU includes teacher computation, higher MFU alone does not establish greater computational efficiency.

4. Performance at equal GPU-hours. We give teacher-free GRPO additional steps within DiffGate’s GPU-hour budget:

SettingGRPO stepsGRPO avg@8GRPO pass@8DiffGate avg@8DiffGate pass@8
Code 0.6B6532.047.332.749.9
Code 1.7B7545.055.744.961.4

DiffGate retains pass@8 gains of 2.6 and 5.7 points at equal GPU-hours, with comparable avg@8. The budget-matched OPD→GRPO comparison appears in Q1(a).

We welcome further questions and respectfully ask you to reconsider your score if these revisions address your concerns.

AuthorsOpenReview ↗

Follow up on Rebuttal

Dear Reviewer zSrk,

We look forward to your feedback on our rebuttal. To address your concerns, we added three-seed evaluations, compute- and step-matched OPD→GRPO comparisons, and equal-GPU-hour comparisons with teacher-free GRPO. We also provided runtime and MFU analyses, optimized teacher scoring, and added a compact ablation table to the main text. Further experiments cover Gemma models, multimodal reasoning, and multi-turn text-to-SQL.

We hope these additions clarify the benefits of DiffGate’s design and address your concerns. We would be happy to discuss any remaining questions.

Thank you,
Authors

Reviewer_nBgmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper addresses the limitations of on-policy distillation and online rl training. The proposed method combines the dense supervision of OPD and true training objective of rl training. The experiments show the effectiveness of their methods.

Critical strengths

The problem is significant, and it is reasonable to combine the advantages of on-policy distillation and reinforcement learning. The paper is relatively well-written.

Critical weaknesses

Combining the advantages of OPD and RL is natural but not novel.

The paper would benefit from comparing the supervision provided by OPD and RL, when do they align? When do they not? Does the RL term really correct the supervision of OPD when OPD term is not reliable? Seems not?

I expect training curves in the main body. Intuitively, I expect the proposed method to have faster improvement in rewards and similar performance at the end of training. Could the author analyze why the final training reward is less than GRPO? The pass@8 of DiffGate seems higher than baselines; does it mean the rl component in the loss term does not fully stimulate training?

Additionally, the OPD term could also help with more effective reasoning for the student, but DiffGate drops the term in the batch with advantage !=0.

Questions for authors

The font size in Figure 2 is a bit small and not easy to read without zoom-in. Also, it seems to be AI-generated.

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for their constructive feedback and for noting that the “problem is significant” and the “paper is relatively well-written.” We address each concern point by point below.

W1: Novelty of combining OPD and RL.

Response: DiffGate makes teacher guidance within GRPO responsive to the student’s needs: it provides stronger support on failed responses in harder groups, gradually reduces that support as the student improves, and limits each token’s teacher contribution to prevent large discrepancies in gradient limits the magnitude of the token-level teacher advantage.

MethodTeacher supervisionHow teacher strength is controlled
KDRLAuxiliary reverse-KL loss on all responsesFixed coefficient
Additive GRPO+OPDBounded teacher advantage on failed responsesConstant weight
DiffGateBounded teacher advantage on failed responsesGroup-dependent weight (1−Rˉ)γ(1-\bar R)^\gamma

Here, Rˉ\bar R is the fraction of correct responses in a sampled group. DiffGate therefore adapts teacher guidance to the student’s current success on each problem. Bounding additionally prevents extreme teacher–student probability discrepancies from producing arbitrarily large teacher advantages. Our contribution is this combined design; we do not claim joint OPD–RL training or failure-only masking individually as novel.

The manuscript evaluates both components:

  • Difficulty weighting: Sec. 5.2 and Appendix E.5, Table 11 report 1.3–2.4-point avg@8 gains over the tested constant-weight baseline with bounded, failure-only guidance.
  • Bounding: Appendix E.4, Table 10 reports 1.0–1.5-point avg@8 gains, alongside reduced gradient spikes.

We have also added direct comparisons with plain KDRL and KDRL-Masking. Together with the controlled ablations, these comparisons support the value of DiffGate’s guidance mechanism beyond simply combining OPD and RL.

W2: Alignment and reliability of OPD and RL supervision.

Response: OPD and GRPO can reinforce or oppose each other. The RL term does not automatically correct an unreliable teacher signal. DiffGate controls when teacher guidance is used and how strongly it contributes through outcome-based selection, difficulty weighting, and bounding.

When do the signals agree? GRPO reinforces tokens in successful responses and suppresses tokens in failed responses within mixed-success groups. OPD instead encourages a sampled token when the teacher assigns it a higher probability than the student, and discourages it otherwise.

Formally, let $\Delta_{i,t}=\log\pi_T(o_{i,t}\mid c_{i,t}) -\log\pi_{\mathrm{old}}(o_{i,t}\mid c_{i,t}).$ Before PPO clipping, the two contributions agree on the sampled-token update when AiGRPOΔi,t>0A_i^{\mathrm{GRPO}}\Delta_{i,t}>0 and conflict when this product is negative.

Response/groupGRPO contributionDiffGate’s teacher contribution
Correct response in a mixed-success groupReinforces sampled tokensRemoved
Failed response in a mixed-success groupSuppresses sampled tokensRetained; agrees when Δ<0\Delta<0, conflicts when Δ>0\Delta>0
All-failure groupZeroRetained with maximal gate weight
All-success groupZeroRemoved

A conflicting signal is not necessarily incorrect. A failed response may contain useful intermediate steps that the teacher encourages, even though GRPO penalizes the response as a whole. Likewise, agreement between the two signals does not establish that an individual token deserves reinforcement or suppression.

Furthermore, we observed that GRPO produces more concentrated token predictions, while DiffGate maintains higher entropy. On mathematics, the final entropy of the 1.7B student was 0.048–0.050 across three GRPO runs, compared with 0.43 for DiffGate. For the 0.6B student, it was approximately 0.20 for GRPO versus 0.51 for DiffGate. This suggests that DiffGate preserves more diversity during generation. More concentrated predictions may improve training reward and average correctness while limiting the range of solutions explored. Our observations are consistent with DiffGate’s higher pass@8, though entropy alone cannot explain the performance difference.

The manuscript provides two relevant ablations:

  • App. E.7, Tab 14: Near the end of training, teacher–student disagreement on correct responses is 1.29–1.72× that on failed responses. Disagreement magnitude alone therefore does not reliably identify errors.
  • App. E.7, Tab.13: At initialization, 13.7–35.6% of groups contain only failed responses. In these groups, GRPO’s reward advantage is zero, while teacher guidance remains available.

DiffGate uses verified response outcomes to control teacher influence. Bounding limits extreme teacher contributions, and difficulty weighting adjusts their strength according to student success.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W3: Training Curves, Final Training Rewards, and Pass@8 Performance.

Response: Figure 4 presents held-out performance curves in the main text, with additional curves in Appendix E.11, Figure 10. Appendix E.8, Figure 9 reports training reward, response length, and entropy. We will make these references more prominent.

Lower training reward does not imply that the RL component is ineffective. In mathematics, GRPO reaches a higher final training reward, but our three-seed evaluation shows that DiffGate achieves higher mean avg@8 and pass@8 across all four model–domain settings. DiffGate combines outcome-based reinforcement with teacher guidance, which can encourage useful tokens within failed responses and produce a different training trajectory. Its lower training reward should therefore be considered alongside these improvements in held-out accuracy and solution coverage.

As shown in Appendix E.11, GRPO’s mathematics avg@8 peaks early and then declines, while DiffGate maintains higher pass@8 across the observed checkpoints. Higher training reward therefore does not necessarily translate into better performance on unseen problems. These observations are consistent with a generalization trade-off, without establishing its underlying cause.

Note: Furthermore, we have expanded our evaluation beyond the Qwen family with Gemma 2 and Gemma 3 experiments, added multimodal reasoning and multi-turn text-to-SQL benchmarks, and included direct comparisons with KDRL and KDRL-Masking please see General Comment. These additional experiments provide broader evidence for DiffGate’s effectiveness across model families and task settings, while clarifying its benefits relative to closely related methods.

W4: Teacher Guidance on Nonzero-Advantage Response.

Response: We would like to clarify that DiffGate retains teacher guidance on failed responses even when their GRPO advantage is nonzero. The gate depends on response correctness and the group’s solve rate wi=(1−Rˉ)γ(1−Ri),w_i=(1-\bar R)^\gamma(1-R_i), where RiR_i indicates whether response ii is correct and Rˉ\bar R is the fraction of correct responses in its group.

Response/groupGRPO advantageTeacher guidance
Failed response in a mixed-success groupNegative, nonzeroRetained, with difficulty weighting
Successful response in a mixed-success groupPositive, nonzeroRemoved
All-failure groupZeroRetained with maximal gate weight
All-success groupZeroRemoved

Thus, failed responses in mixed-success groups receive both RL and teacher supervision. Successful responses receive reward-based reinforcement without an additional teacher-imitation term.

We test whether teacher guidance should also be applied to successful responses in Appendix E.6, Table 12. Both variants below retain difficulty weighting. Failed-only guidance performs better on mathematics, while guidance on all responses performs slightly better on code. These results show that the benefit of the selector is task-dependent.

Q1: Figure 2: Readability

Response: Thank you for pointing this out. The figure was designed and assembled by the authors, with AI assistance used to create some graphical elements. We will redraw Figure 2 with larger fonts and a clearer layout in the camera-ready version.

We would be happy to answer any remaining questions and respectfully ask you to consider raising your score if these revisions satisfactorily address your concerns.

AuthorsOpenReview ↗

Follow up on Rebuttal

Dear Reviewer nBgm,

We look forward to your feedback on our rebuttal. To address your concerns, we clarified DiffGate’s novelty and when OPD and RL supervision agree or conflict, added KDRL and KDRL-Masking comparisons, and provided three-seed results and training-dynamics analysis. We also clarified that teacher guidance remains active on failed responses with nonzero GRPO advantages and expanded evaluation to Gemma models, multimodal reasoning, and multi-turn text-to-SQL.

We hope these additions address your concerns and would be happy to discuss any remaining questions.

Thank you,
Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Comment (Part 1/2)

We thank the reviewers for their time, careful evaluation, and constructive feedback. In response, we have expanded our experiments to include multimodal reasoning, multi-turn text-to-SQL, model families beyond Qwen, additional benchmarks, and direct comparisons with closely related methods. We have also added compute-matched comparisons and runtime analyses. These additions strengthen the empirical support for DiffGate and broaden its evaluation.

1. Different Model-family coverage.

Response: To assess generalization beyond Qwen, we added experiments with Gemma 2 (2B student, 9B teacher) and Gemma 3 (4B student, 12B teacher).

Student / TeacherMethodMBPP+ avg@8MBPP+ pass@8HumanEval+ avg@8HumanEval+ pass@8LCB avg@8LCB pass@8
Gemma 2 2B / 9BBase40.053.030.643.14.49.6
GRPO41.953.231.645.47.511.2
OPD (reverse KL)42.255.530.354.86.313.2
DiffGate44.158.435.455.98.216.3
Gemma 3 4B / 12BBase62.570.852.870.219.727.6
GRPO64.171.455.971.121.229.0
OPD (reverse KL)63.173.458.374.120.231.4
DiffGate65.276.859.776.922.934.8
MethodMinerva avg@8Minerva pass@8OlympiadBench avg@8OlympiadBench pass@8AIME24 avg@8AIME24 pass@8AIME25 avg@8AIME25 pass@8
Base23.843.937.554.99.221.311.723.8
GRPO25.144.738.256.211.123.513.125.1
OPD23.042.237.954.48.421.611.424.7
DiffGate26.146.339.257.810.524.414.526.8

DiffGate outperforms GRPO and reverse-KL OPD on both metrics across all three code benchmarks in both Gemma configurations. Relative to GRPO, gains are 0.7–3.8 points in avg@8 and 5.1–10.5 in pass@8 for Gemma 2, and 1.1–3.8 and 5.4–5.8 points, respectively, for Gemma 3. On mathematics, DiffGate improves pass@8 across all four benchmarks and avg@8 on three, with a 0.6-point decrease on AIME24. These results support DiffGate’s applicability beyond the Qwen family, while showing that gains vary across tasks and metrics.

2. Additional Benchmarks: Multimodal Reasoning and Multi-Turn Text-to-SQL

Response: We have added multimodal reasoning and multi-turn text-to-SQL experiments to test DiffGate beyond single-turn, text-only tasks. We report avg@8 and pass@8 separately and highlight the best student scores in bold.

(a) Multimodal reasoning. We train a Qwen3-VL-2B student with a Qwen3-VL-8B teacher on ViRL39K and evaluate on MathVista, MathVerse, its Vision Only subset, and We-Math. Both models receive the same image inputs and compatible preprocessing, and the teacher scores the student-generated text.

ModelMathVista avg@8MathVista pass@8MathVerse avg@8MathVerse pass@8MathVerse vision-only avg@8MathVerse vision-only pass@8We-Math avg@8We-Math pass@8
Initial 2B student61.879.939.162.037.362.149.975.4
GRPO65.281.244.964.439.462.261.083.9
OPD64.083.645.468.540.864.861.582.0
DiffGate66.983.446.468.743.065.963.486.8

DiffGate achieves the highest avg@8 on all four benchmarks and the highest pass@8 on three. It improves over GRPO by 1.5–3.6 points in avg@8 and 2.2–4.3 points in pass@8. OPD leads only on MathVista pass@8, by 0.2 points.

(b) Multi-turn text-to-SQL. We train a Qwen3-1.7B student using an 8B teacher in the SkyRL-SQL environment. We trained on 7296 problems, a 0.3% sample of SynSQL-2.5M, one pass at batch 128 for 57 steps. We drop the trivially easy queries and any whose gold answer depends on the current time or on a random draw, because those return a different result on every run and could not be graded. We then evaluate on the Spider development and test splits.

ModelSpider dev avg@8Spider dev pass@8Spider test avg@8Spider test pass@8
Initial 1.7B student61.669.961.672.1
GRPO64.873.264.374.2
OPD62.575.562.874.8
DiffGate65.778.766.577.9

DiffGate achieves the highest scores on both metrics and both splits. Compared with GRPO, avg@8 improves by 0.9 points on development and 2.2 points on test, while pass@8 improves by 5.5 and 3.7 points, respectively.

These experiments show that DiffGate’s benefits extend to visual reasoning and multi-turn interaction, with improvements over GRPO in both average correctness and multi-attempt solution coverage.

AuthorsOpenReview ↗

General Comment (Part 2/2)

3. New baselines

Response: We have added plain KDRL and KDRL-Masking as directly related baselines and evaluated GRPO and DiffGate over three random seeds. The table reports avg@8 and pass@8 separately. Three-seed results show mean ± standard deviation; KDRL variants report single-run results.

MethodMath 0.6B avg@8Math 0.6B pass@8Math 1.7B avg@8Math 1.7B pass@8Code 0.6B avg@8Code 0.6B pass@8Code 1.7B avg@8Code 1.7B pass@8
GRPO (3 seeds)12.2±0.127.3±1.019.8±1.434.6±1.031.2±0.847.3±0.843.5±2.256.3±4.4
Plain KDRL12.225.621.033.731.348.343.656.7
KDRL-Masking12.426.119.331.331.247.243.855.8
DiffGate (3 seeds)13.1±0.729.4±1.221.2±0.637.5±1.432.7±0.749.9±0.444.9±0.461.4±0.4

We ran both GRPO and DiffGate with three random seeds and report their mean scores and standard deviations. DiffGate achieves the highest reported mean avg@8 and pass@8, exceeding GRPO by 0.9–1.5 and 2.1–5.1 points, respectively. Its three-seed means also exceed the single-run results of plain KDRL by 0.2–1.4 points in avg@8 and 1.6–4.7 points in pass@8, and KDRL-Masking by 0.7–1.9 and 2.7–6.2 points, respectively.

These comparisons support the benefits of DiffGate beyond joint distillation–RL training and failure-only masking. The ablations in Appendices E.4–E.5 further examine bounding and difficulty weighting.

4. Compute-controlled comparisons and efficiency.

Response: We added comparisons with two-stage OPD→GRPO under matched GPU-hours and training steps. At equal GPU-hours, DiffGate improves avg@8 by 0.5–1.8 points and pass@8 by 1.8–3.2 points across all four settings. With matched steps, it improves avg@8 in all four settings and pass@8 in three. Against teacher-free GRPO at equal GPU-hours, DiffGate retains 2.6- and 5.7-point pass@8 gains on code, with comparable avg@8.

We also profiled runtime and MFU and optimized teacher scoring, achieving 6.3× speedup on mathematics and 10.4× on code. On mathematics, this reduces total runtime to approximately 1.2× that of teacher-free GRPO, with estimated 1.7B end-to-end MFU increasing from 14.6% to 26.0%. See Reviewer zSrk Q1 and Q2 Response.

5. Comparison with previous methods

Response: We compare the methods below to clarify what DiffGate adds. DiffGate adjusts teacher guidance according to each group’s solve rate and bounds the teacher advantage at each token. This gives more guidance to harder groups while limiting extreme teacher contributions.

MethodTeacher signalResponses receiving guidanceTeacher weightingGuidance when all responses fail
GRPONone——No
Reverse-KL OPDReverse-KL objectiveAll responsesNo difficulty weightingYes
Additive GRPO+OPDUnbounded additive teacher advantageAll responsesConstant weightYes
KDRLAuxiliary k2k_2 reverse-KL lossAll responsesConstant weightYes
Uni-OPDOutcome-calibrated distillation returnsCorrect and incorrect responses, with data balancingOutcome-guided margin calibrationBase OPD remains available; margin calibration requires both outcomes
EOPDReverse-KL objective with additional forward KL at high-entropy tokensAll responsesTeacher-entropy-based activation of forward KLYes
DiffGateBounded additive teacher advantageFailed responsesDifficulty-adaptive weightYes, with maximal gate weight

In an all-failure group, GRPO’s reward advantage is zero, but teacher-based methods can still provide supervision. KDRL-Masking above denotes response-level masking; KDRL also studies a separate mask that activates only for all-failure groups.

DiffGate goes beyond deciding whether to apply teacher guidance: it also adjusts how strongly to apply it. Groups with lower solve rates receive stronger guidance, while groups with higher solve rates receive less. Bounding separately limits the magnitude of each token’s teacher advantage.

The manuscript evaluates these components:

  • Difficulty weighting: Appendix E.5, Table 11 reports 1.3–2.4-point avg@8 gains over the tested constant-weight baseline with bounded, failure-only guidance.
  • Bounding: Appendix E.4, Table 10 reports 1.0–1.5-point avg@8 gains, alongside reduced gradient spikes.
  • Response selection: Appendix E.6, Table 12 compares additional all-response and failed-only variants, showing that their benefits vary across settings.
AuthorsOpenReview ↗

Thank you.

We thank Reviewers nBgm and bMgy for raising their scores and for their thoughtful feedback. We appreciate your time and are pleased that our responses addressed most of your concerns. We will incorporate all additional experiments and clarifications from the rebuttal into the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.