- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DiffGate, which combines GRPO with on-policy distillation. It adds teacher guidance only to failed responses, adjusts its strength based on the group solve rate, and uses tanh to limit the size of the teacher signal. The authors evaluate it on mathematics and code tasks with Qwen3 models, report improvements in average pass@8, and analyze difficulty weighting, signal bounding, and training behavior.
Critical strengths
S1: The approach is clear and easy to understand. Adding teacher guidance when the student cannot solve a problem is an understandable approach. It can also provide a learning signal for all-failure groups without changing the basic process of generating student responses.
S2: Some metrics improve. Average pass@8 is higher than GRPO on both mathematics and code, and average code accuracy also improves. The authors include training analyses and ablations, so the evaluation goes beyond final scores.
Critical weaknesses
W1: Directly related methods are missing from the comparison, and the new contribution is not very clear.
Several prior works already combine GRPO and OPD, but the paper does not directly compare with methods such as KDRL and RLAD. In particular, KDRL already studies distillation on incorrect responses and all-failure groups. DiffGate mainly adds difficulty weighting and tanh bounding, which are fairly straightforward changes. Although the authors report some ablations, removing difficulty weighting also increases the total amount of teacher guidance. This leaves it unclear whether the proposed design is better than simply adjusting the teacher weight. I do not see a clear advance over prior methods from the current results.
W2: The results are not convincing enough, and the model coverage is narrow.
The method does not consistently outperform GRPO. Average mathematics accuracy is slightly lower at both student sizes, and both avg@8 and pass@8 on LiveCodeBench are lower at both sizes. Average pass@8 does improve, but this alone does not make the overall results particularly strong. Also, all students and teachers come from Qwen3. Testing several model sizes does not show that the method works across model families, and transfer to science tasks does not answer that question either. I therefore still have doubts about how broadly the method applies.
Questions for authors
Q1: Why are directly related methods such as KDRL, KDRL-Masking, and RLAD not included? Are there results under the same settings that show where DiffGate improves over these methods? This would affect my assessment of the new contribution.
Q2: Was the teacher weight tuned separately for the baseline without difficulty weighting? If the overall strength of teacher guidance is made comparable, does DiffGate still have a clear advantage? I would like to understand whether the gains come from difficulty-based allocation or simply from reducing the teacher's influence.
5 comments
Rebuttal (Part 2/2)
3. Coverage beyond Qwen. To directly address the model-family concern, we added code-generation experiments with Gemma 2 (2B student, 9B teacher) and Gemma 3 (4B student, 12B teacher). See General Comment response.
DiffGate outperforms GRPO on both metrics in every reported Gemma comparison. For Gemma 2, gains are 2.2–3.8 points in avg@8 and 5.2–10.5 points in pass@8. For Gemma 3, gains hold across all three benchmarks, including LiveCodeBench, where avg@8 improves by 1.7 points and pass@8 by 5.8 points.
These experiments directly address the coverage concern: DiffGate’s benefits extend beyond Qwen and beyond pass@8 alone, with improvements in both metrics across the reported Gemma, visual reasoning, and multi-turn SQL evaluations.
Q2. Teacher Weight and Difficulty based Allocation.
Response: We have added lower constant-weight controls to test whether simply reducing teacher strength can reproduce DiffGate’s gains. These experiments complement Appendix E.5, Table 11.
All variants use the same bounded teacher advantage on failed responses. They differ in how that guidance is weighted (setting ):
- Constant-weight controls: Every failed response receives the same coefficient, with .
- DiffGate: The coefficient is , where is the group solve rate. Harder groups therefore receive stronger guidance.
We test a constant coefficient of 1.0 and reduced coefficients of 0.80, 0.70, 0.87, and 0.78 for Math 0.6B, Math 1.7B, Code 0.6B, and Code 1.7B, respectively.
| Method | Math 0.6B avg@8 | Math 0.6B pass@8 | Math 1.7B avg@8 | Math 1.7B pass@8 | Code 0.6B avg@8 | Code 0.6B pass@8 | Code 1.7B avg@8 | Code 1.7B pass@8 |
|---|---|---|---|---|---|---|---|---|
| Constant weight, | 12.4 | 27.4 | 19.4 | 35.6 | 32.6 | 50.3 | 44.5 | 60.7 |
| Constant weight, reduced | 12.0 | 27.5 | 20.6 | 36.3 | 31.8 | 48.2 | 44.9 | 59.6 |
| DiffGate | 13.7 | 29.5 | 21.1 | 39.1 | 33.0 | 49.8 | 45.2 | 61.4 |
DiffGate outperforms the reduced-weight control on both metrics in all four settings, improving avg@8 by 0.3–1.7 points and pass@8 by 1.6–2.8 points. It also exceeds both constant-weight controls on avg@8 in all four settings and on pass@8 in three of four.
The key distinction is that DiffGate adjusts guidance separately for each group, even within the same update. A lower constant weight reduces guidance equally for every failed response; DiffGate concentrates stronger guidance on groups the student finds harder. The results show that the tested constant-weight reductions do not recover DiffGate’s overall performance, supporting the benefit of its adaptive weighting design.
We would be happy to answer any remaining questions and respectfully ask you to consider raising your score if these revisions satisfactorily address your concerns.
Follow up on Rebuttal
Dear Reviewer bMgy,
We look forward to your feedback on our rebuttal. To address your concerns, we added direct comparisons with KDRL and KDRL-Masking, lower constant-weight controls to assess difficulty-based allocation, and three-seed evaluations, including LiveCodeBench. We also clarified DiffGate’s contribution and expanded evaluation to Gemma 2 and Gemma 3, multimodal reasoning, and multi-turn text-to-SQL.
We hope these additions address your concerns about novelty, teacher weighting, and generalization. We would be happy to discuss any remaining questions.
Thank you,
Authors
Rebuttal (Part 1/2)
We thank the reviewer for their constructive feedback and for noting that our approach is “clear and easy to understand” and provides “a learning signal for all-failure groups.” We address each concern point by point below.
Response: We have added direct comparisons with plain KDRL and KDRL-Masking and clarified what DiffGate contributes. KDRL already explores joint GRPO–OPD training. Our contribution is how teacher guidance is weighted and bounded within this setting.
Additional baseline results. DiffGate reports mean ± standard deviation over three seeds; the KDRL variants report single-run results.
DiffGate’s mean scores are higher on both metrics across all four settings. Its pass@8 gains are 1.6–4.7 points over KDRL and 2.7–6.2 points over KDRL-Masking. These are observed improvements; we do not claim statistical significance because the KDRL variants lack multi-seed uncertainty estimates.
What DiffGate adds. A binary mask determines whether a response receives teacher guidance. DiffGate additionally determines how much guidance it receives: the weight (1−Rˉ)γ assigns stronger guidance to groups with lower solve rates and reduces it as the student becomes more successful. Its bounded teacher advantage also limits extreme token-level contributions.
The manuscript evaluates these components separately:
Removing difficulty weighting changes both where guidance is allocated and its overall strength. Table 11 therefore evaluates their combined effect. We address the reviewer’s question about lower constant teacher weights separately in Q2, including the additional controls.
RLAD. We have not included numerical comparisons with RLAD because we could not find an official publicly released implementation. Reimplementing it would require additional validation to ensure a fair comparison.
Response: We have strengthened the evaluation in three ways: multiseed run, additional reasoning tasks, and experiments beyond the Qwen family. The new results show improvements in both average correctness and solution coverage across the additional task and model settings.
1. Updated three-seed results. Our evaluation across three random seeds shows that DiffGate achieves higher mean avg@8 and pass@8 than GRPO, with gains of 0.9–1.5 points in avg@8 and 2.1–5.1 points in pass@8 across the aggregate mathematics and code results at both student sizes. These results strengthen the original findings: DiffGate improves both aggregate accuracy and solution coverage.
We also repeated the LiveCodeBench experiments three times. As shown below, DiffGate achieves higher mean scores on both metrics at both model sizes, although these gains are modest relative to the variability across runs.
2. Broader task coverage. We added multimodal reasoning with a Qwen3-VL-2B student and 8B teacher, and multi-turn text-to-SQL with a Qwen3-1.7B student and 8B teacher. See General Comment response.
DiffGate improves both avg@8 and pass@8 over GRPO in all six evaluations. It also outperforms OPD on both metrics except MathVista pass@8, where OPD leads by 0.2 points. The Spider scores remain unchanged when evaluated only on questions shared across all models—1,032 development and 2,147 test questions—confirming that the gains do not arise from different evaluation subsets.