acceptodds
Under review as a conference paper at ICLR 2027

GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

Abstract

Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose **GIFT** (**G**radient-**I**nformed **F**il**T**ering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component **parallel** to the RL gradient, which measures first-order conflict and provides a **safety** filter, and a component **orthogonal** to it, which quantifies novel directional guidance and provides a measure of **value**. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive **coverage annealing** schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5% on difficult mathematical reasoning tasks. *Code will be available upon publication.*

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Token-Level Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FgnSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GIFT, a method for combining GRPO with on-policy self-distillation (OPSD) to improve LLM reasoning. Instead of applying distillation to all tokens, GIFT analyzes the relationship between GRPO and OPSD gradients in token-level logit space. It prioritizes tokens where the distillation gradient does not conflict with GRPO and provides additional orthogonal directional information. The selected tokens receive the full distillation loss, while the proportion of selected tokens gradually decreases during training, eventually transitioning to pure GRPO.

The authors derive a closed-form scoring rule for efficient token selection and evaluate GIFT on mathematical and logical reasoning tasks across Qwen3 and OLMo models. The results show improvements in average performance over several RL and distillation baselines, with ablation studies examining the effectiveness of the gradient-based selection strategy and progressive removal of teacher supervision.

Critical strengths

Critical Strengths

1. A practical and well-motivated framework for combining RL and distillation.

The paper addresses an important problem in distillation-augmented RL: how to selectively incorporate dense teacher supervision without excessively interfering with outcome-based optimization. The proposed gradient decomposition provides an intuitive way to distinguish conflicting guidance from complementary directional information. Moreover, the closed-form scoring rule avoids additional backward passes, while coverage annealing offers a simple mechanism for progressively removing teacher supervision. These design choices make GIFT a practical and potentially useful framework.

2. Extensive experiments and informative ablation studies.

The paper evaluates GIFT across multiple model scales and architectures (Qwen3 and OLMo-3), covering both mathematical and logical reasoning tasks. The reported results show competitive and generally improved average performance over a diverse set of RL and distillation baselines, particularly on challenging benchmarks such as AIME25. The ablation studies further provide useful evidence for the effectiveness of combining conflict-aware selection, orthogonal residual scoring, and coverage annealing.

Critical weaknesses

Weaknesses

1. Gap between logit-space theoretical guarantees and actual optimization.

The theoretical analysis establishes first-order gradient compatibility only in token-level logit space. However, such compatibility does not necessarily hold in the shared parameter space due to Jacobian transformations and cross-token interactions, as the authors acknowledge in Appendix C.2.

More importantly, the paper provides no direct empirical evidence that GIFT improves parameter-space gradient alignment or that larger orthogonal residuals lead to greater reward improvements. This weakens the central theoretical justification for the proposed geometric filtering strategy, which appears to be a theoretically motivated heuristic rather than a guaranteed improvement to RL optimization.

2. Unclear relative contribution of OPSD and GRPO gradients.

GIFT fixes the distillation coefficient at λ=1\lambda=1, but the actual contribution of OPSD relative to GRPO remains unclear. In particular, the GRPO gradient magnitude depends on ∣Ai∣|A_i|, whereas the proposed selection scores are invariant to the advantage magnitude (Proposition 2). Consequently, OPSD may dominate updates at tokens with small GRPO advantages, regardless of their geometric compatibility.

The paper lacks analysis of the relative gradient norms, their evolution during training, and their dependence on advantage magnitudes. Although selection ablations are provided, it remains unclear to what extent the gains arise from the proposed geometric criterion versus the effective balance between the two objectives. This limits confidence in the mechanism and robustness of the method.

Minor Questions

  1. Gradient weighting consistency. In Eq. (1), did_i is already defined as wi(pi−q~i)w_i(p_i-\tilde q_i), but Eq. (25) in Appendix D appears to multiply did_i by wiw_i again. Is this a notation error? Please clarify the actual implementation.

  2. Zero-advantage tokens. How much of the total OPSD gradient comes from tokens with zero or clipped GRPO gradients? These tokens can still receive distillation supervision under the exact-quota selection mechanism. Reporting their contribution would help clarify the effective interaction between the two objectives.

  3. Missing gradient projection baseline. Given that Theorem 2 emphasizes the value of the orthogonal component of the distillation gradient, why does GIFT perform discrete token selection rather than directly removing conflicting gradient components? A PCGrad-style logit-space projection could eliminate the conflicting component while preserving orthogonal guidance. Could the authors compare GIFT against such a projection-based baseline under comparable supervision budgets? This comparison would help establish whether the proposed token selection and lexicographic scoring are necessary, rather than simply enforcing gradient compatibility through projection.

Questions for authors

See weakness. I am willing to raise my score to raise my score if the authors can adequately address my concerns.

9 comments
AuthorsOpenReview ↗

Response to Reviewer FgnS

Dear Reviewer FgnS:

Thank you for your patience. We are currently finalizing the additional experiments and analyzing the results to thoroughly address the reviewers' feedback. We will upload the revised manuscript, along with a detailed point-by-point response, very soon. Please be assured that we are keeping track of the deadline and will complete the submission on time. Thank you again for your support.

AuthorsOpenReview ↗

Response to Reviewer FgnS (1/5)

We sincerely thank the reviewer for a precise and constructive reading. In the revised manuscript, we ran new parameter-space gradient alignment and norm diagnostics (now Figure 2 and a rewritten Appendix C.4), tracked rollout sparsity and conflict dynamics (Figure 4, Appendix I), implemented and ran the requested PCGrad-style logit-space projection baseline, corrected the notation in Appendix D, and expanded our evaluation to large-scale STEM, Coding, and Multimodal benchmarks (Appendix J). (Note: Equation, proposition, and figure numbers below reference the revised manuscript, with revisions highlighted in blue.)


W1: Logit-space theoretical guarantees vs. parameter-space optimization — now measured directly (New Figure 2, Rewritten Appendix C.4)

The reviewer correctly identified the weakest link in our original submission: we proved first-order compatibility in token-level logit space, acknowledged in Appendix C that it need not transfer globally through the shared-parameter Jacobian, and offered no direct parameter-space measurement. We have now directly measured it and linked both geometric components (ai,bia_i, b_i) to optimization and reward gains:

  • 1. Theoretical Bridge: Exact Per-Token Linearity vs. Cross-Token Blocks (Rewritten Appendix C.4): Let GR=J⊤RG_R = J^\top R and GD=J⊤DmG_D = J^\top D_m denote the parameter gradients of the GRPO and masked OPSD objectives, where JJ is the stacked logit Jacobian (⟨GR,GD⟩=R⊤JJ⊤Dm\langle G_R, G_D \rangle = R^\top J J^\top D_m). At any individual token ii, the parameter-gradient mapping ∇θℓi=Ji⊤∇ziℓi\nabla_{\theta} \ell_i = J_i^\top \nabla_{z_i} \ell_i is linear, so our logit decomposition di=ai∥ri∥2ri+di⊥d_i = \frac{a_i}{\Vert r_i \Vert_2} r_i + d_i^\perp strictly preserves collinearity for all diagonal self-interaction terms (Ji⊤di=ai∥ri∥2∇θℓiGRPO+Ji⊤di⊥J_i^\top d_i = \frac{a_i}{\Vert r_i \Vert_2} \nabla_{\theta} \ell_i^{\mathrm{GRPO}} + J_i^\top d_i^\perp). Discrepancies between logit-space and parameter-space alignment can arise only from off-diagonal cross-token blocks JiJj⊤J_i J_j^\top (i≠ji \neq j). While materializing per-token parameter gradients for online selection requires O(NP)\mathcal{O}(NP) storage and per-token backward passes (intractable at LLM scale, whereas GIFT takes O(M)\mathcal{O}(M) in closed form via Proposition 1), aggregate batch-level parameter gradients (GR,GDG_R, G_D) cost only one backward pass each per step and can be directly measured.
AuthorsOpenReview ↗

Response to Reviewer FgnS (2/5)

  • 2. [NEW] Direct Measurement of Parameter-Space Gradient Alignment (New Figure 2, Qwen3-8B): At every actor step, we log the cosine similarity between the applied total parameter gradient and the model's own outcome-grounded GRPO reference gradient across all trainable parameters (accumulated before clipping, with GD,tG_{D,t} carrying each method's own mask and token weights): cos⁡(Gtotal,t, GR,t),Gtotal,t=GR,t+λGD,t\cos\big(G_{\mathrm{total},t},\,G_{R,t}\big), \qquad G_{\mathrm{total},t}=G_{R,t}+\lambda G_{D,t} Because each method is compared against its own GR,tG_{R,t} at its own parameters, this directly measures how far the auxiliary distillation term deflects that method's outcome-driven update in parameter space:
Parameter-Space Alignment cos⁡(Gtotal,t,GR,t)\cos(G_{\mathrm{total},t}, G_{R,t})Mean (Steps 45–77)Min (Steps 45–77)Late Annealing (Steps 70–77)Post-Exit (t≥78t \ge 78)
GRPO + OPSD (Unfiltered)0.12−0.04-0.04∼0.11\sim 0.11 (≈84∘\approx 84^\circ deflection)0.00–0.23
GIFT (Ours)0.49 (+0.37)0.24 (+0.28)0.60–0.97 (+0.49 to +0.82)1.00 (Exact RL)
  • 3. Why This Improvement is Mechanistic Rather Than Incidental: Two timing features in Figure 2 are decisive. First, the curves are indistinguishable during early training (before ∼\sim Step 20)—exactly as the mechanism predicts, since GIFT applies the teacher broadly during full-coverage warm-up (t≤10t \le 10) when conflicts are lower. Second, the curves separate precisely when coverage annealing prunes conflicting tokens and the token-level conflicting fraction (ai<0a_i < 0) surges as the maturing student discovers valid reasoning paths that diverge from the single reference trace (New Figure 4, Right, where ai<0a_i < 0 climbs to 55%–70%). The parameter-space gap is widest at Steps 70–77 (+0.49 to +0.82 higher cosine), where GIFT's coverage annealing has already driven conflicting-token retention to 0.0% (Figure 7). Thus, the empirical answer to your concern is clear: off-diagonal cross-token blocks JiJj⊤J_i J_j^\top do not wash out our token-local criterion—filtering on sign(ai)\mathrm{sign}(a_i) in logit space preserves aggregate parameter-space alignment that unfiltered distillation destroys. Moreover, at t≥Texit=78t \ge T_{\mathrm{exit}} = 78, GIFT's mask vanishes identically so ∇θLtGIFT=∇θLtGRPO\nabla_\theta \mathcal{L}_t^{\mathrm{GIFT}} = \nabla_\theta \mathcal{L}_t^{\mathrm{GRPO}} (Corollary 2), ensuring final convergence is governed purely by exact parameter-space RL gradients.

  • 4. Linking Orthogonal Residuals (bib_i) and Safety (aia_i) to Verifiable Reward Gains (Existing Table 4 & New Appendix J): We have kept our scope statement in Appendix C.4 precise rather than overselling Figure 2: cosine similarity is a first-order directional statistic, so for the direct link between larger orthogonal residuals (bib_i) and task rewards, we rely on controlled ablations that vary only the ranking quantity under matched coverage schedules (Table 4), alongside new large-scale evaluations across Math/STEM, Coding, and Multimodal reasoning (New Appendix J, Tables 9–11):

    • Isolating bib_i and aia_i under matched schedules (Table 4): Holding the safety filter and schedule κt\kappa_t fixed, prioritizing larger bib_i (GIFT: 78.96% Avg., 71.11% on AIME25) substantially outperforms ranking by parallel agreement aia_i alone (Conflict-only, which ignores bib_i: 77.08% Avg., 65.83% on AIME25, a +5.28 pp gain on AIME25). Conversely, maximizing bib_i or ∥di∥2\Vert d_i\Vert_2 while ignoring sign(ai)\mathrm{sign}(a_i) (Residual-only: 76.97%; Teacher-norm: 76.90%) consistently degrades accuracy.
    • Large-scale corroboration of Theorem 2(2) (New Tables 9–11): Scalar advantage-modulation hybrids (e.g., RLSD) are confined to span(ri)\mathrm{span}(r_i) (bi=0b_i = 0). Across 12,280 STEM problems (Table 9), 1,682 Code-R1 problems (Table 10), and 4 Multimodal benchmarks (Table 11), injecting filtered orthogonal guidance (bi>0,ai≥0b_i > 0, a_i \ge 0) yields consistent, large-scale gains over both pure GRPO and scalar-modulated RLSD:
Evaluation Suite (New Appendix J)BASEGRPOOPSDRLSD (bi=0b_i=0)GIFT (Ours)Gain vs. GRPO / RLSD
7-Bench Math & STEM (N=12,280N=12,280, Table 9)51.3852.31—55.9660.28+7.97 / +4.32 pp
3-Bench Code Generation (N=1,682N=1,682, Table 10)64.5365.6463.6265.7868.12+2.48 / +2.34 pp
4-Bench Multimodal VL (Qwen3-VL-8B, Table 11)60.5861.7359.2560.2762.61+0.88 / +2.34 pp
Reviewer_VwfkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies the tension between outcome-based RLVR and token-level distillation targets during optimization, and proposes GIFT to select which teacher updates to apply alongside the GRPO objective. GIFT decomposes the distillation gradient into components parallel and orthogonal to the RL gradient, yielding a scoring rule to rank token-level guidance. Combined with a coverage annealing schedule that progressively shrinks the number of distilled tokens until training reduces to pure GRPO, GIFT reports improvements on mathematical and logical reasoning benchmarks across several model scales.

Critical strengths

The authors address a critical problem: how to integrate sparse outcome-based RL with dense token-level distillation targets. This is an important and interesting topic.

Critical weaknesses
  1. The paper is motivated by a conflict between the GRPO and distillation gradients, but never shows that this conflict actually occurs or meaningfully hurts training. Also, the theorems look heavy, but my personal impression is that they can be largely compressed, which would clarify the paper and make it easier to read and understand. For example, what is the purpose of stating Theorem 2.2? Is this necessary? Also, theorem 3 reads circular, its stated optimality holds for an objective constructed to match the scoring rule. It would be good if authors explain the necessarity and underlying takeaways for the theoretical claims.
  2. The experimental evidence is not fully convincing. The mathematical reasoning benchmarks are relatively small: AIME24/25 contain only 30 problems each. On AMC23 and AIME24, GIFT is often matched or outperformed by baselines. Besides, why GRPO only matches or even underperforms BASE, especially on Qwen3-8B? This is counterintuitive. It would be good if the authors could provide more information
Questions for authors

Could the authors quantify what negative influence the "conflict" actually brings? e.g., the fraction of tokens with aia_i < 0 during training, and whether excluding them improves accuracy?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Vwfk (1/4)

We sincerely thank Reviewer Vwfk for the insightful feedback. In the revised manuscript (all modifications are highlighted in blue), we have: (1) added direct empirical diagnostics of gradient conflict (ai<0a_i < 0) and GRPO outcome sparsity (Figure 3, Appendix I); (2) streamlined Sections 3.2–3.4 and added explicit baseline-comparison paragraphs; and (3) expanded evaluations to 7 large-scale math/STEM (N=12,280N = 12,280), 3 coding (N=1,682N = 1,682), and 4 multimodal benchmarks (Appendix J).

W1 & Q1: Quantifying gradient conflict (ai<0a_i < 0) and its harm, purpose of Theorems 1 & 2, and non-circularity of Theorem 3

  1. New Empirical Diagnostics: What Figure 3 (Right) measures and why it resolves your concern (Appendix I, lines 1358–1403):
  • What we did in Figure 3 (Right): To directly test whether gradient conflict occurs during training, for every active token i∈Di \in \mathcal{D} with a non-vanishing GRPO gradient (∥ri∥2>ϵg\Vert r_i\Vert_2 > \epsilon_g), we compute the token-level logit gradients ri=∇ziℓiGRPOr_i = \nabla_{z_i}\ell_i^{\text{GRPO}} and di=∇zi(wiℓisKL)d_i = \nabla_{z_i}(w_i\ell_i^{\text{sKL}}) over the full vocabulary V\mathcal{V} and track the signed parallel projection ai=⟨di,r^i⟩a_i = \langle d_i, \hat{r}_i \rangle across training steps. A token is conflicting whenever ai<0a_i < 0 (⟨di,ri⟩<0\langle d_i, r_i \rangle < 0), meaning the teacher's distillation gradient forms an obtuse angle with the student's RL gradient and opposes first-order GRPO descent (Theorem 1)—which occurs whenever RL reinforces a successful student token (Ai>0A_i > 0) that the reference-conditioned teacher suppresses (q~i(ui)≪pi(ui)\tilde{q}_i(u_i) \ll p_i(u_i)), or vice versa.
  • Pervasive conflict in unfiltered training: As shown in Figure 3 (Right) on Qwen3-8B trained on Code-R1, the fraction of active tokens with ai<0a_i < 0 starts near 50% and steadily climbs to 55%–70% in later stages (while the aligned fraction ai≥0a_i \ge 0 drops to 30%–45%), because the improving student increasingly discovers valid code implementations (Ai>0A_i > 0) that diverge syntactically or algorithmically from the teacher's single reference trace. Similarly, on DeepMath-103K, 26.17% of active token updates exhibit ai<0a_i < 0.
  • Quantifying the negative impact of conflict & answering Q1 (Does excluding ai<0a_i < 0 improve accuracy?):
    • Unfiltered conflict hurts training: Leaving this conflicting majority unfiltered directly counteracts RL descent. In code generation (New Table 10), unfiltered OPSD degrades pass@1 below the untuned BASE model (63.62% vs. 64.53%, -0.91 pp, whereas GIFT reaches 68.12%, +4.50 pp over OPSD). In Qwen3-8B math (Existing Tables 3 & 4), unfiltered joint training (All Tokens: 77.49%) and conflict-agnostic selection (Residual-only, which selects by bib_i while ignoring sign(ai)\text{sign}(a_i): 76.97%) underperform GIFT (78.96%) by -1.47 pp and -1.99 pp (-4.72 pp and -4.44 pp on AIME25).
    • Hard exclusion improves accuracy, and warm-up + annealing is optimal: Directly answering Q1: Yes, excluding ai<0a_i < 0 tokens improves accuracy. In Existing Table 4, strictly excluding all ai<0a_i < 0 tokens from Step 1 (Conflict-Free) rebounds accuracy from 76.97% (Residual-only) and 77.49% (All Tokens) to 77.73%. Furthermore, GIFT's exact-quota coverage annealing achieves an additional +1.23 pp gain over hard exclusion (78.96% vs. 77.73%, and +4.00 pp on AIME25: 71.11% vs. 67.11%). During the initial 10-step warm-up, when 70%–90% of RL rollout groups have zero reward variance (Figure 3 Left), admitting mild conflicts (smallest ∣ai∣\vert a_i\vert) provides exploratory scaffolding to escape local optima (removing warm-up H=0H=0 drops accuracy to 78.03% in Table 8). Once RL gradients stabilize, coverage annealing prunes ai<0a_i < 0 tokens first—driving conflicting-token retention from 100% at Step 10 down to 0.0% by Step 40 while retaining 97.4% of non-conflicting tokens (Existing Figure 6, Appendix K).
AuthorsOpenReview ↗

Response to Reviewer Vwfk (2/4)

  1. Streamlining Theorems 1 & 2 and their core purpose (Distinguishing GIFT from prior methods):
    • We substantially compressed Sections 3.2–3.3 and moved all technical derivations to Appendices B–C. In particular, we simplified Theorem 2(2) by removing the heavy product-space formulation H=⨁i∈IR∣V∣\mathcal{H} = \bigoplus_{i \in \mathcal{I}} \mathbb{R}^{\vert\mathcal{V}\vert} and stating the direct token-level identity inf⁡c∈R∥di−cri∥2=∥di⊥∥2=bi\inf_{c \in \mathbb{R}} \Vert d_i - c r_i\Vert_2 = \Vert d_i^\perp\Vert_2 = b_i (Eq. 6).
    • Why Theorems 1 and 2 (including Theorem 2.2) are essential: Our primary objective in formulating Theorems 1 and 2 is to formally differentiate GIFT from the two main families of prior work, which we now highlight via two dedicated comparison paragraphs in Sections 3.2 and 3.3:
    • Theorem 1 vs. Statistical Token-Selection Heuristics (Section 3.2, "Comparison with Statistical Token-Selection Heuristics"): Theorem 1 proves sign(ai)\text{sign}(a_i) is an exact first-order safety filter. Combining Theorem 1 with Proposition 1 (ai=wisign(Ai)⟨pi−q~i,pi−eui⟩/∥pi−eui∥2a_i = w_i \text{sign}(A_i)\langle p_i - \tilde{q}_i, p_i - e_{u_i}\rangle / \Vert p_i - e_{u_i}\Vert_2) proves why unsigned heuristics (TIP, EOPD, RSTG) fail: when a confident student (pi≈euip_i \approx e_{u_i}, low entropy hih_i) reaches a correct answer (Ai>0A_i > 0) via a valid step that diverges from the reference trace (q~i(ui)≪pi(ui)\tilde{q}_i(u_i) \ll p_i(u_i), high DKLD_{\text{KL}}), ai<0a_i < 0 is strictly negative. Heuristics like TIP's DKL(q~i∥pi)(1−hi)D_{\text{KL}}(\tilde{q}_i \Vert p_i)(1 - h_i) assign maximum priority to these exact tokens, penalizing valid student exploration and reversing RL descent.
    • Theorem 2(2) vs. Scalar Advantage-Modulation Hybrids (Section 3.3, "Comparison with Scalar Advantage-Modulation and Reward-Gating Hybrids"): Theorem 2(2) is the exact mathematical separation between GIFT and recent hybrid methods (RLSD, RLCSD, AMR-SD, OPDVR). Those methods incorporate the teacher solely by replacing the scalar advantage AiA_i with a modulated scalar A~i∈R\tilde{A}_i \in \mathbb{R} at the sampled token uiu_i, confining their logit gradient to r~i∝A~i(pi−eui)∈span(ri)\tilde{r}_i \propto \tilde{A}_i(p_i - e_{u_i}) \in \text{span}(r_i) (bi=0b_i = 0). Across all unsampled vocabulary tokens v≠uiv \neq u_i, r~i(v)∝A~ipi(v)\tilde{r}_i(v) \propto \tilde{A}_i p_i(v) scales logits strictly in proportion to the student's own distribution pip_i, completely discarding the teacher's probabilities q~i(v)\tilde{q}_i(v) over alternative reasoning tokens. Theorem 2(2) proves that any scalar advantage-modulation rule incurs an irreducible approximation error of exactly bib_i, explaining why GIFT outperforms RLSD by +5.46 to +7.68 pp on complex multi-step competition math (New Table 9).
  2. Why Theorem 3 is not circular (Revised Section 3.4, lines 280–322):
    • In the initial draft, presenting the lexicographic program immediately before the score sis_i created the impression that the objective was reverse-engineered from sis_i.
    • In Revised Section 3.4, we first motivate the three subset-level allocation criteria from first principles independently of sis_i: given a quota of kk tokens from Em\mathcal{E}_m, a subset A⊆EmA \subseteq \mathcal{E}_m (∣A∣=k\vert A\vert=k) should (i) minimize the number of conflicting tokens admitted N−(A)=∣A∩Cm∣N^-(A) = \vert A \cap \mathcal{C}_m\vert (by Theorem 1), (ii) when early warm-up quotas exceed the non-conflicting pool (k>∣Um∣k > \vert\mathcal{U}_m\vert), minimize total normalized opposition O−(A)=∑i∈A∩Cm∣ai∣O^-(A) = \sum_{i \in A \cap \mathcal{C}_m} \vert a_i\vert, and (iii) among non-conflicting tokens, maximize total orthogonal residual energy E+(A)=∑i∈A∩Um+bi2E^+(A) = \sum_{i \in A \cap \mathcal{U}_m^+} b_i^2 (by Theorem 2).
    • Optimizing a three-tier lexicographic objective over (∣Em∣k)\binom{\vert\mathcal{E}_m\vert}{k} subsets is a constrained combinatorial subset-selection problem. Theorem 3's non-trivial contribution is proving that because the three token regimes—non-conflicting active Um+\mathcal{U}_m^+ (si=bi≥0s_i = b_i \ge 0), neutral zero-gradient Um0\mathcal{U}_m^0 (si=0s_i = 0), and conflicting Cm\mathcal{C}_m (si=−∣ai∣<0s_i = -\vert a_i\vert < 0)—are mutually disjoint and strictly ordered by sign, and because bi↦bi2b_i \mapsto b_i^2 is strictly monotonic on [0,∞)[0, \infty), this combinatorial program is solved exactly in O(∣Em∣log⁡∣Em∣)\mathcal{O}(\vert\mathcal{E}_m\vert\log\vert\mathcal{E}_m\vert) time by a single TopK sort on the scalar sis_i (Eq. 8). Thus, sis_i is the closed-form algorithmic solution, not a circular assumption.
AuthorsOpenReview ↗

Response to Reviewer Vwfk (3/4)

W2: Why pure GRPO stagnates on Qwen3-8B, AMC23/AIME24 comparisons, and large-scale evaluations

  1. New Empirical Diagnostics: Why pure GRPO matches or underperforms BASE on Qwen3-8B (New Figure 3 Left, Appendix I, lines 1341–1357):
    • To explain why pure GRPO barely matches BASE on Qwen3-8B (76.64% vs. 76.67% in Table 1; 74.86% vs. 76.76% on MATH in Table 9), Figure 3 (Left) tracks the rollout outcome distribution across prompt groups throughout pure GRPO training on Qwen3-8B:
    • Severe Group-Level Reward Degeneracy: When training Qwen3-8B on challenging splits (hard DeepMath-103K problems or strict unit tests in Code-R1), 50%–75% of prompt rollout groups fail verification across all sampled responses (All wrong, $ R=0 $), while 10%–35% succeed uniformly (All correct, $ R=1 $). In total, 70%–90% of prompt groups exhibit zero reward variance (All wrong + all correct).
    • Vanishing Gradients & High-Variance Updates: Because GRPO normalizes rewards within each prompt group, zero-variance groups produce identically zero advantages ($ A_i = 0 )andvanishingpolicygradients() and vanishing policy gradients ( r_i = 0 $). Pure GRPO therefore updates the 8B policy using only the remaining 10%–30% of prompts per batch, causing high gradient variance and stagnation on an already heavily post-trained backbone.
    • How GIFT resolves this: GIFT applies full-coverage OPSD scaffolding during the initial warm-up ($ t \le H )andassignszero−gradienttokens() and assigns zero-gradient tokens ( \mathcal{U}_m^0 )neutralpriority() neutral priority ( s_i = 0 $, Corollary 1), supplying dense teacher supervision precisely on the 70%–90% of groups where GRPO's signal vanishes.
Reviewer_B3kEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces GIFT, a novel way of using on-policy self distillation with GRPO to get both the dense teacher guidance and correctness rewards from GRPO. They do this via decomposing the gradients of GIFT into two components which are orthogonal to each other and use it for training.

Critical strengths
  1. The high level idea is understandable and novel.
  2. Ther performance obtained is good.
Critical weaknesses

I have concerns with the paper that have not been addressed

  1. Personally, I feel that the paper requires significant polishing in terms of writing. The current draft reads like an AI influencer blog with a lot of hype but few insights. For instance, I don't understand why are certain terms bold in the draft, when they don't contribute much to the overall theme of the paragraph. This can be felt across sections, and the paper is hard to read and follow.

  2. Following up on the previous point, the theorems are especially hard to interpret and validate because they are not written in the clearest way possible. For instance, why does theorem 2, start like "Under the assumptions of Theorem 1", what are the assumptions that are being referred to? Theorem 1 does not make these clear. Similarly, in the appendix the proofs are not clear. I request the authors to write each theorem down fully, and showcase the message the theorem is trying to convey. The paper in the current form does not do that.

  3. The performance compared to other methods is not too encouraging, moreover the main experiments that are reported are reported on AIME variants, AIME is a high variance dataset due to 30 questions only. Coding is also not reported which is another measurement of the empirical value of this work.

  4. The paper takes a stance against heuristics used to decide token supervision in post-training and why their theoretically motivated gradient cosine selection is superior, however their ablation in Table 3, shows that their selection policy is no different from any other simpler heuristic.

Questions for authors

See weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer B3kE (1/4)

We sincerely thank Reviewer B3kE for the constructive critique. In the revised manuscript (all modifications are highlighted in blue), we have: (1) rewritten Section 3 in a rigorous technical style with all inline bolding removed; (2) made all theorems/proofs fully self-contained, followed immediately by explicit takeaway and baseline-comparison paragraphs; and (3) added large-scale evaluations across 7 math/STEM ($ N = 12,280 ),3coding(), 3 coding ( N = 1,682 $), and 4 multimodal benchmarks (Appendix J).

W1 & W2: Writing style, arbitrary bolding, self-contained theorems, and theoretical takeaways

We treated W1 and W2 together by ensuring our mathematical formalism—rather than rhetorical emphasis—carries the weight of our argument.

  1. Removing distracting bolding & overhauling technical prose (Revised Section 3):
    • We removed all inline bold emphasis throughout Section 3 and the main text, replacing promotional phrasing with a concise, formal register.
    • Section 3.1 now defines every mathematical object before it is used: the token-level clipped GRPO surrogate loss $ f_i(z_i) := \ell_i^{\text{GRPO}}(z_i) $ and its gradient $ r_i = \nabla_{z_i} f_i(z_i) $ (Eq. 1), the sparse forward-KL distillation gradient $ d_i = w_i(p_i - \tilde{q}_i) $ (Eq. 2), the active non-zero RL gradient set $ \mathcal{D} = {i \in \mathcal{I} : \Vert{}r_i\Vert{}_2 > \epsilon_g} $, and the orthogonal decomposition $ (a_i, b_i) $ (Eq. 3), prior to analyzing the combined update $ z_i(\eta) = z_i - \eta(r_i + \lambda d_i) $ in Section 3.2.
  2. Fully self-contained theorems and step-by-step proofs (Revised Sections 3.2–3.4 & Appendices B–C):
    • We eliminated the cross-reference "Under the assumptions of Theorem 1" from Theorem 2. Theorems 1–3 now state every mathematical assumption and notation explicitly within their own environments.
    • In Appendices B and C, we consolidated the closed-form $ \mathcal{O}(M) $ computation and score invariance into Proposition 1 (Eqs. 15–17) and restated Theorems 1–3 in full before providing self-contained, step-by-step proofs.
AuthorsOpenReview ↗

Response to Reviewer B3kE (2/4)

  1. Explicit theorem takeaways & formal differentiation from existing methods:
  • In the revised manuscript, the first paragraph immediately following each theorem states its exact intuitive takeaway.
  • Crucially, our primary purpose in formulating Theorems 1 and 2 is to establish a formal lens that distinguishes GIFT from existing token-selection and hybrid RL-distillation methods. To make this explicit, we added two new discussion paragraphs in Sections 3.2 and 3.3:
  • Theorem 1 Takeaway & "Comparison with Statistical Token-Selection Heuristics" (Section 3.2, lines 212–230): The paragraph immediately following Theorem 1 states its core takeaway: sign(ai)\text{sign}(a_i) is an exact first-order safety filter—tokens with ai<0a_i < 0 oppose local RL descent, whereas ai≥0a_i \ge 0 preserves or accelerates it. We then add a comparison against unsigned statistical heuristics (TIP, EOPD, RSTG) that select tokens via entropy hih_i and divergence DKL(q~i∥pi)D_{\text{KL}}(\tilde{q}_i \Vert p_i). Using our closed-form identity ai=wisign(Ai)⟨pi−q~i,pi−eui⟩/∥pi−eui∥2a_i = w_i \text{sign}(A_i)\langle p_i - \tilde{q}_i, p_i - e_{u_i}\rangle / \Vert p_i - e_{u_i}\Vert_2 (Proposition 1), we show that when a confident student (pi≈euip_i \approx e_{u_i}, low hih_i) reaches a correct outcome (Ai>0A_i > 0) via a valid alternative reasoning step (q~i(ui)≪pi(ui)\tilde{q}_i(u_i) \ll p_i(u_i), high DKLD_{\text{KL}}), the inner product is strictly negative, yielding ai<0a_i < 0. Prioritizing high-DKLD_{\text{KL}}, low-hih_i tokens (such as TIP's DKL(q~i∥pi)(1−hi)D_{\text{KL}}(\tilde{q}_i \Vert p_i)(1 - h_i)) thus systematically misclassifies valid student exploration as error and injects conflicting gradients that reverse RL descent.
  • Theorem 2 Takeaway & "Comparison with Scalar Advantage-Modulation and Reward-Gating Hybrids" (Section 3.3, lines 255–278): The paragraph immediately following Theorem 2 summarizes why bi=∥di⊥∥2b_i = \Vert d_i^\perp\Vert_2 serves as our priority metric among safe tokens (ai≥0a_i \ge 0): Part (1) shows bib_i is the largest first-order distillation gain at zero first-order cost to GRPO; Part (2) shows bi=inf⁡c∈R∥di−cri∥2b_i = \inf_{c \in \mathbb{R}} \Vert d_i - c r_i\Vert_2 is the exact Euclidean distance from did_i to the RL gradient span span(ri)\text{span}(r_i); and Part (3) shows bib_i monotonically increases gradient diversity Δi\Delta_i without destructive cancellation. We then add a comparison to scalar hybrids (RLSD, RLCSD, AMR-SD, OPDVR), which incorporate the teacher solely by replacing AiA_i with a sign-preserving scalar A~i\tilde{A}_i at the sampled token uiu_i. Because their updates take the form r~i∝A~i(pi−eui)∈span(ri)\tilde{r}_i \propto \tilde{A}_i(p_i - e_{u_i}) \in \text{span}(r_i) (bi=0b_i = 0), across all unsampled vocabulary tokens v≠uiv \neq u_i, they scale logits strictly in proportion to the student's own pi(v)p_i(v), discarding teacher probabilities q~i(v)\tilde{q}_i(v) over alternative reasoning branches and incurring an irreducible error of bib_i.
  • Theorem 3 Takeaway (Section 3.4, lines 287–322): Rather than solving a constrained combinatorial program over (∣Em∣k)\binom{\vert\mathcal{E}_m\vert}{k} subsets to minimize conflict count N−(A)N^-(A), minimize conflict opposition O−(A)=∑∣ai∣O^-(A) = \sum \vert a_i\vert, and maximize orthogonal energy E+(A)=∑bi2E^+(A) = \sum b_i^2, Theorem 3 proves that because Um+,Um0,Cm\mathcal{U}_m^+, \mathcal{U}_m^0, \mathcal{C}_m are disjoint and separated by sign, a single O(∣Em∣log⁡∣Em∣)\mathcal{O}(\vert\mathcal{E}_m\vert\log\vert\mathcal{E}_m\vert) TopK sort on the piecewise score sis_i (Eq. 8) recovers the exact lexicographically optimal subset.
AuthorsOpenReview ↗

Response to Reviewer B3kE (3/4)

W3: Benchmark scale, AIME variance, and missing code/broader domain evaluations

Beyond reporting mean@12 on AMC23/AIME24/AIME25 (which evaluates 360 generations per AIME benchmark to reduce single-sample variance), we added three new large-scale evaluation suites in Section 4.2 (lines 468–495) and Appendix J (Tables 9–11) using GIFT's default hyperparameters with zero task-specific tuning:

  1. New Experiments: 7 Large-Scale Math & STEM Benchmarks (N=12,280 problems, Table 9, Appendix J): Evaluating Qwen3-8B (pass@1, %) across 12,280 problems shows that GIFT (60.28% Avg.) outperforms BASE by +8.90 pp, GRPO by +7.97 pp, and the strongest hybrid baseline RLSD by +4.32 pp. On the hardest multi-step competition benchmarks—OlympiadBench, MathOdyssey, and Omni-MATH—GIFT surpasses RLSD by +5.46 pp, +7.18 pp, and +7.68 pp, directly corroborating Theorem 2(2) at scale.
MethodMATH (5,000)GSM8K (1,319)Olympiad (674)Omni-MATH (4,428)MathOdyssey (389)Minerva (272)GPQA (198)7-Bench Avg. (12,280)$ \Delta $ vs. BASE
BASE76.7689.7640.8026.1747.8134.9343.4351.38—
GRPO74.8690.5239.6124.4447.8144.4944.4452.31+0.93
RLSD82.9088.1048.4031.1055.8041.5043.9055.96+4.58
GIFT (Ours)86.9493.4853.8638.7862.9843.0142.9360.28+8.90
  1. New Experiments: Code Generation Benchmarks ( N = 1,682 problems, Table 10, Appendix J):
    We trained Qwen3-8B on verifiable programming problems from Code-R1 and evaluated pass@1 (%) on HumanEval+ ($ 164 ),MBPP+(), MBPP+ ( 378 )understrict−plustestsuites,andBigCodeBench() under strict-plus test suites, and BigCodeBench ( 1,140 ).Becausefunctionalcorrectnessadmitsmanyvalidalgorithmicimplementations,rigidlyimitatingasinglereferencesolutioninunfilteredOPSDpenalizesvalidstudentexploration(causing55). Because functional correctness admits many valid algorithmic implementations, rigidly imitating a single reference solution in unfiltered OPSD penalizes valid student exploration (causing 55%–70% of active tokens to conflict; Figure 3 Right) and drops average accuracy _below_ BASE (63.62% vs. 64.53%, -0.91 pp). By filtering out conflicting updates ( a_i < 0 $) and annealing coverage to zero, GIFT achieves the highest score on every benchmark (68.12% Avg., +3.59 pp over BASE, +2.48 pp over GRPO, and +2.34 pp over RLSD).
MethodHumanEval+ (164)MBPP+ (378)BigCodeBench (1,140)3-Bench Avg. (1,682)$ \Delta $ vs. BASE
BASE79.8871.6942.0264.53—
GRPO81.7173.0242.1965.64+1.11
OPSD80.4970.1140.2663.62-0.91
RLSD84.7672.2240.3565.78+1.25
GIFT (Ours)84.7676.7242.8968.12+3.59
  1. New Experiments: Multimodal Visual Reasoning (Qwen3-VL-8B-Instruct, Table 11, Appendix J):
    Training Qwen3-VL-8B-Instruct on MMFineReason-123K (pass@1, %) shows that unfiltered distillation (OPSD: -1.33 pp, SDPO: -0.83 pp) and scalar modulation (RLSD: -0.31 pp) all degrade below BASE (60.58%) due to cross-modal conflicts, whereas GIFT achieves the best accuracy across all four benchmarks (62.61% Avg., +2.03 pp over BASE).
MethodMMMUMathVistaMathVisionWeMath4-Bench Avg.$ \Delta $ vs. BASE
BASE65.5673.4047.8355.5260.58—
GRPO66.4475.9048.3656.2361.73+1.15
OPSD65.5675.0045.3951.0559.25-1.33
SDPO65.3374.7048.0350.9559.75-0.83
RLSD65.7874.6046.7154.0060.27-0.31
OPDVR66.4476.2048.0356.1061.69+1.11
LUFFY66.2276.1046.7155.3361.09+0.51
GIFT (Ours)66.7876.6049.5457.5262.61+2.03

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal, Discussion Outcomes, and Manuscript Revisions

Dear Area Chairs and Reviewers,

We sincerely thank all reviewers for their constructive feedback and the Area Chairs for their time and service. We are grateful that both Reviewer B3kE and Reviewer Vwfk have explicitly confirmed that our revisions addressed their concerns and have raised their scores, and we have also provided a comprehensive 5-part empirical and theoretical response to address the insightful questions from Reviewer FgnS.

Below is a concise summary of our key additions and revisions (highlighted in blue in the updated PDF):

1. Three New Large-Scale & Multi-Domain Evaluation Suites (>14,000 Test Problems, Section 4.2 & Appendix J): Using GIFT’s default hyperparameters with zero task-specific tuning, we expanded evaluations far beyond AIME:

  • 7 Large-Scale Math & STEM Benchmarks (N=12,280N=12,280, Table 9): On Qwen3-8B across MATH, GSM8K, OlympiadBench, Omni-MATH, MathOdyssey, MinervaMath, and GPQA, GIFT (60.28% Avg.) outperforms BASE by +8.90 pp, GRPO by +7.97 pp, and RLSD by +4.32 pp (with +5.46 to +7.68 pp gains over RLSD on the hardest Olympiad-level sets).
  • 3 Code Generation Benchmarks (N=1,682N=1,682, Table 10): On Qwen3-8B trained on Code-R1 across HumanEval+, MBPP+, and BigCodeBench, unfiltered OPSD drops below BASE (-0.91 pp) due to penalizing valid alternative implementations, whereas GIFT achieves the best score on all benchmarks (68.12% Avg., +3.59 pp over BASE, +2.34 pp over RLSD).
  • 4 Multimodal Reasoning Benchmarks (Qwen3-VL-8B-Instruct, Table 11): Across MMMU, MathVista, MathVision, and WeMath, unfiltered OPSD (-1.33 pp) and RLSD (-0.31 pp) degrade below BASE, while GIFT achieves the highest accuracy across all four sets (62.61% Avg., +2.03 pp over BASE).

2. New Parameter-Space & Training Dynamics Diagnostics (Appendices C.4 & I, Figures 2 & 4):

  • Parameter-Space Gradient Alignment (Figure 2, for FgnS W1): Tracking cos⁡(Gtotal,t,GR,t)\cos(G_{\text{total}, t}, G_{R, t}) across all trainable parameters confirms that token-level logit filtering translates directly to parameter space: unfiltered GRPO+OPSD remains suppressed near 0.12 (frequently <0<0), whereas GIFT rises to 0.49 mean (0.60–0.97 in late annealing) before reaching 1.00 at teacher exit.
  • Quantifying Conflict (ai<0a_i < 0) & GRPO Sparsity (Figure 4, for Vwfk W1/W2): We show that pure GRPO stagnates on 8B models because 70%–90% of rollout groups have zero reward variance (Ai=0A_i=0), while unfiltered OPSD suffers because ai<0a_i < 0 conflicts affect 26.2% of active tokens on DeepMath and 50%–70% on Code-R1.
  • Gradient Norms & Zero-Advantage Shares (for FgnS W2/Q2): Parameter-gradient norms remain balanced (∥gOPSD,t∥2/∥gGRPO,t∥2\Vert g_{\text{OPSD}, t}\Vert_2 / \Vert g_{\text{GRPO}, t}\Vert_2 mean 1.82), with 57.88% of selected updates scaffolding zero-gradient tokens (ri=0r_i=0) and 42.12% guiding active RL tokens.
  • PCGrad-Style Logit Projection Baseline (for FgnS Q3): Projecting conflicting gradients (ai<0a_i < 0) onto di⊥d_i^\perp (70.97% on AIME24/25) underperforms both hard exclusion (Conflict-Free: 71.61%) and GIFT (73.61%, +2.64 pp), as ai<0a_i < 0 indicates teacher context mismatch where keeping di⊥d_i^\perp still pulls off-sample logits toward a divergent reference trace.

3. Theoretical & Presentation Overhaul (Revised Section 3 & Appendices B–D):

  • Self-Contained Rigor (for B3kE & Vwfk): We rewrote Section 3 in a rigorous academic style (removing all inline bolding), simplified Theorem 2(2), made Theorems 1–3 fully self-contained, and fixed minor typos in Table 3 and Appendix D (wiw_i).
  • Explicit Differentiation from Prior Work (Sections 3.2–3.4): We added formal comparisons after Theorems 1–2 proving why statistical heuristics (e.g., TIP) misclassify valid student exploration (Ai>0A_i>0) as error (ai<0a_i<0), why scalar hybrids (RLSD, RLCSD, OPDVR) incur an irreducible error of bib_i, and how a single TopK\text{TopK} sort on sis_i solves the constrained lexicographic allocation objective (Theorem 3).

Thank you again for your time and consideration.

Best regards,
The Authors of Paper 12268

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.