- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GIFT, a method for combining GRPO with on-policy self-distillation (OPSD) to improve LLM reasoning. Instead of applying distillation to all tokens, GIFT analyzes the relationship between GRPO and OPSD gradients in token-level logit space. It prioritizes tokens where the distillation gradient does not conflict with GRPO and provides additional orthogonal directional information. The selected tokens receive the full distillation loss, while the proportion of selected tokens gradually decreases during training, eventually transitioning to pure GRPO.
The authors derive a closed-form scoring rule for efficient token selection and evaluate GIFT on mathematical and logical reasoning tasks across Qwen3 and OLMo models. The results show improvements in average performance over several RL and distillation baselines, with ablation studies examining the effectiveness of the gradient-based selection strategy and progressive removal of teacher supervision.
Critical strengths
Critical Strengths
1. A practical and well-motivated framework for combining RL and distillation.
The paper addresses an important problem in distillation-augmented RL: how to selectively incorporate dense teacher supervision without excessively interfering with outcome-based optimization. The proposed gradient decomposition provides an intuitive way to distinguish conflicting guidance from complementary directional information. Moreover, the closed-form scoring rule avoids additional backward passes, while coverage annealing offers a simple mechanism for progressively removing teacher supervision. These design choices make GIFT a practical and potentially useful framework.
2. Extensive experiments and informative ablation studies.
The paper evaluates GIFT across multiple model scales and architectures (Qwen3 and OLMo-3), covering both mathematical and logical reasoning tasks. The reported results show competitive and generally improved average performance over a diverse set of RL and distillation baselines, particularly on challenging benchmarks such as AIME25. The ablation studies further provide useful evidence for the effectiveness of combining conflict-aware selection, orthogonal residual scoring, and coverage annealing.
Critical weaknesses
Weaknesses
1. Gap between logit-space theoretical guarantees and actual optimization.
The theoretical analysis establishes first-order gradient compatibility only in token-level logit space. However, such compatibility does not necessarily hold in the shared parameter space due to Jacobian transformations and cross-token interactions, as the authors acknowledge in Appendix C.2.
More importantly, the paper provides no direct empirical evidence that GIFT improves parameter-space gradient alignment or that larger orthogonal residuals lead to greater reward improvements. This weakens the central theoretical justification for the proposed geometric filtering strategy, which appears to be a theoretically motivated heuristic rather than a guaranteed improvement to RL optimization.
2. Unclear relative contribution of OPSD and GRPO gradients.
GIFT fixes the distillation coefficient at , but the actual contribution of OPSD relative to GRPO remains unclear. In particular, the GRPO gradient magnitude depends on , whereas the proposed selection scores are invariant to the advantage magnitude (Proposition 2). Consequently, OPSD may dominate updates at tokens with small GRPO advantages, regardless of their geometric compatibility.
The paper lacks analysis of the relative gradient norms, their evolution during training, and their dependence on advantage magnitudes. Although selection ablations are provided, it remains unclear to what extent the gains arise from the proposed geometric criterion versus the effective balance between the two objectives. This limits confidence in the mechanism and robustness of the method.
Minor Questions
-
Gradient weighting consistency. In Eq. (1), is already defined as , but Eq. (25) in Appendix D appears to multiply by again. Is this a notation error? Please clarify the actual implementation.
-
Zero-advantage tokens. How much of the total OPSD gradient comes from tokens with zero or clipped GRPO gradients? These tokens can still receive distillation supervision under the exact-quota selection mechanism. Reporting their contribution would help clarify the effective interaction between the two objectives.
-
Missing gradient projection baseline. Given that Theorem 2 emphasizes the value of the orthogonal component of the distillation gradient, why does GIFT perform discrete token selection rather than directly removing conflicting gradient components? A PCGrad-style logit-space projection could eliminate the conflicting component while preserving orthogonal guidance. Could the authors compare GIFT against such a projection-based baseline under comparable supervision budgets? This comparison would help establish whether the proposed token selection and lexicographic scoring are necessary, rather than simply enforcing gradient compatibility through projection.
Questions for authors
See weakness. I am willing to raise my score to raise my score if the authors can adequately address my concerns.
9 comments
Response to Reviewer FgnS (1/5)
We sincerely thank the reviewer for a precise and constructive reading. In the revised manuscript, we ran new parameter-space gradient alignment and norm diagnostics (now Figure 2 and a rewritten Appendix C.4), tracked rollout sparsity and conflict dynamics (Figure 4, Appendix I), implemented and ran the requested PCGrad-style logit-space projection baseline, corrected the notation in Appendix D, and expanded our evaluation to large-scale STEM, Coding, and Multimodal benchmarks (Appendix J). (Note: Equation, proposition, and figure numbers below reference the revised manuscript, with revisions highlighted in blue.)
W1: Logit-space theoretical guarantees vs. parameter-space optimization — now measured directly (New Figure 2, Rewritten Appendix C.4)
The reviewer correctly identified the weakest link in our original submission: we proved first-order compatibility in token-level logit space, acknowledged in Appendix C that it need not transfer globally through the shared-parameter Jacobian, and offered no direct parameter-space measurement. We have now directly measured it and linked both geometric components () to optimization and reward gains:
- 1. Theoretical Bridge: Exact Per-Token Linearity vs. Cross-Token Blocks (Rewritten Appendix C.4): Let and denote the parameter gradients of the GRPO and masked OPSD objectives, where is the stacked logit Jacobian (). At any individual token , the parameter-gradient mapping is linear, so our logit decomposition strictly preserves collinearity for all diagonal self-interaction terms (). Discrepancies between logit-space and parameter-space alignment can arise only from off-diagonal cross-token blocks (). While materializing per-token parameter gradients for online selection requires storage and per-token backward passes (intractable at LLM scale, whereas GIFT takes in closed form via Proposition 1), aggregate batch-level parameter gradients () cost only one backward pass each per step and can be directly measured.
Response to Reviewer FgnS (2/5)
- 2. [NEW] Direct Measurement of Parameter-Space Gradient Alignment (New Figure 2, Qwen3-8B): At every actor step, we log the cosine similarity between the applied total parameter gradient and the model's own outcome-grounded GRPO reference gradient across all trainable parameters (accumulated before clipping, with carrying each method's own mask and token weights): Because each method is compared against its own at its own parameters, this directly measures how far the auxiliary distillation term deflects that method's outcome-driven update in parameter space:
| Parameter-Space Alignment | Mean (Steps 45–77) | Min (Steps 45–77) | Late Annealing (Steps 70–77) | Post-Exit () |
|---|---|---|---|---|
GRPO + OPSD (Unfiltered) | 0.12 | ( deflection) | 0.00–0.23 | |
GIFT (Ours) | 0.49 (+0.37) | 0.24 (+0.28) | 0.60–0.97 (+0.49 to +0.82) | 1.00 (Exact RL) |
-
3. Why This Improvement is Mechanistic Rather Than Incidental: Two timing features in Figure 2 are decisive. First, the curves are indistinguishable during early training (before Step 20)—exactly as the mechanism predicts, since GIFT applies the teacher broadly during full-coverage warm-up () when conflicts are lower. Second, the curves separate precisely when coverage annealing prunes conflicting tokens and the token-level conflicting fraction () surges as the maturing student discovers valid reasoning paths that diverge from the single reference trace (New Figure 4, Right, where climbs to 55%–70%). The parameter-space gap is widest at Steps 70–77 (+0.49 to +0.82 higher cosine), where GIFT's coverage annealing has already driven conflicting-token retention to 0.0% (Figure 7). Thus, the empirical answer to your concern is clear: off-diagonal cross-token blocks do not wash out our token-local criterion—filtering on in logit space preserves aggregate parameter-space alignment that unfiltered distillation destroys. Moreover, at , GIFT's mask vanishes identically so (Corollary 2), ensuring final convergence is governed purely by exact parameter-space RL gradients.
-
4. Linking Orthogonal Residuals () and Safety () to Verifiable Reward Gains (Existing Table 4 & New Appendix J): We have kept our scope statement in Appendix C.4 precise rather than overselling Figure 2: cosine similarity is a first-order directional statistic, so for the direct link between larger orthogonal residuals () and task rewards, we rely on controlled ablations that vary only the ranking quantity under matched coverage schedules (Table 4), alongside new large-scale evaluations across Math/STEM, Coding, and Multimodal reasoning (New Appendix J, Tables 9–11):
- Isolating and under matched schedules (Table 4): Holding the safety filter and schedule fixed, prioritizing larger (
GIFT: 78.96% Avg., 71.11% on AIME25) substantially outperforms ranking by parallel agreement alone (Conflict-only, which ignores : 77.08% Avg., 65.83% on AIME25, a +5.28 pp gain on AIME25). Conversely, maximizing or while ignoring (Residual-only: 76.97%;Teacher-norm: 76.90%) consistently degrades accuracy. - Large-scale corroboration of Theorem 2(2) (New Tables 9–11): Scalar advantage-modulation hybrids (e.g.,
RLSD) are confined to (). Across 12,280 STEM problems (Table 9), 1,682 Code-R1 problems (Table 10), and 4 Multimodal benchmarks (Table 11), injecting filtered orthogonal guidance () yields consistent, large-scale gains over both pureGRPOand scalar-modulatedRLSD:
- Isolating and under matched schedules (Table 4): Holding the safety filter and schedule fixed, prioritizing larger (
| Evaluation Suite (New Appendix J) | BASE | GRPO | OPSD | RLSD () | GIFT (Ours) | Gain vs. GRPO / RLSD |
|---|---|---|---|---|---|---|
| 7-Bench Math & STEM (, Table 9) | 51.38 | 52.31 | — | 55.96 | 60.28 | +7.97 / +4.32 pp |
| 3-Bench Code Generation (, Table 10) | 64.53 | 65.64 | 63.62 | 65.78 | 68.12 | +2.48 / +2.34 pp |
| 4-Bench Multimodal VL (Qwen3-VL-8B, Table 11) | 60.58 | 61.73 | 59.25 | 60.27 | 62.61 | +0.88 / +2.34 pp |
Response to Reviewer FgnS
Dear Reviewer FgnS:
Thank you for your patience. We are currently finalizing the additional experiments and analyzing the results to thoroughly address the reviewers' feedback. We will upload the revised manuscript, along with a detailed point-by-point response, very soon. Please be assured that we are keeping track of the deadline and will complete the submission on time. Thank you again for your support.