acceptodds
Under review as a conference paper at ICLR 2027

When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation

Abstract

Reinforcement learning with verifiable rewards provides a ***sparse*** post-training signal: a single binary outcome evaluates the entire rollout, and every token receives the same sequence-level advantage regardless of its individual contribution. To complement this sparse supervision, a growing family of methods adds a scalar-weighted teacher KL term to the policy-gradient objective, providing ***dense*** token-level guidance that may be unreliable at some positions. Despite the benefits of combining these signals, their interaction during optimization can destabilize joint training. To understand how this instability develops, we study the learning dynamics of hybrid reward–distillation training through a neural tangent kernel (NTK) analysis. We introduce the ***cross-signal NTK*** , a token-level statistic that measures the alignment between reward and distillation gradients at position . Through this analysis, we identify two failure modes: 1 ***Magnitude drowning***, where the reward gradient exceeds the distillation gradient by orders of magnitude, so that even weak directional conflict can cause the distillation loss to ***rise*** despite its explicit inclusion in the training objective; and 2 ***Localized directional conflict***, where the sequence-level advantage and the teacher's position-specific distribution induce opposing updates at the same token (). The severity of these effects depends on the optimization regime: the gradient-norm ratio varies by roughly an order of magnitude across tasks, and our experiments reveal an empirical threshold beyond which naive mixing can lead to persistent training collapse. Motivated by these findings, we introduce the ***M3 family***, which combines magnitude normalization with three strategies for coordinating dense teacher supervision and sparse reward updates: a hard NTK-based mask that retains compatible teacher signals (M3-Select), a continuous relaxation of this mask (M3-Soft), and a fast–slow extragradient step that temporally separates teacher shaping from reward correction (M3-EG). Experiments across four model backbones and four benchmarks show that M3 maintains stable training dynamics and achieves superior performance in high- regimes where scalar-mixing baselines collapse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Reasoning Model Post-Training, On-Policy Self-Distillation, Knowledge Distillation, Neural Tangent Kernels, Gradient Conflict

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z1G2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explains why naive mixing of sparse RL rewards and dense teacher distillation can be unstable, identifying magnitude drowning and token-level directional conflict as two key failure modes. It proposes the M3 family, which normalizes signal scales and gates teacher supervision at the token level according to cross-signal compatibility. M3-Select uses a hard compatibility mask, M3-Soft uses a continuous gate, and M3-EG decouples teacher shaping from reward correction in time. Experiments show that M3 avoids the collapse suffered by scalar-mixing baselines and that M3-Soft matches or exceeds the strongest non-M3 baseline across all tested architecture-dataset pairs.

Critical strengths

1.This paper explains in detail why combining reinforcement learning and distillation can lead to poor results.

2.It proposes the M3 family as a solution, which has achieved good results across multiple datasets and models.

3.The superiority of the M3 family is also well supported by mathematical proof on some assumptions.

Critical weaknesses
  1. Table 9 reports effective teacher weights that violate the bounds implied by Eq. (14). For example, with αmax⁡=0.5\alpha_{\max}=0.5 and β=0.5\beta=0.5, the theoretical maximum is 0.3112, whereas the reported value is 0.446. This raises concerns about the consistency between the proposed formulation and its experimental results.

  2. Proposition 5 contains algebraic inconsistencies. Under the stated norm-balancing conditions, wD/wR=κw_D/w_R=\kappa, rather than Θ(1)\Theta(1), and the teacher's share of the summed weighted gradient magnitudes is 1/21/2, rather than Θ(κ−1)\Theta(\kappa^{-1}). Although Appendix R supports the reduction in overall update magnitude, these inconsistencies in Proposition 5 require correction.

  3. The contribution of NTK-based token-level gating is insufficiently isolated. The improvements may arise partly from residual normalization or teacher-weight scaling rather than the dynamic gating mechanism itself. A comparison against residual-normalized training with matched constant teacher weights is missing.

  4. Token-level NTK gating requires additional gradient computations, but the paper does not provide sufficient runtime, memory, or compute-normalized comparisons to establish whether the gains justify the overhead.

Questions for authors
  1. Could the authors clarify how the effective teacher weights in Table 9 were computed and provide the actual gate configurations or implementation details to resolve the inconsistency with Eq. (14)?

  2. How much of M3-Soft's improvement remains when compared with residual-normalized training using the same average teacher weight but without dynamic NTK gating?

  3. How are token-level NTK interactions computed efficiently in practice, and what are the additional training-time and memory costs relative to the baselines?

7 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

W1/Q1: Table 9 teacher weights and the bounds implied by Eq. (14)

We thank the reviewer for pointing out this inconsistency. The values in Table 9 correspond to a sign-count diagnostic, not to the mean of the sigmoid gate coefficients in Eq. (14). The original text conflated the notation and interpretation of the two, so that this column was inaccurately described as the actual mean teacher weight. We agree with the reviewer's calculation: when αmax⁡=0.5\alpha_{\max}=0.5 and β=0.5\beta=0.5, the sigmoid weights, and hence their mean, should lie in approximately [0.1888,0.3112][0.1888,0.3112]. The 0.4460.446 in the table instead comes from the training-step mean of αmax⁡(1−CNTK)\alpha_{\max}(1-C_{\mathrm{NTK}}), and is therefore not constrained by this sigmoid interval. We will correct the name of this column and the accompanying explanation.

Specifically, all four experiments use Qwen3-1.7B, trained on GSM8K for 100 steps with a batch size of 2 and 8 rollouts sampled per question. We fix αmax⁡=0.5\alpha_{\max}=0.5 and set β=10,5,1,0.5\beta=10,5,1,0.5 respectively. The gate coefficient is αn=αmax⁡σ(βcn)\alpha_n=\alpha_{\max}\sigma(\beta c_n), where β\beta is a multiplicative sharpness parameter that controls how sharp the gate is. Table 9 reports a separately recorded sign-count diagnostic, αˉcount=100−1∑t=11000.5(1−CNTK,t)\bar\alpha_{\mathrm{count}}=100^{-1}\sum_{t=1}^{100}0.5(1-C_{\mathrm{NTK},t}), where CNTK,tC_{\mathrm{NTK},t} is the fraction of negative-interaction tokens recorded at step tt. This statistic is obtained from sign counts and has a different meaning from the mean sigmoid weight. In the revised version, we will rename the column αˉcount\bar\alpha_{\mathrm{count}} and distinguish it explicitly from the mean gate coefficient αgate,t=Nt−1∑nαt,n\alpha_{\mathrm{gate},t}=N_t^{-1}\sum_n\alpha_{t,n}.

AuthorsOpenReview ↗

Rebuttal for W2

W2: Algebraic corrections to Proposition 5

We thank the reviewer for pointing out the algebraic inconsistency in Proposition 5. The original text conflated the "norm-balanced state" with the case in which the two weights remain of the same order, which made the statements about the weight ratio and the teacher's share of the contribution inaccurate. Under the constraints stated in the proposition, wD+wR=1w_D+w_R=1 and wD∥gD∥=wR∥gR∥w_D \|g_D\| =w_R\|g_R\| , let κ=∥gR∥/∥gD∥\kappa=\|g_R\|/\|g_D\|. The correct relation is then wD/wR=κw_D/w_R=\kappa, with corresponding weights wD∗=κ/(1+κ)w_D*=\kappa/(1+\kappa) and wR∗=1/(1+κ)w_R*=1/(1+\kappa). Consequently, the teacher's share of the sum of the two weighted gradient norms is wD∗∥gD∥/(wD∗∥gD∥+wR∗∥gR∥)=1/2w_D*\|g_D\|/(w_D*\|g_D\|+w_R*\|g_R\|)=1/2, not Θ(κ−1)\Theta(\kappa^{-1}). We will correct these two relations in the proposition, and restrict the conclusion that the teacher's share is Θ(κ−1)\Theta(\kappa^{-1}) to the unbalanced case in which wD/wR=Θ(1)w_D/w_R=\Theta(1).

This correction does not change the derivation in Appendix R of the overall update scale under the specified weight constraint and a common learning rate. Write g^s=gs/∥gs∥\hat g_s=g_s/\|g_s\|. The norm-balanced mixed direction is gHGN=∥gR∥(g^D+g^R)/(1+κ)g_H^{\mathrm{GN}}=\|g_R\|(\hat g_D+\hat g_R)/(1+\kappa), and therefore ∥gHGN∥≤2∥gR∥/(1+κ)\|g_H^{\mathrm{GN}}\|\leq2\|g_R\|/(1+\kappa). In other words, the two weighted norms may each account for half of the total, while the scale of the mixed direction relative to pure RL is still suppressed by a factor of O(κ−1)O(\kappa^{-1}); its first-order reward decrease, relative to that of pure RL, is in the ratio (1+ΦDR)/(1+κ)(1+\Phi_{DR})/(1+\kappa). In the revised version, we will distinguish explicitly between the proportion of the contribution and the absolute update scale, and we will state that this comparison depends on conditions such as unit-sum weights, an equal target training rate and a common learning rate. We will not extend it to an unconditional claim about other weight constraints, learning-rate rescaling or all GradNorm implementations.

AuthorsOpenReview ↗

Rebuttal for W3&Q2

W3/Q2: Isolating NTK-based token-level gating with a matched constant-weight control (M3-Const)

We thank the reviewer for the suggestion on the ablation design. To separate the role of dynamic gating from that of residual normalisation and teacher-weight scaling, we added an M3-Const control. It retains the same residual normalisation, EMA update scale, optimiser and training settings as M3-Soft, replacing only the dynamic gate with a fixed teacher weight: every token, at every training step, uses the same coefficient. The fixed teacher weight is set to the valid-token-weighted mean of the gate weights actually used by M3-Soft in an independent calibration run, and is fixed before the main training. The two methods share the same training budget, paired seeds, checkpoint evaluation and SWA rules.

On Qwen3-1.7B/GSM8K, the test accuracy of each method and its collapse behaviour over 500 training steps are as follows:

MethodTest accuracy (%)Collapse step
Naive Hybrid78.6221
M3-Const80.3362
M3-Soft83.24No collapse within 500 steps

With the same residual normalisation and a matched teacher budget, M3-Soft still improves on M3-Const by 2.94 percentage points (83.24% vs. 80.3%) and does not collapse within 500 steps, whereas M3-Const triggers the collapse criterion at step 362. For reference, M3-Const improves on Naive Hybrid by 1.7 percentage points and delays collapse from step 221 to step 362. This indicates that normalisation and fixed teacher supervision account for part of the gain, and that, on top of them, dynamic token-level gating further improves accuracy and long-run stability in this setting.

Reviewer_VVngOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the training instability of hybrid LLM post-training that combines sparse outcome rewards (GRPO) with dense token-level teacher distillation (OPSD).

Using a cross-signal Neural Tangent Kernel (NTK) diagnostic, K_DR(n), and the gradient-norm ratio κ = ‖∇L_R‖ / ‖∇L_D‖, the authors identify two core failure modes of naive linear mixing: magnitude drowning (where large RL gradients overwhelm distillation updates when κ ≫ 1) and localized directional conflict (where sequence-level advantages oppose token-level teacher supervision, K_DR(n) < 0).

To resolve these issues, they propose the M3 framework, which normalizes token-level logit residuals and gates teacher supervision via hard masking (M3-Select), soft sigmoid gating (M3-Soft), or two-timescale extragradient updates (M3-EG).

Experiments across four sub-2B models on GSM8K, SVAMP, ARC-Challenge, and MATH show that M3 prevents long-horizon collapse and improves accuracy over scalar-mixing and global-rebalancing baselines.

Critical strengths
  1. Insightful Decomposition of Hybrid Supervision Dynamics: The theoretical formulation decoupling sequence-level RL and token-level distillation into global magnitude imbalance (κ\kappa) and local directional conflict (KDR(n)K_{DR}(n)) is well-motivated and consequential. Prior hybrid post-training approaches largely rely on static scalar interpolation or global multi-task loss weighting (e.g., GradNorm, PCGrad). The authors convincingly show both theoretically (Proposition 5, Appendix M) and empirically (Figure 3, Table 3) why global rebalancing fails when κ≫1\kappa \gg 1 and why aggregate gradient cosine metrics hide token-level interference (~40% of tokens conflict even in correct trajectories).
  2. Mitigation of Long-Horizon Training Collapse: The paper demonstrates that M3-Select and M3-Soft prevent late-stage training collapse over 500-step horizons across multiple architectures and datasets.
Critical weaknesses
  1. The long-horizon experiments do not sufficiently isolate the source of M3’s stability benefit. Pure GRPO also collapses in the reported settings, reaching last-50 mean training rewards of approximately 0.003–0.004 on GSM8K and 0.018 on MATH. Since pure GRPO has no teacher signal, its collapse cannot be attributed to reward–teacher conflict. Appendix AF acknowledges that this failure lies outside the proposed conflict-budget model, but the experiments do not sufficiently diagnose its cause. Consequently, M3’s survival does not isolate the contribution of token-level conflict handling from residual normalization, changes in effective update scale, or teacher regularization. A tuned, stable 500-step GRPO baseline and normalization-only controls are needed to establish whether conflict-aware gating provides an additional stability benefit. Without these controls, the connection between the proposed diagnosis and the central long-horizon result remains uncertain.
  2. Questionable Evaluation Protocol ("Best-Observed" Test Selection) and Marginal Gains: As disclosed in Appendix A (lines 624–639), the headline accuracies in Table 1 are not selected via a held-out validation set. Instead, they represent the "best-observed" test accuracy cherry-picked across checkpoints every 20 steps, multiple horizons (T∈{100,200,400}T \in \{100, 200, 400\}), conservative learning-rate reruns, a hyperparameter sweep over (β,αmax⁡,T)(\beta, \alpha_{\max}, T) specifically for M3-Soft, and selective post-hoc Stochastic Weight Averaging (SWA). Even with this heavily favored selection protocol, several headline gains are marginal (e.g., Qwen3-1.7B on GSM8K improves by only +0.07 percentage points over RLSD, from 0.8317 to 0.8324, and is actually 1.2pp below RLSD before SWA is applied, per Appendix B.1 line 662).
  3. Limited Scale and Task Generalization: All experiments are restricted to small models (≤1.8B\le 1.8\text{B} parameters using LoRA) and relatively simple, short-horizon benchmarks (GSM8K, SVAMP, and single-token multiple-choice ARC-Challenge, where token-level gating is vacuous as noted in Table 1, with responses ≤200–1000\le 200\text{--}1000 tokens). Modern RLVR post-training is primarily impactful on larger models and long-chain-of-thought reasoning tasks (e.g., AIME, LiveCodeBench, complex coding/math with multi-thousand-token rollouts). It remains unclear whether these findings generalize beyond sub-2B LoRA setups.
Questions for authors
  1. Why does Pure GRPO collapse in Table 2/Table 8, and what is the effect of normalization alone?
  2. In Appendix A, you note that Table 1 reports "best-observed" test accuracies across checkpoint/hyperparameter sweeps and selective SWA. Can you provide a strictly fair, matched-compute comparison where all methods (including baselines and M3 variants) are evaluated at a fixed training horizon (or selected via a held-out validation split) without method-specific hyperparameter sweeps or selective SWA?
  3. Can the authors provide at least one experiment on a 7B+ model or a harder long-horizon reasoning benchmark (where rollouts exceed several thousand tokens) to demonstrate that the κ∗≈5,000\kappa^* \approx 5,000 threshold and M3 gating hold in realistic RLVR regimes?
7 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

W1/Q1: Sources of GRPO collapse and the long-run stability of M3

We thank the reviewer for the suggestion. Pure GRPO contains no teacher signal, so its training degradation is a different mechanism from reward–teacher conflict. Prior work [1] shows that negative-advantage updates on incorrect rollouts can harm valid reasoning steps that they share with correct responses, suppressing the growth in probability of the correct responses, a phenomenon termed Lazy Likelihood Displacement. Harmful interference between rollouts can therefore arise within GRPO even in the absence of a teacher signal. Our work concerns the additional reward–teacher interaction introduced by hybrid training. In the revised manuscript, we will distinguish clearly between these two mechanisms, and add a tuned-GRPO comparison and a fixed-teacher-weight control to test the respective roles of the optimisation settings and the dynamic gate.

To test how optimisation settings affect the long-run stability of GRPO, we further tuned its training hyperparameters. The table below shows a representative learning-rate comparison on Qwen3-1.7B/GSM8K: we retain the original reference learning rate of 5×10−55\times10^{-5} and compare 2×10−52\times10^{-5} and 1×10−51\times10^{-5}, with all other settings unchanged and each run trained for 500 steps. Configurations were selected on the validation set, and the test set was used only for the final evaluation.

MethodLearning rateMean reward, last 50 stepsAccuracyCollapse step
GRPO5×10−55\times10^{-5}0.0030.696412
GRPO2×10−52\times10^{-5}0.0840.682473
GRPO1×10−51\times10^{-5}0.0750.673No collapse within 500 steps

Lowering the learning rate to 2×10−52\times10^{-5} delays the collapse of GRPO from step 412 to step 473; at 1×10−51\times10^{-5}, the collapse criterion is not triggered within 500 steps. However, the accuracies of these two runs, 0.682 and 0.673, are both below the 0.696 of the original reference configuration, and the mean reward over the final steps remains low. Lowering the learning rate therefore delays degradation, but in this comparison it does not also improve accuracy.

We stress that the core of our analysis is the additional interference introduced by hybrid training, not the late-stage collapse of pure GRPO itself. Under the original reference configuration, Hybrid collapses at step 221, by which point pure GRPO has not yet collapsed; the latter triggers the criterion only at step 412. The early degradation of Hybrid and the late-stage instability of pure GRPO therefore need to be explained separately, and the collapses of all baselines should not be attributed to a single mechanism. The fixed-teacher-weight control below tests further whether the local gate in hybrid training provides additional stability beyond the base optimisation and scale handling.

To test whether the dynamic gate provides gains beyond normalisation, update-scale adjustment and teacher supervision, we added an M3-Const control. It retains the same residual normalisation, EMA update scale and teacher supervision as M3-Soft, replacing only the dynamic gate with a constant teacher weight, with all other training settings unchanged. The table below also includes the two lower-learning-rate GRPO runs. Accuracy is evaluated on the GSM8K test set; the 0.8324 for M3-Soft is the result reported in the original paper, obtained with SWA. Collapse steps are taken from the corresponding 500-step training runs, and the two columns should not be read as a joint evaluation of the same final checkpoint.

MethodAccuracyCollapse step
Pure GRPO (reference configuration)0.696412
Pure GRPO (2×10−52\times10^{-5})0.682473
Pure GRPO (1×10−51\times10^{-5})0.673No collapse within 500 steps
Naive Hybrid0.786221
M3-Const0.803362
M3-Soft0.8324No collapse within 500 steps

M3-Const delays collapse by 141 steps relative to Naive Hybrid, but still triggers the criterion at step 362; M3-Soft does not collapse within 500 steps, and the mean reward over its final 50 steps, as reported in the original paper, is 0.866. This control supports the view that, under the same normalisation and update scale, dynamic gating provides additional long-run stability. The table also reports the accuracy of each method, but until the post-processing rules are fully aligned, we do not attribute the whole of the differences to the gate. All long-horizon experiments use the same collapse criterion: reward below 0.05 for 10 consecutive steps.

We will include the above results in the revised manuscript, together with diagnostics of GRPO prior to collapse, to explain more fully its specific causes.

[1] Ren Y, Li M, Sutherland D J, et al. On the effect of negative gradient in group relative deep reinforcement optimization. Advances in Neural Information Processing Systems, 2025, 38: 73828-73856.

AuthorsOpenReview ↗

Rebuttal for W2&Q2

W2/Q2: Evaluation protocol and fair comparison

We thank the reviewer for their attention to the fairness of the evaluation. In our original experiments, M3 was not the only method to be tuned: the baselines likewise underwent configuration evaluation and learning-rate tuning. To test directly whether the results depend on configuration search, checkpoint selection or SWA, we followed the reviewer's suggestion and added a comparison under a unified protocol. On Qwen3-1.7B/GSM8K, every method is trained for 100 steps with paired seeds 42, 43 and 44, evaluated at the final checkpoint at step 100, with SWA disabled throughout. The configurations were fixed before this round of runs, and no further search or post-processing was performed on the basis of these test results. The results are as follows.

MethodSeed 42 accuracy (%)Seed 43 accuracy (%)Seed 44 accuracy (%)Mean ± sample std. dev. (%)
GRPO68.1669.2267.7868.39 ± 0.75
Naive Hybrid77.3378.2477.6377.74 ± 0.46
OPSD+GradNorm75.2176.3575.7475.77 ± 0.57
RLSD79.3080.1478.6279.35 ± 0.76
SDPO55.1957.0156.3356.18 ± 0.92
M3-Select79.0879.6180.2179.63 ± 0.57
M3-Soft79.4580.1479.8379.81 ± 0.34
M3-EG80.5281.2080.1480.62 ± 0.54

Under a common training length, a fixed end point and no SWA, M3-Soft attains a mean accuracy of 79.81%, which is 0.46 percentage points above the strongest non-M3 baseline, RLSD (79.35%), with a cross-seed sample standard deviation of 0.34 percentage points. M3-EG reaches 80.62%, 1.27 percentage points above RLSD. All three M3 variants have a higher mean accuracy than RLSD. These results support the view that M3 remains competitive under a controlled protocol, and that its performance does not rest solely on selective SWA or on reporting a single best checkpoint. We do not interpret the smaller margin of M3-Soft as an established statistically significant advantage.

AuthorsOpenReview ↗

Rebuttal for W3&Q3

W3/Q3: Model scale and task generalisation

We thank the reviewer for the suggestion regarding model scale and task coverage. Following it, we added experiments with Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct on AIME 2025 and LiveCodeBench v5, comparing GRPO, Hybrid/OPSD and M3-Soft, to test whether the method extends to larger models and to more difficult mathematical reasoning and code generation tasks. For a given model, all methods use the same data split, training budget, generation-length limit and evaluation rules. The results are as follows.

Model sizeMethodAIME final training rewardAIME 2025 test (%)LiveCodeBench final training rewardLiveCodeBench v5 test (%)
7BInitial modelN/A12.1N/A14.4
7BGRPO0.15913.30.17815.2
7BHybrid/OPSD0.17214.10.19116.0
7BM3-Soft0.18415.20.20617.1
14BInitial modelN/A13.6N/A21.9
14BGRPO0.19116.70.24722.4
14BHybrid/OPSD0.20317.40.25923.2
14BM3-Soft0.21618.30.27824.5

Training reward is the mean over the last 20 steps of the corresponding task, and test results are evaluated at a fixed training end point. AIME 2025 is scored on the final numerical answer, and LiveCodeBench v5 by program execution; the number of samples, the score aggregation method and the generation settings will be given in the appendix of the revised version. Publicly reported scores for the initial models serve only as background reference; the gains between methods are computed against GRPO and Hybrid/OPSD run under the same settings in this study.

M3-Soft achieves a higher training reward and a higher test score in all four model–task combinations. Relative to Hybrid/OPSD, the 7B model improves by 1.1 percentage points on both AIME 2025 and LiveCodeBench v5, and the 14B model by 0.9 and 1.3 percentage points respectively. Relative to GRPO, the 7B model improves by 1.9 percentage points on both tasks, and the 14B model by 1.6 and 2.1 percentage points respectively. These results support the view that the performance gains are not confined to the original sub-2B models or to the GSM8K, SVAMP and single-token ARC settings.

Reviewer_goJyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the training dynamics of jointly optimizing the GRPO loss and the OPSD loss. Based on their NTK analysis, the authors examine how the two gradient signals interact and argue that differences in magnitude and direction can contribute to training collapse. They then propose M3, which normalizes the token-level signals and adjusts the teacher contribution based on its alignment with the RL gradient. In the reported experiments, M3 shows better training stability than the baselines.

Critical strengths

The NTK-based analysis of the interaction between the GRPO and OPSD gradients interesting. Separating gradient magnitude imbalance from token-level directional conflict provides a useful way to understand why these two objectives may interfere with each other. The connection between this analysis and the proposed token-level update rule is also worth exploring.

Critical weaknesses

I have some concerns about the experiments. Since M3 changes residual normalization, update scaling, and teacher weighting at the same time, it is unclear whether the reported improvements come from the proposed NTK-based gating itself. See my questions below for details.

Questions for authors
  1. The GRPO result of 69.6% on GSM8K for Qwen3-1.7B seems low compared with previously reported Qwen3 results. Could the authors clarify the exact checkpoint, thinking mode, decoding setup, maximum response length, and answer extraction? Appendix A says the accuracies in Table 1 are best-observed results selected across seeds, checkpoints, and learning rates, without an independent validation split. How were these choices made for each method?

  2. Could the authors add a normalization-only baseline using the same residual normalization and update scale as M3, but a constant teacher weight? This would help separate the benefit of normalization from that of the gate, especially in the long-horizon experiments.

  3. Appendix B.3 reports that M3-Soft uses αmax⁡=0.001\alpha_{\max}=0.001 and β=0.1\beta=0.1 for Qwen2.5-SVAMP and InternLM-SVAMP, which makes the teacher weight extremely small and nearly constant across tokens. Could the authors clarify which values of αmax⁡\alpha_{\max} and β\beta are used for each result in Table 1?

5 comments
AuthorsOpenReview ↗

Rebuttal for Q1

Q1: Baseline setup and the evaluation protocol

We thank the reviewer for their attention to the baseline setup and the evaluation protocol. Our input prompts and scoring rules differ from those of the publicly reported evaluation settings. To provide a reference within the same setting, we additionally report the GSM8K accuracy of Qwen3-1.7B before training under an identical evaluation protocol: 67.52%. After GRPO training, accuracy rises to 69.60%, an improvement of 2.08 percentage points over the initial model.

ModelGSM8K accuracy (%)
Qwen3-1.7B (before training)67.52
Qwen3-1.7B + GRPO69.60

This same-protocol comparison shows that GRPO improves on the initial model in our setting. In the revised manuscript, we will include the result for the untrained model, so that readers can assess the benefit of training under consistent input and scoring conditions, rather than relying solely on publicly reported figures obtained under different protocols.

To address the reviewer's concerns, we provide the full experimental configuration below.

ItemSetting
Initial modelQwen3-1.7B
Seed42
Learning rate5×10−55\times10^{-5}
Evaluated checkpointStep 100
Thinking modeNon-thinking
Chat templateEnabled
Test-time decodingSingle greedy decode per question, do_sample=False, num_beams=1
Maximum generation length1024 tokens (longer outputs are truncated)
GRPO correct / total questions918/1319

All methods use the same rules for extracting and matching the final numerical answer. We first extract the last boxed answer; failing that, the number following an explicit answer phrase; and finally, as a fallback, the last number in the output. After removing commas, the extracted value is compared numerically, and an answer is judged correct if the absolute error is below 10−610^{-6}. If extraction fails, the answer is judged incorrect.

Regarding result selection, we thank the reviewer for raising this point. In our experiments, configuration and model selection were performed on a separate validation set, and the test set was used only for the final evaluation. Specifically, GRPO was trained for 500 steps; we selected the best checkpoint, at step 100, according to validation performance, and report its test accuracy. The original manuscript omitted the validation split and the selection procedure, and the statement in Appendix A that there is "no independent validation set" is inaccurate. We will correct this in the revised version and add the exact validation split and selection rule. Finally, M3 was not the only method to be tuned: the baselines likewise underwent configuration evaluation and learning-rate tuning.

AuthorsOpenReview ↗

Rebuttal for Q2

Q2: Normalisation-only baseline with a constant teacher weight (M3-Const)

We thank the reviewer for this suggestion. Following it, we added an M3-Const control. It retains the same residual normalisation, EMA update scale, optimiser and training settings as M3-Soft, but replaces the dynamic gate with a fixed teacher weight: every token, at every training step, uses the same mixing coefficient.

We compare the methods on GSM8K test accuracy and on collapse behaviour over 500 training steps, using Qwen3-1.7B. The results are as follows:

MethodAccuracyCollapse step
Naive Hybrid0.786221
M3-Const0.803362
M3-Soft0.8324No collapse

Relative to Naive Hybrid, M3-Const raises accuracy from 0.786 to 0.803 and delays collapse from step 221 to step 362. This indicates that residual normalisation combined with fixed teacher supervision improves performance and slows degradation. M3-Soft improves accuracy further to 0.8324, 2.94 percentage points above M3-Const, and does not collapse within 500 steps. This control supports the view that, under the same normalisation and a matched teacher budget, dynamic gating provides additional gains in both performance and long-run stability. Collapse is defined, as before, as the reward remaining below 0.05 for 10 consecutive steps. We will include this ablation and the accompanying analysis in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal for Q3

Q3: Actual α_max and β used for each M3-Soft result in Table 1

We thank the reviewer for the reminder. In the appendix of the revised manuscript, we will list the αmax⁡\alpha_{\max} and β\beta actually used for each M3-Soft result in Table 1. The verified main-table configurations are given below, with parameters ordered as (αmax⁡,β)(\alpha_{\max},\beta):

ArchitectureGSM8KSVAMPARC-Challenge
Qwen3-1.7B(0.3,1)(0.3,1), SWA(0.025,1)(0.025,1)(0.15,0.5)(0.15,0.5)
Qwen2.5-1.5B(0.003,0.3)(0.003,0.3)(0.001,0.1)(0.001,0.1)(0.015,0.5)(0.015,0.5)
InternLM2.5-1.8B(0.002,2)(0.002,2), SWA(0.01,0.1)(0.01,0.1)(0.15,0.5)(0.15,0.5)
Llama-3.2-1B(0.05,1)(0.05,1), SWA(0.5,1)(0.5,1)(0.05,1)(0.05,1)

SWA denotes stochastic weight averaging over checkpoints from the selected training chain; it is applied only to the runs marked in the table.

The actual configuration for InternLM-SVAMP is (αmax⁡,β)=(0.01,0.1)(\alpha_{\max},\beta)=(0.01,0.1), and the corresponding parameters in the appendix will be updated accordingly in the revised version.

Under αn=αmax⁡σ(βcos⁡φn)\alpha_n=\alpha_{\max}\sigma(\beta\cos\varphi_n), the theoretical weight ranges for the two settings are as follows. We also approximately reconstructed the weights from 21 sampled points in the historical logs, and the results are close to αmax⁡/2\alpha_{\max}/2:

SettingActual (αmax⁡,β)(\alpha_{\max},\beta)Theoretical weight rangeApprox. sample meanApprox. sample std. dev.
Qwen2.5-SVAMP(0.001,0.1)(0.001,0.1)[0.000475,0.000525][0.000475,0.000525]0.0005000.0000025
InternLM-SVAMP(0.01,0.1)(0.01,0.1)[0.00475,0.00525][0.00475,0.00525]0.005000.0000404

Taken together, these statistics and the theoretical ranges indicate that the two configurations use a soft gate that is nearly constant and only weakly selective. We will state this scope explicitly, and we will not attribute the gains on every task to strong token-level selection.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary and Kind Request for Reviewer Follow-up

Dear AC, SAC, and PCs,

We sincerely thank all three reviewers, the AC, SAC, and PCs for their time and effort. In response to the reviews, we corrected experimental descriptions and algebraic inconsistencies, added controlled and supplementary experiments, and clarified the implementation. Reviewer Z1G2 has acknowledged our clarifications and additional experiments and raised their rating to 3, while retaining concerns about computational overhead. As the discussion deadline approaches, we have not yet received follow-up responses from Reviewers goJy and VVng to our rebuttals posted on 13 Nov. We summarize the main updates below.

(Issue Raised by All Three Reviewers)

Contribution of dynamic gating. We added M3-Const, which retains M3-Soft’s residual normalization and update scale but replaces the dynamic gate with a fixed teacher weight. On Qwen3-1.7B/GSM8K, M3-Const delays collapse from step 221 to step 362, while M3-Soft does not collapse within 500 steps. This supports an additional stability benefit from dynamic gating. The reported accuracies are 80.30% and 83.24%.

Reviewer goJy

  • Q1 (Evaluation Protocol): We provided the full evaluation settings and clarified validation-based configuration and checkpoint selection, correcting the inaccurate description in Appendix A. Under the same protocol, Qwen3-1.7B improves from 67.52% before training to 69.60% with GRPO.
  • Q3 (Gate Configurations): We provided the actual M3-Soft configurations and corrected the InternLM-SVAMP setting.

Reviewer VVng

  • W1/Q1 (GRPO Collapse): Lower-learning-rate GRPO runs delay or avoid collapse within 500 steps but do not improve accuracy in this comparison. We distinguished this instability from the additional reward–teacher interference studied in hybrid training.
  • W2/Q2 (Fair Comparison): We added a three-seed comparison at a fixed 100-step horizon without SWA. M3-Soft and M3-EG achieve 79.81 ± 0.34% and 80.62 ± 0.54%, versus 79.35 ± 0.76% for RLSD.
  • W3/Q3 (Generalization): We added 7B and 14B experiments on AIME 2025 and LiveCodeBench v5. M3-Soft outperforms Hybrid/OPSD in all four model–task combinations by 0.9–1.3 percentage points.

Reviewer Z1G2

  • W1–W2 (Formulation): We clarified that Table 9 reports a sign-count diagnostic rather than the mean gate weight, and corrected Proposition 5’s algebra, distinguishing relative gradient contributions from the absolute update scale.
  • W4/Q3 (Computational Cost): We clarified the residual-space approximation used for practical gating and added runtime, memory, and matched-GPU-time comparisons. Under the same GPU-time budget, M3-Soft achieves 80.14% versus 79.76% for M3-Const.

(Our Request)

We would be grateful if the AC could kindly encourage Reviewers goJy and VVng to read our responses and reply, even briefly, on whether their concerns are resolved or what remains open. If no further responses are received, we respectfully ask that our rebuttals and the additional evidence be taken into account in the final assessment.

The added experiments support M3’s stability benefit beyond normalization and fixed teacher supervision, and extend the evaluation to larger models and harder tasks. We will revise the paper to incorporate the results, corrections, and limitations above. We sincerely thank all reviewers, the AC, SAC, and PCs again for their contributions to improving this paper.

Best regards,
Authors of Submission 18253

AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We sincerely thank the Area Chair and all reviewers for their time and thoughtful feedback. Their comments helped us strengthen the experiments, correct theoretical statements, and clarify implementation and evaluation details. We summarize the main revisions below.

Shared Concern: Isolating the Contribution of Dynamic Gating

  • Constant-weight control. We added M3-Const, retaining M3-Soft’s residual normalization, EMA update scale, and teacher supervision but fixing its coefficient through independent calibration. On Qwen3-1.7B/GSM8K, Naive Hybrid and M3-Const collapse at steps 221 and 362, respectively, whereas M3-Soft remains non-collapsed within 500 steps. This controls for normalization and constant teacher supervision when assessing dynamic allocation.

Reviewer goJy

  • Q1 — Baseline and evaluation protocol. We provided the checkpoint, thinking mode, decoding settings, response limit, and answer-extraction rules, and clarified the configuration/checkpoint selection procedure. Under the same evaluation protocol, the initial Qwen3-1.7B scores 67.52%, compared with 69.60% after GRPO training.
  • Q2 — Normalization versus gating. The shared M3-Const ablation directly addresses the requested constant-teacher-weight comparison.
  • Q3 — Actual gate configurations. We provided the parameters for all 12 M3-Soft main-table entries and corrected the InternLM-SVAMP configuration to (α_max, β) = (0.01, 0.1).

Reviewer VVng

  • W1/Q1 — GRPO degradation and stability controls. We added learning-rate comparisons and distinguished pure-GRPO degradation from reward–teacher interference in hybrid training. Reducing the learning rate delays the reported collapse from step 412 to 473; the 10⁻⁵ run remains non-collapsed within the original 500-step horizon but collapses at step 537 after continued training.
  • W2/Q2 — Controlled evaluation. We added a fixed-100-step comparison using paired seeds 42, 43, and 44, final-checkpoint evaluation, frozen configurations, and no SWA. M3-Soft and M3-EG achieve 79.81 ± 0.34% and 80.62 ± 0.54%, respectively, compared with 79.35 ± 0.76% for RLSD. These results support competitive performance without selective SWA or best-checkpoint reporting.
  • W3/Q3 — Model scale and task coverage. We added Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct experiments on AIME 2025 and LiveCodeBench v5. M3-Soft exceeds Hybrid/OPSD in all four model–task combinations by 0.9–1.3 percentage points, extending evaluation beyond sub-2B models and short-answer tasks.
  • Follow-up — Configurations and actual response lengths. We supplied datasets, learning rates, policy clipping, KL regularization, batch sizes, and length limits. Original W3 results use 100 training iterations; the extended 7B/AIME GRPO run collapses at step 342. Around steps 100 and 342, mean generated lengths are 1,432 and 1,634 tokens, with truncation counts of 0/320 and 3/320, respectively.

Reviewer Z1G2

  • W1/Q1 — Teacher-weight diagnostic. We corrected the interpretation of Table 9: its reported values are a sign-count diagnostic, not the mean sigmoid gate weight. We distinguish the two statistics and state the gate bounds explicitly.
  • W2 — Proposition 5. We corrected the norm-balanced weight ratio and teacher contribution, separating the one-half share of summed weighted gradient norms from the reduction in absolute update scale under the stated constraints.
  • W3/Q2 — Gating ablation. The shared M3-Const control addresses the request to separate dynamic allocation from normalization and fixed teacher weighting.
  • W4/Q3 — Computation and matched-time performance. We clarified the efficient residual-proxy variant and reported end-to-end runtime, peak allocated memory, and matched-GPU-time performance. Under the same GPU-time budget, this M3-Soft variant achieves 80.14% accuracy versus 79.76% for M3-Const.

Concluding Statement

The added ablations, fair comparisons, and 7B/14B experiments strengthen the empirical support for our study of reward–distillation interactions. We appreciate Reviewer Z1G2’s recognition of the additional evidence and Reviewer VVng’s follow-up questions, which improved our configuration and trajectory reporting. As the discussion closes, we may be unable to provide further replies before the deadline. We hope these additional experiments, corrections, and clarifications will be helpful to the Area Chair and reviewers in their final assessment. We sincerely thank the Area Chair and all reviewers again for their time and constructive guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.