- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explains why naive mixing of sparse RL rewards and dense teacher distillation can be unstable, identifying magnitude drowning and token-level directional conflict as two key failure modes. It proposes the M3 family, which normalizes signal scales and gates teacher supervision at the token level according to cross-signal compatibility. M3-Select uses a hard compatibility mask, M3-Soft uses a continuous gate, and M3-EG decouples teacher shaping from reward correction in time. Experiments show that M3 avoids the collapse suffered by scalar-mixing baselines and that M3-Soft matches or exceeds the strongest non-M3 baseline across all tested architecture-dataset pairs.
Critical strengths
1.This paper explains in detail why combining reinforcement learning and distillation can lead to poor results.
2.It proposes the M3 family as a solution, which has achieved good results across multiple datasets and models.
3.The superiority of the M3 family is also well supported by mathematical proof on some assumptions.
Critical weaknesses
-
Table 9 reports effective teacher weights that violate the bounds implied by Eq. (14). For example, with and , the theoretical maximum is 0.3112, whereas the reported value is 0.446. This raises concerns about the consistency between the proposed formulation and its experimental results.
-
Proposition 5 contains algebraic inconsistencies. Under the stated norm-balancing conditions, , rather than , and the teacher's share of the summed weighted gradient magnitudes is , rather than . Although Appendix R supports the reduction in overall update magnitude, these inconsistencies in Proposition 5 require correction.
-
The contribution of NTK-based token-level gating is insufficiently isolated. The improvements may arise partly from residual normalization or teacher-weight scaling rather than the dynamic gating mechanism itself. A comparison against residual-normalized training with matched constant teacher weights is missing.
-
Token-level NTK gating requires additional gradient computations, but the paper does not provide sufficient runtime, memory, or compute-normalized comparisons to establish whether the gains justify the overhead.
Questions for authors
-
Could the authors clarify how the effective teacher weights in Table 9 were computed and provide the actual gate configurations or implementation details to resolve the inconsistency with Eq. (14)?
-
How much of M3-Soft's improvement remains when compared with residual-normalized training using the same average teacher weight but without dynamic NTK gating?
-
How are token-level NTK interactions computed efficiently in practice, and what are the additional training-time and memory costs relative to the baselines?
7 comments
Rebuttal for W2
W2: Algebraic corrections to Proposition 5
We thank the reviewer for pointing out the algebraic inconsistency in Proposition 5. The original text conflated the "norm-balanced state" with the case in which the two weights remain of the same order, which made the statements about the weight ratio and the teacher's share of the contribution inaccurate. Under the constraints stated in the proposition, and , let . The correct relation is then , with corresponding weights and . Consequently, the teacher's share of the sum of the two weighted gradient norms is , not . We will correct these two relations in the proposition, and restrict the conclusion that the teacher's share is to the unbalanced case in which .
This correction does not change the derivation in Appendix R of the overall update scale under the specified weight constraint and a common learning rate. Write . The norm-balanced mixed direction is , and therefore . In other words, the two weighted norms may each account for half of the total, while the scale of the mixed direction relative to pure RL is still suppressed by a factor of ; its first-order reward decrease, relative to that of pure RL, is in the ratio . In the revised version, we will distinguish explicitly between the proportion of the contribution and the absolute update scale, and we will state that this comparison depends on conditions such as unit-sum weights, an equal target training rate and a common learning rate. We will not extend it to an unconditional claim about other weight constraints, learning-rate rescaling or all GradNorm implementations.
Rebuttal for W3&Q2
W3/Q2: Isolating NTK-based token-level gating with a matched constant-weight control (M3-Const)
We thank the reviewer for the suggestion on the ablation design. To separate the role of dynamic gating from that of residual normalisation and teacher-weight scaling, we added an M3-Const control. It retains the same residual normalisation, EMA update scale, optimiser and training settings as M3-Soft, replacing only the dynamic gate with a fixed teacher weight: every token, at every training step, uses the same coefficient. The fixed teacher weight is set to the valid-token-weighted mean of the gate weights actually used by M3-Soft in an independent calibration run, and is fixed before the main training. The two methods share the same training budget, paired seeds, checkpoint evaluation and SWA rules.
On Qwen3-1.7B/GSM8K, the test accuracy of each method and its collapse behaviour over 500 training steps are as follows:
| Method | Test accuracy (%) | Collapse step |
|---|---|---|
| Naive Hybrid | 78.6 | 221 |
| M3-Const | 80.3 | 362 |
| M3-Soft | 83.24 | No collapse within 500 steps |
With the same residual normalisation and a matched teacher budget, M3-Soft still improves on M3-Const by 2.94 percentage points (83.24% vs. 80.3%) and does not collapse within 500 steps, whereas M3-Const triggers the collapse criterion at step 362. For reference, M3-Const improves on Naive Hybrid by 1.7 percentage points and delays collapse from step 221 to step 362. This indicates that normalisation and fixed teacher supervision account for part of the gain, and that, on top of them, dynamic token-level gating further improves accuracy and long-run stability in this setting.
Rebuttal for W1&Q1
W1/Q1: Table 9 teacher weights and the bounds implied by Eq. (14)
We thank the reviewer for pointing out this inconsistency. The values in Table 9 correspond to a sign-count diagnostic, not to the mean of the sigmoid gate coefficients in Eq. (14). The original text conflated the notation and interpretation of the two, so that this column was inaccurately described as the actual mean teacher weight. We agree with the reviewer's calculation: when αmax=0.5 and β=0.5, the sigmoid weights, and hence their mean, should lie in approximately [0.1888,0.3112]. The 0.446 in the table instead comes from the training-step mean of αmax(1−CNTK), and is therefore not constrained by this sigmoid interval. We will correct the name of this column and the accompanying explanation.
Specifically, all four experiments use Qwen3-1.7B, trained on GSM8K for 100 steps with a batch size of 2 and 8 rollouts sampled per question. We fix αmax=0.5 and set β=10,5,1,0.5 respectively. The gate coefficient is αn=αmaxσ(βcn), where β is a multiplicative sharpness parameter that controls how sharp the gate is. Table 9 reports a separately recorded sign-count diagnostic, αˉcount=100−1∑t=11000.5(1−CNTK,t), where CNTK,t is the fraction of negative-interaction tokens recorded at step t. This statistic is obtained from sign counts and has a different meaning from the mean sigmoid weight. In the revised version, we will rename the column αˉcount and distinguish it explicitly from the mean gate coefficient αgate,t=Nt−1∑nαt,n.