- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper conducts a controlled 20-seed study of GRPO on Qwen2.5-Math-1.5B trained for one epoch on MATH. At the final checkpoints, directional Hessian and Fisher curvature show negative associations with test pass@1, while training reward and gradient norm show weak correlations. Reference KL has positive Spearman but near-zero Pearson correlations. Motivated by this, it proposes KL-SAM: a SAM-style perturbed GRPO loss plus a penalty on the KL between the clean policy and the policy at the perturbed point, both using the same first-order perturbation epsilon = rho*g/||g||. A second-order expansion shows the two terms contribute a directional Hessian and a directional Fisher term along epsilon (Theorem 5.2), and an informal CMI-based bound relates trajectory-Fisher curvature to the generalization gap. For efficiency the perturbation is applied at the output layer following logits-SAM. On Qwen2.5-Math-1.5B/7B and DeepSeek-R1-Distill-Qwen-1.5B with GRPO and DR.GRPO, KL-SAM improves average pass@1 over five math benchmarks by about 2.1 points over the baseline and 0.9 points over logits-SAM, with ~4% step-time overhead.
Critical strengths
- The controlled 20-seed study is useful evidence on variation across GRPO runs under a fixed configuration. Comparing directional curvature with training reward, gradient norm, and reference KL helps distinguish their associations with test performance. The method is simple, its reported step-time overhead is modest, and the writing is clear.
Critical weaknesses
The main evaluation selects checkpoints using the test benchmarks themselves. This weakens the evidence for generalization, especially given the modest average gain over logits-SAM. Table 6 partly addresses this: final-checkpoint results strengthen KL-SAM’s lead to 0.70/1.67 points under GRPO/DR.GRPO, but only for Qwen2.5-Math-1.5B. The other four settings are single runs. Final or validation-selected checkpoints, additional training seeds, and uncertainty estimates for the paired differences are needed to establish robustness. This evaluation issue is my main reason for leaning toward rejection.
The theoretical link to the implemented algorithm remains incomplete. Theorem 5.2 gives a local Hessian–Fisher interpretation, but Theorem 4.1 concerns parameter differences between policies trained on datasets differing in one example. KL-SAM instead penalizes sensitivity along an output-head gradient perturbation. The paper does not establish that reducing this quantity controls the directions appearing in the generalization bound. Clarifying the parameter subspace and estimator used in Table 1, and testing whether the implemented local-KL penalty predicts performance across the 20 seeds, would strengthen the connection.
Questions for authors
-
Can you report final or validation-selected checkpoint results with multiple training seeds for the remaining model–objective settings? Does the average gain over logits-SAM persist?
-
What parameter subspace and estimator were used for the Fisher measurements in Table 1? At a fixed perturbation radius, does the implemented output-head local-KL penalty also correlate with test pass@1 across the 20 runs?
-
All methods reuse the baseline-selected learning rate. Would independently tuning logits-SAM’s learning rate and perturbation radius under a comparable validation-search budget change the comparison?
-
Please report the selected checkpoint steps and best-versus-final gaps for each method and seed. Does the identical KL-SAM GRPO mean in Tables 2 and 6 reflect the final checkpoint being selected in every seed?
3 comments
Supplementary Experimental Results
Table 7 (Appendix D): Selected checkpoints for Qwen2.5-Math-1.5B over three seeds. Numbers indicate training steps; Final denotes the end of training.
| Objective | Method | Seed 42 | Seed 0 | Seed 1 |
|---|---|---|---|---|
| GRPO | Baseline | 200 | 200 | 400 |
| GRPO | logits-SAM | Final | 300 | 400 |
| GRPO | KL-SAM | Final | Final | Final |
| GRPO | KL-only | 200 | 400 | Final |
| DR.GRPO | Baseline | 300 | 400 | Final |
| DR.GRPO | logits-SAM | 400 | Final | 300 |
| DR.GRPO | KL-SAM | Final | 400 | Final |
| DR.GRPO | KL-only | 300 | 300 | Final |
Table 8 (Appendix D; Avg. only): Best-checkpoint pass@1 (%) for DeepSeek-R1-Distill-Qwen-1.5B over three seeds (mean ± SD).
| Objective | Method | Avg. pass@1 (%) |
|---|---|---|
| GRPO | Baseline | 41.97 ± 1.67 |
| GRPO | logits-SAM | 41.61 ± 2.14 |
| GRPO | KL-SAM | 44.57 ± 1.80 |
| DR.GRPO | Baseline | 41.46 ± 1.36 |
| DR.GRPO | logits-SAM | 43.45 ± 1.17 |
| DR.GRPO | KL-SAM | 44.52 ± 0.79 |
Table 9 (Appendix D; Avg. only): Final-checkpoint pass@1 (%) for DeepSeek-R1-Distill-Qwen-1.5B over three seeds (mean ± SD).
| Objective | Method | Avg. pass@1 (%) |
|---|---|---|
| GRPO | Baseline | 40.37 ± 3.54 |
| GRPO | logits-SAM | 36.86 ± 2.81 |
| GRPO | KL-SAM | 42.13 ± 1.97 |
| DR.GRPO | Baseline | 40.15 ± 2.58 |
| DR.GRPO | logits-SAM | 43.38 ± 1.26 |
| DR.GRPO | KL-SAM | 44.24 ± 0.74 |
Table 11 (Appendix D; Avg. only): Final-checkpoint pass@1 (%) for Qwen2.5-Math-7B (one seed).
| Objective | Method | Avg. pass@1 (%) |
|---|---|---|
| GRPO | Baseline | 42.70 |
| GRPO | logits-SAM | 45.05 |
| GRPO | KL-SAM | 44.97 |
| DR.GRPO | Baseline | 41.27 |
| DR.GRPO | logits-SAM | 44.62 |
| DR.GRPO | KL-SAM | 45.97 |
Table 13 (Appendix D): Paired differences in final-checkpoint Avg. pass@1 (percentage points; mean ± SD over three matched training seeds).
| Model | Comparison | GRPO | DR.GRPO |
|---|---|---|---|
| Qwen2.5-Math-1.5B | KL-SAM − Baseline | +3.64 ± 1.70 | +1.28 ± 2.04 |
| Qwen2.5-Math-1.5B | KL-SAM − logits-SAM | +0.70 ± 0.43 | +1.67 ± 0.60 |
| Qwen2.5-Math-1.5B | KL-SAM − KL-only | +1.90 ± 1.32 | +3.16 ± 3.80 |
| DeepSeek-R1-Distill-Qwen-1.5B | KL-SAM − Baseline | +1.76 ± 3.65 | +4.09 ± 1.94 |
| DeepSeek-R1-Distill-Qwen-1.5B | KL-SAM − logits-SAM | +5.27 ± 1.78 | +0.86 ± 1.61 |
Table 14 (Appendix D): Best–final checkpoint gap in Avg. pass@1 (percentage points; lower is better).
| Model | Objective | Seeds | Baseline | logits-SAM | KL-SAM |
|---|---|---|---|---|---|
| Qwen2.5-Math-1.5B | GRPO | 3 | 1.79 | 0.52 | 0.00 |
| Qwen2.5-Math-1.5B | DR.GRPO | 3 | 0.76 | 1.01 | 0.32 |
| DeepSeek-R1-Distill-Qwen-1.5B | GRPO | 3 | 1.60 | 4.75 | 2.44 |
| DeepSeek-R1-Distill-Qwen-1.5B | DR.GRPO | 3 | 1.31 | 0.07 | 0.28 |
| Qwen2.5-Math-7B | GRPO | 1 | 0.00 | 1.26 | 0.95 |
| Qwen2.5-Math-7B | DR.GRPO | 1 | 2.80 | 0.00 | 0.00 |
| Mean across six settings | — | — | 1.38 | 1.27 | 0.67 |
| Best checkpoint = final (runs) | — | 14 | 3/14 | 6/14 | 9/14 |
Thank you for the additional final-checkpoint results and training seeds, which address my main concern about test-based checkpoint selection; although the gains over logits-SAM vary across settings and the theoretical connection remains limited, I am raising my score to 3.
We thank the reviewer for the detailed and constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns about checkpoint selection, evaluation robustness, the theoretical connection, and hyperparameter tuning. All revisions are highlighted in blue in the updated manuscript. We address each concern below.
1. Final-checkpoint robustness and additional training seeds.
Our best-checkpoint reporting follows a common evaluation practice in RLVR/GRPO research, and the same checkpoint-selection protocol is applied consistently to all methods, rather than favoring KL-SAM. Our original submission already reported three-seed final-checkpoint results for Qwen2.5-Math-1.5B (Table 6). We have now extended DeepSeek-R1-Distill-Qwen-1.5B from one to three training seeds under both GRPO and DR.GRPO, and added final-checkpoint evaluations for DeepSeek-1.5B and Qwen2.5-Math-7B (Tables 9 and 11). Thus, final-checkpoint results are now available for all six model–objective settings, without test-based checkpoint selection.
The improvement over logits-SAM persists and becomes larger under final-checkpoint reporting: averaged across six settings, KL-SAM improves Avg. pass@1 by +1.63 percentage points over logits-SAM and +2.96 points over the baseline, compared with +1.03 and +2.25 points under best-checkpoint reporting. Crucially, Table 13 reports final-checkpoint seed-matched differences (mean ± SD) over three seeds. For KL-SAM minus logits-SAM, these are +0.70 ± 0.43 / +1.67 ± 0.60 points on Qwen2.5-Math-1.5B and +5.27 ± 1.78 / +0.86 ± 1.61 points on DeepSeek-1.5B, under GRPO/DR.GRPO respectively. The 7B experiments remain single-seed due to computational constraints.
2. Curvature estimator and connection to the implemented local KL.
We have clarified the measurement protocol in Table 1. The directional Hessian and Fisher quantities were estimated by finite differences of the GRPO loss and policy KL in the full parameter space, using a perturbation radius of 10−4 and rollout-sampled contexts.
To directly test the implemented regularizer, we additionally measured output-head local KL along the gradient perturbation direction across the same 20 runs, using a fixed perturbation radius of ρ=10−3 for all seeds. Its Pearson/Spearman correlations with average test pass@1 are -0.476/-0.394, and with MATH500 pass@1 are -0.456/-0.344 (Section 6.4). These results demonstrate that the local-KL quantity used by KL-SAM also exhibits a negative association with test performance.
Regarding the theoretical connection, Theorem 4.1 provides an information-theoretic generalization motivation involving selector-swap parameter differences, which require counterfactual training and are not directly available during optimization. We do not assume that these directions align with the SAM gradient perturbation. Instead, Proposition 5.1 and Theorem 5.2 characterize the implemented shared-perturbation mechanism and its directional Hessian–Fisher regularization. We have revised Section 5 to make this distinction explicit.
3. Learning-rate and perturbation-radius tuning.
GRPO training is sensitive to the learning rate, which can substantially affect optimization dynamics and generalization performance. To ensure a controlled comparison and isolate the contribution of each regularization method, we use the same baseline-selected learning rate for all methods within each objective, rather than introducing differences due to method-specific learning-rate choices. Meanwhile, we independently tune the perturbation radius for each SAM variant using the same validation grid (Table 4), rather than forcing logits-SAM to use the radius selected for KL-SAM, thereby ensuring a fair comparison with respect to perturbation-radius selection.
4. Selected checkpoints and best-to-final gaps.
We now provide selected checkpoint steps for each method and seed in Tables 7, 10, and 12, together with best-to-final performance gaps in Table 14. Yes, the identical Qwen2.5-Math-1.5B GRPO mean for KL-SAM in Tables 2 and 6 is because the final checkpoint was selected for all three seeds, as explicitly shown in Table 7.
Across six settings, KL-SAM has the smallest mean best-to-final gap: 0.67 points, compared with 1.27 for logits-SAM and 1.38 for the baseline. Its final checkpoint is selected in 9/14 runs, compared with 6/14 for logits-SAM and 3/14 for the baseline. Together with the expanded final-checkpoint and paired-seed evaluations, these results directly address the concern that KL-SAM's reported gains depend on test-based checkpoint selection.