acceptodds
Under review as a conference paper at ICLR 2027

Dual-Curvature Implicit Regularization for GRPO via Sharpness-Aware Minimization

Abstract

Group Relative Policy Optimization (GRPO) has become a widely used algorithm for reinforcement learning with verifiable rewards (RLVR). Common training recipes adopt small learning rates and extended training schedules, while increasing the learning rate can degrade test performance. To investigate which training quantities are associated with generalization performance, we conduct a controlled 20-seed study of GRPO under a fixed training configuration. The strongest associations with test accuracy are observed among the directional Hessian and Fisher curvature metrics, whereas training reward and gradient norm show weak correlations. Motivated by these findings, we propose KL-SAM, a fully first-order optimization algorithm that combines Sharpness-Aware Minimization (SAM)-style perturbations with a local policy-KL regularizer to target objective and policy sensitivity through a single shared perturbation. A local expansion reveals directional Hessian and Fisher contributions, providing an interpretation as implicit dual-curvature regularization. Experiments on models ranging from 1.5B to 7B parameters across multiple objectives and benchmarks demonstrate the effectiveness of our approach. We will release our code upon publication.

Primary area:
optimization
Keywords:
Optimization, Sharpness-Aware Minimization, Curvature Regularization, Large Language Models, Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GkxBOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper conducts a controlled 20-seed study of GRPO on Qwen2.5-Math-1.5B trained for one epoch on MATH. At the final checkpoints, directional Hessian and Fisher curvature show negative associations with test pass@1, while training reward and gradient norm show weak correlations. Reference KL has positive Spearman but near-zero Pearson correlations. Motivated by this, it proposes KL-SAM: a SAM-style perturbed GRPO loss plus a penalty on the KL between the clean policy and the policy at the perturbed point, both using the same first-order perturbation epsilon = rho*g/||g||. A second-order expansion shows the two terms contribute a directional Hessian and a directional Fisher term along epsilon (Theorem 5.2), and an informal CMI-based bound relates trajectory-Fisher curvature to the generalization gap. For efficiency the perturbation is applied at the output layer following logits-SAM. On Qwen2.5-Math-1.5B/7B and DeepSeek-R1-Distill-Qwen-1.5B with GRPO and DR.GRPO, KL-SAM improves average pass@1 over five math benchmarks by about 2.1 points over the baseline and 0.9 points over logits-SAM, with ~4% step-time overhead.

Critical strengths
  1. The controlled 20-seed study is useful evidence on variation across GRPO runs under a fixed configuration. Comparing directional curvature with training reward, gradient norm, and reference KL helps distinguish their associations with test performance. The method is simple, its reported step-time overhead is modest, and the writing is clear.
Critical weaknesses

The main evaluation selects checkpoints using the test benchmarks themselves. This weakens the evidence for generalization, especially given the modest average gain over logits-SAM. Table 6 partly addresses this: final-checkpoint results strengthen KL-SAM’s lead to 0.70/1.67 points under GRPO/DR.GRPO, but only for Qwen2.5-Math-1.5B. The other four settings are single runs. Final or validation-selected checkpoints, additional training seeds, and uncertainty estimates for the paired differences are needed to establish robustness. This evaluation issue is my main reason for leaning toward rejection.

The theoretical link to the implemented algorithm remains incomplete. Theorem 5.2 gives a local Hessian–Fisher interpretation, but Theorem 4.1 concerns parameter differences between policies trained on datasets differing in one example. KL-SAM instead penalizes sensitivity along an output-head gradient perturbation. The paper does not establish that reducing this quantity controls the directions appearing in the generalization bound. Clarifying the parameter subspace and estimator used in Table 1, and testing whether the implemented local-KL penalty predicts performance across the 20 seeds, would strengthen the connection.

Questions for authors
  1. Can you report final or validation-selected checkpoint results with multiple training seeds for the remaining model–objective settings? Does the average gain over logits-SAM persist?

  2. What parameter subspace and estimator were used for the Fisher measurements in Table 1? At a fixed perturbation radius, does the implemented output-head local-KL penalty also correlate with test pass@1 across the 20 runs?

  3. All methods reuse the baseline-selected learning rate. Would independently tuning logits-SAM’s learning rate and perturbation radius under a comparable validation-search budget change the comparison?

  4. Please report the selected checkpoint steps and best-versus-final gaps for each method and seed. Does the identical KL-SAM GRPO mean in Tables 2 and 6 reflect the final checkpoint being selected in every seed?

3 comments

We thank the reviewer for the detailed and constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns about checkpoint selection, evaluation robustness, the theoretical connection, and hyperparameter tuning. All revisions are highlighted in blue in the updated manuscript. We address each concern below.

1. Final-checkpoint robustness and additional training seeds.

Our best-checkpoint reporting follows a common evaluation practice in RLVR/GRPO research, and the same checkpoint-selection protocol is applied consistently to all methods, rather than favoring KL-SAM. Our original submission already reported three-seed final-checkpoint results for Qwen2.5-Math-1.5B (Table 6). We have now extended DeepSeek-R1-Distill-Qwen-1.5B from one to three training seeds under both GRPO and DR.GRPO, and added final-checkpoint evaluations for DeepSeek-1.5B and Qwen2.5-Math-7B (Tables 9 and 11). Thus, final-checkpoint results are now available for all six model–objective settings, without test-based checkpoint selection.

The improvement over logits-SAM persists and becomes larger under final-checkpoint reporting: averaged across six settings, KL-SAM improves Avg. pass@1 by +1.63 percentage points over logits-SAM and +2.96 points over the baseline, compared with +1.03 and +2.25 points under best-checkpoint reporting. Crucially, Table 13 reports final-checkpoint seed-matched differences (mean ± SD) over three seeds. For KL-SAM minus logits-SAM, these are +0.70 ± 0.43 / +1.67 ± 0.60 points on Qwen2.5-Math-1.5B and +5.27 ± 1.78 / +0.86 ± 1.61 points on DeepSeek-1.5B, under GRPO/DR.GRPO respectively. The 7B experiments remain single-seed due to computational constraints.

2. Curvature estimator and connection to the implemented local KL.

We have clarified the measurement protocol in Table 1. The directional Hessian and Fisher quantities were estimated by finite differences of the GRPO loss and policy KL in the full parameter space, using a perturbation radius of 10−410^{-4} and rollout-sampled contexts.

To directly test the implemented regularizer, we additionally measured output-head local KL along the gradient perturbation direction across the same 20 runs, using a fixed perturbation radius of ρ=10−3\rho=10^{-3} for all seeds. Its Pearson/Spearman correlations with average test pass@1 are -0.476/-0.394, and with MATH500 pass@1 are -0.456/-0.344 (Section 6.4). These results demonstrate that the local-KL quantity used by KL-SAM also exhibits a negative association with test performance.

Regarding the theoretical connection, Theorem 4.1 provides an information-theoretic generalization motivation involving selector-swap parameter differences, which require counterfactual training and are not directly available during optimization. We do not assume that these directions align with the SAM gradient perturbation. Instead, Proposition 5.1 and Theorem 5.2 characterize the implemented shared-perturbation mechanism and its directional Hessian–Fisher regularization. We have revised Section 5 to make this distinction explicit.

3. Learning-rate and perturbation-radius tuning.

GRPO training is sensitive to the learning rate, which can substantially affect optimization dynamics and generalization performance. To ensure a controlled comparison and isolate the contribution of each regularization method, we use the same baseline-selected learning rate for all methods within each objective, rather than introducing differences due to method-specific learning-rate choices. Meanwhile, we independently tune the perturbation radius for each SAM variant using the same validation grid (Table 4), rather than forcing logits-SAM to use the radius selected for KL-SAM, thereby ensuring a fair comparison with respect to perturbation-radius selection.

4. Selected checkpoints and best-to-final gaps.

We now provide selected checkpoint steps for each method and seed in Tables 7, 10, and 12, together with best-to-final performance gaps in Table 14. Yes, the identical Qwen2.5-Math-1.5B GRPO mean for KL-SAM in Tables 2 and 6 is because the final checkpoint was selected for all three seeds, as explicitly shown in Table 7.

Across six settings, KL-SAM has the smallest mean best-to-final gap: 0.67 points, compared with 1.27 for logits-SAM and 1.38 for the baseline. Its final checkpoint is selected in 9/14 runs, compared with 6/14 for logits-SAM and 3/14 for the baseline. Together with the expanded final-checkpoint and paired-seed evaluations, these results directly address the concern that KL-SAM's reported gains depend on test-based checkpoint selection.

AuthorsOpenReview ↗

Supplementary Experimental Results

Table 7 (Appendix D): Selected checkpoints for Qwen2.5-Math-1.5B over three seeds. Numbers indicate training steps; Final denotes the end of training.

ObjectiveMethodSeed 42Seed 0Seed 1
GRPOBaseline200200400
GRPOlogits-SAMFinal300400
GRPOKL-SAMFinalFinalFinal
GRPOKL-only200400Final
DR.GRPOBaseline300400Final
DR.GRPOlogits-SAM400Final300
DR.GRPOKL-SAMFinal400Final
DR.GRPOKL-only300300Final

Table 8 (Appendix D; Avg. only): Best-checkpoint pass@1 (%) for DeepSeek-R1-Distill-Qwen-1.5B over three seeds (mean ± SD).

ObjectiveMethodAvg. pass@1 (%)
GRPOBaseline41.97 ± 1.67
GRPOlogits-SAM41.61 ± 2.14
GRPOKL-SAM44.57 ± 1.80
DR.GRPOBaseline41.46 ± 1.36
DR.GRPOlogits-SAM43.45 ± 1.17
DR.GRPOKL-SAM44.52 ± 0.79

Table 9 (Appendix D; Avg. only): Final-checkpoint pass@1 (%) for DeepSeek-R1-Distill-Qwen-1.5B over three seeds (mean ± SD).

ObjectiveMethodAvg. pass@1 (%)
GRPOBaseline40.37 ± 3.54
GRPOlogits-SAM36.86 ± 2.81
GRPOKL-SAM42.13 ± 1.97
DR.GRPOBaseline40.15 ± 2.58
DR.GRPOlogits-SAM43.38 ± 1.26
DR.GRPOKL-SAM44.24 ± 0.74

Table 11 (Appendix D; Avg. only): Final-checkpoint pass@1 (%) for Qwen2.5-Math-7B (one seed).

ObjectiveMethodAvg. pass@1 (%)
GRPOBaseline42.70
GRPOlogits-SAM45.05
GRPOKL-SAM44.97
DR.GRPOBaseline41.27
DR.GRPOlogits-SAM44.62
DR.GRPOKL-SAM45.97

Table 13 (Appendix D): Paired differences in final-checkpoint Avg. pass@1 (percentage points; mean ± SD over three matched training seeds).

ModelComparisonGRPODR.GRPO
Qwen2.5-Math-1.5BKL-SAM − Baseline+3.64 ± 1.70+1.28 ± 2.04
Qwen2.5-Math-1.5BKL-SAM − logits-SAM+0.70 ± 0.43+1.67 ± 0.60
Qwen2.5-Math-1.5BKL-SAM − KL-only+1.90 ± 1.32+3.16 ± 3.80
DeepSeek-R1-Distill-Qwen-1.5BKL-SAM − Baseline+1.76 ± 3.65+4.09 ± 1.94
DeepSeek-R1-Distill-Qwen-1.5BKL-SAM − logits-SAM+5.27 ± 1.78+0.86 ± 1.61

Table 14 (Appendix D): Best–final checkpoint gap in Avg. pass@1 (percentage points; lower is better).

ModelObjectiveSeedsBaselinelogits-SAMKL-SAM
Qwen2.5-Math-1.5BGRPO31.790.520.00
Qwen2.5-Math-1.5BDR.GRPO30.761.010.32
DeepSeek-R1-Distill-Qwen-1.5BGRPO31.604.752.44
DeepSeek-R1-Distill-Qwen-1.5BDR.GRPO31.310.070.28
Qwen2.5-Math-7BGRPO10.001.260.95
Qwen2.5-Math-7BDR.GRPO12.800.000.00
Mean across six settings——1.381.270.67
Best checkpoint = final (runs)—143/146/149/14

Thank you for the additional final-checkpoint results and training seeds, which address my main concern about test-based checkpoint selection; although the gains over logits-SAM vary across settings and the theoretical connection remains limited, I am raising my score to 3.

Reviewer_2keXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is motivated by the generalization problem at larger learning rates within limited training budgets. The author first conduct a diagostic analysis using a fixed training setup, and show that the LLM is showing a better generalization performance on the test set when the policy is at a parameter point of lower local curvature quantities. Given these empirical observations, a new regularization mechanism is proposed to futher include SAM in the reference regularization term. The empirical experiments demonstrate that this new regularization, KL-SAM, is effective paired up with both GRPO and DR.GRPO, with some useful ablation studies comparing KL-SAM against CAPO and logits-SAM.

Critical strengths
  1. This paper is well-written in a smooth narrative throughout, making it easy to read.
Critical weaknesses
  1. The motivation behind this paper is weak. If the underlying problem solved is the stability issue given large learning rate for GRPO training, I am wondering whether this work presents enough contribution.
  2. The proposed method is lack of novelty. It is presented as a naive combination of SAM and GRPO.
  3. This comparison between KL-SAM and previous works require explicit discussions. Moreover, while the empirical comparison against logits-SAM is included, this prior work [1] has already demonstrated SAM can effectively enhance alignment results in LLM alignment. This highlights a need for the comparison analysis between DPO+SAM, and GRPO+SAM.

[1] Luo, Haocheng, et al. "Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization." arXiv preprint arXiv:2603.18258 (2026).

Questions for authors
  1. As DPO inherits the reference KL regularization from standard RLHF, how is DPO+SAM different from your approach?
  2. When you do the diagnostic experiments, have you tried tuning the clip ratios to solve the identified instability issue?
  3. While the claimed contribution is enabling generalization at larger learning rates within limited training budgets, can you demonstrate KL-SAM is effectively improving the trade-off between limited training budgets and alignment results?
2 comments

We thank the reviewer for the constructive feedback. We have revised the manuscript in response to the reviewer's comments, with all revisions highlighted in blue. We would like to clarify the motivation and contributions of KL-SAM, particularly its distinction from DPO+SAM and its relevance to generalization under limited GRPO training budgets. We address each concern below.

1. Motivation: beyond training stability at large learning rates.

Our primary objective is improving generalization to unseen reasoning problems within a limited GRPO training budget, rather than merely preventing optimization instability. As discussed in Section 4, higher training reward does not necessarily translate into better test performance. To investigate this gap, we conduct a controlled 20-seed GRPO study and find that directional Hessian and Fisher curvature exhibit stronger negative associations with test performance than conventional training metrics such as reward and gradient norm (Table 1). These GRPO-specific findings motivate regularizing both objective and policy sensitivity. Figure 1 further illustrates that the baseline and KL-SAM can achieve comparable training rewards while exhibiting different test-performance trajectories.

2. Novelty and distinction from DPO+SAM.

We recognize that logits-SAM [1] has already demonstrated the effectiveness of SAM for DPO-based LLM alignment. However, logits-SAM was originally developed to address likelihood displacement in DPO, whereas KL-SAM is motivated by our controlled 20-seed investigation of generalization in GRPO. Our contribution is not merely transferring SAM from DPO to GRPO: motivated by these GRPO-specific curvature findings, we introduce a complementary local policy-KL regularizer to address both objective and policy sensitivity.

To clarify the relationship between these methods, DPO+logits-SAM [1] applies output-head SAM perturbations to the DPO preference objective, whereas GRPO+logits-SAM applies the same perturbation strategy to the GRPO objective. KL-SAM goes beyond both by adding a clean-to-perturbed policy KL term, DKL(πθ∥πθ+ϵ)D_{\mathrm{KL}}(\pi_\theta \| \pi_{\theta+\epsilon}), along the shared SAM perturbation. Unlike the reference-policy dependence in DPO or the reference KL commonly used in RLHF, this local KL directly regularizes policy sensitivity to parameter perturbations. Proposition 5.1 and Theorem 5.2 characterize its additional directional Fisher contribution alongside SAM's directional Hessian contribution.

Crucially, we evaluate logits-SAM as a separate comparison method alongside the unregularized GRPO/DR.GRPO baseline. We apply the perturbation strategy of [1] under the same GRPO/DR.GRPO training objectives, and setting the local-KL coefficient in KL-SAM to zero recovers this logits-SAM variant. Thus, rather than relying on cross-task comparisons between DPO and GRPO, we directly isolate the contribution of our proposed regularizer under matched training objectives. The component ablations in Table 2 show that KL-SAM outperforms both logits-SAM and KL-only under GRPO and DR.GRPO. Across six model–objective settings, KL-SAM improves average pass@1 by +1.03 percentage points over logits-SAM and +2.25 points over the unregularized baseline under best-checkpoint reporting. At the final checkpoint, these gains increase to +1.63 and +2.96 points, respectively.

3. Clip-ratio tuning.

We did not conduct a separate clip-ratio sweep. However, CAPO [2] investigates this issue and finds that tighter clipping can alleviate policy collapse at the cost of reduced accuracy (Appendix I, Figure 7). Our focus is instead on improving generalization through local objective and policy sensitivity regularization, rather than solely addressing training instability through clipping. We therefore keep the clipping configuration fixed across methods.

4. Generalization under limited training budgets.

Figure 1 directly examines the relationship between training progress and test performance. On Qwen2.5-Math-1.5B under GRPO, KL-SAM reaches approximately 40.0% average test pass@1 within one epoch (469 steps), exceeding the 37.7% final performance of a baseline trained for 2,000 steps with a smaller learning rate of 10−610^{-6}. Meanwhile, the training-reward curves indicate that the gain is not simply explained by higher training reward. KL-SAM also incurs only 4.0% additional step time and 2.3% peak memory usage relative to the unregularized baseline (Section 6.4). These results support the practical value of KL-SAM for improving generalization within limited training budgets.

[2] Melo, Luckeciano C., Alessandro Abate, and Yarin Gal. "Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning." International Conference on Learning Representations (ICLR), 2026.

Thank you for the detailed response and clarifications. I appreciate the explanation of the additional local policy-KL term and the matched comparisons with logits-SAM.

However, my main concerns about novelty and the significance of the contribution remain. As acknowledged in the rebuttal, logits-SAM has already demonstrated the benefits of SAM for LLM alignment. A different motivation and the GRPO setting do not, by themselves, resolve this overlap.

I remain unconvinced that the proposed extension constitutes a sufficiently substantial advance over prior work. I therefore maintain my score of 2 weak rejection.

Reviewer_GxFfOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KL-SAM, KL-SAM combines logits-SAM with local policy-KL regularization to improve GRPO generalization. Experiments show moderate gains with limited computational overhead, but the theoretical and empirical contributions remain insufficiently established.

Critical strengths

The curvature–generalization correlation study provides useful empirical insights.

The method is simple, computationally efficient, and improves performance across several settings.

Critical weaknesses

KL-SAM mainly adds a local KL penalty to existing logits-SAM. The dual-curvature interpretation follows from standard Taylor expansions. While the paper provides empirical correlations and theoretical motivation for considering both Hessian and Fisher curvature, these observations do not establish why their joint regularization is necessary or why local KL along the SAM perturbation should improve generalization beyond logits-SAM. The Taylor expansion characterizes the objective but offers limited insight into the mechanism underlying the additional gains.

The generalization bound concerns dataset-induced parameter changes, while the algorithm regularizes gradient-direction curvature. Their connection is not established.

Improvements over logits-SAM are modest (0.91 points on average), with insufficient multi-seed evaluations and unmatched perturbation radii.

Selecting checkpoints based on test benchmark performance introduces potential selection bias.

Questions for authors

Refer weakness

4 comments

We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns about the theoretical interpretation, the incremental benefit over logits-SAM, evaluation robustness, and hyperparameter fairness. Revisions are highlighted in blue in the updated manuscript.

1. Motivation and contribution of dual-curvature regularization.

While logits-SAM was originally developed to address likelihood displacement in DPO, KL-SAM is motivated by a distinct empirical investigation of generalization in GRPO, rather than simply extending SAM to another training objective. Our motivation comes from the controlled 20-seed GRPO study (Table 1), where directional Hessian and Fisher curvature exhibit stronger negative correlations with test performance than conventional metrics such as training reward and gradient norm. These findings motivate jointly regularizing objective and policy sensitivity. While the Taylor expansions themselves are standard, our contribution lies in the shared-perturbation construction characterized by Proposition 5.1 and Theorem 5.2: the local KL term preserves the first-order SAM perturbation while introducing directional Fisher regularization alongside SAM's directional Hessian contribution, without a separate perturbation search or explicit curvature estimation.

Diagnostics and ablations support the benefit of local KL. Our new 20-seed analysis shows that the implemented output-head local KL, measured along the SAM perturbation at a fixed radius of ρ=10−3\rho=10^{-3}, is negatively correlated with test pass@1 (Pearson/Spearman: -0.476/-0.394, Section 6.4). Furthermore, KL-SAM outperforms both logits-SAM and KL-only under GRPO and DR.GRPO (Table 2), supporting the complementary contribution of local policy-KL regularization beyond SAM alone. Figure 2 provides additional evidence of the intended curvature-regularization effect.

2. Connection between the generalization bound and the algorithm.

Theorem 4.1 establishes an information-theoretic generalization bound involving Fisher curvature along selector-swap parameter differences under the supersample construction. These directions require counterfactual training and are unavailable during optimization. We therefore adopt the loss-gradient direction as a computationally tractable direction for regularizing local sensitivity, motivated by the empirical correlations in Table 1 and the shared-perturbation construction. Although this direction is not guaranteed to align with the selector-swap directions, Theorem 4.1 provides a generalization-oriented motivation, while Proposition 5.1 and Theorem 5.2 characterize the local regularization implemented by KL-SAM. We have revised the corresponding discussion in Section 5 to make this distinction explicit.

3. Improvements over logits-SAM, multi-seed evaluation, and perturbation-radius fairness.

We have strengthened the empirical evaluation by extending DeepSeek-R1-Distill-Qwen-1.5B from one to three training seeds under both GRPO and DR.GRPO. Under best-checkpoint reporting, KL-SAM achieves 44.57 ± 1.80 vs. 41.61 ± 2.14 for logits-SAM under GRPO, and 44.52 ± 0.79 vs. 43.45 ± 1.17 under DR.GRPO (Table 8). With these additional runs, KL-SAM achieves an average improvement of +1.03 percentage points over logits-SAM and +2.25 points over the baseline across six model–objective settings.

Regarding perturbation radii, to ensure a fair comparison, we independently tuned ρ\rho for logits-SAM rather than forcing it to use the radius selected for KL-SAM. Both methods were tuned over the same search grid, ρ∈{10−3,5×10−3,10−2}\rho \in \{10^{-3}, 5\times10^{-3}, 10^{-2}\}, using a validation subset disjoint from MATH500. This allows logits-SAM to use its own preferred radius rather than a potentially suboptimal configuration inherited from KL-SAM (Table 4).

4. Test-based checkpoint selection.

Our best-checkpoint reporting follows a common evaluation practice in RLVR/GRPO research. Importantly, the same checkpoint-selection protocol was applied consistently to all methods, rather than favoring KL-SAM. Moreover, our original submission already included three-seed final-checkpoint results for Qwen2.5-Math-1.5B (Table 6). We have now added final-checkpoint evaluations for DeepSeek-R1-Distill-Qwen-1.5B and Qwen2.5-Math-7B under both GRPO and DR.GRPO (Tables 9 and 11), covering all six model–objective settings.

Importantly, without test-based checkpoint selection, KL-SAM achieves larger average gains of +1.63 percentage points over logits-SAM and +2.96 points over the baseline, compared with +1.03 and +2.25 points under best-checkpoint reporting. We additionally report selected checkpoint steps (Tables 7, 10, and 12) and best-to-final performance gaps (Table 14). These results demonstrate that KL-SAM's improvements are not an artifact of test-based checkpoint selection.

AuthorsOpenReview ↗

Supplementary Experimental Results

Table 8 (Appendix D; Avg. only): Best-checkpoint pass@1 (%) for DeepSeek-R1-Distill-Qwen-1.5B over three seeds (mean ± SD).

ObjectiveMethodAvg. pass@1 (%)
GRPOBaseline41.97 ± 1.67
GRPOlogits-SAM41.61 ± 2.14
GRPOKL-SAM44.57 ± 1.80
DR.GRPOBaseline41.46 ± 1.36
DR.GRPOlogits-SAM43.45 ± 1.17
DR.GRPOKL-SAM44.52 ± 0.79

Table 9 (Appendix D; Avg. only): Final-checkpoint pass@1 (%) for DeepSeek-R1-Distill-Qwen-1.5B over three seeds (mean ± SD).

ObjectiveMethodAvg. pass@1 (%)
GRPOBaseline40.37 ± 3.54
GRPOlogits-SAM36.86 ± 2.81
GRPOKL-SAM42.13 ± 1.97
DR.GRPOBaseline40.15 ± 2.58
DR.GRPOlogits-SAM43.38 ± 1.26
DR.GRPOKL-SAM44.24 ± 0.74

Table 11 (Appendix D; Avg. only): Final-checkpoint pass@1 (%) for Qwen2.5-Math-7B (one seed).

ObjectiveMethodAvg. pass@1 (%)
GRPOBaseline42.70
GRPOlogits-SAM45.05
GRPOKL-SAM44.97
DR.GRPOBaseline41.27
DR.GRPOlogits-SAM44.62
DR.GRPOKL-SAM45.97

Dear reviewer, the discussion period is about to close. Please read the authors' response and let us know briefly which of your concerns remain. If you have any further questions for the authors, please post them soon so the authors have time to respond.

Thank you for the additional experiments and clarification. Nevertheless, I remain concerned about the limited methodological novelty and the insufficient connection between the analysis and the proposed algorithm, so I prefer to maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We thank all reviewers for their constructive feedback. In response to the concerns raised during the review process, we have revised the manuscript and conducted additional experiments. All revisions are highlighted in blue in the updated manuscript.

1. Expanded evaluation and robustness.

We extend the DeepSeek-R1-Distill-Qwen-1.5B experiments from one to three training seeds under both GRPO and DR.GRPO (Table 3, with detailed statistics in Table 8). We also add final-checkpoint evaluations for DeepSeek-1.5B and Qwen2.5-Math-7B (Tables 9 and 11), complementing the existing Qwen2.5-Math-1.5B results (Table 6). In addition, we report seed-matched performance differences with standard deviations (Table 13), selected checkpoint steps (Tables 7, 10, and 12), and best-to-final checkpoint gaps (Table 14).

2. Additional diagnostics and clarifications.

We add a 20-seed analysis of the implemented output-head local KL and its correlation with test performance (Section 6.4). We also clarify the parameter subspace and finite-difference estimators used in the curvature diagnostics (Section 4.2), the theoretical scope of the generalization analysis and its connection to the implemented algorithm (Section 5).

We acknowledge that the Qwen2.5-Math-7B experiments remain single-seed due to computational constraints. Detailed responses to reviewer-specific concerns are provided in the corresponding discussion threads.

We thank the reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.