acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Gains and Capability Stability in Short-Horizon GRPO: A Paired KL Audit

Abstract

Does a reference-policy KL penalty protect capabilities during reasoning reinforcement learning, once target-task learning and training exposure are measured together? We compare six 300-update LoRA runs of Qwen2.5-1.5B-Instruct, pairing three seeds across KL coefficients zero and 0.02. Greedy accuracy on 300 held-out GSM8K questions rises from 58.7% to 68.7–72.3%. Meanwhile, likelihood accuracy remains at 77.7–78.2% on 600 BoolQ questions (77.8% initially) and 73.2–74.6% on 299 ARC-Challenge questions (74.6% initially). The paired KL effects on final BoolQ change are , , and percentage points. Thus this coefficient has no consistent protective effect across the three seeds. Item-level predictions distinguish aggregate stability from preservation of individual decisions: two KL runs finish at the initial BoolQ accuracy despite four or five gains canceling the same number of losses. An exact enumeration of compatible pairings shows how much statistical evidence is lost when only accuracy totals are retained. We also specify why a penalty measured on arithmetic responses does not bound changes on another prompt distribution. The evidence supports substantial arithmetic learning with small changes on two likelihood tasks in this short-horizon setting, while leaving broader preservation and stronger KL interventions unresolved.

Primary area:
reinforcement learning
Keywords:
reasoning reinforcement learning, GRPO, capability retention, KL regularization, paired evaluation, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WY72OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper audits whether a reference-policy KL penalty preserves non-target capabilities during short-horizon reasoning RL. It compares paired Qwen2.5-1.5B LoRA runs with β=0\beta=0 and $\beta=0.02$ finding substantial GSM8K improvement but little change on BoolQ and ARC-Challenge, with no consistent retention benefit from KL. The paper also argues that aggregate accuracy can hide item-level gains and losses, and that KL measured on the training distribution does not guarantee preservation on unrelated evaluation distributions.

Critical strengths
  1. The paired-seed design is clean and the paper is careful not to overclaim from three training seeds.
  2. The distinction between aggregate score stability and item-level prediction stability is useful and clearly presented.
  3. The paper is unusually explicit about the limitations of its KL estimator, evaluation protocols, and scope of the conclusions.
Critical weaknesses
  1. Limited novelty. The main conclusions are largely expected: task-distribution KL does not guarantee cross-task preservation, and marginal accuracies do not determine paired prediction changes. The paper offers neither a new algorithm nor a particularly surprising empirical finding.
  2. Weak KL intervention. Only β=0\beta=0 and β=0.02\beta=0.02 are tested, with little justification, and the paper itself reports that the two conditions do not produce consistently separated divergence diagnostics. This makes the null KL result difficult to interpret.
  3. Very limited empirical regime. A single 1.5B model, q/v-only rank-8 LoRA, 300 updates, GSM8K training, and only BoolQ/ARC retention evaluation are too narrow to support a meaningful conclusion about capability retention in modern reasoning RL.
  4. Several design choices, including the model scale, LoRA setup, KL coefficient, training horizon, and use of partial GSM8K/BoolQ evaluation subsets, are insufficiently motivated.
Questions for authors
  1. Why was β=0.02\beta=0.02 chosen as the only nonzero KL coefficient, especially given that it does not produce consistent separation in the reported divergence diagnostic?
  2. Why should this short, low-rank 1.5B adaptation regime be expected to exhibit enough capability drift for a negative retention result to be informative?
  3. How were the 300 GSM8K and 600 BoolQ evaluation examples selected, and why were the full evaluation splits not used?
2 comments

Thank you for the questions. We expanded the coefficient comparison and measured probability changes on the fixed evaluation questions.

Larger KL coefficients

The original 0.02 coefficient was a fixed setting. We added 0.1 and 0.2, five and ten times that value, with three paired seeds each. All six new runs completed 300 updates using the original training code, data, and LoRA configuration.

Mean final accuracy across three seeds:

KL coefficientGSM8KBoolQARC-Challenge
Initial model58.67%77.83%74.58%
070.11%77.83%74.02%
0.0270.44%77.78%74.14%
0.170.22%77.67%73.91%
0.268.56%77.67%74.58%

At 0.2, mean GSM8K accuracy is 1.56 percentage points below no KL, although still 9.89 points above initialization. All three ARC runs match the initial accuracy; BoolQ remains slightly below baseline.

Measured non-target drift

We normalize the saved answer-option log scores with a softmax and compare initialization with update 300 on identical questions. Relative to no KL, coefficient 0.2 reduces mean option TV from 0.01039 to 0.00838 on BoolQ and from 0.01262 to 0.01148 on ARC, though not in every seed.

The unchanged ARC scores also contain answer turnover. Each 0.2 run finishes at the initial 223/299, but the three runs lose 3, 2, and 3 previously correct answers, respectively, offset by equal gains. The short adaptation therefore produces substantial arithmetic learning and measurable non-target changes even when aggregate accuracy is stable.

Evaluation selection

The 300 GSM8K test questions and 600 BoolQ validation questions are selected by sorting SHA-256 hashes of a fixed task-specific prefix plus the original row index. Selection uses neither model predictions nor training outcomes. ARC includes all 299 validation questions.

The same items are evaluated at updates 0, 50, 100, 200, and 300. Fixed subsets keep repeated evaluation tractable, particularly GSM8K generation, and preserve item-level pairing across runs. Selected IDs and data hashes are saved in the protocol.

In this 1.5B, rank-8 setting, coefficients 0.1 and 0.2 produce lower mean option-distribution drift than no KL, but do not improve retention accuracy on both tasks. At 0.2, ARC's aggregate score is preserved alongside lower arithmetic accuracy and offsetting changes in individual answers.

Thank you for the additional experiments and clarifications. The coefficient sweep and deterministic evaluation-subset selection address some of my original concerns.

However, my main concerns remain unresolved. The paper's central observations are largely established or expected from prior work on KL regularization and capability retention, and the additional experiments do not establish a substantially new scientific insight.

Although stronger KL coefficients reduce measured non-target distributional drift, they still do not demonstrate consistent retention benefits. More importantly, the conclusions remain confined to a single 1.5B model, a highly constrained LoRA adaptation regime, and a narrow set of retention tasks. The new results do not substantially strengthen the generality or significance of the findings.

I therefore maintain my original rating.

Reviewer_ZMpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits a short-horizon GRPO configuration using Qwen2.5-1.5B-Instruct with rank-8 LoRA. Three paired seeds compare reference-penalty coefficients 0 and 0.02 over 300 updates on GSM8K. The same held-out examples are evaluated at fixed checkpoints on GSM8K, BoolQ, and ARC-Challenge. Target accuracy rises by 10–13.7 percentage points, while the two non-target tasks show small net changes and no consistent benefit from the tested KL coefficient. Per-example transition tables distinguish gains and losses concealed by aggregate accuracy. The paper also enumerates the exact McNemar p-values compatible with aggregate counts and explains why controlling divergence on one prompt distribution does not by itself guarantee retention on another.

Critical strengths
  1. The experiment makes its evidence auditable. Matched seeds, unchanged evaluation examples, a fixed endpoint, exact counts, and per-example gains/losses make it possible to distinguish training variation from evaluation uncertainty. For example, the first unregularized GSM8K run improves by 30 correct answers through 52 gains and 22 losses. This is more informative than reporting only an accuracy increase, and the authors correctly keep the training seed as the replication unit.

  2. The interpretation is appropriately restrained. The paper separates absence of a consistent observed KL advantage from evidence of equivalence, and distinguishes sampled token penalties from a guarantee about another prompt distribution. These distinctions make the case study useful as a reproducible methodological example.

Critical weaknesses
  1. The substantive advance over existing retention studies and paired-evaluation methodology is limited. Chen et al., Retaining by Doing (ICML 2026, Section 4.1 and Figure 6), already compare GRPO with and without reference KL across model families and tasks, including Qwen2.5-1.5B, and find similar retention in most settings. The present work adds a more transparent paired audit of one short run regime, but the gain/loss decomposition and exact-binomial analysis do not yet produce a distinct conclusion about when KL protects capabilities. Paired contingency-table evaluation is also established in the NLP testing literature cited by the paper. The contribution would be stronger with a clear finding that this audit reveals beyond documenting the limitations of aggregate reporting. Chen et al.; Dror et al..

  2. The chosen intervention gives little leverage for understanding KL protection. Table 2 reports small sampled penalties in both arms, with the regularized value actually larger for one paired seed; these are also measured on each arm's own trajectories. Meanwhile, BoolQ has only 4–10 transitions and the non-target accuracy changes are small. Consequently, the comparison cannot distinguish a regime where reference KL is unnecessary from one where the tested coefficient produces too little effective separation to reveal protection. The paper acknowledges this limitation, so it does not invalidate the narrow result, but it substantially limits the new scientific insight supporting acceptance.

Questions for authors
  1. Relative to Chen et al.'s KL ablation, what specific empirical or methodological conclusion does the paired audit establish that was previously unresolved? In particular, is there a case where the actual transition information materially changes the scientific interpretation rather than only tightening the compatible p-value range?
  2. Do the saved checkpoints or existing diagnostics establish a meaningful divergence difference between the two arms on a common evaluation distribution? Such evidence would help distinguish an informative absence of retention benefit from a weak intervention. This asks about evidence available for the reported experiment; a broader coefficient or horizon sweep would be future work.
1 comment

Thank you for the questions. We analyzed the original checkpoints on a common evaluation distribution and completed six runs at larger KL coefficients.

Common-prompt divergence

For the same 600 BoolQ and 299 ARC questions, we normalize the saved answer-option log scores with a softmax and compute KL from the initial to the final option distribution. On ARC, coefficient 0.02 produces larger mean KL than no regularization in all three paired seeds:

Paired seedNo KL penaltyCoefficient 0.02
10.0031670.003266
20.0017270.002206
30.0021630.002222

The logged training-trajectory penalty was lower in the regularized arm for two of these seeds. On BoolQ, option KL decreases in two seeds and increases in one. A lower sampled training penalty therefore does not correspond to consistently lower drift on the monitored tasks.

Stronger coefficients

We added coefficients 0.1 and 0.2, each with three seeds and the same 300-update protocol. Mean final accuracy is:

CoefficientGSM8KBoolQARC
070.11%77.83%74.02%
0.0270.44%77.78%74.14%
0.170.22%77.67%73.91%
0.268.56%77.67%74.58%

At 0.2, mean initial-to-final option TV falls from 0.01039 to 0.00838 on BoolQ and from 0.01262 to 0.01148 on ARC relative to no KL. The decreases are not uniform across seeds. ARC accuracy returns to baseline in every run, while mean GSM8K accuracy is lower and BoolQ does not improve.

What the paired answers reveal

In two original regularized runs, BoolQ finishes at its initial 467/600 through five gains and five losses, and four gains and four losses. In the new 0.2 runs, ARC finishes at its initial 223/299 in all three seeds through 3/3, 2/2, and 3/3 gains/losses. These are cases where a zero net accuracy change includes losses of previously correct answers, not merely cases where pairing narrows a p-value range.

Chen et al.'s KL ablation in Retaining by Doing establishes that retention can occur without reference KL. Our additional results distinguish three observations within the same runs: the training penalty can decrease while non-target option KL increases; larger coefficients can reduce mean option drift without improving both retention scores; and an unchanged score can contain offsetting gains and losses. Together, they explain why aggregate retention scores and training-trajectory penalties alone do not determine whether the original predictions were preserved.

Reviewer_aAnhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether a reference-policy KL penalty helps preserve performance on non-target tasks during short-horizon reasoning reinforcement learning. Using LoRA on Qwen2.5-1.5B-Instruct, the paper compares KL penalty coefficients β=0\beta=0 and β=0.02\beta=0.02 across three paired random seeds. It jointly examines GSM8K learning gains, performance on BoolQ and ARC-Challenge, and item-level correctness changes, while analyzing the paired information lost when only aggregate accuracy counts are retained.

Critical strengths

The paper constructs paired experiments with identical initialization, training prompt schedules, and generation-seed schedules, while fixing the number of updates. This helps reduce interference from differences in training settings. Recording item-level gains and losses also has practical value, as it distinguishes “unchanged accuracy” from “preservation of previously correct decisions.” The conclusions explicitly distinguish training-seed uncertainty from evaluation-item uncertainty. Overall, the experimental setup, procedures, and conclusions are relatively rigorous.

Critical weaknesses
  1. Only β=0\beta=0 and β=0.02\beta=0.02 are considered in experiments. Therefore, the applicability of the conclusions requires further evaluation.

  2. This paper focuses on how reinforcement learning affects mathematical learning gains and performance on reading comprehension and science question answering. What would be on other task?

  3. The paper only evaluates the model’s performance on reading comprehension and science question answering, but lacks an analysis of why performance on these tasks remains largely stable. Quantitative analyses of changes in option scores and distributional divergence on these tasks would help distinguish actual output stability from probability changes obscured by the accuracy metric.

  4. The conclusions are difficult to generalize to models of different sizes and versions.

Questions for authors

I believe more experiments are needed to verify the conclusions, including models of varying scales, different types of problems, and different model families.

2 comments

Thank you for the suggestions. We completed six additional runs at KL coefficients 0.1 and 0.2 and measured changes in the saved answer-option scores.

Each coefficient uses the same three seeds, training schedule, and 300-update endpoint. Mean final accuracy is:

KL coefficientGSM8KBoolQARC-Challenge
Initial model58.67%77.83%74.58%
070.11%77.83%74.02%
0.0270.44%77.78%74.14%
0.170.22%77.67%73.91%
0.268.56%77.67%74.58%

At 0.2, all three runs match the initial ARC accuracy. Mean GSM8K accuracy is 1.56 percentage points below the no-KL result, while BoolQ does not improve.

To examine changes hidden by accuracy, we normalize each question's answer-option log scores with a softmax and compute total variation distance between initialization and update 300. All runs use the same 600 BoolQ and 299 ARC questions. Values below average over questions and then seeds.

KL coefficientBoolQ TVARC TV
00.010390.01262
0.020.009710.01305
0.10.009110.01220
0.20.008380.01148

Coefficients 0.1 and 0.2 reduce mean option-distribution drift relative to no KL, though not in every seed. At 0.02, mean drift decreases on BoolQ but increases on ARC.

Stable scores also conceal changed answers. At 0.2, the three ARC runs all finish at the initial 223/299, but lose 3, 2, and 3 previously correct answers, respectively. Equal numbers of gains cancel these losses.

At coefficients 0.1 and 0.2, mean option-distribution drift is lower than without KL, but accuracy retention does not improve on both tasks. Even exact preservation of the ARC score includes changes in which questions are answered correctly.

These results concern arithmetic training of Qwen2.5-1.5B. Reverse training directions and other model sizes are not evaluated here.

Thanks for your experiments and your effort! The authors have addressed some of my concerns, but they have not conducted experiments on larger LLMs, such as 3B or 7B models. Therefore, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.