- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper audits whether a reference-policy KL penalty preserves non-target capabilities during short-horizon reasoning RL. It compares paired Qwen2.5-1.5B LoRA runs with and $\beta=0.02$ finding substantial GSM8K improvement but little change on BoolQ and ARC-Challenge, with no consistent retention benefit from KL. The paper also argues that aggregate accuracy can hide item-level gains and losses, and that KL measured on the training distribution does not guarantee preservation on unrelated evaluation distributions.
Critical strengths
- The paired-seed design is clean and the paper is careful not to overclaim from three training seeds.
- The distinction between aggregate score stability and item-level prediction stability is useful and clearly presented.
- The paper is unusually explicit about the limitations of its KL estimator, evaluation protocols, and scope of the conclusions.
Critical weaknesses
- Limited novelty. The main conclusions are largely expected: task-distribution KL does not guarantee cross-task preservation, and marginal accuracies do not determine paired prediction changes. The paper offers neither a new algorithm nor a particularly surprising empirical finding.
- Weak KL intervention. Only and are tested, with little justification, and the paper itself reports that the two conditions do not produce consistently separated divergence diagnostics. This makes the null KL result difficult to interpret.
- Very limited empirical regime. A single 1.5B model, q/v-only rank-8 LoRA, 300 updates, GSM8K training, and only BoolQ/ARC retention evaluation are too narrow to support a meaningful conclusion about capability retention in modern reasoning RL.
- Several design choices, including the model scale, LoRA setup, KL coefficient, training horizon, and use of partial GSM8K/BoolQ evaluation subsets, are insufficiently motivated.
Questions for authors
- Why was chosen as the only nonzero KL coefficient, especially given that it does not produce consistent separation in the reported divergence diagnostic?
- Why should this short, low-rank 1.5B adaptation regime be expected to exhibit enough capability drift for a negative retention result to be informative?
- How were the 300 GSM8K and 600 BoolQ evaluation examples selected, and why were the full evaluation splits not used?
2 comments
Thank you for the additional experiments and clarifications. The coefficient sweep and deterministic evaluation-subset selection address some of my original concerns.
However, my main concerns remain unresolved. The paper's central observations are largely established or expected from prior work on KL regularization and capability retention, and the additional experiments do not establish a substantially new scientific insight.
Although stronger KL coefficients reduce measured non-target distributional drift, they still do not demonstrate consistent retention benefits. More importantly, the conclusions remain confined to a single 1.5B model, a highly constrained LoRA adaptation regime, and a narrow set of retention tasks. The new results do not substantially strengthen the generality or significance of the findings.
I therefore maintain my original rating.
Thank you for the questions. We expanded the coefficient comparison and measured probability changes on the fixed evaluation questions.
Larger KL coefficients
The original 0.02 coefficient was a fixed setting. We added 0.1 and 0.2, five and ten times that value, with three paired seeds each. All six new runs completed 300 updates using the original training code, data, and LoRA configuration.
Mean final accuracy across three seeds:
At 0.2, mean GSM8K accuracy is 1.56 percentage points below no KL, although still 9.89 points above initialization. All three ARC runs match the initial accuracy; BoolQ remains slightly below baseline.
Measured non-target drift
We normalize the saved answer-option log scores with a softmax and compare initialization with update 300 on identical questions. Relative to no KL, coefficient 0.2 reduces mean option TV from 0.01039 to 0.00838 on BoolQ and from 0.01262 to 0.01148 on ARC, though not in every seed.
The unchanged ARC scores also contain answer turnover. Each 0.2 run finishes at the initial 223/299, but the three runs lose 3, 2, and 3 previously correct answers, respectively, offset by equal gains. The short adaptation therefore produces substantial arithmetic learning and measurable non-target changes even when aggregate accuracy is stable.
Evaluation selection
The 300 GSM8K test questions and 600 BoolQ validation questions are selected by sorting SHA-256 hashes of a fixed task-specific prefix plus the original row index. Selection uses neither model predictions nor training outcomes. ARC includes all 299 validation questions.
The same items are evaluated at updates 0, 50, 100, 200, and 300. Fixed subsets keep repeated evaluation tractable, particularly GSM8K generation, and preserve item-level pairing across runs. Selected IDs and data hashes are saved in the protocol.
In this 1.5B, rank-8 setting, coefficients 0.1 and 0.2 produce lower mean option-distribution drift than no KL, but do not improve retention accuracy on both tasks. At 0.2, ARC's aggregate score is preserved alongside lower arithmetic accuracy and offsetting changes in individual answers.