- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper observes that during RLVR training, there are unconstrained properties which can change/drift across training runs. The paper focuses on one such property: failure disclosure, which is where a model explicitly discloses its failure at the task. The authors note that, across many RLVR runs, the spread of failure disclosure behavior is more varied than the spread of task correctness. The authors explore some different causes of this spread. They also propose a fix for this spread, effectively reintroducing a KL-based drift penalty from the original reference policy.
Critical strengths
- The authors appear to have run extensive experiments; however, I do not find the questions nor the findings significant enough to outweigh the weaknesses below.
Critical weaknesses
- Limited significance and novelty
- The main finding of the paper is that a behavior not directly constrained during training can drift across RL runs; I find this observation to not be very surprising or meaningful.
- The authors primarily study this drift in a KL-free RL setup, even though a KL penalty is the standard and well-known tool for preventing drift. In fact the authors re-introduce the KL penalty (albeit only on failed rollouts) as the fix for this problem.
- Overall, I think the experiments and observations in this manuscript could be condensed into a concise blog post, but I do not think they warrant a conference-paper-level contribution to the field.
- Selective reporting in favor of the main result
- The abstract makes the claim that "Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy." This is supported by the evidence presented in the body, which displays results for Qwen2.5-1.5B, OLMo-2-1B, Qwen-7B, and Qwen2.5-32B-Instruct (Figure 2). However, if my understanding is correct, it seems not to be supported by the other models, which are relegated to the appendix, and only mentioned starting in the discussion section ("Smaller boundary experiments with other model families such as Gemma and Phi do not show the same large spread").
- LLM-smelling jargon makes the paper hard to read and understand
- Overall, the text reads as heavily AI-generated, and this makes it hard for a human reader to comprehend.
- Some examples of LLM-smelling jargon that confused me in my reading: "failure-heavy probe", "primary probe", "boundary evidence", "shelter".
Questions for authors
- Why was KL-free training chosen for the primary experiments, beyond matching Dr. GRPO/DAPO? How large is the disclosure/solve variability ratio when the KL penalty is always used in RL runs?
- Can you report results for Gemma, Phi-4, and distilled Qwen beyond the limited analysis in the appendix?
2 comments
I have read the response from the authors, and maintain my original recommendation.
Author Response to Reviewer NfJb
Thank you for the review! We appreciate the concerns about novelty, generalization, and clarity, but disagree that our main finding simply shows that unconstrained behaviors can drift under RL.
1. Novelty and significance
RL drift, seed sensitivity, and KL regularization are well known, and we cite and mentions this work in our introduction and related works. We do not claim to discover generic drift or invent a new KL algorithm.
Our contribution is showing that models trained with the same objective can achieve similar task performance while differing dramatically in whether they admit objectively verifiable failures. We then investigate when and how this happens.
• The main effect is variability, not simply deterioration. Across 20 Qwen-1.5B runs, disclosure ranges from 0.26 to 0.90 while solve stays between 0.34 and 0.49 (SD ratio 3.93). Mean disclosure remains close to its pre-RL level (0.63 vs. 0.68), with ten runs above and ten below that reference. We replicate the pattern in OLMo-2-1B (20 runs, ratio 7.51) and a second reasoning task (20 runs, ratio 7.89).
• Small training differences can produce large reporting differences. Changing execution configurations at a fixed seed yields disclosure rates from 0.19 to 0.85, while identical configurations replay bit-for-bit. Restoring the same mid-RL training state and changing only future sampling randomness can yield rates from 0.11 to 0.88.
• How behavior is specified also matters. In our neutral controls, the same phrase varies from 0 to 0.90 across 20 runs when weakly specified, but becomes nearly perfectly reproducible when tied to an explicit prompt cue. This supports a broader link between behavioral constraints and stability.
This matters for evaluation: runs that look similar on task benchmarks may behave very differently when they fail. Knowing that behavior could drift does not tell us which behaviors vary, by how much, or whether standard evaluations would reveal it. To our knowledge, prior work has not established this combination of findings. We welcome pointers to any prior empirical work that directly demonstrates this.
2. KL-free training and standard KL
KL-free training is not an artificial or outdated setting: prominent reasoning-RL methods such as Dr. GRPO and DAPO omit reference KL by default. We study it because outcome rewards do not directly constrain failure reporting.
More importantly, limiting policy drift is not the same as making failure reporting reproducible across runs. KL penalizes departures from a reference policy; it neither directly rewards admitting failures nor guarantees preservation across runs. Whether it prevents our observed variability must be tested, not assumed.
We test it directly in the main paper (Section 5.1): across 20 runs with standard all-token KL, disclosure SD falls from 0.189 to 0.069 but remains 2.42 times the solve-rate SD, while mean solve falls by about 8.4 percentage points relative to unanchored training. Hence, standard KL helps, but neither removes the relative variability nor comes without a capability cost in this setting.
We do not claim KL is novel. We measure a specific, safety-relevant reproducibility problem and test how much a familiar intervention (in this case, KL anchoring) addresses it.
3. Model-family results
We do not claim the effect is universal. Our main finding is supported by independent 20-run replications in Qwen and OLMo, established open-weight model families relevant to RL post-training. This is not an isolated result from one model or seed.
Appendix C also reports weaker or absent effects in Phi-4 and distilled Qwen, alongside mixed Gemma results where task performance also varies substantially. These results help establish boundaries rather than contradicting the positive replications.
Showing that this problem exists, can be substantial, and recurs across multiple relevant model families and tasks is already an important empirical contribution. The phenomenon need not appear in every setting to matter, especially when models with similar task performance may report failures very differently.
4. Writing and presentation
We appreciate the readability feedback. We disclosed AI assistance and substantially reviewed and edited the manuscript ourselves before submission. We agree that some terms and explanations could be clearer, and will address this in the next revision.
We hope this clarifies why our paper goes beyond generic RL drift: it measures and investigates a specific failure of behavioral reproducibility relevant to model evaluation.