acceptodds
Under review as a conference paper at ICLR 2027

When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning

Abstract

Outcome-based reinforcement learning can produce models with similar task performance but very different ways of communicating about their mistakes. We study failure disclosure: whether a model admits that an attempted solution failed rather than staying silent or presenting it as successful. Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy. The pattern extends to a second reasoning task and stabilized PPO, persists at 7B, and also appears in an instruction-conditioned 32B setting. We also find that small floating-point and sampling differences during training can redirect reporting behavior even when the task objective and earlier training history are held fixed. Additional tests show that failure disclosure is not a single decision: Checking the answer, entering a report, and completing the admission can separate, and the weak point depends on the task and response format. Further, experiments with neutral controls show more broadly that behaviors left weakly constrained by training are especially likely to vary across runs, of which failure disclosure is an example. We can reduce variability in failure disclosure by discouraging the model from drifting from its starting policy on failed, well-formed responses. This makes reporting substantially more consistent, though its effect on task performance depends on the setting. Stable task accuracy therefore does not guarantee stable safety-relevant behavior: Researchers should measure these behaviors directly across runs and design training methods that keep them reliable.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language models, reinforcement learning, AI safety, alignment, failure disclosure, monitorability, reproducibility, behavioral stability, post-training, underspecification, honesty

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NfJbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that during RLVR training, there are unconstrained properties which can change/drift across training runs. The paper focuses on one such property: failure disclosure, which is where a model explicitly discloses its failure at the task. The authors note that, across many RLVR runs, the spread of failure disclosure behavior is more varied than the spread of task correctness. The authors explore some different causes of this spread. They also propose a fix for this spread, effectively reintroducing a KL-based drift penalty from the original reference policy.

Critical strengths
  • The authors appear to have run extensive experiments; however, I do not find the questions nor the findings significant enough to outweigh the weaknesses below.
Critical weaknesses
  • Limited significance and novelty
    • The main finding of the paper is that a behavior not directly constrained during training can drift across RL runs; I find this observation to not be very surprising or meaningful.
    • The authors primarily study this drift in a KL-free RL setup, even though a KL penalty is the standard and well-known tool for preventing drift. In fact the authors re-introduce the KL penalty (albeit only on failed rollouts) as the fix for this problem.
    • Overall, I think the experiments and observations in this manuscript could be condensed into a concise blog post, but I do not think they warrant a conference-paper-level contribution to the field.
  • Selective reporting in favor of the main result
    • The abstract makes the claim that "Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy." This is supported by the evidence presented in the body, which displays results for Qwen2.5-1.5B, OLMo-2-1B, Qwen-7B, and Qwen2.5-32B-Instruct (Figure 2). However, if my understanding is correct, it seems not to be supported by the other models, which are relegated to the appendix, and only mentioned starting in the discussion section ("Smaller boundary experiments with other model families such as Gemma and Phi do not show the same large spread").
  • LLM-smelling jargon makes the paper hard to read and understand
    • Overall, the text reads as heavily AI-generated, and this makes it hard for a human reader to comprehend.
    • Some examples of LLM-smelling jargon that confused me in my reading: "failure-heavy probe", "primary probe", "boundary evidence", "shelter".
Questions for authors
  • Why was KL-free training chosen for the primary experiments, beyond matching Dr. GRPO/DAPO? How large is the disclosure/solve variability ratio when the KL penalty is always used in RL runs?
  • Can you report results for Gemma, Phi-4, and distilled Qwen beyond the limited analysis in the appendix?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer NfJb

Thank you for the review! We appreciate the concerns about novelty, generalization, and clarity, but disagree that our main finding simply shows that unconstrained behaviors can drift under RL.

1. Novelty and significance

RL drift, seed sensitivity, and KL regularization are well known, and we cite and mentions this work in our introduction and related works. We do not claim to discover generic drift or invent a new KL algorithm.

Our contribution is showing that models trained with the same objective can achieve similar task performance while differing dramatically in whether they admit objectively verifiable failures. We then investigate when and how this happens.

• The main effect is variability, not simply deterioration. Across 20 Qwen-1.5B runs, disclosure ranges from 0.26 to 0.90 while solve stays between 0.34 and 0.49 (SD ratio 3.93). Mean disclosure remains close to its pre-RL level (0.63 vs. 0.68), with ten runs above and ten below that reference. We replicate the pattern in OLMo-2-1B (20 runs, ratio 7.51) and a second reasoning task (20 runs, ratio 7.89).

• Small training differences can produce large reporting differences. Changing execution configurations at a fixed seed yields disclosure rates from 0.19 to 0.85, while identical configurations replay bit-for-bit. Restoring the same mid-RL training state and changing only future sampling randomness can yield rates from 0.11 to 0.88.

• How behavior is specified also matters. In our neutral controls, the same phrase varies from 0 to 0.90 across 20 runs when weakly specified, but becomes nearly perfectly reproducible when tied to an explicit prompt cue. This supports a broader link between behavioral constraints and stability.

This matters for evaluation: runs that look similar on task benchmarks may behave very differently when they fail. Knowing that behavior could drift does not tell us which behaviors vary, by how much, or whether standard evaluations would reveal it. To our knowledge, prior work has not established this combination of findings. We welcome pointers to any prior empirical work that directly demonstrates this.

2. KL-free training and standard KL

KL-free training is not an artificial or outdated setting: prominent reasoning-RL methods such as Dr. GRPO and DAPO omit reference KL by default. We study it because outcome rewards do not directly constrain failure reporting.

More importantly, limiting policy drift is not the same as making failure reporting reproducible across runs. KL penalizes departures from a reference policy; it neither directly rewards admitting failures nor guarantees preservation across runs. Whether it prevents our observed variability must be tested, not assumed.

We test it directly in the main paper (Section 5.1): across 20 runs with standard all-token KL, disclosure SD falls from 0.189 to 0.069 but remains 2.42 times the solve-rate SD, while mean solve falls by about 8.4 percentage points relative to unanchored training. Hence, standard KL helps, but neither removes the relative variability nor comes without a capability cost in this setting.

We do not claim KL is novel. We measure a specific, safety-relevant reproducibility problem and test how much a familiar intervention (in this case, KL anchoring) addresses it.

3. Model-family results

We do not claim the effect is universal. Our main finding is supported by independent 20-run replications in Qwen and OLMo, established open-weight model families relevant to RL post-training. This is not an isolated result from one model or seed.

Appendix C also reports weaker or absent effects in Phi-4 and distilled Qwen, alongside mixed Gemma results where task performance also varies substantially. These results help establish boundaries rather than contradicting the positive replications.

Showing that this problem exists, can be substantial, and recurs across multiple relevant model families and tasks is already an important empirical contribution. The phenomenon need not appear in every setting to matter, especially when models with similar task performance may report failures very differently.

4. Writing and presentation

We appreciate the readability feedback. We disclosed AI assistance and substantially reviewed and edited the manuscript ourselves before submission. We agree that some terms and explanations could be clearer, and will address this in the next revision.

We hope this clarifies why our paper goes beyond generic RL drift: it measures and investigates a specific failure of behavioral reproducibility relevant to model evaluation.

I have read the response from the authors, and maintain my original recommendation.

Reviewer_97M7OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript examines whether models trained with outcome-based reinforcement learning consistently acknowledge failed attempts. The question is relevant to evaluating model behavior beyond task accuracy, and the experiments provide substantial evidence for the main observation.

Critical strengths

Strengths

  1. The manuscript distinguishes task success from communicating a failed attempt. Models with similar accuracy may give users different information when they are wrong. The introduction demonstrates clearly and provides a focused motivation for the study.

  2. Strong evidence from repeated experiments. The primary experiment compares multiple independent Qwen2.5-1.5B training runs and measures variation in both solving and failure disclosure. Reporting varies more than task performance. Experiments with other settings also help establish the finding’s scope and boundaries.

  3. The study not only compares random seeds, but also changes execution configurations while holding the seed fixed, and separately continue training. Identical-configuration replays provide an additional control.

  4. The study uses failure-conditional reference anchoring to reduce variation in reporting across training runs and test it across tasks and model sizes.

Critical weaknesses

Weakness:

  1. Although the execution-configuration experiment shows that floating-point changes can redirect disclosure at one fixed seed, the six configurations were not repeated across seeds. So it is unclear that how common it is. Suggest to repeat the configuration sweep across seeds.

  2. The reporting-route interventions cannot determine whether a silent model recognized its error and failed to express it, or never recognized it. Suggest to add an independent error-recognition probe to clarify the failing stage and strengthen the proposed mechanism.

  3. A learned response format may affect the primary effect size and report-entry mechanism. Suggest to add a matched within-model template ablation.

  4. Reference anchoring reduces cross-run disclosure variability, but its practical advantage remains unclear. Failure-conditional and all-token anchoring are not matched for total regularization. Suggest compare approaches at matched capability and constraint budgets.

  5. Training steps are few. Most headline comparisons stop at 300 training steps, while a few continuations to 600 or 1,200 steps. Suggest do more repeated long-horizon experiments to explore whether disclosure variability persists, disappears, or changes form as training proceeds.

  6. The experimental findings are based on a single LLM family (Qwen). Suggest to test in other LLM families to improve the generalization of the findings.

Questions for authors

Please see my weakness points.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 97M7

Thank you for recognizing the strengths of our repeated-training experiments, controlled execution comparisons, and anchoring results. We would like to clarify a few points, especially regarding model-family generalization.

1. We already test a second model family

The review states that our experiments are limited to Qwen. However, Figure 2A and Tables 11–12 report 20 independent OLMo-2-1B retrainings, which reproduce the main finding.

In OLMo, disclosure ranges from 0.072 to 0.903 while solve stays between 0.375 and 0.492. The disclosure-to-solve SD ratio is 7.51, compared with 3.93 for Qwen. This is a substantial cross-family replication in the main paper, and Appendix C reports further model-family results. Hence, our primary experiments already include more model families, and our central finding is not only established for Qwen.

2. Execution differences across seeds

We agree that testing execution changes across more seeds helps assess how widespread this sensitivity is. Our existing records contain an additional six-seed analysis that was not in the submitted paper. Applying one microbatch/gradient-accumulation change across six Qwen-1.5B seeds changes disclosure by approximately 0.18–0.34 in four of the six seeds at step 300, in both directions. All six show substantial differences at step 150.

This complements our submitted fixed-seed experiment and bit-exact replay checks. These results sufficiently demonstrate that execution sensitivity is not limited to the originally reported seed; we will include these additional seed results in the next manuscript revision.

3. Error recognition and reporting stages

We agree that these experiments cannot tell whether a model internally recognizes a failure but does not report it, or fails to recognize it. We explicitly acknowledge that limitation in Section 4.2 and the Limitations section. Our claim concerns observable stages: checking, reaching a reporting opportunity, and completing an admission. An independent recognition probe would be useful future work, but is not required for that behavioral finding. We leave more in-depth interpretability work to follow-up studies.

4. Response formats and KL anchoring

We agree that response format matters; Appendix E includes paragraph-transition, report-prefix, and formulation controls. They support format sensitivity, though they are not a fully matched training-format ablation.

We also compare failure-conditional with standard all-token KL in Section 5.1 (details in Appendix H). Both reduce variability, and we explicitly acknowledge the unmatched regularization budgets. Our claim is that reference anchoring can reduce reporting variability, not that failure-conditional KL is a better algorithm. A matched-budget comparison would answer that separate question, but is not needed to establish our main empirical finding or phenomenon.

5. Longer training

We mainly examine reproducibility at a fixed training budget, not whether differences persist indefinitely. In practice, RL training uses finite compute budgets, and models are evaluated and selected at specific checkpoints. Even if models eventually converged after much longer training, this would not invalidate the large differences in failure reporting we observe across otherwise comparable runs at 300 steps.

Moreover, longer training is not necessarily a solution. We already report experiments through 600 and 1200 steps (Appendix J.1), with mixed results: some settings retain substantial variability, while others converge toward low disclosure. Greater consistency at the cost of losing the desired reporting behavior would not solve the underlying problem. Thus, our experiments already establish a meaningful fixed-budget reproducibility problem, regardless of what might happen at much longer horizons.

We hope these clarifications, especially the main-text OLMo replication and existing multi-seed results, help address the concerns about generalization and experimental support.

I thank the authors for the response. I maintain my rating.

Reviewer_LZTwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failure disclosure in language models: whether a model acknowledges that it failed a task instead of giving an incorrect answer without warning.

The authors show that models trained with reinforcement learning can achieve similar task accuracy while differing substantially in how often they report failures. Experiments across multiple models and training configurations reveal significant variation in disclosure behavior. The paper also proposes a KL-based regularization method to reduce this variation.

Critical strengths
  1. The paper highlights that similar task accuracy does not guarantee consistent failure-reporting behavior, an important issue for reliable language models.

  2. The main finding is supported by multiple training runs, different models, and controlled experiments that rule out several simple explanations.

  3. KL regularization reduces disclosure variability, and the authors provide detailed results, confidence intervals, and negative findings.

Critical weaknesses
  1. The main finding is convincing under KL-free GRPO, with additional replications on OLMo-2-1B and the shortest-path task (Table 12). However, the abstract extends the conclusions to stabilized PPO and to the 7B and 32B models, where the evidence is more limited. For example, PPO shows much smaller disclosure variability (SD 0.047 vs. 0.185; Table 11), while the 7B and 32B experiments also show considerable variation in task accuracy. The claim that the weak reporting stage depends on the task and response format rests on unquantified grid experiments and only two 7B runs (Table 20). Additionally, the claimed causal effect of credit assignment comes from a three-pair pilot, and larger follow-up studies with 15–20 pairs did not confirm the improvement (Appendix G). These broader claims should be narrowed to match the available evidence.

  2. The grid experiments used to support the checking-stage and response-format claims provide no quantitative results, model details, or run counts (Table 20; Appendix E). The PPO experiment is also insufficiently described. Table 3 gives only basic settings and does not explain what makes the method "stabilized" or specify the value model, advantage estimation, clipping, or evaluation probe. Similarly, the 32B experiment lacks details about the instruction, the frozen detector, and the blind-read calibration procedure (Table 10). Some main-text claims also cite tables that do not contain the stated results; for example, the Figure 1 caption cites Table 19 for a median SD that the table does not report. Although the core experiments are well documented, these missing details make it difficult to verify the broader conclusions.

  3. The proposed method applies KL regularization only to failed, well-formed responses. However, all-token KL regularization, essentially the standard GRPO KL term, reduces disclosure variability at least as much (SD ratio 0.363 vs. 0.472; Table 29). In this single 1.5B Countdown comparison, failure-specific KL has a smaller impact on task accuracy, preserving about 2.4 percentage points more performance. As the paper acknowledges, the comparison is not budget-matched because the proposed method regularizes only 57–60% of tokens at the same penalty strength (Appendix H). Slower task learning also explains part of the reduced variability in 1.5B Countdown, though not the 7B result, where accuracy improved. A budget-matched comparison is needed to show whether failure-specific regularization offers a meaningful advantage over standard KL.

Questions for authors
  1. Can the authors clarify which conclusions are supported by large-scale experiments and which remain preliminary, particularly for PPO, larger models, and reporting-stage separation?

  2. Can the missing quantitative results for the grid experiments be provided, along with complete descriptions of the PPO and 32B settings?

  3. Why do the larger follow-up experiments not reproduce the initial causal credit-assignment effect?

  4. Can the authors compare failure-specific and all-token KL regularization under matched effective regularization budgets?

  5. Does the variation in failure disclosure persist with longer training, beyond the current 300-step evaluation?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer LZTw

Thank you for the detailed review and for recognizing the strength of our main finding! We appreciate the suggestions and would like to clarify a few points.

1. Main findings versus supporting experiments

Our central result is supported by four 20-run independent sets of retrainings: two Qwen-1.5B Countdown, one OLMo-2-1B, and one shortest-path. The latter two reproduce the main pattern, with disclosure-to-solve SD ratios of 7.51 and 7.89 (Tables 10–12).

Our PPO, 7B, 32B, and reporting-stage experiments explore other settings, possible explanations, and boundaries, rather than serving as equally large replications. PPO has a smaller absolute disclosure spread but a ratio of 3.55 under the primary evaluation; the larger models show reporting variation alongside greater variation in task performance. We agree these distinctions could be clearer in the abstract.

Given the substantial computational cost of independent RL retrainings, particularly for larger models, we do not think every supporting experiment needs the same 20-run scale. They address different questions, while our main finding already has extensive independent replication.

2. Missing details and reporting-stage experiments

We agree that some methods could be described more fully. To clarify the specific points raised:

• PPO: TRL 1.7.0; one PPO epoch per batch instead of four; a value network initialized from the SFT checkpoint with a separate head; GAE (γ=1\gamma=1, λ=0.95\lambda=0.95); policy/value clipping at 0.2; no reference KL. Eight runs of 300 batches, evaluated on the primary probe.

• 32B: Qwen2.5-32B-Instruct, without additional SFT. The fixed instruction reads: "It is acceptable to fail. If no valid expression reaches the target, state clearly that you could not find one, then give your best attempt in the answer tags." The detector was frozen beforehand; blinded calibration used model readers, not human annotators.

• Grids: Our existing SFT-based diagnostics (not RL replications) show balanced accuracy of 0.77–0.999 in a 7B checking setup, yet no spontaneous reports across 1,497–2,010 own failures. This supports a distinction between prompted checking and spontaneous reporting, not a claim about internal error recognition.

We also appreciate the Figure 1 correction: the median within-cell SD is discussed in Appendix D.2, not Table 19.

3. Credit-assignment experiments

We agree the three-pair pilot is too small to establish a general mechanism. Larger follow-ups (up to 20 pairs) did not consistently reduce cross-run variability, as reported in Section 4.4 and Appendix G. This suggests local credit assignment alone is not a reliable solution in our tested settings, motivating KL anchoring as a broader way to preserve behavior. Neither our main finding nor our KL anchoring results depend on this credit-assignment pilot.

4. Failure-conditional versus all-token KL

We agree that all-token KL narrows variability at least as much, and that our comparison cannot establish which scope performs better at an equal regularization budget. Section 5.1 already makes this distinction.

Across 20 runs, disclosure SD falls from 0.189 without KL to 0.089 with failure-conditional KL and 0.069 with all-token KL. Failure-conditional KL preserves more solve accuracy at the tested coefficient but applies less regularization. A retrospective analysis of archived logs finds its realized penalty is about 78% of the all-token penalty, so these results do not establish an efficiency advantage.

Our claim is that reference anchoring can reduce reporting variability, not that failure-conditional KL is a better algorithm. A matched-budget comparison addresses a separate question, not our central finding.

5. Longer training

We study reproducibility at a fixed training budget, not whether differences persist indefinitely. In practice, RL uses finite compute budgets, and models are evaluated and selected at specific checkpoints. Even if runs eventually converged, that would not erase the large differences in reporting we observe at 300 steps.

Longer training is also not necessarily a solution. We report experiments through 600 and 1,200 steps (Appendix J.1), with mixed results: variability persists in some settings, while others converge toward low disclosure. Greater consistency from losing the desired reporting behavior would not solve the underlying problem.

Thank you again for the constructive suggestions. We believe our main finding is well supported and agree that some supporting analyses can be described more clearly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

We thank the reviewers for their time and feedback. We appreciate their recognition of the importance of studying failure reporting, and would like to clarify a few points that came up across the reviews.

1. Our main contribution goes beyond the general observation that RL can cause behavioral drift. Across four 20-run retrainings spanning Qwen and OLMo and two reasoning tasks, failure disclosure varies much more than task performance. In our primary experiment, mean disclosure stays relatively close to its pre-RL level (0.63 vs. 0.68), yet individual runs end up with very different disclosure rates (0.26–0.90). We also show that changing low-level execution details at a fixed seed, or future sampling randomness after restoring the same training state, can produce large differences in reporting.

Importantly, knowing that a behavior could drift does not tell us how much it actually varies, whether it varies more than task performance, or whether standard evaluations would reveal these differences. Our findings show that models with similar task performance can behave very differently when they fail, which has important implications for model evaluation and oversight.

2. The paper provides extensive evidence beyond the main result. We include independent replications, checks of the disclosure detector and evaluation setup, controlled training experiments, behavioral controls, and anchoring interventions. The additional experiments explore when the effect occurs, what may explain it, and where its boundaries lie. They are not each intended to be another full 20-run replication, especially given the substantial computational cost of independent RL retrainings. Our central finding is already supported by multiple large, independent retraining runs.

3. KL anchoring is not intended as a new optimization algorithm, nor does its existence make our findings trivial. We directly compare failure-conditional and standard all-token KL in Section 5.1. Both reduce reporting variability, with different tradeoffs in task performance, and we acknowledge that their regularization budgets are not matched. However, even with standard all-token KL, disclosure remains about 2.4 times more variable than solve rate in our tested setting. Thus, KL helps but does not fully resolve the problem. Knowing beforehand that KL can limit policy drift does not establish how well it preserves this particular behavior across independent training runs, which we empirically investigate.

We appreciate the suggestions for clearer writing and documentation, and address the specific concerns in our individual responses. We will incorporate this feedback into the next manuscript revision. We hope these clarifications help convey the strength of our main empirical contribution and the intended scope of our supporting experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.