acceptodds
Under review as a conference paper at ICLR 2027

Training Language Models to Fail Gracefully

10 views

Abstract

Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to *fail gracefully*: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with **GracefulRL**, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, reliability, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CszHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to train LLMs to fallback when they are unlikely to succeed. The paper first shows that simply assigning an intermediate reward could easily lead to a model that always falls back. To address this challenge, they proposed GracefulRL by using a log-probability objective to encourage both task success and error avoidance. Experiments show that the method can reduce errors while largely preserving task performance. The paper addresses an important problem and provides an interesting analysis of the challenges in learning fallback behavior.

Critical strengths
  1. The motivation of this paper is clear and the problem is practical. LLM should have a better understanding of its capability boundary.
  2. The proposed method is supported by theoretical analysis and experiments.
Critical weaknesses
  1. GracefulRL still collapses to always falling back in SafeMaze when fewer solutions are sampled during training. This raises concerns about its reliability on harder tasks, where successful trajectories may be even harder to obtain.
Questions for authors
  1. Can GracefulRL be applied to agentic tasks? Does multi-turn interaction affect the behavior of fallback?
  2. If we prompt LLM to fail gracefully once it is unsure about the solution, does GracefulRL still have a clear advantage over prompt engineering?
  3. How sensitive is GracefulRL to the fallback reward? Should we conduct tuning for each new task?
3 comments

We thank the reviewer for their feedback. We address their concerns below:

W1 and Q3. Behavior in the low rollout regime and sensitivity

GracefulRL is substantially more stable than RLOO/GRPO across different settings, and its sensitivity to the rollout budget follows a predictable pattern: higher fallback rewards require more rollouts. Specifically, for SafeMaze at c=0.8c=0.8, GracefulRL collapses with 16 and 32 rollouts but performs well with 64 rollouts as mentioned in Appendix C.4. For lower values of cc, 16 rollouts is enough. Intuitively, this follows from the fact that for NN rollouts, the population objective is only an approximation to JGracefulRL=(1−c)log⁡pS+clog⁡(1−pE)J_{GracefulRL} = (1-c)\log p_S+c\log(1-p_E) where the approximation gets tighter with a higher NN.

As cc increases, the objective places greater relative emphasis on avoiding errors, making it increasingly important to accurately estimate the success-promoting component. With a limited rollout budget, this component may be poorly estimated, particularly when successful trajectories are rare. Conclusively, our algorithm is stabler and needs less tuning compared to RLOO/GRPO, and even when it does, there is a predictable recipe.

Q1. Agentic tasks

GracefulRL is already evaluated on multi-turn, interactive environments, including both SafeMaze and Minesweeper. In particular, Minesweeper requires the LLM agent to make a sequence of decisions while interacting with the environment, where an incorrect action can lead to failure. These experiments demonstrate that GracefulRL can effectively learn when to fall back in sequential decision-making settings, rather than being limited to single-turn tasks. More generally, applying GracefulRL to other agentic tasks requires defining an appropriate fallback behaviour for the environment, such as abstaining from an action, requesting human intervention, or reverting code changes. Given a suitable fallback state and corresponding outcome-based rewards, we expect GracefulRL to generalize to a broader range of agentic tasks.

AuthorsOpenReview ↗

Rebuttal continued

Q2. Prompt engineering

We thank the reviewer for raising this question. While prompt engineering can encourage models to abstain when uncertain, we find that prompting alone is insufficient to reliably elicit fallback behavior from models trained solely to maximize correctness. To investigate this, we conducted additional experiments on AIME using a 4B model trained with standard RLVR on correctness, without explicit fallback training. We evaluated five different prompting strategies: the GracefulRL prompting, confidence thresholding, explicit scoring instructions, error penalties, and self-verification. We varied the fallback reward cc from 0 to 0.8, with 360 generations per setting (90 problems × 4 rollouts).

Across all prompting strategies and reward settings, the model almost never abstains, with fallback rates remaining below 2% in every setting. Even when explicitly instructed to consider the cost of incorrect answers or verify its solution before responding, the model overwhelmingly attempts an answer. This suggests that simply instructing a model to abstain, even with explicit incentives for doing so, may not be sufficient to induce reliable fallback behavior after standard RLVR training.

Results are in this table. Each cell reports Success / Fallback / Error (%) over 360 generations (90 AIME problems × 4 rollouts).

cGracefulRLThresholdScoringPenaltySelf-check
0.018.9 / 0.0 / 81.119.2 / 0.0 / 80.817.8 / 1.1 / 81.117.8 / 0.0 / 82.219.4 / 0.0 / 80.6
0.218.9 / 0.0 / 81.118.3 / 0.6 / 81.118.6 / 1.1 / 80.316.1 / 0.3 / 83.616.7 / 0.0 / 83.3
0.420.3 / 0.0 / 79.718.6 / 0.3 / 81.117.2 / 1.4 / 81.416.9 / 0.3 / 82.817.8 / 0.0 / 82.2
0.618.3 / 0.0 / 81.719.7 / 0.3 / 80.016.1 / 1.1 / 82.817.5 / 0.0 / 82.515.8 / 0.0 / 84.2
0.818.3 / 0.0 / 81.717.5 / 0.3 / 82.219.2 / 1.9 / 78.918.3 / 0.3 / 81.415.8 / 0.3 / 83.9

These findings are consistent with prior work. AbstentionBench [1] reports that reasoning fine-tuning degrades abstention performance by 24% on average, while OpenAI's o3 System Card [2] reports higher hallucination rates for o3 than o1 despite improved reasoning capabilities. These findings suggest that stronger general reasoning capabilities do not necessarily translate into more reliable abstention behavior.

We believe that prompt engineering is a valuable and complementary approach, as it can leverage models' existing capabilities to encourage appropriate fallback behavior. However, rather than relying solely on prompting to elicit this behavior, we believe that explicitly training models to acquire the capability would be better. Such training allows us to directly optimize the tradeoff between task success and costly errors, potentially leading to more robust and reliable behavior. This is particularly important in risk-sensitive applications, where incorrect decisions may have significant consequences.

[1] Kirichenko, P., Ibrahim, M., Chaudhuri, K., and Bell, S. J. (2025). AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. NeurIPS 2025 (Datasets and Benchmarks Track). https://arxiv.org/abs/2506.09038

[2] OpenAI (2025). OpenAI o3 and o4-mini System Card. Section 3.3: Hallucinations. https://openai.com/index/o3-o4-mini-system-card/

Dear Reviewer,

We believe that we have answered all your questions. We would appreciate if you could engage with us. We’ll be happy to respond to any further questions you might have!

Reviewer_VPzGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GracefulRL, which optimizes a weighted sum of the log-probabilities of success and non-error outcomes to mitigate policy collapse under ternary rewards for success, safe fallback, and error. Experiments on maze navigation, mathematical reasoning, and multi-turn Minesweeper demonstrate selective fallback behavior while largely preserving task-solving ability, yielding improved success–error trade-offs.

Critical strengths
  1. The research problem is practically meaningful. Learning to fall back safely when task completion is unreliable addresses the cost of failure beyond success rate alone, making it relevant to model and agent reliability.
  2. The experiments cover diverse task settings. Evaluations on maze navigation, mathematical reasoning, and multi-turn Minesweeper provide initial evidence of applicability across different models and interaction settings.
Critical weaknesses
  1. The problem positioning needs further clarification. The paper should more thoroughly discuss the relationship between safe fallback and confidence expression. Confidence expression allows users to decide whether to accept an answer according to their risk preferences, whereas direct abstention delegates more decision-making to the model. Although conditional prompts can adjust the model’s tendency to fall back, the trade-offs in human control and applicability remain insufficiently discussed.

  2. The algorithmic novelty is limited. The authors acknowledge that the objective is a special case of TailRL under ternary rewards, with smoothing as the main algorithmic modification. The paper should clarify the substantive advances in optimization analysis and empirical findings beyond prior work.

  3. Coverage of related work and experimental comparisons is limited. Confidence-expression methods receive limited discussion, and comparisons with these methods are largely restricted to RLCR with thresholding. Methods addressing the conflict between accuracy and calibration, such as DCPO, are not included. The paper also discusses abstention-learning methods such as TIAR without directly comparing against them. Including these baselines in the mathematical reasoning abstention setting would better establish the method’s advantages over related approaches.

  4. Validation across model scales is limited. The language model experiments primarily use Qwen3 models with 1.7B and 4B parameters. Adding experiments with an 8B model would strengthen the evidence for effectiveness at larger scales.

Questions for authors

please see Weaknesses

3 comments

We thank the reviewer for their feedback. We address their concerns below:

W1. Positioning w.r.t confidence expression

Confidence expression and graceful failure are distinct capabilities, each with its unique use cases.

Confidence expression focuses on communicating the model's uncertainty, allowing users or downstream systems to make decisions based on their risk preferences. This is particularly useful in human-in-the-loop settings, where users can independently decide whether to trust a model's output.

Graceful fallback, on the other hand, focuses on learning how an agent should act under uncertainty. This distinction is particularly important in autonomous and sequential decision-making environments. As an example, in maze navigation, expressing confidence about reaching the goal does not by itself determine what actions it should take to reach the goal or the safe square. Beyond estimating uncertainty, the agent must also learn the appropriate decision-making policy.

We will clarify this distinction more strongly in the revised paper beyond what’s already done in Section 6. We also wish to point out that our work is not about arguing if either is better, and both have their separate use cases.

W2. Novelty beyond the algorithm

We would like to clarity, that despite close connections to TailRL at the algorithmic level (except the smoothed estimator), our primary contributions lie in identifying and explaining a previously underexplored optimization pathology, providing theoretical insights, and demonstrating how these insights enable reliable fallback behavior in practice.

While TailRL is motivated by improving Best-of-(k) performance through tail-likelihood optimization, our work studies the problem through a different lens: understanding the optimization dynamics of learning safe fallback behavior. In particular, we establish the following theoretical results:

  1. Vanishing gradients for standard policy gradients (Proposition 1): We prove that ∣∇θJPG∣=O(1−po)|\nabla_\theta J_{\mathrm{PG}}|=O(1-p_o) as po→1p_o\to1, for any outcome o∈Success, Fallback, Erroro\in{\text{Success, Fallback, Error}}. This provides a theoretical explanation for why standard PG methods can collapse to fallback-only policies.
  2. Non-vanishing gradients for GracefulRL (Proposition 2): We prove that ∣∇θJGracefulRL∣=Θ(1)|\nabla_\theta J_{\mathrm{GracefulRL}}|=\Theta(1) as po→1p_o\to1 for o∈Fallback, Erroro\in{\text{Fallback, Error}}, under the stated regularity and non-degeneracy assumptions. This establishes why the GracefulRL objective preserves an optimization signal near suboptimal saturated policies.
  3. Extension to GRPO (Appendix A.4): We additionally establish that GRPO suffers from gradient suppression near saturated outcomes, with an O(1−po)O(\sqrt{1-p_o}) bound, showing that this pathology is not restricted to REINFORCE/RLOO.

These optimization-theoretic insights into likelihood-based objectives are, to our knowledge, not established in prior work and constitute valuable scientific contributions in their own right. Beyond their theoretical significance, they provide practical guidance on when likelihood-based objectives may be preferable—or even necessary—to overcome limitations of established approaches such as GRPO.

Beyond the theoretical analysis, our experimental evidence establishes several new findings:

  1. Standard policy-gradient methods can collapse to always falling back despite correctly specified reward preferences;
  2. GracefulRL can learn selective fallback policies in interactive, multi-turn environments purely from outcome level rewards;
  3. Models can be trained to condition their fallback behavior on different risk preferences while preserving task-solving capabilities; and
  4. In Minesweeper, we further demonstrate that learning safe fallback can improve both task success and error rates.
AuthorsOpenReview ↗

Rebuttal continued

W3. Coverage of related work and experimental comparisons is limited

For further experimental comparison, we tried tuning the reward for our conditioned training on the LLM scale and found the method to be unstable with the 1.7B and 4B models, with the coverage on abstention vanishing after ~250 steps for the 1.7B models and ~850 steps for the 4B model. These results again hints at the pathology associated with vanilla policy-gradient methods wherein training can be unstable, with training runs collapsing, sometimes after many steps. Results are available in Appendix D.8 of the updated manuscript.

On DCPO: DCPO is a method to improve the calibration of the verbalized confidences of models, whilst also improving accuracy. We attempted to reproduce DCPO using its publicly released implementation on Qwen3-1.7B-Base and Qwen3-4B-Base. In our setting, training became unstable, with rewards collapsing to zero and the model predicting zero confidence across queries. Upon investigating the implementation, we identified a discrepancy between the published calibration reward, which uses absolute error, and the released code, which uses squared error (https://github.com/icip-cas/DCPO/blob/2ebb9770d33fc045fff45329a542a476cfcd8e13/verl/workers/reward_manager/hybrid.py#L203). As these objectives induce different optimization dynamics, we cannot confidently interpret these unsuccessful runs as a faithful evaluation of the published method. We report the observed behavior in Appendix D.9 and acknowledge that establishing a reliable comparison requires further investigation. A faithful implementation of this baseline would require re-implementing the codebase and finding other potential issues, along with tuning hyperparameters which is not feasible during a short rebuttal period. Our results are available in Appendix D.9 of the updated manuscript.

On TIAR: TIAR is a method for abstention where the advantages for fallback are tuned based on the current success and error rates. LLM scale experiments are quite compute intensive, therefore we implement TIAR on the SafeMaze environment and observe a collapse where the model resorts to a constant fallback rate of 100% for all values of the λ\lambda hyperparameter. We also theoretically justify it in Section C.7 of the updated manuscript, where we show that the TIAR objective also suffers from vanishing gradients near fallback saturation, just like GRPO and RLOO. Our work is exactly trying to give a theoretical justification behind this issue and provides a principled fix.

W4. Validation across the 8B scale

We ran our algorithm at the 8B scale using the same hyperparameters used at the 1.7B and 4B scale, without any separate tuning and have added the results to Appendix D.10 of the updated manuscript. We observe similar behavior as we did on the 1.7B and 4B scales, where in the model during training selectively learns to abstain on problems which are difficult with little loss to performance.

For instance, on the AIME dataset, at c=0c=0, the model gets 23.8% accuracy with 75.9% error rates, but at the most cautious c=0.8c=0.8 level, the model gets only 11% error rate with 18.7% accuracy rate. Therefore, the error rate goes down approximately 64.9% with a loss of 5.1% in accuracy. At the 8B scale, we observe that the difference with respect to RLCR+Thresholding reduces wherein we get higher success rates at lower levels of caution but slightly worse success rates at higher levels of caution. Improving the success rates at higher values of caution is left for future work.

Dear Reviewer,

We believe that we have answered all your questions. We would appreciate if you could engage with us. We’ll be happy to respond to any further questions you might have!

Reviewer_b3LROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GracefulRL, a reinforcement learning approach that encourages language models to safely fall back when task success is unlikely, rather than producing incorrect answers. The author find out the previous approachs may cuase the model to be trapped into providing safe answers both empirically and theoretically. To address this issue, they use the training objective equivalent to a specialization of TailRL. Experiments on maze navigation, mathematical reasoning, and multi-turn Minesweeper demonstrate that GracefulRL can substantially reduce errors while largely preserving task success, and in some settings improve both success and error rates.

Critical strengths
  1. The analysis on the failing pattern of normal approch is intuitive and easy to understand. The authors also provide theoretical justifications to it.
  2. The specially designed maze experiments is intuitive and useful. Using game-like setting give us another view besides experiments on LLMs.
Critical weaknesses
  1. The baseline method RLOO seems like a RLOO version of truthRL. I'm wondering why the authors chose to use RLOO instead of GRPO, which is the setup used in TruthRL. I think that will give a better comparison to previous work.
  2. Frankly speaking both the method and the theoretical proves are like an extension of exsiting work tailRL. Applying tailRL to the special setting here. Here is a question for the authors, what general take way or theoretical contribution of this paper beyond being a special case of tailRL?
  3. There are a few related work in the line of truthRL and Abstain-R1, the authos may consider descussing them (doesn't mean the authors need to compare them as baselines.) "Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor" "Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks".
Questions for authors

See weakness. I'm willing to reconsider the score if weaknesses are fairly addressed.

4 comments

Dear Reviewer b3LR,

Thank you for your feedback. We address your concerns below:

W1. GRPO vs. RLOO

We thank the reviewer for raising this point. We would like to clarify that our findings regarding training possibly collapsing hold for both GRPO and RLOO, as supported by the following experimental and theoretical evidence.

  1. Existing GRPO experiments: The SafeMaze experiments in Figure 3(b) and Appendix C.6 already use GRPO, matching the setup in TruthRL.
  2. Additional GRPO experiments: We have repeated the LLM-scale experiments in Figure 4(a) using GRPO instead of RLOO and updated the figure accordingly. We observe qualitatively similar behavior across both algorithms, with both exhibiting collapse toward fallback.
  3. Theoretical analysis: We have additionally extended our theoretical analysis to GRPO in Appendix A.4. Specifically, we establish that the gradient norm is upper-bounded by O(1−po)O(\sqrt{1-p_o}) as po→1p_o \to 1, where pop_o denotes the probability of outcome oo. For comparison, our corresponding bound for RLOO is O(1−po)O(1-p_o). Thus, despite the different normalization schemes, both estimators suffer from vanishing gradients near outcome saturation.

Together, these results demonstrate that the collapse phenomenon is not specific to RLOO. Rather, it persists under GRPO, both theoretically and empirically, supporting our broader argument that the optimization pathology arises from the underlying training objective.

W2. Contributions

While the final optimization objective is indeed a specialization of TailRL to ternary rewards, we would like to emphasize that our contributions are not, in fact, the objective itself. They are:

  1. Theoretical analysis of optimization dynamics. TailRL motivates its objective from the perspective of improving upper-tail coverage and Best-of-(k) performance. In contrast, our work studies the optimization dynamics of learning fallback behavior and establishes new theoretical results. Specifically, we prove that (i) the gradient norm of the policy gradient objective vanishes as O(1−po)O(1-p_o) when the policy saturates on any outcome (o), whereas (ii) the gradient of the GracefulRL objective remains Θ(1)\Theta(1) near undesirable fallback or error saturation, under appropriate regularity and non-degeneracy assumptions. These results establish a previously unexplored connection between likelihood-based objectives and vanishing optimization signals near suboptimal policies.

  2. General formulation and experimental breadth. Our work studies graceful failure as a general decision-making problem, rather than restricting it to uncertainty estimation or abstention in single-step prediction. Our formulation also doesn’t confine us to just language models and we investigate three distinct instantiations of the idea: navigating to a safe square in Maze navigation, abstaining on mathematical reasoning problems, and making sequential decisions in a multi-turn agentic environment (Minesweeper). Importantly, these results demonstrate that models can learn when and how to fall back from outcome-level rewards, without requiring explicit supervision specifying when fallback is appropriate.

  3. Novel experimental design and findings. To the best of our knowledge, our risk-conditioned training setup for mathematical reasoning in LLMs is novel. It enables a single model to adapt its behavior to different levels of risk sensitivity, providing a practical mechanism for controllable deployment without training separate models for each preference. Furthermore, our findings in Minesweeper reveal that improving graceful failure does not necessarily require sacrificing task success. In particular, we demonstrate that training for graceful failure can simultaneously increase success rates and reduce error rates, challenging the assumption that safety and task performance must always be in tension.

AuthorsOpenReview ↗

Rebuttal continued

W3. Discussion of Related Work

We appreciate the reviewer bringing these works to our attention and will discuss and cite them in the revised manuscript.

  1. Che et al., Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: This work studies abstention collapse under error-penalized RL and shows that both the reward gradient and the KL regularization gradient can vanish as the abstention policy saturates. This observation is related to our analysis of vanishing policy gradients. However, their proposed solution separates confidence estimation from abstention, training the model to produce mandatory confidence reports using a proper scoring rule. In contrast, GracefulRL directly optimizes the policy over success, fallback, and error outcomes using a log-barrier objective, without requiring an explicit confidence estimator. Furthermore, our framework extends beyond answer abstention to sequential decision-making tasks where fallback itself is a learned behavior.
  2. Tsvilodub et al., Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models: This work investigates improving abstention by encouraging efficient reasoning about whether a task contains sufficient information to be answered. Our work differs in both its formulation and scope. We consider abstention as one instantiation of the broader problem of graceful failure and is not confined to LLMs, which also encompasses taking safe actions in navigation and making risk-sensitive decisions in multi-turn agentic environments such as Minesweeper.

Dear Reviewer,

We believe that we have answered all your questions. We would appreciate if you could engage with us. We’ll be happy to respond to any further questions you might have!

I changed my score and hope this work to get accepted. I think this work is interesting and solid.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.