- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to train LLMs to fallback when they are unlikely to succeed. The paper first shows that simply assigning an intermediate reward could easily lead to a model that always falls back. To address this challenge, they proposed GracefulRL by using a log-probability objective to encourage both task success and error avoidance. Experiments show that the method can reduce errors while largely preserving task performance. The paper addresses an important problem and provides an interesting analysis of the challenges in learning fallback behavior.
Critical strengths
- The motivation of this paper is clear and the problem is practical. LLM should have a better understanding of its capability boundary.
- The proposed method is supported by theoretical analysis and experiments.
Critical weaknesses
- GracefulRL still collapses to always falling back in SafeMaze when fewer solutions are sampled during training. This raises concerns about its reliability on harder tasks, where successful trajectories may be even harder to obtain.
Questions for authors
- Can GracefulRL be applied to agentic tasks? Does multi-turn interaction affect the behavior of fallback?
- If we prompt LLM to fail gracefully once it is unsure about the solution, does GracefulRL still have a clear advantage over prompt engineering?
- How sensitive is GracefulRL to the fallback reward? Should we conduct tuning for each new task?
3 comments
Rebuttal continued
Q2. Prompt engineering
We thank the reviewer for raising this question. While prompt engineering can encourage models to abstain when uncertain, we find that prompting alone is insufficient to reliably elicit fallback behavior from models trained solely to maximize correctness. To investigate this, we conducted additional experiments on AIME using a 4B model trained with standard RLVR on correctness, without explicit fallback training. We evaluated five different prompting strategies: the GracefulRL prompting, confidence thresholding, explicit scoring instructions, error penalties, and self-verification. We varied the fallback reward from 0 to 0.8, with 360 generations per setting (90 problems × 4 rollouts).
Across all prompting strategies and reward settings, the model almost never abstains, with fallback rates remaining below 2% in every setting. Even when explicitly instructed to consider the cost of incorrect answers or verify its solution before responding, the model overwhelmingly attempts an answer. This suggests that simply instructing a model to abstain, even with explicit incentives for doing so, may not be sufficient to induce reliable fallback behavior after standard RLVR training.
Results are in this table. Each cell reports Success / Fallback / Error (%) over 360 generations (90 AIME problems × 4 rollouts).
| c | GracefulRL | Threshold | Scoring | Penalty | Self-check |
|---|---|---|---|---|---|
| 0.0 | 18.9 / 0.0 / 81.1 | 19.2 / 0.0 / 80.8 | 17.8 / 1.1 / 81.1 | 17.8 / 0.0 / 82.2 | 19.4 / 0.0 / 80.6 |
| 0.2 | 18.9 / 0.0 / 81.1 | 18.3 / 0.6 / 81.1 | 18.6 / 1.1 / 80.3 | 16.1 / 0.3 / 83.6 | 16.7 / 0.0 / 83.3 |
| 0.4 | 20.3 / 0.0 / 79.7 | 18.6 / 0.3 / 81.1 | 17.2 / 1.4 / 81.4 | 16.9 / 0.3 / 82.8 | 17.8 / 0.0 / 82.2 |
| 0.6 | 18.3 / 0.0 / 81.7 | 19.7 / 0.3 / 80.0 | 16.1 / 1.1 / 82.8 | 17.5 / 0.0 / 82.5 | 15.8 / 0.0 / 84.2 |
| 0.8 | 18.3 / 0.0 / 81.7 | 17.5 / 0.3 / 82.2 | 19.2 / 1.9 / 78.9 | 18.3 / 0.3 / 81.4 | 15.8 / 0.3 / 83.9 |
These findings are consistent with prior work. AbstentionBench [1] reports that reasoning fine-tuning degrades abstention performance by 24% on average, while OpenAI's o3 System Card [2] reports higher hallucination rates for o3 than o1 despite improved reasoning capabilities. These findings suggest that stronger general reasoning capabilities do not necessarily translate into more reliable abstention behavior.
We believe that prompt engineering is a valuable and complementary approach, as it can leverage models' existing capabilities to encourage appropriate fallback behavior. However, rather than relying solely on prompting to elicit this behavior, we believe that explicitly training models to acquire the capability would be better. Such training allows us to directly optimize the tradeoff between task success and costly errors, potentially leading to more robust and reliable behavior. This is particularly important in risk-sensitive applications, where incorrect decisions may have significant consequences.
[1] Kirichenko, P., Ibrahim, M., Chaudhuri, K., and Bell, S. J. (2025). AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. NeurIPS 2025 (Datasets and Benchmarks Track). https://arxiv.org/abs/2506.09038
[2] OpenAI (2025). OpenAI o3 and o4-mini System Card. Section 3.3: Hallucinations. https://openai.com/index/o3-o4-mini-system-card/
Dear Reviewer,
We believe that we have answered all your questions. We would appreciate if you could engage with us. We’ll be happy to respond to any further questions you might have!
Rebuttal
We thank the reviewer for their feedback. We address their concerns below:
W1 and Q3. Behavior in the low rollout regime and sensitivity
GracefulRL is substantially more stable than RLOO/GRPO across different settings, and its sensitivity to the rollout budget follows a predictable pattern: higher fallback rewards require more rollouts. Specifically, for SafeMaze at c=0.8, GracefulRL collapses with 16 and 32 rollouts but performs well with 64 rollouts as mentioned in Appendix C.4. For lower values of c, 16 rollouts is enough. Intuitively, this follows from the fact that for N rollouts, the population objective is only an approximation to JGracefulRL=(1−c)logpS+clog(1−pE) where the approximation gets tighter with a higher N.
As c increases, the objective places greater relative emphasis on avoiding errors, making it increasingly important to accurately estimate the success-promoting component. With a limited rollout budget, this component may be poorly estimated, particularly when successful trajectories are rare. Conclusively, our algorithm is stabler and needs less tuning compared to RLOO/GRPO, and even when it does, there is a predictable recipe.
Q1. Agentic tasks
GracefulRL is already evaluated on multi-turn, interactive environments, including both SafeMaze and Minesweeper. In particular, Minesweeper requires the LLM agent to make a sequence of decisions while interacting with the environment, where an incorrect action can lead to failure. These experiments demonstrate that GracefulRL can effectively learn when to fall back in sequential decision-making settings, rather than being limited to single-turn tasks. More generally, applying GracefulRL to other agentic tasks requires defining an appropriate fallback behaviour for the environment, such as abstaining from an action, requesting human intervention, or reverting code changes. Given a suitable fallback state and corresponding outcome-based rewards, we expect GracefulRL to generalize to a broader range of agentic tasks.