acceptodds
Under review as a conference paper at ICLR 2027

RISE: Recovery-Informed Scaling via Events for Tool-Using Language Models

Abstract

Tool-using language models operate on mutable external state, where detecting an anomaly does not ensure that a subsequent action restores progress. We study Failed Recovery Decision: an incorrect decision during recovery with an attributable adverse outcome. Its retrospective adjudication is separated from the Recovery-risk Disagreement Signal, which uses public execution evidence. Building on this separation, \method uses public evidence between completed rollouts to direct fresh executions, retain an accepted trajectory, and construct source-constrained event credit from eligible candidate comparisons. Selection determines which execution to retain; bilateral credit identifies structures to preserve or avoid and issues to recheck. Thus, a rejected proposal can guide a later execution without replacing the accepted trajectory. Across four benchmarks and three models, \method improves over Best-of- under a shared four-rollout limit, including increases from 82.73 to 91.37 on AppWorld Test-Challenge scenario completion and from 70.68 to 76.96 on OccuBench PASS@1 with GPT-5.6-Sol.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool-Using LLMs, Stateful Agents, Test-Time Scaling, Recovery Decisions, Event-Based Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fhHzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose RISE, a test-time scaling framework that combines complete-trajectory acceptance with event-level recovery guidance

Critical strengths

1.This paper has the least AI-generated content in my reviewed papers. I can tell the authors devote effort to writing. 2.The results and performance gain are compelling, ablation study shows that all components contribute to the performances. 3.The problem itself is interesting, recovering can not offer self-improvement, and agents just make the same mistake maybe after the context window does not contain the history of failure .

Critical weaknesses

1.First of all, the text on the figures and charts can not be selected. Are these png files? Should better use the PDF format.

2.Latex mistakes. Page 13, Statebench is wrongly cited with “??”. Compiling the paper should at least results in a warning. Some parts of writing are not clear. What is “a frontier”, a model combining public evidence, current credit, and adapter-maintained history? Also, “To extract information from the candidate that was rejected,” this sentence is not clear: what is “information”? steps of the trajectories? Correct statement? Why extracting information from rejected ones? Authors should add a sentence for explanation, like “rejected candidates also contain useful information”.

I know that there are a lot of posts on social media complaining reviewers are ignorant, me as an author complain very often as well. But I really think the ability of demonstrating the work precisely and clearly to reviewers also matters.

3.DeepSeek-V4.1-Flash is used for evaluations on its own trajectories, which can induce self-preference biases.

4.Two baselines are tested, with only one test-time scaling method, more should be added. Some related work is missing:

[1]Correct Predictions, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis. ArXiv 2604.14121.

5.The authors should do comparisons between your work and others in the Related work Section, to better emphasize your motivation, advantages and contributions.

6.What is a “Recover need Fw”? Does it mean If the agent does not recover from current step, it will definitely fail? I know that some errors will not lead to failure so they are not that important, but please cite those work and talk about the concepts you defined.

7.The author claims that “we are the first to characterize failed recovery as a distinct failure mode in tool-using language models.”However, for the topic without detailed comparisons, I think the paper Failure as a process: An anatomy of cli coding agent trajectories studies the similar problem.

Questions for authors

See weeknesses.

3 comments
AuthorsOpenReview ↗

Evaluation Procedures, Recovery Guidance, and Related Work

Q1: Why is figure text unselectable?

A1: We created the figures in PowerPoint and exported them as PNG raster images. The labels are part of the images, so they cannot be selected as text in the PDF.

Q2: What is the unresolved StateBench citation on page 13?

A2: The StateBench citation in Appendix B (p. 13, line 698) displays as “??” due to a citation formatting error. We will correct it in the revised manuscript.

Q3: What is a frontier, and why learn from a rejected trajectory?

A3: A frontier is the instruction that tells the agent what to check in its next rollout. The adapter builds it from observable risk cues, feedback from prior comparisons, and relevant history (Sec. 3.2). A rejected rollout can reveal a specific action to avoid, such as retrying an authenticated call without passing the credential. RISE uses the retained trajectory to identify patterns to preserve and the rejected one to identify patterns to avoid. The feedback records structures such as tool operations, argument keys, and result categories. The next rollout starts in a fresh environment and obtains its own argument values. For instance, guidance can identify the credential argument that a call needs, and the agent obtains the credential for that execution (Secs. 3.2 and 5.2). Appendix D also gives a StateBench example in which changing the rejected trajectory changes which read operation is marked for preservation, while the selected trajectory stays the same. In Appendix D, RISE keeps B and rejects C, and uses the comparison to give D new feedback and an issue to check.

Q4: Could DeepSeek evaluation favor its own trajectories?

A4: AppWorld's official environment checks determine whether each returned trajectory meets the task and scenario goals. Those outcomes yield TGC and SGC. On StateBench, DeepSeek judges task requirements and UX, and separate rules check state requirements. On OccuBench, DeepSeek simulates the environment and verifies task completion. RISE and Best-of-4 use the same final evaluation procedure within each of these benchmarks.

Q5: Which additional test-time scaling baseline is most informative?

A5: We ran a Reflexion-style baseline on StateBench with the same DeepSeek model, tasks, seeds, and four-attempt cap. If the first attempt fails, the agent reads a reflection on that failure before its second attempt. PASS⁵ measures completion across all five independent runs (Table 1), and UX is rated on a 1–5 scale.

MethodPASS@1PASS⁵UX
Reflexion-style70.40%46.67%4.0154
RISE (Table 1)72.67%48.67%4.1499

RISE's point estimates are higher by 2.27 pp, 2.00 pp, and 0.1345, respectively.

Q6: How does CRAFT relate to RISE?

A6: CRAFT synthesizes a reasoning trace from the steps that candidate traces agree on. RISE uses public tool events to guide a fresh execution and returns one executed trajectory. CRAFT

Q7: How should Related Work clarify the motivation and contribution?

A7: Best-of-N selects among independent executions. Reflexion-style passes a critique into a retry. RISE uses recovery cues and source-linked action patterns to guide a fresh tool execution, while keeping a complete trajectory as the current selection. This design addresses a specific failure pattern: several attempts can overlook the same prerequisite even after encountering the same error. Appendix E gives two further points of comparison. SE-Agent revises earlier trajectories, and PIVOT uses discrepancies between plans and executions to inform acceptance. RISE uses public tool events both to locate a recovery issue and to derive feedback from a comparison between completed executions. A5 tests this design against reflection-based retry on StateBench under the same four-attempt cap.

Q8: Does a recovery need Fw imply inevitable task failure without recovery?

A8: In Eq. 1, Fw denotes a local repair need. For example, a failed authentication step can block a required read, while the task may still succeed through a later repair or another route. The local evidence might be a failed call, inconsistent state, or a problem reported by the user (Sec. 2). For example, an expected business rejection may simply be the correct response to an invalid request. FRD requires three further conditions: an actual repair attempt, an incorrect decision, and an attributable adverse consequence. A later success can therefore coexist with an earlier FRD. Failure as a Process

Q9: How does RISE build on prior work on failed recovery?

A9: Zhao et al. analyze how failures arise, evolve, and recover in CLI coding-agent trajectories. RISE's contribution is an episode-level criterion for attributing an adverse outcome to a repair decision, together with an algorithm that uses observable recovery cues to guide a fresh tool rollout. Failure as a Process

For question 4, I think the author does not understand self-preference biases, instead trying to convince me that DeepSeek is an eligible model to be a LLM judge….

AuthorsOpenReview ↗

DeepSeek’s self-preference

Thank you for clarifying. DeepSeek’s self-preference can influence the scores on metrics it judges. RISE and Best-of-4 are evaluated using the same procedure, so their scores provide a comparison under that shared protocol. The observed gap may reflect both differences in task performance and the judge’s preferences. On AppWorld, the official environment checks determine TGC and SGC.

Reviewer_4f2eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

RISE is a test-time scaling method for tool-using agents that targets failed recoveries (the agent notices an error but fixes it wrong). Instead of Best-of-N's independent attempts, it runs attempts one after another, guiding each with risk signals and keep/avoid patterns from earlier attempts, and keeps the best one. With 4 attempts on StateBench, AppWorld, OccuBench and DeepPlanning across 3 models, it beats Best-of-N on point estimates.

Critical strengths
  • The problem is real, agents often detect an error and then fix it wrong (the AppWorld example in Fig. 1, logging in and then retrying without the token, is a good illustration).
  • The argument that Best-of-N only helps with random failures and not with repeated ones is intuitive, and using feedback between attempts to target repeated failures makes sense.
Critical weaknesses
  • The gains over Best-of-N are not significant. The only confidence intervals are in the appendix (Table 7, DeepSeek only), and RISE vs Best-of-N includes zero on every benchmark: +0.53 [-2.67, 3.73] on StateBench, +1.57 [-3.66, 6.81] on OccuBench, +1.68 [-0.96, 4.08] on AppWorld. The main tables only show point estimates.
  • Compute is matched only in number of attempts. RISE is sequential (each attempt waits for the previous one) while Best-of-N can run in parallel, so wall-clock time is roughly 4x. RISE also adds selector, credit and guidance LLM calls between attempts, and longer prompts. Time, tokens and cost are never reported, so the "equal cost" claim doesn't hold.
  • Weak baselines. Only a single attempt and Best-of-N (N=4) are compared. RISE is basically retrying with structured feedback, but no iterative method is compared.
  • Hand-built for each benchmark. The risk signal, event view, comparison rule and credit rule are custom per benchmark (Table 6), so it's not clear how well it transfers to new ones.
  • Hard to read, too much jargon ("Recovery-risk Disagreement Signal", "source-constrained bilateral event credit", "incumbent", "frontier", "material disagreement") for a fairly simple idea: retry with feedback about what went wrong, keep the best attempt.
Questions for authors
  1. Can you report wall-clock time, tokens and cost for RISE and Best-of-N, and compare at matched tokens or latency?
  2. How does RISE compare to iterative methods like Reflexion or Self-Refine under the same budget?
2 comments
AuthorsOpenReview ↗

Paired Results, Benchmark Adapters, and Iterative Baselines

Q1: How certain are the gains over Best-of-N?

A1: On DeepPlanning Travel with GPT-5.6-Sol, RISE scores higher than Best-of-N on Composite in both languages, with paired 95% confidence intervals above zero. Each language has 120 pairs. Differences and intervals are in score points.

ComparisonChineseEnglish
RISE−Vanilla+9.74 [8.75,10.75]+10.03 [9.09,10.98]
RISE−Best-of-N+6.09 [5.20,6.96]+6.46 [5.55,7.33]
Best-of-N−Vanilla+3.65 [2.68,4.64]+3.57 [2.62,4.52]

In Table 4, RISE raises Composite from 40.32 to 46.41 in Chinese and from 42.11 to 48.57 in English.

Table 7 reports task completion differences for DeepSeek. All three DeepSeek intervals for RISE vs. Best-of-N include zero: StateBench +0.53 pp [−2.67,3.73], OccuBench +1.57 pp [−3.66,6.81], and AppWorld Test-Challenge TGC +1.68 pp [−0.96,4.08]. On that AppWorld split, Best-of-N vs. Vanilla also includes zero (+0.24 pp [−2.16,2.88]).

Q2: How do sequential execution and longer prompts affect the comparison?

A2: RISE uses earlier results to guide later attempts. Best-of-4 can run its independent candidates in parallel within a task, but our DeepSeek AppWorld experiment executed them serially and parallelized across tasks. A fourfold latency ratio assumes full within-task parallelism, similar rollout durations, and negligible overhead. The experiments in Sec. 4 match the four-rollout cap. A6 reports token and call counts for generation, selection, and feedback.

Q3: Why compare only against Vanilla and Best-of-N?

A3: We added a StateBench Reflexion-style baseline with the same four-attempt cap (A7). On OccuBench, Table 5 holds the RISE loop fixed and varies the feedback. PASS@1 drops from 63.35% to 60.73% when a free-form summary of the two candidates replaces structured feedback. Removing the explicit recovery target yields 53.14%, and withholding event credit yields 57.07%. Best-of-N scores 61.78% in this OccuBench comparison. The full method exceeds all three feedback variants at the same rollout cap. The StateBench comparison in A7 tests a separate reflection-based retry procedure.

Q4: Which parts of RISE are benchmark-specific?

A4: All four benchmarks use the same search loop (Algorithm A.1). Each adapter defines how to represent events, flag risks, decide when to compare candidates, and construct feedback (Table 6). StateBench compares the content and order of transactions and constructs credit by rule. AppWorld compares ordered API signatures. OccuBench compares transaction content and order in a StateBench-compatible view of actions and observations. DeepPlanning checks for differences in ordered tool, argument-key, and result-category tuples or in final-answer text. OccuBench, AppWorld, and DeepPlanning share the process of checking the source of each nominated event and reducing it to its structural pattern. Porting RISE requires defining these adapter components for the new tool interface.

Q5: What does RISE add to retrying with feedback?

A5: Fig. 1 shows why another attempt alone can be insufficient: the agent logs in after an authentication failure, then retries without the new credential. RISE tells the next attempt what to check. If the new trajectory differs in a way the adapter considers relevant, RISE compares the two executions to decide which one to retain and which action patterns to preserve or avoid on the next attempt.

Q6: Can you compare tokens, cost, and wall-clock time, including at matched budgets?

A6: We evaluated both methods on the same 417 DeepSeek AppWorld Test-Challenge tasks, with up to four complete rollouts per task. For the requests included in the comparison, RISE used 384,796,560 tokens and 25,756 calls, versus 379,257,918 tokens and 28,154 calls for Best-of-4 (+1.46% tokens, −8.52% calls). The totals cover generation, selection, and associated feedback. RISE achieved 89.69% TGC versus 88.01% (+1.68 pp, 95% CI [−0.96,4.08]). Across 1,668 candidates per method, cumulative candidate execution time was 1,241,428 s for RISE and 1,628,464 s for Best-of-4. These totals count overlapping and reused candidates.

Q7: How does RISE compare with Reflexion or Self-Refine under the same budget?

A7: On StateBench, Reflexion-style uses the same DeepSeek model, tasks, seeds, and four-attempt cap as RISE. If the first attempt fails, the agent reads a reflection on that failure before its second attempt. PASS⁵ measures completion across all five independent runs (Table 1), and UX is rated on a 1–5 scale.

MethodPASS@1PASS⁵UX
Reflexion-style70.40%46.67%4.0154
Best-of-4 (Table 1)72.13%46.67%4.1428
RISE (Table 1)72.67%48.67%4.1499

Compared with Reflexion-style, RISE's point estimates are higher by 2.27 pp in PASS@1, 2.00 pp in PASS⁵, and 0.1345 in UX. Best-of-4 and Reflexion-style both score 46.67% on PASS⁵, while RISE reaches 48.67% under the same attempt cap.

Thank you for the detailed response. The token and call counts on AppWorld clearly show that RISE does not use extra cost compared to Best-of-N (+1.46% tokens, -8.52% calls), so I withdraw that concern, although it would be good to see this for the other benchmarks and models too. The DeepPlanning results with GPT-5.6-Sol are also statistically significant, which is a nice addition. However, this is the only setting where the gains over Best-of-N are significant, all three DeepSeek comparisons still include zero, and the paper claims improvements across all benchmarks and models. Since the data is available, CIs for the other models and benchmarks should be reported. The new Reflexion-style baseline is only on StateBench with one model and no CIs, and RISE vs Best-of-N there is still +0.53 [-2.67, 3.73]. Moreover, in the OccuBench ablation every feedback variant performs worse than Best-of-N, which suggests that the sequential loop only works when the full benchmark-specific setup is in place, and the response confirms that porting RISE requires defining new adapters for each tool interface, so I am still unsure how it generalizes.

My main concerns are still not addressed. The response does not report the wall-clock latency per task, only the cumulative candidate execution time, and running Best-of-N serially is an implementation choice, it can be parallelized. Using your own numbers, RISE takes around 50 minutes per task (1,241,428 s / 417 tasks) with attempts that must run one after another, while an average Best-of-4 candidate takes around 16 minutes (1,628,464 s / 1,668 candidates) and the four can run in parallel, so RISE is still several times slower. Also, the question of what happens with different N is not answered at all. Best-of-N can easily scale to 32 or 64 parallel samples, while RISE would need 32 or 64 sequential rounds, and the paper only tests N=4. Without a comparison of latency and of different N, it is not clear when RISE would be preferred over Best-of-N, so I will keep my score.

Reviewer_beybOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RISE directs fresh tool-use executions using recovery-risk evidence, maintains an accepted trajectory, and extracts source-constrained feedback from candidate comparisons. It distinguishes retrospectively adjudicated recovery failures from signals available during execution and evaluates the resulting search procedure against independent sampling under a four-rollout limit.

Critical strengths

The separation between retrospective failure labels and online evidence is conceptually useful and avoids directly feeding evaluation labels into search. Results span four benchmarks and three models, while the internal ablations examine recovery focus and credit construction. Appendix B provides paired uncertainty estimates, and Appendix C explicitly acknowledges that failure witnesses do not establish exhaustive negative labels; these qualifications improve the transparency of the evaluation.

Critical weaknesses

Matching complete-rollout counts does not establish the claimed efficiency comparison in terms of total inference compute. Frontier generation, candidate assessment, and feedback processing can consume additional resources, even when credit nominations share a selector response. Without total token, call, or latency accounting, the results establish an advantage under a rollout-count constraint but leave the benefit at comparable total cost unresolved.

The strength of the recovery-specific evidence is also uncertain. All three reported DeepSeek comparisons against Best-of-N in Table 7 have confidence intervals containing zero; this does not negate the larger point estimates for other models, but those results lack the corresponding uncertainty analysis. The FRD screening additionally lacks independent blinded agreement, and “no witness” is not a verified negative. Consequently, reduced detected FRD counts do not yet establish the magnitude of a reduction in actual recovery failures. The mechanism is promising, but comparable-cost effectiveness and the recovery-specific interpretation remain insufficiently established. Existing-run analysis and clarification of the audit procedure could change the assessment.

Questions for authors
  1. Analysis of existing logs: What total inference costs include guidance and selection overhead, and how do they compare with Best-of-N? This would determine whether the rollout-matched improvement also supports the efficiency claim.
  2. Clarification: How were FRD witnesses adjudicated, and what existing safeguards make detection comparable across methods? Given that “no witness” is not a verified negative, this would affect how strongly reduced witness counts support a recovery-specific improvement.
2 comments
AuthorsOpenReview ↗

Inference Usage and Evidence on Recovery Failures

Q1: Does matching four rollouts establish an efficiency advantage?

A1: Section 4 caps both methods at four complete rollouts per task. Best-of-4 uses those slots for independent candidates. RISE uses one initial rollout and up to three guided proposals. A3 reports the measured token and call counts. RISE uses evidence from completed rollouts to identify what the next attempt should check. In Sec. 5.2, the guidance flags a failed authenticated read. The next attempt consults the tool specification, supplies the required credential, and passes all official checks. The adapter first checks for differences in the public event views (Sec. 3.3). On AppWorld, this check uses ordered API signatures. If the views meet the adapter's equivalence criterion, RISE keeps the current trajectory and skips the selector call. Its response also nominates event credit on OccuBench, AppWorld, and DeepPlanning. StateBench assigns credit by rule (Appendices A and D).

Q2: What do the paired results and FRD screening establish?

A2: Appendix C counts returned trajectories with a detected Failed Recovery Decision (FRD). The table reports the supplementary screening results from Table 8.

BenchmarkVanillaBest-of-NRISE
StateBench40/75039/75022/750
OccuBench15/38211/38210/382
AppWorld Test-Challenge68/41740/4173/417

On all three benchmarks, RISE returns fewer trajectories with a detected FRD than Best-of-N. Appendix C also reports completion on tasks where a baseline had a detected FRD. On the 40 AppWorld tasks where screening detected FRD for Best-of-N, RISE completes 39 tasks and Best-of-N completes 36. On the 68 tasks where screening detected FRD for Vanilla, RISE completes 65 and Vanilla completes 66. The OccuBench ablation separately tests the role of recovery guidance under a fixed rollout budget. Withholding the explicit recovery target lowers PASS@1 from 63.35% to 53.14%. Blocking event credit from reaching later attempts lowers it to 57.07%. These results support the value of carrying targeted recovery information into later rollouts in that setting.

Table 7 addresses overall task completion. On AppWorld Test-Challenge, the TGC difference is +1.68 pp (95% CI [−0.96,4.08], 10,000 scenario-grouped paired bootstrap resamples). Among the 417 paired task outcomes, 18 favor RISE, 11 favor Best-of-N, and 388 agree. For DeepSeek, the RISE vs. Best-of-N intervals include zero on all three benchmarks. On StateBench the difference is +0.53 pp (95% CI [−2.67,3.73]), and on OccuBench it is +1.57 pp (95% CI [−3.66,6.81]). On the same DeepSeek AppWorld split, the interval for Best-of-N vs. Vanilla also includes zero (+0.24 pp, 95% CI [−2.16,2.88]). In the GPT-5.6-Sol DeepPlanning Travel analysis (120 pairs per language), RISE's Composite score exceeds Best-of-N by 6.09 points in Chinese (95% CI [5.20,6.96]) and 6.46 points in English (95% CI [5.55,7.33]).

Q3: How much inference did RISE and Best-of-4 use?

A3: On the same 417 DeepSeek-v4.1-Flash AppWorld Test-Challenge tasks, both methods had a cap of four complete rollouts.

MethodInput tokensOutput tokensCalls
Best-of-4374,165,0105,092,90828,154
RISE380,247,9784,548,58225,756

Across the requests included in this comparison, RISE used 1.46% more tokens and made 8.52% fewer calls. The totals include candidate generation, selection, and related guidance and feedback. The net change in token use combines 6,082,968 more input tokens with 544,326 fewer output tokens. On these tasks, RISE achieved 89.69% TGC and Best-of-4 achieved 88.01% (+1.68 pp, 95% CI [−0.96,4.08]).

Q4: How did we identify FRD, and how comparable is detection across methods?

A4: We screened the final trajectory for each matched task and seed using the same criteria across methods (Appendix C). A trajectory counts once if it contains an episode with a need to recover, an actual recovery attempt, a decision that conflicts with the available evidence, and an adverse outcome attributable to that decision. The screen starts with a visible recovery action, such as a retry, a corrective call, or a verification step. On AppWorld, an API failure or traceback helps locate the relevant episode. The evidence for the repair and its consequence can span multiple calls (Appendix C). The label also requires evidence linking the repair decision to the adverse outcome, even if the task later succeeds.

Dear Reviewer,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.