- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose RISE, a test-time scaling framework that combines complete-trajectory acceptance with event-level recovery guidance
Critical strengths
1.This paper has the least AI-generated content in my reviewed papers. I can tell the authors devote effort to writing. 2.The results and performance gain are compelling, ablation study shows that all components contribute to the performances. 3.The problem itself is interesting, recovering can not offer self-improvement, and agents just make the same mistake maybe after the context window does not contain the history of failure .
Critical weaknesses
1.First of all, the text on the figures and charts can not be selected. Are these png files? Should better use the PDF format.
2.Latex mistakes. Page 13, Statebench is wrongly cited with “??”. Compiling the paper should at least results in a warning. Some parts of writing are not clear. What is “a frontier”, a model combining public evidence, current credit, and adapter-maintained history? Also, “To extract information from the candidate that was rejected,” this sentence is not clear: what is “information”? steps of the trajectories? Correct statement? Why extracting information from rejected ones? Authors should add a sentence for explanation, like “rejected candidates also contain useful information”.
I know that there are a lot of posts on social media complaining reviewers are ignorant, me as an author complain very often as well. But I really think the ability of demonstrating the work precisely and clearly to reviewers also matters.
3.DeepSeek-V4.1-Flash is used for evaluations on its own trajectories, which can induce self-preference biases.
4.Two baselines are tested, with only one test-time scaling method, more should be added. Some related work is missing:
[1]Correct Predictions, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis. ArXiv 2604.14121.
5.The authors should do comparisons between your work and others in the Related work Section, to better emphasize your motivation, advantages and contributions.
6.What is a “Recover need Fw”? Does it mean If the agent does not recover from current step, it will definitely fail? I know that some errors will not lead to failure so they are not that important, but please cite those work and talk about the concepts you defined.
7.The author claims that “we are the first to characterize failed recovery as a distinct failure mode in tool-using language models.”However, for the topic without detailed comparisons, I think the paper Failure as a process: An anatomy of cli coding agent trajectories studies the similar problem.
Questions for authors
See weeknesses.
3 comments
For question 4, I think the author does not understand self-preference biases, instead trying to convince me that DeepSeek is an eligible model to be a LLM judge….
DeepSeek’s self-preference
Thank you for clarifying. DeepSeek’s self-preference can influence the scores on metrics it judges. RISE and Best-of-4 are evaluated using the same procedure, so their scores provide a comparison under that shared protocol. The observed gap may reflect both differences in task performance and the judge’s preferences. On AppWorld, the official environment checks determine TGC and SGC.
Evaluation Procedures, Recovery Guidance, and Related Work
Q1: Why is figure text unselectable?
A1: We created the figures in PowerPoint and exported them as PNG raster images. The labels are part of the images, so they cannot be selected as text in the PDF.
Q2: What is the unresolved StateBench citation on page 13?
A2: The StateBench citation in Appendix B (p. 13, line 698) displays as “??” due to a citation formatting error. We will correct it in the revised manuscript.
Q3: What is a frontier, and why learn from a rejected trajectory?
A3: A frontier is the instruction that tells the agent what to check in its next rollout. The adapter builds it from observable risk cues, feedback from prior comparisons, and relevant history (Sec. 3.2). A rejected rollout can reveal a specific action to avoid, such as retrying an authenticated call without passing the credential. RISE uses the retained trajectory to identify patterns to preserve and the rejected one to identify patterns to avoid. The feedback records structures such as tool operations, argument keys, and result categories. The next rollout starts in a fresh environment and obtains its own argument values. For instance, guidance can identify the credential argument that a call needs, and the agent obtains the credential for that execution (Secs. 3.2 and 5.2). Appendix D also gives a StateBench example in which changing the rejected trajectory changes which read operation is marked for preservation, while the selected trajectory stays the same. In Appendix D, RISE keeps B and rejects C, and uses the comparison to give D new feedback and an issue to check.
Q4: Could DeepSeek evaluation favor its own trajectories?
A4: AppWorld's official environment checks determine whether each returned trajectory meets the task and scenario goals. Those outcomes yield TGC and SGC. On StateBench, DeepSeek judges task requirements and UX, and separate rules check state requirements. On OccuBench, DeepSeek simulates the environment and verifies task completion. RISE and Best-of-4 use the same final evaluation procedure within each of these benchmarks.
Q5: Which additional test-time scaling baseline is most informative?
A5: We ran a Reflexion-style baseline on StateBench with the same DeepSeek model, tasks, seeds, and four-attempt cap. If the first attempt fails, the agent reads a reflection on that failure before its second attempt. PASS⁵ measures completion across all five independent runs (Table 1), and UX is rated on a 1–5 scale.
RISE's point estimates are higher by 2.27 pp, 2.00 pp, and 0.1345, respectively.
Q6: How does CRAFT relate to RISE?
A6: CRAFT synthesizes a reasoning trace from the steps that candidate traces agree on. RISE uses public tool events to guide a fresh execution and returns one executed trajectory. CRAFT
Q7: How should Related Work clarify the motivation and contribution?
A7: Best-of-N selects among independent executions. Reflexion-style passes a critique into a retry. RISE uses recovery cues and source-linked action patterns to guide a fresh tool execution, while keeping a complete trajectory as the current selection. This design addresses a specific failure pattern: several attempts can overlook the same prerequisite even after encountering the same error. Appendix E gives two further points of comparison. SE-Agent revises earlier trajectories, and PIVOT uses discrepancies between plans and executions to inform acceptance. RISE uses public tool events both to locate a recovery issue and to derive feedback from a comparison between completed executions. A5 tests this design against reflection-based retry on StateBench under the same four-attempt cap.
Q8: Does a recovery need Fw imply inevitable task failure without recovery?
A8: In Eq. 1, Fw denotes a local repair need. For example, a failed authentication step can block a required read, while the task may still succeed through a later repair or another route. The local evidence might be a failed call, inconsistent state, or a problem reported by the user (Sec. 2). For example, an expected business rejection may simply be the correct response to an invalid request. FRD requires three further conditions: an actual repair attempt, an incorrect decision, and an attributable adverse consequence. A later success can therefore coexist with an earlier FRD. Failure as a Process
Q9: How does RISE build on prior work on failed recovery?
A9: Zhao et al. analyze how failures arise, evolve, and recover in CLI coding-agent trajectories. RISE's contribution is an episode-level criterion for attributing an adverse outcome to a repair decision, together with an algorithm that uses observable recovery cues to guide a fresh tool rollout. Failure as a Process