acceptodds
Withdrawn from ICLR 2027

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measure recovery by comparing which action is selected as best across independent runs, a quantity known as set agreement. However, set agreement is a purely ordinal measure that records which action wins without reflecting the absolute level of performance. When all actions fail, they tie at zero reward, and independent runs produce the same tied set with high probability, creating an illusion of stability that masks near-zero recovery success. We formalize this limitation through a set-path symmetry result, proving that for equal-cost Bernoulli actions the success probabilities (0.9, 0.8) and (0.2, 0.1) yield identical best-action-set distributions at every sample size. No procedure based solely on which action wins can distinguish these two regimes. We further prove that certifying exact population ties is impossible in finite time, and that the assignment of outcomes to checkpoints carries information beyond marginal outcome distributions. The pooled success probability is the missing scalar that resolves the ordinal ambiguity. Experiments on 864 frozen RecoveryBench episodes and two planning cohorts totaling 3,456 responses confirm the theoretical predictions. Agreement and held-out quality can move in opposite directions, and permuting checkpoint-to-action bindings changes 8 to 13% of cell-level conclusions. Based on these findings, we propose reporting four diagnostic quantities: agreement, all-zero fraction, held-out success, and pooled success.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, failure recovery

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_EjksOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that set agreement, the frequency with which independent runs select the same best recovery action at a failed agent checkpoint, can be high even when recovery rarely succeeds, because universal failure produces ties. It proves a set-path symmetry for two equal-cost Bernoulli actions, together with results on certifying ties and best actions, and evaluates agreement alongside held-out success on frozen RecoveryBench episodes, two planning cohorts with three GPT-5.6 models, and a Qwen3-8B panel. It recommends reporting agreement together with the all-zero fraction, held-out success and pooled success.

Critical strengths
  1. The paper raises a practical caution that is worth making: stability numbers such as set agreement are easy to over-read on hard tasks where most attempts fail, and the two-action coupling in Section 4.1 illustrates this concisely.
Critical weaknesses
  1. I noticed that the main text appears to run beyond the ICLR 2027 page limit. The Author Guidelines state that at submission "the main text should be 9 pages or fewer" and that "Papers with main text beyond the page limit will be desk-rejected", and here the end of the Limitations paragraph in Section 6 continues onto page 10 (lines 486-489). As I understand the guidelines, only the references, appendices and the reproducibility, ethics and AI-use statements are exempt, so I believe this would fall under the desk-rejection policy. I also had some concerns about the scope of the theory and the interpretation of the experiments, but since this issue appears decisive, I have kept my review focused on it.
Questions for authors
  1. [Clarification] I may have misjudged where the main text ends, so I would be grateful if the authors could confirm whether the text on page 10 was intended to fall within the 9-page limit.
Reviewer_pf3tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that set agreement, the frequency with which independent runs select the same best recovery action, does not reflect absolute recovery success. Its main formal result is a set-path symmetry: for two equal-cost Bernoulli actions, (p1,p2)(p_1, p_2) and (1−p2,1−p1)(1-p_2, 1-p_1) produce the same distribution of best-action sets. The paper also states that exact ties cannot be certified in finite time, and that permuting checkpoint-to-action bindings can change cell-level conclusions. Experiments cover 12 RecoveryBench checkpoints with three GPT-5.6 models, two 24-checkpoint planning cohorts, and a Qwen3-8B panel. Based on these, the paper recommends reporting agreement, all-zero fraction, held-out success, and pooled success together.

Critical strengths
  1. The observation that shared failure inflates agreement is clearly illustrated on RecoveryBench (Table 1). There, all agreement for Luna at k=1k=1 comes from all-zero comparisons.

  2. The four recommended diagnostics are cheap to compute from data that is already collected.

Critical weaknesses
  1. The submission violates the page limit. The ICLR 2027 Author Guidelines require the main text to be at most 9 pages at submission and state that papers beyond the limit will be desk-rejected.

  2. The contribution is thin and the experiments do not support the claims. The core point, that an ordinal metric does not capture absolute success, is expected; the symmetry proof is a one-line coupling (Y1†=1−Y2Y^\dagger_1 = 1 - Y_2), and the tie-certification limit is standard in the best-arm identification literature the paper cites. On RecoveryBench, 487 of 864 episodes are model-protocol failures and only 364 reach the verifier (Table 49), so the 1.4% success and 85.4% agreement mostly reflect a harness where most episodes never run, not recovery behavior. The Qwen3-8B panel is cited in the introduction as evidence that the pattern generalizes, yet its agreement does not decline with kk (change +0.014+0.014, interval [−0.135,0.167][-0.135, 0.167], Table 46). The permutation result (4 and 6 of 48 cells changed) only shows that scrambling data changes conclusions, and the claim that this exceeds typical measurement noise is unsupported. Finally, the 30+ page appendix contains material unrelated to the main claims (release gates, dispatcher logic, EC2 retrieval) and refers to an undefined system "OrchLoop" (Appendix C), which suggests content assembled from another project.

Questions for authors
  1. Section 1 says set agreement is a widely used stability metric in RecoveryBench and ToolMaze. Where in those benchmarks is set agreement used as a primary metric or interpreted as evidence of action quality? If benchmarks already report success rates, the problem addressed here may not exist in practice.

  2. What do Table 1 and the Spearman correlations in Table 50 look like when restricted to the 364 episodes with native verifier observations? If the high-agreement, low-success pattern weakens there, the main RecoveryBench result reflects protocol failures in the harness rather than a property of set agreement.

  3. In the Qwen3-8B panel, agreement does not decline as kk grows, while held-out success is 37.9%. How is this consistent with the tie-resolution mechanism the paper proposes, and on what basis does the introduction say this panel confirms the pattern generalizes?

Reviewer_pHXQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the limitations of "Set Agreement", which is used to evaluate the capability of LLM agent failure recovery. It theoretically shows that the identical best-action set could appear from different success probabilities. Through the experiments, it also shows that high agreement does not guarantees the successful recovery. Based on these findings, it suggests additional diagnostic metrics for more reliable evaluation.

Critical strengths
  • The paper addresses the important problem: reliably evaluting LLM agent's failure recovery capability.
  • It provide both theoretical and empirical analyses.
  • Extensive experiment support the theoretical analyses.
  • It suggests practical evaluation guideline without additional data collection.
Critical weaknesses

Page limit violation observed

  • It is obvious that high agreement does not guarantee high performance. Agreement is just a metric for evaluation stability, not actual recovery quality. The research question lacks novelty and significance.

  • It claim that existing works misinterpret the high set agreement leads to high recovery quality. However, this claim is not sufficiently justified. So, it is unclear how important the problem is in the research field.

  • RecoveryBench relies on only 12 checkpoints, which indicates the limited diversity and generalization.

  • The proposed solution is just additionally using widely used metrics, which is more like a suggestion rather than solution.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.