acceptodds
Under review as a conference paper at ICLR 2027

Same Agent, Different Conclusion: Why Recovery-Method Evaluation Is Underspecified

Abstract

Tool-using agents recover from execution failures by retrying a call or switching tools, which is what lets one finish in a complex environment. Every comparison of such methods is scored by a rule that either counts the calls or ignores them, and neither case is safe. Where they are counted, a repair that should have rescued a failure turns a correct episode into a wrong one: a tool returns empty, the agent retries, and the extra call breaks the match against the gold answer. The effect is not incidental: a reference agent, the middleware it ships with, and a benchmark's own checker compose into it, and emptying half the tool returns takes it from to while emitting more calls; the identical runs scored with recovery calls excluded show no drop. Where they are not counted, the winner depends instead on how accuracy is traded against tokens: across episodes over nine recovery methods and five benchmarks, four standard ways of making that trade select multiple winners. Both are decidable before an experiment runs: which case applies is stated in the scoring rule, and the weight at which a ranking changes is a function of two columns. A recovery comparison has to state what counts as success, and what success is worth against cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM tool-using agents, Error recovery, Efficiency evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CNfoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how evaluation rules affect comparisons of recovery methods for tool-using agents. It distinguishes two questions: whether additional recovery calls directly affect correctness scores, and how accuracy should be traded against token cost. The authors formalize conditions under which additional recovery calls reduce scores under call-sensitive evaluation rules. They also derive an upper bound on the cost weight at which the accuracy-maximizing method ceases to be uniquely optimal under an additive accuracy–cost objective. The experiments rescore identical BFCL recovery traces with recovery calls included or excluded, and examine how cost preferences affect method recommendations across two models, nine methods, and five benchmarks. The results show that both scoring rules and cost metrics can affect the conclusions of recovery-method comparisons.

Critical strengths
  1. A useful distinction between scoring rules and cost preferences. Scoring rules determine whether recovery is counted as successful, while cost preferences determine which method is preferable. This distinction has practical and explanatory value: it helps separate effects of the scoring rule from effects of the accuracy–cost trade-off. It also supports a useful reporting recommendation: recovery comparisons should explicitly state their success criterion and cost preference.

  2. Broad empirical coverage. The evaluation covers two models, nine methods, and five benchmarks, reporting both accuracy and token cost. The range of settings and evaluation perspectives strengthens the empirical basis and credibility of the findings.

Critical weaknesses
  1. The cost metrics are not calibrated to the same cost preference, weakening the claim that the ratio metric has a blind spot. The weights in the additive and ratio objectives have different meanings, so sweeping the same numerical interval does not represent the same accuracy–cost trade-off. The paper explains why the ratio form is insensitive within its chosen parameter window, but does not establish that the two metrics are evaluated under equivalent cost preferences.

  2. The BFCL experiment establishes a scoring penalty without independently verifying successful task recovery. It clearly shows that additional recovery calls lower call-matching scores, but does not establish that an injected fault causes an actual task failure that recovery subsequently resolves. The evidence therefore supports the existence of a scoring penalty, but does not substantiate the stronger interpretation that a successfully repaired task is incorrectly judged to have failed. The paper should clarify the scope of this stress test and whether its findings warrant criticism of BFCL’s original design.

Questions for authors
  1. How should the claimed blind spot be interpreted after matching cost preferences? How should the conclusions of Section 4.3 be interpreted when the additive and ratio objectives are calibrated to represent the same accuracy–cost preference? Is the claim that the existing parameterization has limited sensitivity within the chosen numerical window, or that the ratio metric recommends inappropriate methods under a shared deployment objective? If the latter, please specify that objective and provide the parameter calibration or utility analysis needed to support the claim.

  2. What evidence shows that the penalized recovery rescues tasks that would otherwise fail? Does the BFCL probe include an independent final-answer or environment-state checker to verify that recovery rescues tasks that would otherwise fail? If so, please identify the corresponding recovery-on and recovery-off results under the same faults. If not, please clarify which conclusions concern call-matching scores alone and which concern actual task completion, and identify the evidence supporting the latter.

Reviewer_hqXLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose how to measure the recovery method for tool-use agent. The author believe there are two necessary settings missing from current assessment. First is whether the scoring rule should penalizes additional tool calls. The section is how to balance the importance between accuracy and token cost. The conclusion could be opposite if we don't pre-determine the two settings.

In the paper, the scoring rule has been categorized in two types. One is taking the extra tool calling into final measurements and the other one is only measure the outcome. The author prove that the first rule will not give recovery action any bonus. As for the second rule, author use a linear equation to consolidate accuracy and token cost into a value function.

The experiments are conducted on five benchmarks. In the first experiment, the authors use 80 held-out tasks from BFCL and inject empty tool responses to trigger recovery actions. In the second experiment, the authors compare nine recovery methods. The evaluation coers 50,050 episodes across five benchmarks and two models.

There are three main conclusions:

  • Exact tool calling matching is a measuring method prefer to the recovery actions.
  • There is no universal method to balance the trade off between accuracy and token cost
  • The ratio-based efficiency metrics are too sensitive to token cost
Critical strengths
  • The visual presentation is a strength of the paper. The figures show the pipeline clearly, and the tables report the results together with the statistical tests in a form that is easy to check
  • The author point out a good point that a measurement which can fairly treat recovery actions need more attentions from research community
Critical weaknesses
  • Overall, this paper is hard to follow from understanding the title to abstract. The abstract does not clearly explain the research problem, the motivation, or the proposed approach. Instead, it introduces several concepts and conclusions without sufficient context or logical connections

  • The novelty of this paper is limited. The author point out a practical point to conduct further research. Unfortunately, the authors do not provide a sufficiently deep investigation into the underlying mechanisms behind these observations, nor do they propose a concrete solution to address the identified evaluation problems. The conclusions from this research does not go much beyond the plain observations

Questions for authors

No

Reviewer_Wqn3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that tool-agent recovery evaluation methods are underspecified in two ways. First, the scoring rule can penalize recovery. Second, the scorer reads only the answer. To address these issues, they conduct fault-injection and large-scale experiments that account for recovery attempts.

Critical strengths
  1. The paper targets the fundamental ability of AI agents.
  2. The paper conducts a few ablation and control evaluations that enrich its findings.
Critical weaknesses
  1. The framing of these benchmarks as lacking is misleading. Recovery is not their goal, and stating that as a problem misses the point of bulding/ desgining new pipeline/ framwork for evaluation.
  2. Not familiar with all the benchmark details, But, for tau-bench, the state matching is the main metric, and for tools, the check is that the gold actions are contained, not an exact match. So, not sure if this is also accurate for the other benchmarks.
  3. The central result in Proposition 1 is correct by definition. BFCL does not execute calls, so the “emptied return” is fictional, and the evidence for a real-world problem uses one benchmark that the authors recognize is artificial.
  4. The α census is a formalation, not a measurement, and the α=0 class is overclaimed. State-based checkes are alredy mesured retry implicitly.
  5. The best method depending on the cost is the standard Pareto-frontier point (Kapoor et al.).
Questions for authors
  1. 2585 tasks × 9 methods × 2 models = 46,530, not 50,050. The 3,520 gap is unexplained.
  2. Small fixes: a. In the intro, “benchmarks (...)” cites methods, not benchmarks. b. “The inversion” appears before it is defined. c. “non-weight on cost” should be “non-negative weight.”

Comments on the paper

1 comment

Thank you for the additional experiments and analysis. They addressed some of my concerns. I’ll keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.