acceptodds
Under review as a conference paper at ICLR 2027

Same Trajectories, Different Credit: Scoring-Rule Dependence in Tool-Fault Recovery Evaluation

Abstract

After a tool call fails and keeps failing, an agent can stop, do the parts of the task the failure did not block, report the failure, or assert the outcome it was asked for. A benchmark’s continuation rule decides which of these its score credits, so a higher pass rate does not say which behaviours an intervention suppressed. ToolMaze ships two such checks and evaluates its single-path category c1 with the stricter one; we build matched scorers differing only in the forbidden set and re-score identical stored trajectories. That one change moves the pass rate by 11 to 47 points in every arm, and lowers the benefit credited to a recovery message by 6.6 and 16.0 points on two models. A pre-registered replication on sixty held-out tasks reproduces the level shift but not that reduction: score levels are rule-dependent throughout, whereas the attenuation does not replicate across task sets. Which later sub-tasks count as blocked is not specification-free: three plausible tests label the same 62 relations differently, and the same tool can assert the failed step’s outcome in one branch and report the failure in another. Measured apart, the behaviours diverge: the pass rate rises by 33 points while structured-field agreement with a fault-free run on independent sub-tasks falls from 79.4% to 39.7%, and final answers report the failure more often. We recommend reporting the continuation rule with the score, and release the replay protocol, the pre-specified adjudication rubric and the labels.

Primary area:
datasets and benchmarks
Keywords:
tool-augmented LLM agents, agent evaluation, benchmark construct validity, tool-fault recovery, scoring rules, trajectory re-scoring, measurement validity, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jHWgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how scoring rules affect tool-fault recovery evaluation on ToolMaze. It replays agent executions and re-scores stored trajectories under different continuation rules, examining changes in pass rates and the measured benefits of recovery prompts. It also compares definitions of task dependence and separately measures independent-task execution and failure reporting.

Critical strengths

None

Critical weaknesses
  1. The paper offers limited novelty and does not establish a substantial contribution. The work largely consists of incremental modifications and analyses of ToolMaze’s existing scoring rules. Higher pass rates after relaxing the set of forbidden tools are largely a direct consequence of the rule change. The contribution beyond prior work on rescoring trajectories is unclear. The paper also provides no independent evidence showing which rule better measures recovery quality in practice. The reduction in the estimated benefit of recovery prompts does not replicate on the remaining 60 tasks. This limits the generality of the result.

  2. The paper is poorly written and organized, making its central argument difficult to follow. A relatively straightforward question is obscured by numerous experimental conditions, dependence definitions, metrics, and repeated qualifications. Important explanations are scattered across the main text and lengthy appendices, with insufficient separation between the central findings and supporting analyses. Even after careful reading, it takes substantial effort to identify the paper’s actual contribution and determine which conclusions are adequately supported. The presentation substantially weakens the paper’s clarity and persuasiveness.

Questions for authors

What is the substantive new contribution beyond modifying and analyzing ToolMaze’s existing scoring rules? What evidence shows that this contribution generalizes to other evaluation settings and helps assess practical recovery quality more accurately?

Reviewer_DLAhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how the continuation rule of a tool-fault recovery benchmark determines the credit given to agent behavior after a permanently failing tool call. On ToolMaze category c1, the authors replay logged episodes to the fault-injected call, branch the result under four runtime texts (N, W, R, RB), run each branch to the agent's final answer, and re-score the identical stored trajectories with matched scorers that differ only in the forbidden set: the shipped strict rule versus a data-flow rule built from the benchmark's own dependency function. On tasks 001–040 with Claude Haiku 4.5 and gpt-4.1-mini, swapping the rule moves pass rates by 18.7–47.4 points and lowers the credited RB−N gain by 6.6 and 16.0 points; a pre-registered replication on tasks 041–100 reproduces a smaller level shift (11.1–12.0 points) but not the attenuation (+0.9 [−1.9, 4.6]). The paper further shows that three operationalizations of sub-task dependence label the same 62 relations differently, that a rule keyed on tool names cannot separate notifications asserting success from those reporting failure, and that pass rate, structured-field agreement on independent sub-tasks, and failure reporting in final answers move in different directions. It recommends reporting the continuation rule with the score.

Critical strengths
  1. Exemplary measurement design. Holding trajectories byte-identical while changing one scoring condition isolates the evaluator's contribution. The paired task-clustered intervals, pre-registration, transparent reporting of the failed replication, and the correction log (Appendix M) make the reported effects credible. This is why I have no substantive soundness concerns.
  2. The reading of seven benchmarks' scoring code (Appendix C) shows that none defines which later sub-tasks a failure blocks. This supports the recommendation beyond ToolMaze.
Critical weaknesses
  1. The robust result is close to definitional, and the substantive one does not replicate. Relaxing the forbidden set can only turn failures into passes, so the 11–47-point level shift across all measured arms follows from construction. The claim that carries new knowledge — that the rule changes the credited effect of an intervention — fails the authors' own pre-registered replication (+0.9 [−1.9, 4.6] on tasks 041–100 vs. +16.0 [8.0, 25.0] on 001–040), and within c1 the rule does not change which runtime text ranks best (Limitations). Where the attenuation is large, it is also largely anticipated by the intervention text: on gpt-4.1-mini, 14.6 of the 16.0 points arise between R and RB (Table 9), i.e., from adding a clause that tells the agent to stop and report, which is exactly what the strict rule rewards. The same clause plausibly also explains why structured-field agreement on independent sub-tasks is lower under RB than under N while the pass rate is higher (Table 3); on gpt-4.1-mini, the R→RB step alone accounts for 28.0 of the 34.7-point drop in continuation (Table 11). The main takeaway is therefore "the rule can matter on some task sets," which I find insufficient for the main track.

  2. The evidential base is narrow for a general recommendation. The rule-swap re-scoring covers one category (c1) of one benchmark, permanent faults only, two small models, and 40 (+60) tasks; the c2 replication (Appendix E) and the transient-fault branches (Appendix H) do not swap the continuation rule, and the authors note that the held-out tasks and the c2 check "test the reading rather than establish generality" (Limitations). The dependence and call-content labels come from one author and one language model, and the final-answer codes from a single author. Together with very dense, heavily hedged prose that relies on 13 appendices, the paper reads more as a careful audit of one benchmark than as a generalizable contribution.

Questions for authors
  1. Appendix E shows the ranking of texts inverting on c2, but there both the tasks and the rule change. Is there any setting — another benchmark, or an intervention whose text does not itself encode the strict rule's target behavior — where swapping the continuation rule alone changes the ranking or the sign of an intervention effect? Such evidence would most directly change my assessment.
  2. Can you state a criterion, checkable before scoring, that predicts when a rule change will alter an intervention effect (e.g., from what the intervention instructs), beyond the post hoc observation that the two arms were reclassified unequally?
  3. Re-scoring a second benchmark family with the same protocol would substantially strengthen the generality of the claims.
Reviewer_ca6MOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper studies how scoring rules affect an evaluation of tool-fault recovery in LLM agents. The authors evaluate same trajectories on different scoring rules, showing that different scoring rules significantly affect both recovery performance and interventions. The paper further investigates ambiguity in defining tool dependencies and finds the importance of explicitly specifying scoring rules when interpreting agent recovery benchmarks.

Critical strengths
  • The target problem and motivation is clear.
  • The authors conducted systematic experiments for showing that different scoring rules can influence the same trajectories.
  • The paper provides practical suggestions about the importance of explicitly defining scoring rule on agent benchmarks and the necessity of carefully interpreting the evalution results.
Critical weaknesses
  • The research question is not interesting because it sounds obvious. It is obvious that benchmark evaluation scores depend on the evaluation scoring rule. The research question provides no interesting or new insights.

  • The study relies heavily on a specific category of the ToolMaze dataset. Generalization to other benchmarks and failure types was not examined.

  • Lack of novelty. It seems like a domain-specific empirical case study of existing works on Construct Validity. Compared to Rollout Cards, there are very marginal technical contributions and novelty, since they take the same approach.

  • Only two small LLMs (not even recent models) are evaluated.

Questions for authors

See Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.