acceptodds
Under review as a conference paper at ICLR 2027

Invocation-Level Reliability of Tool-Using Agents

Abstract

An early tool-call error can corrupt every downstream input even when an agent subsequently applies the right local rule. We isolate this effect with matched evaluations under a correct baseline history and the model's own free-running history. For tasks of depth , their pooled invocation rates and define a fit-free relative propagation loss . Across five open-weight models on controlled routing tasks, the two 7-8B models lose about 68% of baseline capability by depth 6; stronger models mostly remain at ceiling. Our main result is a construct-validity condition for trajectory scoring. Let be the canonical next target after a divergence and the agent-observable history. If has conditional guessing probability at most given , then any fixed-gold scorer credits the agent with probability at most , regardless of its local competence. Gold-agreement severity is therefore driven to a scorer-imposed boundary, and observed reconvergence cannot measure behavioral recovery. Fixed reference alone is not sufficient: the result applies when the canonical target becomes unreachable or unpredictable, and excludes reconstructible targets and evaluators that accept alternative states or milestone paths. Our hidden-constant tasks instantiate the condition: 0 of 869 corrupted-context calls match gold, and 0 of 580 opportunities reconverge. Conditional-on-state replay instead asks whether a call correctly continues from the state the agent actually holds; it yields interior severity estimates of 0.149 and 0.316 and requires no new model queries. Alternative absolute-gap and error-amplification summaries give the same qualitative depth trend.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, tool use, function calling, error propagation, evaluation methodology, benchmark construct validity, agentic reliability, trajectory scoring

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zU46OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether invocation-level evaluation can distinguish deviation from a reference trajectory from a loss of local competence after an early tool-use error. It compares baseline performance under correct histories with free-running performance under the model's own histories on matched tasks, defining relative propagation loss as Ld=1−gˉd/pˉdL_d=1-\bar g_d/\bar p_d. Its central result is a conditional construct-validity statement: when the canonical post-divergence target is difficult to predict from the observable history, reference agreement is bounded by the target's guessing probability and provides limited information about local competence. The authors apply conditional-on-state replay to cached trajectories and examine five open-weight models on synthetic routing tasks, supplemented by targeted posterior predictive checks, error injection, and assigned state repair. The paper distinguishes correct local continuation, return to the canonical state, and completion of the original task.

Critical strengths

1. The paper identifies an important issue in interpreting agent evaluations. An upstream error can cause subsequent locally correct actions to remain inconsistent with the reference trajectory. Later reference mismatches therefore do not directly establish new local decision errors. Section 5.3 states the relevant condition through conditional guessing probability and discusses exceptions involving reconstructible targets, resets, and alternative valid paths. This gives a concrete basis for interpreting trajectory scores without dismissing all fixed-reference evaluations.

2. The paired protocol and cached replay make the different factors easier to examine separately. The two arms use matched tasks and the same retry budget. The primary depth-level estimates use paired task-level bootstrap intervals, and the paper reports absolute gaps and error amplification alongside relative loss. Conditional-on-state replay changes the argument-scoring target on the same cached outputs without new model queries, directly showing how the scoring regime changes the interpretation of identical behavior. The paper also makes clear that tool selection was already scored against the actual state in the original protocol.

3. The diagnostics and negative findings offer useful methodological lessons. Appendix C shows a fit passing an aggregate posterior predictive check while failing a check targeted at corrupted contexts, illustrating what aggregate fit can miss. Appendix F separately reports post-repair reference agreement and original-task completion, without treating improved reference agreement as clear evidence of task recovery. The discussion of ceiling effects, parser issues, and unexecuted experimental conditions also helps establish the limits of the findings.

Critical weaknesses

1. The claimed roughly 2.9-fold observational overestimate appears to mix different estimands. Figure 4 defines severity as the relative loss 1−P(correct∣corrupted)/P(correct∣clean)1-P(\mathrm{correct}\mid\mathrm{corrupted})/P(\mathrm{correct}\mid\mathrm{clean}), with an observational estimate of 0.316 for allam. However, Table 7 reports corrected/corrupted accuracies of 0.411/0.304 and an effect of 0.108, which appears to be an absolute accuracy difference. Using the displayed rounded rates, relative severity would be approximately 1−0.304/0.411≈0.2601-0.304/0.411\approx0.260. Thus, dividing 0.316 by 0.108 appears to compare a relative loss with an absolute difference. The resulting claim occurs in the contributions, Section 5.4, and Appendix E, and should be recomputed under a common definition or removed. Even after aligning normalization, the state and position distributions of the observational and intervention samples need to be comparable before their difference can quantify bias.

2. The generator-level justification of the theoretical condition and its relationship to prior work need further detail. The proposition holds under its stated condition, but Section 5.3 sets ϵk=10−5\epsilon_k=10^{-5} under a conditional-uniformity assumption without deriving it from the complete generation process. For (a⋅x+b) mod M(a\cdot x+b)\bmod M, hidden constants alone do not exclude information revealed by previous input-output observations. The constant distribution, tool reuse, and observable history need to be specified. Related Work should also discuss Exposure Bias versus Self-Recovery (EMNLP 2021), which already compares data prefixes with model-generated prefixes and warns that retaining a fixed reference after changing the prefix can mislead evaluation. The contribution is better positioned around the formal tool-use condition, executable scoring, and diagnostic evidence, with a clearer account of what these add to the earlier analysis.

3. The empirical evidence remains concentrated in narrow task and model conditions. The primary tasks involve parity routing and number copying, stronger models are mostly at ceiling, and the injection and recovery experiments use only allam-2-7b. This model is itself anti-correlated with the routing rule, so the selection-channel effect may depend on parity distributions and tool presentation. Section 7 acknowledges that conditional-on-state scoring does not directly transfer to BFCL multi-turn tasks whose correct actions depend on user intent. The evidence therefore supports a controlled evaluation diagnosis but does not establish its prevalence in realistic agents. In addition, Appendix E tests multiple injection positions on the same task collection but resamples pairs; if a source task contributes multiple pairs, dependence between those pairs needs to be accounted for.

Questions for authors

1. Does Table 7 report an absolute effect or relative severity? Please recompute the observational and intervention estimates and intervals under a common definition. After aligning the estimand and accounting for state and position distributions, does the roughly 2.9-fold overestimate remain supported? Otherwise, which conclusions about causal effects still hold?

2. Could the authors specify constant sampling, tool reuse, and history visibility, and use these rules to derive or bound ϵk(Hk)\epsilon_k(H_k)? Please also explain what new conclusions or capabilities the tool-use setting adds beyond existing exposure-bias prefix comparisons and conditional-reference evaluation.

3. Could the authors add a check with non-copy arguments or randomized tool presentation on an additional reproducible model that is not at ceiling? For the existing injection data, please specify the number of pairs contributed by each source task and provide task-cluster bootstrap intervals where tasks are shared, to assess whether the intervals and effect directions remain robust.

Reviewer_BBbFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper studies how earlier tool-call errors affect later calls, using matched correct-history and free-running evaluations. Main point is that matching canonical trajectory can stop measuring local competence once gold target becomes unpredictable after divergence. Authors propose conditional-on-state scoring and test it on synthetic routing tasks with five models.

Critical strengths

Paper clearly separates local call correctness, agreement with reference, and final task success. This matters because one upstream error should not automatically make every later action look locally incompetent. Error-injection and repair experiments are useful checks, and authors are fairly direct about what these experiments cannot establish.

Critical weaknesses

The “2.9× inflation” claim seems to compare different quantities. Figure 4 defines severity as relative loss, but Table 7 reports 0.108, which matches absolute accuracy difference, 0.411 − 0.304. Using relative definition gives 1− 0.304/0.411 \approx 0.26, much closer to observational 0.316. Current comparison does not establish claimed magnitude of bias.

Empirical support is narrow. Most informative results come from two small models on parity routing with copied arguments, while stronger models mostly hit ceiling. Main proposition is a straightforward guessing-probability bound; broader contribution depends on showing this scoring problem matters in realistic tasks, where obtaining correct state-conditional target is also harder.

Questions for authors
  1. Why is causal severity in Table 7 an absolute difference while observational severity is relative? After using same definition, does evidence still support upward bias?
  2. App. E has 316 pairs from 52 tasks. Does bootstrap keep all injection positions from same task together? If not, please report task-level intervals.
  3. Can authors demonstrate conditional scoring on a task with transformed arguments or a more realistic state transition, beyond parity routing and value copying?
Reviewer_ZSa4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Tool-call errors can propagate through an agent’s execution, affecting downstream results even when subsequent actions correctly follow from the agent’s current state. This paper studies error propagation by comparing invocation correctness under two conditions: a correct reference history and the model’s own execution history. It introduces propagation metrics and evaluates five open-weight models on synthetic routing tasks, finding substantial losses on two smaller models while stronger models largely remain at ceiling. Its central finding concerns evaluation: when an earlier error makes the canonical target unpredictable from the agent’s observable history, agreement with that target cannot reliably measure local competence or recovery. The paper recommends checking whether the canonical target remains reconstructible from the agent’s observable history before interpreting post-error gold agreement as a measure of local competence.

Critical strengths

Terminal rewards provide limited insight into where errors arise and how they affect subsequent actions. The paper’s focus on invocation-level reliability addresses this gap by distinguishing local tool-use competence from the downstream consequences of earlier errors, making it timely and practically relevant to the evaluation of tool-using agents.

Critical weaknesses
  1. Presentation and clarity. I found the paper difficult to follow. It reads more like a technical report than a focused research paper, introducing numerous concepts and diagnostics before establishing their purpose. The abstract uses terms such as “conditional guessing probability” and “gold-agreement severity” without sufficient intuitive explanation. The relationship between task depth d and the number of tool calls should also be made explicit earlier. Section 3.2 introduces clean and poisoned contexts, origin-specific recovery rates, and the severity parameter pi, but provides limited intuition for these quantities or the transitions represented by Equations (2) and (3). Although some definitions are present, their connection to the evaluation protocol and central claims remains difficult to understand. Similarly, “size-M space” in Section 3.3 could simply be introduced as the finite set of possible tool outputs.

  2. Limited empirical scope. The empirical evidence comes from a narrow family of synthetic integer-valued tasks and associated control and recovery variants. The strongest propagation results are concentrated in two smaller models, while stronger models largely remain at ceiling. These experiments provide a controlled demonstration, but do not establish how frequently the identified scoring problem arises in realistic tool-use settings or whether conditional-on-state scoring is practical and informative there. Evaluation in a stateful benchmark with more realistic tool interactions would strengthen the paper’s practical significance.

  3. Unclear significance beyond the expected propagation effect. The observation that an early tool-call error can affect subsequent execution is unsurprising. The paper’s more distinctive contribution is its argument that canonical gold agreement can fail to measure local competence after divergence, together with its proposed conditional-on-state scoring. However, the paper does not yet convincingly establish how much additional diagnostic value these contributions provide in practice. A concrete demonstration that they reveal an otherwise hidden failure mode or change an evaluation conclusion in a realistic setting would make the contribution more compelling.

Questions for authors
  1. The introduction states that “a single wrong call early in the chain corrupts the inputs to every downstream call.” This appears stronger than the conditions established later in the paper, since agents may detect an error, retry an action, or restore an earlier state. How prevalent is persistent error propagation in realistic tool-use tasks? Could the authors provide concrete examples and quantitative evidence using contemporary frontier models on established benchmarks to better substantiate the practical motivation?

  2. The paper acknowledges alternative valid trajectories and distinguishes local competence from recovery of the original goal. However, how should the baseline/free-running gap L_d be interpreted when an agent follows a different valid path or recovers after an earlier mistake? Since the primary analysis combines state-conditional tool selection with canonical argument agreement, could the authors clarify when this gap represents a meaningful loss of capability and when it primarily reflects divergence from the reference trajectory? A worked example involving alternative solutions or successful recovery would help.

  3. Section 3.1 defines both evaluation arms over the same number of calls d. How would the protocol handle tasks in which free-running execution requires additional calls for retries, diagnosis, or recovery, or completes successfully in fewer calls? Is equal trajectory length a restriction of the current synthetic setting, or can the proposed metrics accommodate variable-length executions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.