acceptodds
Desk rejected at ICLR 2027

The Tool-Agent Observability Boundary: Predicting Recoverable Tool-Call Failures Before Any Recovery Run

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Adding more validation to a tool-using agent, whether schemas, entity checks, ranges, or self-consistency, feels like it should always help. It does not, and which failures it can reach is decidable in advance. Tool-output reliability is governed by an observability boundary: some corruptions leave evidence in the returned observation, and some cannot. A wrong-entity, partial, or malformed response violates a formal invariant and is recoverable. A stale-valid observation, a real past state of the same entity, is schema-valid, id-consistent, in-enum, and in-range. We prove a no-go theorem: no gate in the non-temporal invariant family (schema, id-echo, enum, range) separates it from a fresh record, so more reasoning or a larger model provably cannot cross the boundary. A freshness sweep confirms this: recovery falls to 0% when the verifier returns only same-cache evidence. Our contribution is to make the boundary predictable rather than to assert it. LOCO assigns per-observation causal criticality labels by leave-one-corruption-out replay, and shows that the obvious structural proxies fail unpredictably: read→write recall ranges from 0.0 to 1.0 across domains, so the boundary cannot be eyeballed. StructAudit compiles admission predicates from tool schemas and clean examples alone, then emits a frozen coverage map that places each corruption class on the correct side a priori and label-free, at 87.9% per-break agreement across three causal domains and predicate-level precision 1.000 with recall 0.700 on retail. FreshVERIFY supplies the fresh read where the map predicts a freshness limit. Measured against the strongest cheap gate rather than the weakest, one invariant does most of the work: on gpt-oss-120B, id-echo alone matches or exceeds the full predicate family on recovery, and it is the most robust cheap gate net of lost clean rescues; where the fuller family recovers more, it does so at higher rescue-loss, and declared-schema validation trails both. This corroborates the boundary rather than undercutting it, since the recoverable corruptions are exactly the entity-observable ones. The resulting operating rule for tool-output admission: validate what is observable, verify what is not, and do not expect in-trajectory reasoning to manufacture missing freshness. Whether the same informational constraint governs verification signals more broadly remains open; we settle it where it can be measured.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, tool use, agent reliability, tool-output verification, observability, failure recovery, agent evaluation, schema validation, no-go theorem, tau-bench

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pr7VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the tool-call failures of agents. It aims to make the boundary of observability predictable, distinguishing recoverble corruptions from others. It proposes LOCO for casual failure attribution, StructAudit for compiling admission predicates, and FreshVERIFY for obtaining extra evidence when local verification fails. It conducts experiments to show the effectiveness of StructAudit.

Critical strengths
  1. Tackles an important problem of tool-call failures in agents
Critical weaknesses
  1. The writing is hard to follow. It seems that this paper first conducts an empirical study, and designs a tool based on it. But it is unclear how LOCO, StructAudit, and FreshVERIFY work together. It is also unclear how FreshVERIFY works. I would recommend the authors to deliver the core idea in a precise, direct, and concise manner. I would also suggest the authors to organize the paper structure in a similar way as other papers in this venue.

  2. The evaluation does not show the generalizability of the proposed methods. The LLMs are limited to QWen models. The benchmarks are focused on airline and telecom settings, and have very limited numbers of tasks. The tool-invocation information is also missing. Therefore, it is hard to validate the capability of StructAudit with the given evaluation results. In addition, StructAudit has a lower recovery rate than id-echo alone in some cases.

Questions for authors
  1. What is the high-level idea of the proposed solutions?

  2. What is the design of FreshVERIFY?

Reviewer_eZwYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper sorts corrupted tool observations into two kinds. Wrong-entity, partial and malformed records leave evidence in the observation. Stale-valid records, which are real past states of the same entity, leave none. There are three contributions. LOCO is a seeded, entity-keyed corruption harness on τ-bench retail/airline and a ported τ²-bench telecom domain. Its leave-one-corruption-out replay, with a no-op control, labels the observation behind each failure. A no-go theorem states that Boolean combinations of schema/type, id-echo, enum and range predicates cannot separate fresh from stale-valid records. StructAudit compiles such predicates from clean records into a gate and a frozen per-tool "coverage map"; FreshVERIFY supplies fresh reads. Experiments use Qwen2.5-14B/32B and gpt-oss-120B. StructAudit beats declared-schema validation (8/8 and 6/6 retail cells) but does not dominate an id-echo check alone. Recovery falls to 0% when the verifier returns cached values. The map matches the observed recovering mechanism on 29/33 (binary) or 24/33 (three-class) breaks.

Critical strengths
  1. A clean counterfactual measurement protocol (LOCO, §3–§5, p4). Corruptions are keyed by (seed, task, entity, occurrence), so suppressing one leaves the others bit-identical. A no-op control passes in every seed. Break-recovery is reported apart from rescue-loss, and gates are compared against a cost-matched random flagger (§4). The resulting per-observation causal labels are the paper's most reusable asset; for instance, 43/51 = 84% of retail breaks are single-attributable (p4, L209–210). AgenTracer [1] should be cited, but LOCO's entity-keyed, single-observation labels are a real refinement.
  2. Candid reporting that gives a practical result (§8.1–8.2). The authors concede that StructAudit's lift "is thus attributable chiefly to the id-echo predicate" (p6, L320–321) and that "we do not claim StructAudit dominates id_match" (end of §8.1). They also report the stricter 72.7% beside 87.9% (p8). The takeaway is trustworthy. On gpt-oss, id-echo alone reaches 42.7% mean recovery, against 38.0% for the full family and 23.1% for declared schemas (Table 2). Gating on cached evidence is net-negative (p5, L255–258).
Critical weaknesses

1. The no-go theorem is definitional once its assumptions are stated, and the paper claims more for it than it proves.

  • Proof. Theorem 1/2 (p5, L244–245; p13, L680–686) covers only Boolean combinations of four predicate types, even though A.1 defines a formal gate as any function of (q, H, o) (p13, L652–655). The proof also relies on two assumptions it does not state:

    • Every past state passes the compiled predicates. Lemma 2 just asserts that past values are "drawn from the same domains the compiler mined, so they are in-enum / in-range" (p13, L670–671). App. C's 0.95-presence and name-lexicon heuristics (p14, L728–733) do not guarantee this.
    • Fresh records are never flagged: "p also admits the fresh, valid o" (p13, L689).

    Stated precisely, the theorem says gates that read only fields on which o and o′ agree cannot tell them apart. That is immediate; the paper itself calls it "intuitive" (p5, L248).

  • Overreach. The abstract says "so more reasoning or a larger model provably cannot cross the boundary" (p1, L19–22). A.4 says "a more powerful in-trajectory gate does not help" (p14, L706–707). But an LLM reasoning over (q, H, o) is not built from F. §6 Scope also concedes an "in-trajectory escape" via a reliable last-seen value or timestamp (p5, L264 ff.). The cited corroboration fails too (p3, L143–145). STALE [2] studies agents that have the invalidating evidence and fail to use it, which is a reasoning failure, not an information limit.

  • Evidence. The sweep that "confirms this" (p1, L21–23) pools corruption types. For seed 42 at qc0.5, LOCO labels Fig. 2's 12 breaks as 7 wrong-entity, 3 partial and 2 stale (p4, L208, L212–213). So 10 of those 12 breaks are not stale-valid, and the 0% at p_fresh = 0 shows only that a cached re-fetch fixes nothing; it does not isolate stale-valid records. No reasoning, self-verification or larger-model condition is tested at same-cache, although Fig. 1 (right) claims all three still give 0%.

  • Why it matters. The paper names "the boundary itself (the causal recoverability map and the no-go theorem)" as its contribution (p3, L146–147).

  • Suggestion. State the assumption. Then either drop "provably" for reasoning and larger models, or test model-based gates on stale-only breaks at p_fresh = 0 (self-verification, or an LLM judge with access to H; cf. [3]).

2. The predictive evidence is small, partly circular and has no baseline, and the method's gain comes mostly from id-echo.

  • Circularity. The coverage map is computed from the predicates the gate itself runs ("wrong_entity→formal iff an echo predicate exists", p14, L738). It is scored against an "observed cheapest-recovering mechanism" (§8.3, p7, L377) that is never defined, so agreement on single-type breaks is expected by construction. There is no baseline on the 33 breaks (e.g., stale→freshness, all else→formal) and no intervals.
  • Classes. Fig. 4 uses four classes, but §8.3 scores three. "Content" is undefined, and no break is ever observed as "unobservable".
  • Proxy. The abstract's read→write contrast (recall 0.0 to 1.0) uses an undefined predictor. Its estimates come from a single seed and tiny cells (airline: 3 and 5 breaks, p16).
  • Method and 32B. Invariant inference "is not itself novel" (p9, L467), and the gain comes chiefly from id-echo (p6, L320–321). The paper claims replication on Qwen-32B (p3, L109; p9, L457–458), but shows no StructAudit-vs-naive_schema result for it. Fig. 5's labels are id_match, schema, l1, l2, l1l2, random and verify_all.
  • Suggestion. Define the mechanism and the predictor. Add the type-only baseline, per-domain counts with intervals, multi-seed proxy estimates, and the 32B comparison.
Questions for authors
  1. Does "provably cannot cross" (p1, L19–22) rest on Theorem 1 alone? If so, how does it cover gates that read H, such as one that checks a read against the agent's own earlier write? Such a check is a temporal invariant that needs a "reliable last-seen value", which is the case §6 Scope itself calls the "in-trajectory escape". What assumption makes Lemma 2 hold for the compiled predicates? Withdrawing the claim and stating the assumption would resolve my soundness concern; the result would still be definitional.
  2. Report Fig. 2 on stale-only breaks, and run one model-based gate (self-verification, or an LLM judge with access to H) on them at p_fresh = 0. If that gate stays at chance over enough breaks, "informational rather than computational" (p2, L104) becomes an empirical finding, and I would raise Soundness and Informativeness.
  3. How is the "observed cheapest-recovering mechanism" determined? What are the "content" class and the read→write predictor? How do the 33 breaks split by domain, and what agreement does a corruption-type-only rule reach on them? If the map clearly beats that rule, I would weight the predictive claim more.

References

[1] Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, Shuicheng Yan. "AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?" ICLR 2026 (arXiv:2509.03312).

[2] Hanxiang Chao, Yihan Bai, Rui Sheng, Tianle Li, Yushi Sun. "STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?" arXiv:2605.06527, 2026.

[3] Jimin Sun, So Yeon Min, Yingshan Chang, Yonatan Bisk. "Tools Fail: Detecting Silent Errors in Faulty Tools." EMNLP 2024 (arXiv:2406.19228).

[4] Austin Wright, Henry H. Andrews, Ben Hutton. "JSON Schema Validation: A Vocabulary for Structural Validation of JSON." IETF Internet-Draft draft-bhutton-json-schema-validation-00 (JSON Schema 2020-12), 2020.

[5] Ruipeng Wang et al. "AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition." ICML 2026 (arXiv:2602.11348).

Reviewer_oug2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the observability boundary of corrupted tool responses in LLM agents. It introduces LOCO for causal failure attribution, proves that non-temporal structural predicates cannot distinguish stale-valid observations from fresh ones, and proposes StructAudit to automatically compile validation rules and predict recoverability without task labels. FreshVERIFY complements local validation with fresh external evidence. Experiments across three agent domains and multiple LLMs evaluate recovery effectiveness and prediction accuracy.

Critical strengths

S1. Clear problem formulation and theoretical insight. The paper distinguishes locally detectable errors from stale-valid observations and formalizes the limitations of non-temporal structural validation.

S2. Causally grounded failure analysis. LOCO uses controlled counterfactual replay and deterministic corruption injection to identify failure-critical observations, providing more informative attribution than aggregate task success.

S3. Systematic empirical characterization. The evaluation spans multiple domains and model families, includes lightweight baselines, and reports recovery–rescue-loss tradeoffs rather than recovery alone.

Critical weaknesses

W1. The observability map does not provide an actionable online verification policy. StructAudit predicts which corruption classes are detectable but cannot identify which individual schema-valid observations are stale. Verifying every mutable-field response may approach Verify-All costs, whereas validating only flagged responses misses stale-valid failures. Moreover, FreshVERIFY relies on oracle-like fresh reads without evaluating realistic verification costs or shared-cache failures [1]. The paper lacks an end-to-end selective verification strategy with measured recovery–latency tradeoffs.

W2. Limited practical advantage over strong lightweight baselines. StructAudit's improvement over naive schema validation largely comes from ID consistency checks. On gpt-oss-120B, ID Match alone achieves higher recovery (42.7% vs. 38.0%, Table 2), while additional predicates increase rescue-loss on Qwen-14B. Comparisons with ID Match + Schema and stronger contract-based validators [2], including end-to-end latency and matched verification costs, are needed to justify the additional complexity.

W3. Insufficient evidence for predictive generality and statistical reliability. The coverage map is designed around the same four predefined corruption types used in evaluation, potentially making recoverability prediction easier than in realistic settings. The three-domain evaluation contains only 33 causal breaks, with 87.9% binary but 72.7% exact three-class agreement. Furthermore, significance tests pooling repeated seeds and corruption rates may overlook sample dependence. Broader fault types, as explored in [3], larger samples, and dependence-aware statistical analysis would strengthen the claims.

[1] Zheng et al. Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failures. arXiv:2609.10969, 2026. https://arxiv.org/abs/2609.10969

[2] Liu et al. ToolGate: Contract-Grounded and Verified Tool Execution for LLMs. Findings of ACL, 2026. https://aclanthology.org/2026.findings-acl.470/

[3] Mazumder and Lia. AgentCheck: A Reproduce–Intervene–Mitigate Workbench for LLM Agents over MCP. EMNLP System Demonstrations, 2026. https://aclanthology.org/2026.emnlp-demos.64/

Questions for authors

Given that stale-valid responses pass all local predicates, how can FreshVERIFY be selectively triggered without approaching Verify-All costs, particularly when independent fresh evidence is unavailable or unreliable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.