acceptodds
Under review as a conference paper at ICLR 2027

Tracepidia: Diagnosing Failures Behind Failures in Agent Trajectories

Abstract

Agents can fail long-horizon tasks not only because of the model's own limitations but also because of the tools, the execution environment, or other components of the system. Correcting such a failure requires identifying the responsible component and how its behavior affected the outcome. Existing approaches label the responsible agent and the error step from the agent's trajectory. Such a label cannot tell a tool fault from a model fault, and the trajectory does not show when a tool or the context manager changes what the model receives. To this end, we introduce Tracepidia, which consists of a benchmark, an auditor, and a toolkit for agent failure diagnosis. The benchmark contains 2,332 samples on 71 Terminal-Bench 2.1 tasks. Each sample injects one fault into a passing trajectory in which every step succeeds, so its responsible component, failure category, and error step are known. The auditor names these three from the trajectory and the telemetry that the agent harness emits while it runs, and the toolkit records this telemetry without changes to the agent. On held-out tasks, the auditor raises macro-F1 over failure categories from 0.315 to 0.424 against a direct judge that uses the same LLM. The gain is largest on samples that still pass the verifier, where no failed test shows what went wrong. Across 890 sessions of ten agent configurations, the auditor finds a failure in 57.10% of the sessions that pass the verifier. A stronger model lowers the share of sessions with a failure, and higher reasoning effort does not.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent trajectories, benchmark, coding agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u7zZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies component-level failure diagnosis for LLM agents. Beyond identifying the erroneous step, it asks which system component is responsible for a failure, such as the model, a tool, the context manager, the environment, or the grader (8 components and 41 categories in total). The authors argue that a trajectory alone cannot tell whether a tool tampered with its output or the model ignored a correct output, and that runtime telemetry is needed to compare a tool's raw output with what the model actually received.

The paper makes four contributions. (1) A benchmark: starting from 71 healthy trajectories on Terminal-Bench 2.1, a single fault is injected at step k and the live model continues the run, which after filtering by a GPT-6-sol judge yields 2,332 labeled samples. (2) An auditor, a three-stage LLM diagnosis pipeline with nomination, classification, and localization. (3) A toolkit with an OpenTelemetry-based recorder and a visualization dashboard. (4) An audit of 890 real sessions across 10 agent configurations.

The auditor raises macro-F1 over failure categories from 0.315 to 0.424, and 57.1% of verifier-passing sessions contain at least one incident.

Critical strengths
  1. The problem is valuable and the idea is well motivated. The paper moves failure attribution from which agent or which step to which component, and the example in Figure 1 clearly shows the limitation of trajectory-only analysis. Comparing raw outputs with what the model actually received can, in principle, reveal faults that are invisible in the trajectory.
  2. The sample generation pipeline is well designed. The prefix is replayed exactly, every observation after the fault comes from real execution, and edited trajectories are re-executed in a fresh container for verification, so no observation is fabricated.
  3. The appendix is thorough. It reports task-level bootstrap confidence intervals, paired differences, replicate runs, and full prompts, which makes the results easy to check and reproduce.
Critical weaknesses

W1. The central claim, that telemetry is needed to diagnose failures caused by components other than the model, is weakly supported.

  1. The claim that some faults can be identified only through telemetry rests on only 15 test samples: 9 samples of context corruption from 4 tasks and 6 samples of call or response mistranslation from 3 tasks (Table 13).
  2. These faults are injected through the authors' instrumented proxy and tool wrappers, and the telemetry checks target exactly this injection mechanism. It is unclear whether detection generalizes to naturally occurring faults. The real runs in Section 4 do not use these proxies and wrappers, so detection of these two fault types is never validated in a realistic setting.
  3. The benchmark rarely tests components other than the model. Of all samples, 90.18% are attributed to the model, and on the test set, always predicting the model already gives 86.0% component accuracy (559/650), leaving only 91 test samples with another responsible component. The three largest categories are shallow tool-call errors that together account for 69.38% of the benchmark, and all base trajectories come from a single model and harness (GPT-5.6-terra with Codex CLI).

W2. The advantage of the auditor over the baseline is not reliably established.

  1. Against the direct judge (Table 9), the 95% intervals for category accuracy and for all four Step@δ metrics include zero in both runs. Macro-F1 improves significantly only in the first run (+11.0 points, interval +1.0 to +17.0). In the second run the gain is +6.9 points, and its interval (−2.1 to +13.8) includes zero. The main text acknowledges this, but the abstract and the contribution list report only the higher run.
  2. The first run of the full auditor is best on only one of the eight metrics in Table 1 (Top-3). Removing the calibration rules gives higher Comp. (0.892) and Macro-F1 (0.436), and removing nomination gives higher Step@1/2/5. In Table 14, removing telemetry, nomination, or worked examples does not significantly lower macro-F1. The only significant effects are those of the worked examples on category accuracy (2.6 points) and of the calibration rules on Top-3 accuracy (4.9 points).
  3. Two runs of the same system differ by 4.0 points in macro-F1, about the same as the effect of removing nomination, telemetry, or worked examples (3.7 to 5.9 points). Only two runs of the auditor and one run of the direct judge are reported, on a test set of 30 tasks.
  4. The only baseline is a direct judge using the same LLM, and it receives neither the telemetry nor the per-test results. The comparison therefore cannot separate the effect of the extra information from that of the method itself. The auditor is also evaluated with only one LLM, and none of the diagnosis methods discussed in the related work, such as AgentRx (arXiv:2602.02475), TrajAudit (arXiv:2605.26563), and CHIEF (arXiv:2602.23701), is compared experimentally.
Questions for authors
  1. Given that the official OpenAI changelog lists the release of GPT-6 Sol on September 22, 2026, three days before the full-paper deadline, when was the quality control of the 3,452 samples performed, which exact model version was used, and was a different judge model used at any stage?
  2. Since the runs in Section 4 were recorded without the proxy and tool wrappers and therefore capture neither what the LLM received nor the raw tool outputs, can the toolkit record these differences on an unmodified harness, and if not, how would the auditor detect context corruption or mistranslation in practice?
  3. Of the 1,120 samples excluded from the benchmark, how many were excluded because the run did not complete, because the judge decided that the failure did not occur, or because the judge could not determine the outcome, and how does the category distribution change after filtering?
  4. Does the auditor's advantage over the direct judge remain significant when each system is run at least three times and the direct judge receives the same telemetry and per-test results?
  5. How does your data construction differ from Aegis (arXiv:2509.14295), which also injects faults into successful trajectories to construct labels?
  6. Given that the auditor flags 33.33% of the healthy controls and that all incidents were repaired in 68.51% of the passing sessions with incidents, have you checked its diagnoses on the 890 sessions against human annotation, and what share of passing sessions still contains an unresolved incident?
  7. As a suggestion for future work, would you consider evaluating telemetry-based detection on naturally occurring tool and context faults, and comparing the auditor with existing methods such as AgentRx (arXiv:2602.02475), TrajAudit (arXiv:2605.26563), or DoVer (arXiv:2512.06749) and with other auditor LLMs?
Reviewer_Qs23OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses a critical pain point in LLM‑agent debugging: agent task failures cannot always be attributed to the LLM model itself. Faults may originate from tool wrappers, context managers, execution environments or grading subsystems, which standard trajectory‑only diagnosis cannot disambiguate. The authors present TRACEPIDIA, a complete three‑part solution for agent failure diagnosis. The paper further conducts large‑scale audits over 890 real agent sessions across ten agent configurations without artificial fault injection. A key empirical finding is that 57.10% of verifier‑passing sessions contain detectable incidents. Stronger base models reduce incident rates, yet higher reasoning effort primarily improves post‑hoc repair rather than preventing root‑cause faults.

Critical strengths
  • Strength 1: The problem formulation and benchmark fill a critical gap for agent failure attribution. Existing agent‑failure benchmarks mostly rely solely on agent‑facing execution traces and cannot separate model‑originated bugs from system‑component faults. The injection‑based ground‑truth construction, paired with boundary telemetry records, enables evaluation for failures invisible from agent‑side logs. The large fraction of latent failures (samples pass verifier despite injected faults) reflects real‑world agent debugging scenarios.
  • Strength 2: Large‑scale empirical study reveals non‑trivial practical insights for agent evaluation. Real‑world audits on 890 sessions produce actionable observations: final pass rates mask abundant intermediate incidents; increased reasoning effort improves repair capacity instead of eliminating root failures; harness choice reshapes failure profiles and repair rates. These results challenge the common practice of evaluating agents purely using final pass‑rate metrics and highlight the necessity of tracing intermediate execution states.
Critical weaknesses
  • Weakness 1: Overall diagnosis performance remains moderate, together with non‑negligible false‑positive rates on fault‑free controls. The best macro‑F1 reaches only 0.424 for multi‑category diagnosis. On healthy control trajectories with zero injected faults, the auditor flags failures for 33.33% controls. Many misclassifications confuse malformed tool‑call faults with functional implementation defects. While telemetry helps for trace‑invisible failures, general‑case category‑level diagnosis accuracy is far from production‑ready. This limits confidence when applying the auditor for real‑world unlabeled agent logs.
  • Weakness 2: Limited generalization scope of the benchmark and evaluation. All benchmark samples originate from Terminal‑Bench CLI coding agents. Fault injection patterns and the 41‑category taxonomy are tailored for command‑line tool‑using agents. There is no validation showing whether the auditor and failure taxonomy transfer to other agent families, e.g., web‑browsing agents, multi‑agent systems, or long‑horizon planning agents.
Questions for authors

None

Reviewer_35JqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that standard agent trajectories cannot distinguish model errors from system-level faults because they record only the post-processed observations delivered to LLMs. The authors construct a counterfactual fault-injection benchmark from sanitized container replays and propose a multi-stage auditing pipeline to localize failures. While the container-verified replay mechanism is well-designed, the dataset is overwhelmingly skewed toward trivial model syntax errors, the complex auditing pipeline fails to outperform a fair baseline on failed tasks, and a high false-positive rate undermines the empirical findings on uninjected runs.

Critical strengths
  1. The benchmark construction pipeline sanitizes passing runs into error-free base trajectories via container-verified edits before injecting single faults during live execution, which cleanly isolates ground-truth causal responsibility from earlier recovered detours.

  2. Boundary telemetry captures the exact difference between raw tool or client payloads and the context delivered to the model, which makes invisible infrastructure failures directly observable.

Critical weaknesses
  1. The baseline comparison is unfair because the direct judge is denied the in-context examples, per-test verifier outputs, and boundary telemetry given to the full auditor. The ablations show that the entire span-nomination stage adds no value beyond run-to-run noise, while removing the worked examples wipes out the category accuracy gains. The full auditor even underperforms the unassisted baseline on tasks where the verifier fails.

  2. The dataset is heavily skewed toward the language model, as over 90% of the benchmark consists of model-side errors like bad syntax and hallucinated tools. This imbalance means the paper's core claim about catching invisible boundary faults rests on just 15 test samples for context corruption and tool mistranslation combined. A sample of that size is too small to support broad conclusions about system-level diagnosis.

  3. The empirical study of uninjected agent runs relies on a noisy auditor that flags a third of clean control trajectories as faulty. Those wild runs were also recorded without the proxy and tool wrappers needed to capture boundary telemetry. These limitations make it hard to trust the reported incident rates in passing sessions, especially without human validation.

Questions for authors
  1. How does the single-pass direct judge perform when given the exact same worked examples, per-test verifier results, and boundary telemetry provided to your auditor?

  2. Why does the multi-stage auditor perform worse than the unassisted baseline on both category accuracy and step localization when the task verifier fails?

  3. Did the authors conduct any human validation on the uninjected runs to estimate how many of the reported incidents in passing sessions are false positives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.