- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies component-level failure diagnosis for LLM agents. Beyond identifying the erroneous step, it asks which system component is responsible for a failure, such as the model, a tool, the context manager, the environment, or the grader (8 components and 41 categories in total). The authors argue that a trajectory alone cannot tell whether a tool tampered with its output or the model ignored a correct output, and that runtime telemetry is needed to compare a tool's raw output with what the model actually received.
The paper makes four contributions. (1) A benchmark: starting from 71 healthy trajectories on Terminal-Bench 2.1, a single fault is injected at step k and the live model continues the run, which after filtering by a GPT-6-sol judge yields 2,332 labeled samples. (2) An auditor, a three-stage LLM diagnosis pipeline with nomination, classification, and localization. (3) A toolkit with an OpenTelemetry-based recorder and a visualization dashboard. (4) An audit of 890 real sessions across 10 agent configurations.
The auditor raises macro-F1 over failure categories from 0.315 to 0.424, and 57.1% of verifier-passing sessions contain at least one incident.
Critical strengths
- The problem is valuable and the idea is well motivated. The paper moves failure attribution from which agent or which step to which component, and the example in Figure 1 clearly shows the limitation of trajectory-only analysis. Comparing raw outputs with what the model actually received can, in principle, reveal faults that are invisible in the trajectory.
- The sample generation pipeline is well designed. The prefix is replayed exactly, every observation after the fault comes from real execution, and edited trajectories are re-executed in a fresh container for verification, so no observation is fabricated.
- The appendix is thorough. It reports task-level bootstrap confidence intervals, paired differences, replicate runs, and full prompts, which makes the results easy to check and reproduce.
Critical weaknesses
W1. The central claim, that telemetry is needed to diagnose failures caused by components other than the model, is weakly supported.
- The claim that some faults can be identified only through telemetry rests on only 15 test samples: 9 samples of context corruption from 4 tasks and 6 samples of call or response mistranslation from 3 tasks (Table 13).
- These faults are injected through the authors' instrumented proxy and tool wrappers, and the telemetry checks target exactly this injection mechanism. It is unclear whether detection generalizes to naturally occurring faults. The real runs in Section 4 do not use these proxies and wrappers, so detection of these two fault types is never validated in a realistic setting.
- The benchmark rarely tests components other than the model. Of all samples, 90.18% are attributed to the model, and on the test set, always predicting the model already gives 86.0% component accuracy (559/650), leaving only 91 test samples with another responsible component. The three largest categories are shallow tool-call errors that together account for 69.38% of the benchmark, and all base trajectories come from a single model and harness (GPT-5.6-terra with Codex CLI).
W2. The advantage of the auditor over the baseline is not reliably established.
- Against the direct judge (Table 9), the 95% intervals for category accuracy and for all four Step@δ metrics include zero in both runs. Macro-F1 improves significantly only in the first run (+11.0 points, interval +1.0 to +17.0). In the second run the gain is +6.9 points, and its interval (−2.1 to +13.8) includes zero. The main text acknowledges this, but the abstract and the contribution list report only the higher run.
- The first run of the full auditor is best on only one of the eight metrics in Table 1 (Top-3). Removing the calibration rules gives higher Comp. (0.892) and Macro-F1 (0.436), and removing nomination gives higher Step@1/2/5. In Table 14, removing telemetry, nomination, or worked examples does not significantly lower macro-F1. The only significant effects are those of the worked examples on category accuracy (2.6 points) and of the calibration rules on Top-3 accuracy (4.9 points).
- Two runs of the same system differ by 4.0 points in macro-F1, about the same as the effect of removing nomination, telemetry, or worked examples (3.7 to 5.9 points). Only two runs of the auditor and one run of the direct judge are reported, on a test set of 30 tasks.
- The only baseline is a direct judge using the same LLM, and it receives neither the telemetry nor the per-test results. The comparison therefore cannot separate the effect of the extra information from that of the method itself. The auditor is also evaluated with only one LLM, and none of the diagnosis methods discussed in the related work, such as AgentRx (arXiv:2602.02475), TrajAudit (arXiv:2605.26563), and CHIEF (arXiv:2602.23701), is compared experimentally.
Questions for authors
- Given that the official OpenAI changelog lists the release of GPT-6 Sol on September 22, 2026, three days before the full-paper deadline, when was the quality control of the 3,452 samples performed, which exact model version was used, and was a different judge model used at any stage?
- Since the runs in Section 4 were recorded without the proxy and tool wrappers and therefore capture neither what the LLM received nor the raw tool outputs, can the toolkit record these differences on an unmodified harness, and if not, how would the auditor detect context corruption or mistranslation in practice?
- Of the 1,120 samples excluded from the benchmark, how many were excluded because the run did not complete, because the judge decided that the failure did not occur, or because the judge could not determine the outcome, and how does the category distribution change after filtering?
- Does the auditor's advantage over the direct judge remain significant when each system is run at least three times and the direct judge receives the same telemetry and per-test results?
- How does your data construction differ from Aegis (arXiv:2509.14295), which also injects faults into successful trajectories to construct labels?
- Given that the auditor flags 33.33% of the healthy controls and that all incidents were repaired in 68.51% of the passing sessions with incidents, have you checked its diagnoses on the 890 sessions against human annotation, and what share of passing sessions still contains an unresolved incident?
- As a suggestion for future work, would you consider evaluating telemetry-based detection on naturally occurring tool and context faults, and comparing the auditor with existing methods such as AgentRx (arXiv:2602.02475), TrajAudit (arXiv:2605.26563), or DoVer (arXiv:2512.06749) and with other auditor LLMs?