- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PROBEAGENT, a framework for automatically discovering and diagnosing agent failure modes. Its goal is to distinguish two kinds of failures: those caused by the difficulty of a particular task, and errors the agent repeats across different tasks.
The framework has three stages:
- Task construction and diagnosis. Starting from benchmark seed tasks, an LLM rewrites them into new executable tasks (probes). Each new task must pass three checks: the reference solution passes, doing nothing fails, and an independent review finds no shortcuts or loopholes. When the target agent fails, an LLM reads the trajectory and writes a failure explanation consisting of the erroneous action, where it occurs, the triggering condition τ (the task condition under which the error arises), and a corrective action.
- Recurrence testing. The framework constructs several new tasks (siblings) that preserve τ while changing the domain, data, tools, or output format, and runs the same agent on them again. An LLM judges whether the agent makes the same error, and the error is then classified as stable agent behavior or as specific to the original task.
- Grouping. Failures are grouped by their causes, and a one-sentence description is generated for each group.
The experiments cover 22 model–harness configurations and 10 benchmarks, yielding 1,244 error records and 536 failure-mode descriptions. The automatic evaluation reports 88.5% agreement. In the human evaluation, annotators accept 90.0% of the descriptions and correctly reject 83.3% of similar distractor descriptions.
Critical strengths
- The problem is valuable and the idea is clear. The paper separates "the task itself is hard" from "the agent repeatedly makes the same error". Rather than stopping at post-hoc explanation, it treats each explanation as a hypothesis, constructs new tasks to test it, and forms a "diagnose → construct tasks → diagnose again" loop. This goes a step beyond purely post-hoc attribution.
- Quality control during task construction is careful. Across 1,451 construction rounds, the independent review rejects a further 638 candidates that had already passed the execution checks, and shortcuts that bypass the task requirements are found in 565 rounds. This shows that "it runs" does not mean "it tests the right thing", which is also useful for research on the validity of agent benchmarks.
- The experiments are broad and the appendix is candid. The experiments cover many models and harnesses. The appendix honestly reports the trade-off in grouping granularity, groups for which no description could be generated, timeout differences, and environment defects. The ablation of the grouping constraint (78.0% → 91.8%) is convincing.
Critical weaknesses
W1. The recurrence test, a main addition over ProbeLLM, is neither validated nor sufficient to identify the cause of a failure.
- The trigger τ, the siblings built from it, and the recurrence verdict are all produced by GPT-5.6-sol (Appendix A). The paper does not check whether τ is correct or stable across repeated analyses of the same failure, whether siblings actually preserve τ, or whether recurrence verdicts agree with humans, and sibling builds are not reported separately in the admission statistics. Generated environments are often flawed: the manual audit finds a wrong expected value or a format ambiguity in 6 of 20 environments (Table 9), and Appendix A.1 notes that executable controls alone miss semantic drift, such as a GAIA question turned into an unrelated code-editing task.
- Recurrence on siblings does not show that τ is the cause. No sibling holds τ fixed while controlling task length or budget, and no intervention, such as giving the agent the corrective action, tests whether the error disappears. Timeouts are a plausible confound: Terminus 2 with Sonnet 5 times out on 46% of rollouts versus 12–13% under the other three harnesses (Appendix D.2), and a missing deliverable caused by a timeout would also count as recurrence. In addition, a sibling counts as reproducing if any one of its rollouts is judged a recurrence (Eq. 4), and a task-specific verdict only means that the error did not recur in the siblings tested.
- Few failure modes are supported by the recurrence test. Only 119 of 239 explanations reached sibling testing, and 20 met the agent-behavior criterion (Figure 3). Of the 565 induced groups, 404 (71.5%) have no linked sibling test, and 361 (63.9%) contain only one failure (Table 6).
W2. The evaluation measures agreement within the pipeline, and there is no baseline.
- Held-out failures are assigned to a group by the induction judge and confirmed by the same judge against the group's first member, and no-match outputs and failed calls are excluded (Appendix A.2). Since every role uses GPT-5.6-sol, the 88.5% and 91.8% figures measure the self-consistency of one model rather than diagnostic accuracy.
- The human evaluation uses 2 annotators and 30 failures and reports no inter-annotator agreement (Appendix C.5). Annotators judge whether a description matches the failure as specified by the pipeline; no one checks whether the failure explanation itself, namely the erroneous action, its location, τ, and the corrective action, is correct.
- The only comparison is with a fixed eight-category taxonomy (Figure 2), which shows finer granularity rather than correctness. There is no comparison with ProbeLLM (arXiv:2602.12966), which the paper extends, or with a simple baseline that asks an LLM to summarize failure modes. Although Eq. 1 defines the objective in terms of comparing search policies, no alternative policy is evaluated, whereas ProbeLLM itself reports comparisons with prior probing methods.
Questions for authors
- Under the same budget, how does PROBEAGENT compare with ProbeLLM (arXiv:2602.12966), both in failure discovery (τ-preserving siblings versus ProbeLLM's local perturbation of failing seeds) and in grouping (ProbeLLM's induction versus density-constrained grouping on the same failures)? Since Eq. 1 is defined over search policies, could you also compare with at least one alternative policy, such as exploration without sibling tests?
- In the held-out evaluation, how many failures were assigned to a group, how many received no match, and how many calls failed? Could humans annotate a sample of failures without seeing the pipeline's diagnosis, so that you can report the accuracy of the failure explanations (erroneous action, location, τ, corrective action), the agreement of the pairwise and recurrence judgments with human labels, and the agreement between annotators?
- How do you verify that a sibling preserves τ? Could you report the admission results of sibling builds separately, manually audit a sample of siblings, and report whether repeated analyses of the same failure yield a consistent τ?
- Does the recurrence judge require the sibling rollout to fail? How do the verdicts in Figure 3 change when timed-out rollouts are excluded, when recurrence must occur in a majority of rollouts, or when siblings are matched to the source probe in length and budget, and does the error disappear when the corrective action is given to the agent as a hint?
- Since Table 5 lists 13 seed tasks from 8 suites, how are the 14 seed tasks and 10 benchmarks in Section 4.1 counted? Could you also release the prompts and code, and give one end-to-end case each for an agent-behavior and a task-specific verdict, covering the seed task, the probe, the siblings with their instructions and checkers, the trajectories, and the verdicts?
- As a suggestion for future work, would you consider testing whether the discovered failure modes can be used to improve agents, for example through fine-tuning or prompting, as ProbeLLM does for LLMs?