acceptodds
Under review as a conference paper at ICLR 2027

Automated Discovery of Failure Modes in Agentic AI Systems

Abstract

Multi-agent systems composed of large language model-driven agents are increasingly used in real-world applications, yet evaluating their reliability remains challenging. Compared with single-model systems, their decentralized, multi-step, and stochastic behavior complicates failure analysis. We present a framework for automated identification of failure modes in agentic AI systems with limited or no historical interaction data. Our framework generates domain-specific evaluation scenarios under structured categorization of statistical tests to uncover system weaknesses. Our framework is evaluated against a real-world multi-agent customer-service system, enabling realistic failure-mode detection. The results have shown that our framework identifies failure modes missed by state-of-the-art evaluation approaches.

Primary area:
generative models
Keywords:
Agent, Muti-agent System, Agent Evaluation, Large Language Model, Generative AI, Agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UqNuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents an approach to discover failure modes in agentic AI systems. The framework consists of four stages, namely, Failure Case Canonicalization that uses LLM judge to produce a concise error summary from the failure traces, Coarse-Grained Failure Pattern Induction for clustering those canonical failures using LLMs, Auto-stop Hierarchical Refinement for recursively refining the failure cluster into sub-clusters using LLMs, and Statistical Validation that generates synthetic test cases for each of the sub-clusters at the leaf level and conducts A/B hypothesis testing against a baseline success rate to identify statistically significant failure modes. Further, mitigation is conducted in two ways, using Claude Opus 4.6 and human refinement. The framework is primarily tested with a conversational multi-agent system with a synthetically generated user simulator and test cases for customer service. Results show that the pipeline helps improve the success rate from 74.9% to 85.3%.

Critical strengths
  1. The framework is practically useful and works even without historical trajectories. It provides failure categories, helps identify statistically significant failure modes, and automated mitigation.
  2. The framework is general as demonstrated by applying it to τ\tau-bench agents.
  3. Results show that the overall framework helps improve task success on customer service tasks.
Critical weaknesses
  1. No baseline comparison against existing error attribution or failure mitigation work. The abstract claims that the framework identifies failure modes missed by capability benchmarks and LLM judge-based evaluation approaches but this is not supported by results.
  2. The first three stages in the pipeline are LLM-based. It would help to add accuracy (e.g., precision, recall) of each stage. There is some lightweight review of failures as stated in Appendix N but no details about the rigor of the human validation process is provided.
  3. The results are based on a single run of a single model. As stated in the paper, there is high level of uncertainty by changing the model. Thus, it would help to see results consistently improve over multiple runs and generalize across models.
Questions for authors
  1. How are the n=100 test cases generated? Are they targeted to try to test the particular failure category? Then, for the newly generated traces, do you re-label if the same failure occurs or just any failure?
  2. What is the error distribution after mitigation strategies are applied?
Reviewer_B8FcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a trace-driven framework for automatically discovering failure modes in agentic AI systems. The authors (1) canonicalize failed traces via an LLM judge, (2) induce coarse-grained failure scenarios, (3) refine them hierarchically with an auto-stop criterion, and (4) statistically validate each candidate mode via A/B testing against a baseline with multiple-hypothesis-testing correction, separating systematic failures from stochastic noise. They then use the discovered modes to drive an automated, human-guided mitigation loop. On a customer-service multi-agent system, focusing on statistically validated modes raises success from 74.9% to 85.3% and outperforms indiscriminate correction (+7.4 pp vs. +4.0 pp); the framework also generalizes across deterministic and dynamic-orchestration architectures (τ-bench, DROP) and to held-out sets.

Critical strengths

• Existing benchmarks measure single-agent "capability," and task-level success rates treat failures as isolated outcomes; multi-agent failures mostly arise from emergent interaction, long-horizon state, and stochasticity. The paper targets the real gap of "production reliability evaluation," and the motivation holds. Meanwhile, the positioning is clear, explicitly distinguishing "population-level, label-free failure-mode discovery" from per-trace failure attribution (Who&When, FAMAS, CHIEF) and from agent self-evolution.

• Uses A/B testing against a baseline with Bonferroni / Holm / BH multiple-hypothesis-testing correction (Table 7) to separate systematic weaknesses from stochastic noise.

• Validated across three architecture classes (deterministic-workflow MAS, dynamic-orchestration DROP, single-agent τ-bench retail/airline), and mitigation remains effective on unseen held-out sets.

Critical weaknesses

• The evidence is almost entirely synthetic: the so-called "real customer-service system" is self-built, and all 1,100 trajectories and statistical re-tests come from the LLM user simulator, with no real user logs or production data; "real-world" in the abstract is overstated and ecological validity is weak.

• The statistical validation risks circularity: Stage 4 significance is tested by regenerating samples from the same simulator, so it verifies consistency with the generator's distribution, not true failure prevalence.

• The main experiments lack a strong-baseline comparison; the abstract claims to discover failure modes missed by capability benchmarks and LLM-judge evaluation, but no head-to-head quantitative comparison against a strong LLM-as-a-judge baseline is given in the main text.

• The pipeline itself relies heavily on LLM judges—canonicalization, clustering, and refinement all use Gemini 3 Flash—without reporting inter-annotator agreement or human validation of canonicalization quality.

• Compute/cost is not accounted for: heavy use of Gemini 3 Flash and Claude Opus 4.6, with n=100 re-tests per scenario, and scalability to large-scale production traces is entirely undiscussed.

Questions for authors
  1. All 1,100 trajectories and every statistical re-test come from an LLM user simulator, with no real user logs or production data. Could "real-world customer-service system" be restated more accurately as a "simulated production-style system"? On what basis does the framework claim to identify failure modes in real deployments when no real traces are used?
  2. Stage 4 significance is tested by regenerating n=100 samples for each leaf failure scenario from the same data generator. Does this test verify consistency with the generator's distribution, or true failure prevalence? Could a small human-labeled ground-truth set (even a few dozen traces) be used to break the loop and confirm the discovered failure modes actually exist?
  3. The abstract claims to discover failure modes missed by capability benchmarks and LLM-judge evaluation, but no head-to-head quantitative comparison against a strong LLM-as-a-judge baseline or a failure-attribution method (e.g., Who&When / FAMAS) appears.
Reviewer_Eh1kOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper focus on the practical scenario on MAS system improvment. It proposes a framework for identifying recurring failure modes from large scale MAS trajectories and leveraging them for system-level improvement. The framework combines LLM-based failure attribution, hierarchical failure mode detection, and statistical validation to identify significant failure patterns and prioritize mitigation. The authors demonstrate automated code repair guided by the identified failure modes and further investigate the limitations of LLM-as-a-Judge and evaluation uncertainty introduced by user simulation.

Critical strengths
  • The targeted scenario is meaningful and practical. The paper proposes a realistic,well-designed, working-well framework for MAS system improvement. The methodology for extracting failure modes sounds solid.

  • Downstream repair performance is good.

  • Experiments are extensive. Generalization beyond MAS and hidden failure attribution in MAS is intriguing and insightful.

Critical weaknesses
  • While the authors emphasized the "production-scale agents" in their study, this was insufficiently justified in the draft. It would be better to clarify in what ways the proposed setting is more "production-scale" than other benchmarks. Honestly, the scale of the constructed benchmark looks similar to that of other benchmarks, and the tasks in other benchmarks also reflect real-world user queries. Therefore, the claim sounds a bit overstated.

  • The trajectories were generated from 62 scenarios. How diverse are the trajectories? If they are not diverse enough, might it be easier to identify recurring failure modes?

  • The related work is insufficient. It covers only Who&When and MAST, which is limited. Some recent works like DoVer[1], MARS [2], FlowFixer [3] (diagnosis-guided repair), and AutoSaddler [4] (harness optimization) also leverage failure analysis and execution feedback to improve the system. So, the claim that existing works did not address system improvement based on identified failures sounds overstated. In my opinion, instead of making such a claim, the authors could compare instance-level repair (existing works) and recurring failure-mode-driven system repair (this work) and clarify the unique advantages of the proposed method and the research gap it bridges.

  • In the statistical validation, 100 synthetic test cases were generated for each leaf-level failure scenario. However, it is unclear how the test cases were generated to reproduce the specific failure mode. Furthermore, a low task success rate for a scenario does not guarantee the reproducibility of the failure mode. Task failures may consist of not only the failure mode in the leaf node but also other failure modes. So, instead of relying on a simple task failure rate, the authors should verify the reproducibility of the targeted failure mode in the generated trajectories.

[1] DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [2] MARS: Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search [3] FlowFixer: Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference [4] AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.