acceptodds
Under review as a conference paper at ICLR 2027

MAADBench: The Refreshable Paradigm For Anomaly Detection In Multi-agent Systems

Abstract

Recent studies report LLM-based multi-agent systems (MAS) fail at rates of 41%–87%, yet to our knowledge, no benchmark to date supports systematic anomaly detection (AD) for them. Building MAS AD benchmarks is hard because they must remain valid as LLM systems evolve: tasks may leak into training data and thus be memorized by LLMs, traces and anomaly patterns expire as backbones evolve, and labels must be provided reliably for each refresh. To address these challenges, we present **MAADBench** (**MA**: multi-agent; **AD**: anomaly detection), the first refreshable MAS AD benchmark designed for diverse evolving LLM backbones underlying the agents. MAADBench combines (1) sampled-and-coupled generative tasks over a -task space to mitigate task leakage, (2) refreshable trace generation under configurable LLM backbones, and (3) automated provision of cost-free, deterministic step-level labels for fine-grained AD evaluation. Beyond offering the paradigm itself, we run MAADBench with five SOTA LLM backbones, and release the dataset **MAADBench-Full** with 5,200 step-labeled traces. Benchmarking 25 AD methods on MAADBench dataset reveals substantial limitations in current approaches: they rely heavily on supervision, struggle with subtle MAS-specific anomalies, and lack robustness across LLM backbones. These gaps point to a rich research agenda for MAS-specific anomaly detection, with MAADBench providing a systematic and refreshable testbed for method development and evaluation. We open-source MAADBench-Full at <https://huggingface.co/datasets/hww123/MAADBench-full>.

Primary area:
datasets and benchmarks
Keywords:
Multi-agent systems, Anomaly Detection, LLM, Benchmark, Long-lived

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RvsFOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The central subject in this paper is about failure in multi-agent LLM systems. Authors have listed few main problems with available datasets that have been created for detecting these types of failures, for example traces in the data set which are generated by a specific model become stale when the model changes or evolve, and also limitations in labeling of the steps in those datasets which is because of the high labeling cost and also the fact that most of the available labels don't have enough granularity, or for example another problem which has been mentioned by the authors is the leakage of tasks into the training data. All these points motivated the authors to propose MAADBench which tries to address these issues by replacing a fixed dataset with a benchmark generator. In this framework, more complex tasks are being generated in a bottom-up manner. These complex tasks in MAADBench are composed of what authors are calling atomic tasks and are covering different areas, like coding, math, etc. The atomic tasks have known answers and are composed in a escape room sort of setting. This makes the labeling of each step has a clear label which freely comes from those atomic tasks labels. This structure also makes the whole pipeline repeatable relatively easily for different types of models. Then the authors have used their framework to generate a dataset of 5200 traces, and investigated different types of anomalies and also benchmarked several anomaly detection options and reported their findings.

Critical strengths

The main point of strength in this paper for me is the structure of the pipeline. The bottom-up format that enables the tasks to be generated out of atomic steps with almost free step-level labeling costs and clear and deterministic labels for each step, which also can be regenerated easily even if the backbone model changes or evolves, is an interesting setting and nice idea. In addition, I think the evaluation provided by the authors is broad in different aspects, like number of traces, backbones, benchmarked anomaly detection methods, etc.

Critical weaknesses

My main critic about this paper is that although it is claiming that it is targeting the anomaly detection in multi-agent systems, the structure and design of the proposed pipeline is not really covering some important aspects of failure in multi-agent systems. For example, in multi-agent systems some of the main reasons of failure could be issues in joint reasoning and negotiations among agents, or for example agents duplicate other agents' tasks, etc. In the proposed framework in this paper, these important multi-agent aspects of failure haven't been covered properly, because the settings puts the agents in a setting which each agent does its own task and hands over the result to the next one, which doesn't provide a free-form of communication which could potentially result in those aforementioned types of miscommunication failures that exists in multi-agent systems. The only way of interaction that I could find in the pipeline was handing over the outcome to next agent in a fixed format along a chain that has been predetermined, which could potentially lead to passing the wrong value to the next agent and this wrong value then be propagated in the chain, but I couldn't find other forms of interaction failures like misunderstanding, negotiation and joint reasoning issues, etc. So it is hard to accept the claim that the provided pipeline really covers anomalies of multi-agent systems in general. A more accurate explanation is that it mainly generates errors coming out of single-agent failures which are then propagated in the pipeline, instead of multi-agent failure aspects that come from interactive part of multi-agent systems. I still think the proposed pipeline is interesting and valuable, but the scope and framing of the contribution needs to be adjusted and explained in more accurate way.

Questions for authors
  • As I mentioned in the weakness section, in my opinion, the main issue is not properly addressing the sort of failures in multi-agent systems which come from a free-form of interaction and communication. Could authors elaborate on this point and clarify whether in their set up any labeled failure comes out of these free format communications, rather than passing a wrong value from a single agent failure to the other agents down the chain? I see that in figure 9 authors have reported MAST’s inter-agent categories like ignored input, but my understanding is that these labels mainly correspond to the cases that coming from a single agent's error, not from an interaction error. If authors also agree that there is no such a failure mode in the current set up, I would suggest that authors either clarify and clearly explain this point in the paper or add some sort of real interaction failures.

  • My understanding is that authors have used the true number of anomalies in their test set to set the threshold for their accuracy and F1 calculations. If this understanding is correct, I think since this number is not known in real settings, I think further explanations on how the F1 score and accuracy changes with the threshold value in validation set is needed.

  • I think there are some inconsistencies in the evaluation that need to be addressed. For example Claude Opus 4.8 contributes 400 traces at one temperature, but the other backbones contribute 400 traces at each of 3 temperatures. However, comparisons (for example in Table 2) seem to be conducted side by side without analyzing the effect of this difference. Could authors elaborate more on the impact of this inconsistency on the conclusions related to cross-backbone aspects?

6 comments

We thank the reviewer for the positive assessment of MAADBench's bottom-up construction, deterministic labeling, and broad experimental evaluation. We address the comments as below.

W1. Failures arising from free-form inter-agent communication and coordination

Response: In our environment, agents interact through structured handoffs. Thus, failures raised from free-form negotiation and joint deliberation are not covered. We now state this explicitly in our revised paper in the Discussion section (Appendix B).

We would like to clarify, however, that the inter-agent labels do not describe errors made by a single agent in isolation. Our ignored-input cases (FM-2.5) are handoff failures. More precisely, an upstream agent produces a correct output and passes it on, but the downstream agent ignores or replaces it. We verify upstream correctness before assigning this label; if the upstream message itself is wrong, the case is labeled as FM-5 (now called Downstream Failures in the revision) instead. The failure therefore arises between agents: correct information is sent but not used by the receiver, which matches MAST’s definition of inter-agent misalignment. They are genuine communication failures that cannot arise in a single-agent system – albeit narrower than negotiation or joint-reasoning failures. We have added this clarification into Appendix F for Figure 9.

We also want to explain why we chose this scope. Structured MAS is a significant paradigm in its own right. Well-known systems and frameworks such as MetaGPT, ChatDev, CrewAI, and LangGraph illustrate how predefined workflows can coordinate multiple agents with controllable, auditable workflows.

We chose structured interaction to make task requirements and action-level checks explicit and reproducible. Extending the benchmark to free-form communication would require additional interaction specifications and validated labeling procedures, which remains an important and challenging direction for future work.

Revision: We have added a short discussion about future work to extend MAADBench to free-form communication in the Discussion section (Appendix B), and a further explanation of the ignored-input cases (FM-2.5) with other failure modes in Appendix F.

W2. How does threshold selection affect F1 and other threshold-dependent metrics?

Response: Consistent with prior benchmarks [1], threshold-independent AUROC is our primary metric in the main text. Therefore, threshold selection does not affect our main conclusions.

For completeness, we also report F1, accuracy, and balanced accuracy in the Appendix. Since these metrics require a decision threshold, we followed standard practice [2–5] and set it using the anomaly ratio. However, we agree that reporting them under validation-selected thresholds further strengthens the analysis.

To address the reviewer's suggestion, we now extended our evaluation to 18 detectors with native thresholding capabilities and ran each over five seeds. This analysis focuses on non-LLM detectors, as including LLM-based detectors would incur substantially greater computational costs. We adopt the following evaluation methodology. Following the room-disjoint split protocol, we further separate 20% of the training set for validation while keeping the same prior held-out test set. This yielded a non-overlapping 64/16/20 train/validation/test split. We selected thresholds maximizing validation F1 or accuracy respectively and applied each to the test set (instead of the a priori ratio-based threshold). Table A below reports the numbers. Due to space constraints, we show only the best-performing methods (by AUROC) from each supervision/family group here. However, full results of all methods are available in Table 12 (Appendix H) of the revised manuscript.

Table A. F-1 and Accuracy with threshold selected by anomaly ratio (Ratio-F-1/Ratio-Acc ) versus threshold selected by Validation set (Val-F-1/Val-Acc ) . Results are means across seeds;

SupervisionMethodFamilyF1 (Ratio-F-1 → Val-F1)Acc (Ratio-Acc → Val-Acc)AUROC
—RandomRandom.184→.287.717→.823.502
UnsupervisedIsolation ForestTabular.243→.306.739→.823.550
One-ClassOC-SVMTabular.255→.314.745→.823.592
DOMINANTGraph.225→.456.734→.812.713
BlindGuardMAS-specific.134→.300.696→.819.440
Semi-SupervisedDeepSADTabular.462→.471.813→.841.773
GGADGraph.175→.302.715→.822.522
G-Safe (semi-sup)MAS-specific.223→.417.841→.844.735
SupervisedXGBoostTabular.551→.560.845→.859.855
G-SafeguardMAS-specific.476→.511.846→.849.804

Analysis of new results in Table A. The above comparison shows that thresholds selected on the validation set yield higher F1 and accuracy than thresholds based on the true anomaly ratio in the test set (the protocol used in our original evaluation). This finding is consistent with prior work showing that the F1-optimal threshold depends on a detector's attainable F1 score, rather than on the prevalence of the positive class [6]. Intuitively, setting the threshold so that the number of predicted anomalies matches the expected number of true anomalies constrains precision and recall to be equal, which need not coincide with the threshold that maximizes F1. In short, using the test anomaly ratio did not inflate our previously reported results; if anything, the original evaluation protocol was conservative. As AUROC is a threshold-independent metric, it is unaffected by the proposed threshold-selection methodology. Therefore, the AUROC results reported in the main paper remain unchanged.

Revision: We have added the full results and analysis into Table 12 in Appendix H.

[1] Han et al. ADBench. 2022.
[2] Zong et al. Deep Autoencoding Gaussian Mixture Model. ICLR 2018.
[3] Bergman & Hoshen. Classification-Based Anomaly Detection. ICLR 2020.
[4] Shenkar & Wolf. Internal Contrastive Learning for Tabular Anomaly Detection. ICLR 2022.
[5] Dai et al. Deep Noise Evaluation for Tabular Anomaly Detection. AAAI 2025. [6] Lipton et al. Optimal Thresholding of Classifiers to Maximize F1 Measure. ECML-PKDD 2014.

W3. How does the unequal number of temperature settings across backbones affect comparisons?

Response: We thank the reviewer for raising this question. The difference arises from an API constraint, namely, Claude Opus 4.8 supports only one single temperature. In short, the unequal temperature coverage does not change our cross-backbone conclusions: 1) The success-rate trends in Table 2 hold at every tested temperature, and 2) As noted in Section 4, all detection results use MAADBench-Eval, which contains 400 (equal number of) traces per backbone at a single temperature. We elaborate on the two points below.

Task success rates (Extending Table 2). The original Table 2 averages each non-Opus backbone over its three temperatures. To allow direct comparison, we now also report results for each temperature separately (Table 8 in the Appendix):

LLMTemp# RoomEasyMediumHardExtremeOverallPuzzle Attempted / TotalPuzzle SR
Claude Opus 4.8-40070.0%63.0%29.0%30.0%48.0%805 / 100061.0%
Claude Sonnet 4.60.040053.0%43.0%9.0%23.0%32.0%692 / 100042.6%
0.340052.0%31.0%14.0%12.0%27.3%679 / 100039.5%
0.640052.0%32.0%10.0%23.0%29.2%694 / 100042.1%
DeepSeek-R10.040060.0%39.0%12.0%18.0%32.2%643 / 100038.6%
0.340069.0%38.0%12.0%22.0%35.2%692 / 100044.3%
0.640071.0%49.0%13.0%26.0%39.8%719 / 100048.8%
GPT-4.10.040047.0%21.0%4.0%0.0%18.0%509 / 100021.4%
0.340046.0%20.0%2.0%0.0%17.0%507 / 100021.2%
0.640046.0%16.0%3.0%0.0%16.2%505 / 100020.5%
GPT-5.40.040061.0%27.0%11.0%1.0%25.0%563 / 100030.2%
0.340062.0%29.0%11.0%0.0%25.5%566 / 100030.7%
0.640057.0%25.0%9.0%0.0%22.8%560 / 100029.2%

The broad cross-backbone trends remain consistent. Namely, Opus outperforms every evaluated configuration, while DeepSeek-R1 and Sonnet outperform GPT-5.4 and GPT-4.1 at all tested temperatures. While temperature setting has a higher impact for certain models (DeepSeek), it does not overturn the conclusions.

Detector training. Table 2 reports task success rates on MAADBench-Full, which covers all temperature settings. All detection results, in contrast, are computed on MAADBench-Eval, which selects a single temperature per backbone (temperature 0, or the default for Opus 4.8). Each backbone therefore contributes exactly 400 traces to MAADBench-Eval, so detector training and evaluation are balanced across backbones.

Revision: We have added the extended Table 2 (Table 8, Appendix F) into the manuscript. At the beginning of Section 4, we further stressed that the data (MAADBench-Eval) used in the actual evaluation has a balanced portion (400 traces each) for each LLM backbone.

Reviewer_XsKFOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MAADBench, a refreshable benchmark for anomaly detection in LLM-based multi-agent systems. It combines combinatorial task generation, configurable LLM backbones, and deterministic step-level labeling. The authors release 5,200 labeled traces and evaluate 25 anomaly detection methods, highlighting challenges in supervision, backbone robustness, and anomaly propagation.

Critical strengths
  • The paper addresses an important problem and proposes a useful framework for generating labeled multi-agent execution traces.

  • The evaluation covers diverse anomaly detection methods, supervision levels, and LLM backbones, providing useful empirical observations.

Critical weaknesses
  • Appendix G.1 splits data at the action level rather than the trajectory level, allowing correlated actions and overlapping execution contexts to appear in both training and testing. This may inflate detection performance and undermine generalization claims.

  • Despite the claimed (10^{37}) task space, the benchmark relies on a fixed escape-room workflow with limited agent roles and reasoning domains. Combinatorial diversity does not necessarily imply diversity in multi-agent behaviors or failure mechanisms. Furthermore, deterministic oracle-based labeling ensures reproducibility but does not establish semantic correctness or completeness. The lack of independent label validation and evaluation on realistic multi-agent workflows limits the benchmark's broader applicability.

  • Lack of human validation for oracle-generated labels. The paper relies entirely on deterministic oracle-based labeling without reporting an independent human audit.

  • The central claims regarding refreshability and anomaly propagation lack convincing experimental support. Although traces are generated using five LLM backbones, the paper does not evaluate generalization to unseen backbones or model-version shifts. Testing on backbone-specific subsets is insufficient to demonstrate robustness under benchmark refreshes. Similarly, the propagation analysis relies on sequential failure correlations and identifies root causes primarily through failure order, without controlled interventions to establish causal relationships. These limitations weaken the empirical support for the paper's main claims.

Questions for authors

See the weaknesses above. In particular, can the authors provide trajectory-disjoint evaluation, independent label validation, and cross-backbone generalization experiments?

3 comments

We thank the reviewer for recognizing the importance of a generalizable benchmark for MAS anomaly detection and the breadth of our evaluation.

W1. Room-disjoint Train-test split

Response: Indeed, our evaluation reported in the paper already use room-disjoint splitting across LLM backbones. We apologize that Appendix G.1 did not describe this sufficiently clearly. To clarify, we collect trajectories across backbones, and perform an approximately 80/20 train–test split by room ID. Thus, all traces associated with a given room or any of its subtasks (including traces generated by different LLM backbones) are assigned exclusively to either training or testing, never both. This rules out a shortcut in which a detector memorizes a subtask’s answer from a training trace and reuses it on a test trace from the same room.

Revision: We have added this clarification into Section 4.1 and Appendix G.1.

W2. Limited workflow diversity despite the large combinatorial task space.

Response: We first clarify that the purpose of the 103710^{37} combinatorial task space is to mitigate task leakage through fresh task generation, not to diversify multi-agent behaviors. Importantly, combinatorial diversity alone does not establish diversity in agent interactions nor failure mechanisms, and we do not present task-space size as evidence of such coverage. More broadly, while MAS can encompass many agent organizations, workflows, and communication protocols, we explicitly acknowledge in Appendix B that MAADBench does not exhaustively cover this design space. However, broader coverage alone does not address two fundamental challenges: costly labeling and benchmark obsolescence as models evolve. We therefore prioritize longevity through refreshability, establishing a foundation for sustained evaluation and future expansion. Extending coverage remains important, provided new extensions preserve refreshability to maintain lasting benchmark value.

W2 & W3. Deterministic labeling versus semantic correctness and independent validation.

Response: We thank the reviewer for this question. We would like to clarify that not relying on human annotation is a deliberate design choice of MAADBench. Human labels would have to be redone for every new backbone, which would reintroduce exactly the bottleneck the benchmark is designed to remove. Instead, each task by design is constructed with predefined, verifiable correctness criteria, so that every label can be produced automatically by a deterministic procedure (which we call “oracle”).

Accordingly, MAADBench provides a deterministic labeling algorithm, not a fixed collection of correct label artifacts. The correctness of this algorithm is established through two complementary mechanisms. (1) Ground-truth sources. Some atomic tasks are drawn from established benchmarks, such as GSM-Hard and LiveCodeBench, which provide already validated ground-truth answers. (2) Expert-verified composition rules. Five coauthors with MAS expertise verified the rules used to propagate these ground truths through composed workflows. For example, when the output of a GSM-Hard task serves as input to a downstream agent, the corresponding gold label for the derived subtask is computed according to these verified rules. During development, we applied preliminary versions of the rules to pilot traces spanning all five backbones, inspected the resulting oracle labels, and jointly resolved edge cases, iteratively refining the rules reported in Appendix D.5. This process validated and debugged the deterministic labeling algorithm itself, rather than producing benchmark labels. Once finalized, the algorithm generates labels automatically for all traces, including those from previously unseen backbones, without further human intervention.

Revision: We have added the explanation of this procedure into Appendix D.5.

W4 (a). Refreshability and cross-backbone generalization.

Response: We would like to distinguish two related but distinct properties: benchmark refreshability and detector robustness.

Refreshability is a property of the benchmark: MAADBench’s execution traces and labels can be recollected for new LLM backbones, without requiring manual annotation.

Robustness is a property of an individual detector: whether a detector trained on one set of traces performs well on traces from another source, such as an unseen backbone.

MAADBench’s contribution and objective is refreshability of the benchmark, and not robustness of detectors. MAADBench does not guarantee the robustness of the detectors it evaluates.

What refreshability does provide is the ability to ask such questions. Static benchmarks with fixed traces cannot test a detector on a newly released backbone, whereas MAADBench can generate labeled traces for it on demand. However, we are glad to conduct an illustration robustness experiment to demonstrate the refreshability of MAADBench.

Transfer protocol. We hold out Claude Opus 4.8 and compare two training conditions on an identical test set: the Opus traces from the shared, room-disjoint test split. ”Opus included“ trains on all backbones. “Opus held” out removes Opus from training and validation. Because the test set is fixed, any gap reflects only whether Opus was seen during training, not differences in test difficulty. Mean AUC over SVM, XGBoost and G-Safeguard (5 seeds) is 0.772 ± 0.018 with Opus included vs. 0.778 ± 0.007 held out.

Table B. AUROC on Opus 4.8 test traces (mean ± std over 5 seeds).

MethodOpus includedOpus held out
SVM.729 ± .029.766 ± .011
XGBoost.807 ± .017.797 ± .009
G-Safeguard.780 ± .025.770 ± .019
Mean.772 ± .018.778 ± .007

The purpose of this experiment is to demonstrate how MAADBench can be used to evaluate detector behavior under varying conditions. It is not intended to establish or compare detector robustness. Here, Opus 4.8 serves as a proxy for an unseen backbone; for any newly released backbone, the same experiment can be run directly after using MAADBench to generate and label its traces.

Revision: We have included the above discussion and experiment into Appendix H, Table 10 as an illustration of evaluations enabled by MAADBench’s refreshability.

W4 (b). Anomaly propagation and causal interpretation.

Response: Our work targets anomaly detection in MASs rather than root-cause analysis or causal attribution. We agree that causal attribution requires counterfactual reasoning and controlled re-execution, which are beyond our scope. Instead, our analysis examines how anomaly detectability varies across workflow stages, with a particular focus on early-stage anomalies, which may be especially important to identify in real-world deployments to help prevent downstream execution issues. Such observations may also serve as useful inputs to subsequent root-cause or causal analyses, although they do not themselves constitute causal evidence. To avoid overinterpretation, we have replaced the term “root causes” with “early-stage anomalies” and “propagation” with “co-occurrence” throughout the revised manuscript.

Revision: We have updated the terminology throughout the paper and clarified this distinction, especially in Sections 3 and 4.4.

Dear Reviewer, As the discussion period draws to a close, we would greatly appreciate your feedback on whether our responses and additional experiments address your concerns. We would be happy to clarify any remaining questions. Thank you for your time and consideration.

Reviewer_Y1c8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this work composes atomic tasks with verifiable answers into dependency-linked escape-room tasks, generates multi-agent traces, and labels anomalies at action and task levels. Data can be regenerated when the backbone changes. this work collects 5,200 traces from five backbones, compares 25 detectors, and analyzes performance across actions, models, and failure positions. Its main contribution is a regenerable environment with automatic anomaly labels.

Critical strengths

1.Jointly generating verifiable atomic tasks and dependencies allows new backbone traces to be annotated without repeated manual step-level labeling. The action-level oracle supplies a reproducible correctness standard for the defined structured tasks and enables study beyond final task success. 2. Five backbones and 25 methods cover multiple supervision settings, tabular/graph approaches, and LLM detectors. Separating action correctness from final puzzle outcomes helps show why local errors need not imply overall failure and why stronger task-solving models need not produce easier-to-detect anomalies.

Critical weaknesses
  1. The split design is my main concern. Appendix G.1 randomly assigns 80% of actions to training and 20% to testing, while features include all outputs up to the current action. Highly related actions and shared prefixes from one trace can therefore occur in both splits; the same rooms are also reused across backbones. This does not establish transfer to new rooms or backbones. Appendix G.2 additionally uses the known test anomaly count to set thresholds, giving F1/recall access to information unavailable in ordinary testing. Threshold-free AUROC is unaffected by this latter issue.

  2. The propagation labels support a weaker interpretation than the root-cause discussion. Appendix Table 8 labels a later error as propagation whenever the same puzzle already has an upstream error; Section 4.4 also assigns Origin/Middle/Downstream by order. These rules locate errors in a sequence but do not distinguish consequences of an earlier error from independent new mistakes. The claim that true root causes are harder to detect is therefore not directly established by these labels.

Questions for authors
  1. Do the main results persist when splitting by base room ID, keeping all traces from that room across backbones in one split? Please also select thresholds on validation data. This reevaluation would directly affect my confidence in the benchmark results.

  2. Have the authors checked which downstream errors persist after correcting upstream outputs? Without such evidence, could propagation and Origin conclusions be restricted to the temporal definitions actually used?

2 comments

We thank the reviewer for recognizing the value of deterministic action-level verification, regenerable trajectories, and the distinction between local action errors and final task outcomes.

W1/Q1. Train-test split

Response: Indeed, our evaluation reported in the paper already use room-disjoint splitting across LLM backbones. We apologize that Appendix G.1 did not describe this sufficiently clearly. To clarify, we collect trajectories across backbones, and perform an approximately 80/20 train–test split by room ID. Thus, all traces associated with a given room or any of its subtasks (including traces generated by different LLM backbones) are assigned exclusively to either training or testing, never both. This rules out a shortcut in which a detector memorizes a subtask’s answer from a training trace and reuses it on a test trace from the same room.

Revision: We have added this clarification into Section 4.1 and Appendix G.1.

W1/Q1. Thresholding analysis

Response: Consistent with prior benchmarks [1], threshold-independent AUROC is our primary metric in the main text. Therefore, threshold selection does not affect our main conclusions.

For completeness, we also report F1, accuracy, and balanced accuracy in the Appendix. Since these metrics require a decision threshold, we followed standard practice [2–5] and set it using the anomaly ratio. However, we agree that reporting them under validation-selected thresholds further strengthens the analysis.

To address the reviewer's suggestion, we now extended our evaluation to 18 detectors with native thresholding capabilities and ran each over five seeds. This analysis focuses on non-LLM detectors, as including LLM-based detectors would incur substantially greater computational costs. We adopt the following evaluation methodology. Following the room-disjoint split protocol, we further separate 20% of the training set for validation while keeping the same prior held-out test set. This yielded a non-overlapping 64/16/20 train/validation/test split. We selected thresholds maximizing validation F1 or accuracy, respectively, and applied each to the test set. Table A below reports the numbers. Due to space constraints, we show only the best-performing methods (by AUROC) from each supervision/family group here. However, full results of all methods are available in Table 12 (Appendix H) of the revised manuscript.

Table A. F-1 and Accuracy with threshold selected by anomaly ratio (Ratio-F-1/Ratio-Acc ) versus threshold selected by Validation set (Val-F-1/Val-Acc ) . Results are means across seeds;

SupervisionMethodFamilyF1 (Ratio-F-1 → Val-F1)Acc (Ratio-Acc → Val-Acc)AUROC
—RandomRandom.184→.287.717→.823.502
UnsupervisedIsolation ForestTabular.243→.306.739→.823.550
One-ClassOC-SVMTabular.255→.314.745→.823.592
DOMINANTGraph.225→.456.734→.812.713
BlindGuardMAS-specific.134→.300.696→.819.440
Semi-SupervisedDeepSADTabular.462→.471.813→.841.773
GGADGraph.175→.302.715→.822.522
G-Safe (semi-sup)MAS-specific.223→.417.841→.844.735
SupervisedXGBoostTabular.551→.560.845→.859.855
G-SafeguardMAS-specific.476→.511.846→.849.804

Analysis of new results in Table A. The above comparison shows that thresholds selected on the validation set yield higher F1 and accuracy than thresholds based on the true anomaly ratio in the test set (the protocol used in our original evaluation). This finding is consistent with prior work showing that the F1-optimal threshold depends on a detector's attainable F1 score, rather than on the prevalence of the positive class [6]. Intuitively, setting the threshold so that the number of predicted anomalies matches the expected number of true anomalies constrains precision and recall to be equal, which need not coincide with the threshold that maximizes F1. In short, using the test anomaly ratio did not inflate our previously reported results; if anything, the original evaluation protocol was conservative. As AUROC is a threshold-independent metric, it is unaffected by the proposed threshold-selection methodology. Therefore, the AUROC results reported in the main paper remain unchanged.

Revision: We have added the full results and analysis into Table 12 in Appendix H.

[1] Han et al. ADBench. 2022.
[2] Zong et al. Deep Autoencoding Gaussian Mixture Model. ICLR 2018.
[3] Bergman & Hoshen. Classification-Based Anomaly Detection. ICLR 2020.
[4] Shenkar & Wolf. Internal Contrastive Learning for Tabular Anomaly Detection. ICLR 2022.
[5] Dai et al. Deep Noise Evaluation for Tabular Anomaly Detection. AAAI 2025. [6] Lipton et al. Optimal Thresholding of Classifiers to Maximize F1 Measure. ECML-PKDD 2014.

W2/Q2. Anomaly propagation and root cause

Response: Our work targets anomaly detection in MASs rather than root-cause analysis or causal attribution. We agree that causal attribution requires counterfactual reasoning and controlled re-execution, which are beyond our scope. Instead, our analysis examines how anomaly detectability varies across workflow stages, with a particular focus on early-stage anomalies, which may be especially important to identify in real-world deployments to help prevent downstream execution issues. Such observations may also serve as useful inputs to subsequent root-cause or causal analyses, although they do not themselves constitute causal evidence. To avoid overinterpretation, we have replaced the term “root causes” with “early-stage anomalies” and “propagation” with “co-occurrence” throughout the revised manuscript.

Revision: We have updated the terminology throughout the paper and clarified this distinction, especially in Sections 3 and 4.4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the AC, PC, and reviewers for their constructive feedback on MAADBench. All three reviewers recognized MAADBench’s core contributions, including its deterministic action-level labeling framework, refreshable traces that eliminate the need for repeated manual annotation, and a broad experimental evaluation (5 LLM backbones, 25 anomaly detectors).

The remaining points are few and mainly concern details that were under-emphasized due to space constraints. We summarize them below; together with the revised manuscript and new results, we believe they fully address the reviewers’ questions.

1. Concerns that were misunderstandings, already handled in the original paper

  • Train/test leakage (Y1c8 -W1/Q1, XsKF-W1). Our experiments already use a room-disjoint split across all backbones: every trace of a room, from any backbone, falls either entirely in train or entirely in test. Appendix G.1 described this insufficiently, and we have now clarified it.

  • Threshold dependence (Y1c8-W1/Q1, RvsF-W2). All main results and conclusions use threshold-free AUROC. The threshold-affected metrics were all in the supplementary appendix, which followed the standard protocol of the prior literature (yet see below for additional experiments we conducted).

  • Backbone imbalance (RvsF-W3). All detection experiments use MAADBench-Eval, which is balanced by design, namely, it contains exactly 400 traces per backbone. The temperature difference only concerns the success rates in Table 2, which are computed on a larger dataset MAADBench-Full.

  • Inter-agent labels (RvsF-W1). Our ignored-input labels only capture the downstream anomalies with correct upstream agent input. They therefore capture genuine handoff failures between agents, not propagated single-agent errors.

2. New experiments requested

  • Validation-selected thresholds (Y1c8-W1/Q1, RvsF-W2) (18 detectors × 5 seeds, 64/16/20 split): compared with the original ratio-based thresholding, these new validation thresholds yield higher F1 and accuracy. The original protocol was therefore conservative, not inflated.

  • Held-out-backbone evaluation (XsKF-W4) (Opus excluded from training): illustrates the cross-backbone studies that MAADBench’s refreshability enables.

3. Conceptual clarifications

  • Refreshability vs. robustness (XsKF-W4): We clarify that refreshability is a property of the benchmark, while cross-backbone robustness is a property of the individual detectors. MAADBench claims the former, not the latter: it enables researchers to study detector robustness but does not claim or guarantee it.

  • Scope (RvsF-W1, XsKF-W2): MAADBench targets structured multi-agent workflows, a major paradigm (e.g., MetaGPT, ChatDev, LangGraph). Free-form negotiation is acknowledged as future work.

  • Early-stage anomalies vs. root causes (Y1c8-W2/Q2, XsKF-W5): our analysis characterizes where anomalies occur in a trace, not causal attribution. We revised the terminology accordingly.

  • Label validity (XsKF-W3): MAADBench provides a correct labeling algorithm rather than a fixed set of label artifacts. Its correctness rests on ground truths from established benchmarks and on labeling rules verified by experts.

4. Main changes incorporated into the revised manuscript, primarily into the appendix

  • Clarified the split protocol (Section 4.1, Appendix G.1) and the 400-traces-per-backbone design of MAADBench-Eval (Section 4). Added the validation-based threshold analysis (Appendix H).

  • Revised the terminology on early-stage anomalies and co-occurrence (Sections 3, 4.4), and further explained the automatic labeling procedure (Appendix D.5).

  • Added discussions of scope and of free-form communication as future work (Appendix B), and clarified the criterion for inter-agent labels (Appendix F).

Taken together, we believe the revisions and new results comprehensively address the reviewers’ concerns and further strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.