- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The central subject in this paper is about failure in multi-agent LLM systems. Authors have listed few main problems with available datasets that have been created for detecting these types of failures, for example traces in the data set which are generated by a specific model become stale when the model changes or evolve, and also limitations in labeling of the steps in those datasets which is because of the high labeling cost and also the fact that most of the available labels don't have enough granularity, or for example another problem which has been mentioned by the authors is the leakage of tasks into the training data. All these points motivated the authors to propose MAADBench which tries to address these issues by replacing a fixed dataset with a benchmark generator. In this framework, more complex tasks are being generated in a bottom-up manner. These complex tasks in MAADBench are composed of what authors are calling atomic tasks and are covering different areas, like coding, math, etc. The atomic tasks have known answers and are composed in a escape room sort of setting. This makes the labeling of each step has a clear label which freely comes from those atomic tasks labels. This structure also makes the whole pipeline repeatable relatively easily for different types of models. Then the authors have used their framework to generate a dataset of 5200 traces, and investigated different types of anomalies and also benchmarked several anomaly detection options and reported their findings.
Critical strengths
The main point of strength in this paper for me is the structure of the pipeline. The bottom-up format that enables the tasks to be generated out of atomic steps with almost free step-level labeling costs and clear and deterministic labels for each step, which also can be regenerated easily even if the backbone model changes or evolves, is an interesting setting and nice idea. In addition, I think the evaluation provided by the authors is broad in different aspects, like number of traces, backbones, benchmarked anomaly detection methods, etc.
Critical weaknesses
My main critic about this paper is that although it is claiming that it is targeting the anomaly detection in multi-agent systems, the structure and design of the proposed pipeline is not really covering some important aspects of failure in multi-agent systems. For example, in multi-agent systems some of the main reasons of failure could be issues in joint reasoning and negotiations among agents, or for example agents duplicate other agents' tasks, etc. In the proposed framework in this paper, these important multi-agent aspects of failure haven't been covered properly, because the settings puts the agents in a setting which each agent does its own task and hands over the result to the next one, which doesn't provide a free-form of communication which could potentially result in those aforementioned types of miscommunication failures that exists in multi-agent systems. The only way of interaction that I could find in the pipeline was handing over the outcome to next agent in a fixed format along a chain that has been predetermined, which could potentially lead to passing the wrong value to the next agent and this wrong value then be propagated in the chain, but I couldn't find other forms of interaction failures like misunderstanding, negotiation and joint reasoning issues, etc. So it is hard to accept the claim that the provided pipeline really covers anomalies of multi-agent systems in general. A more accurate explanation is that it mainly generates errors coming out of single-agent failures which are then propagated in the pipeline, instead of multi-agent failure aspects that come from interactive part of multi-agent systems. I still think the proposed pipeline is interesting and valuable, but the scope and framing of the contribution needs to be adjusted and explained in more accurate way.
Questions for authors
-
As I mentioned in the weakness section, in my opinion, the main issue is not properly addressing the sort of failures in multi-agent systems which come from a free-form of interaction and communication. Could authors elaborate on this point and clarify whether in their set up any labeled failure comes out of these free format communications, rather than passing a wrong value from a single agent failure to the other agents down the chain? I see that in figure 9 authors have reported MAST’s inter-agent categories like ignored input, but my understanding is that these labels mainly correspond to the cases that coming from a single agent's error, not from an interaction error. If authors also agree that there is no such a failure mode in the current set up, I would suggest that authors either clarify and clearly explain this point in the paper or add some sort of real interaction failures.
-
My understanding is that authors have used the true number of anomalies in their test set to set the threshold for their accuracy and F1 calculations. If this understanding is correct, I think since this number is not known in real settings, I think further explanations on how the F1 score and accuracy changes with the threshold value in validation set is needed.
-
I think there are some inconsistencies in the evaluation that need to be addressed. For example Claude Opus 4.8 contributes 400 traces at one temperature, but the other backbones contribute 400 traces at each of 3 temperatures. However, comparisons (for example in Table 2) seem to be conducted side by side without analyzing the effect of this difference. Could authors elaborate more on the impact of this inconsistency on the conclusions related to cross-backbone aspects?
6 comments
W2. How does threshold selection affect F1 and other threshold-dependent metrics?
Response: Consistent with prior benchmarks [1], threshold-independent AUROC is our primary metric in the main text. Therefore, threshold selection does not affect our main conclusions.
For completeness, we also report F1, accuracy, and balanced accuracy in the Appendix. Since these metrics require a decision threshold, we followed standard practice [2–5] and set it using the anomaly ratio. However, we agree that reporting them under validation-selected thresholds further strengthens the analysis.
To address the reviewer's suggestion, we now extended our evaluation to 18 detectors with native thresholding capabilities and ran each over five seeds. This analysis focuses on non-LLM detectors, as including LLM-based detectors would incur substantially greater computational costs. We adopt the following evaluation methodology. Following the room-disjoint split protocol, we further separate 20% of the training set for validation while keeping the same prior held-out test set. This yielded a non-overlapping 64/16/20 train/validation/test split. We selected thresholds maximizing validation F1 or accuracy respectively and applied each to the test set (instead of the a priori ratio-based threshold). Table A below reports the numbers. Due to space constraints, we show only the best-performing methods (by AUROC) from each supervision/family group here. However, full results of all methods are available in Table 12 (Appendix H) of the revised manuscript.
Table A. F-1 and Accuracy with threshold selected by anomaly ratio (Ratio-F-1/Ratio-Acc ) versus threshold selected by Validation set (Val-F-1/Val-Acc ) . Results are means across seeds;
| Supervision | Method | Family | F1 (Ratio-F-1 → Val-F1) | Acc (Ratio-Acc → Val-Acc) | AUROC |
|---|---|---|---|---|---|
| — | Random | Random | .184→.287 | .717→.823 | .502 |
| Unsupervised | Isolation Forest | Tabular | .243→.306 | .739→.823 | .550 |
| One-Class | OC-SVM | Tabular | .255→.314 | .745→.823 | .592 |
| DOMINANT | Graph | .225→.456 | .734→.812 | .713 | |
| BlindGuard | MAS-specific | .134→.300 | .696→.819 | .440 | |
| Semi-Supervised | DeepSAD | Tabular | .462→.471 | .813→.841 | .773 |
| GGAD | Graph | .175→.302 | .715→.822 | .522 | |
| G-Safe (semi-sup) | MAS-specific | .223→.417 | .841→.844 | .735 | |
| Supervised | XGBoost | Tabular | .551→.560 | .845→.859 | .855 |
| G-Safeguard | MAS-specific | .476→.511 | .846→.849 | .804 |
Analysis of new results in Table A. The above comparison shows that thresholds selected on the validation set yield higher F1 and accuracy than thresholds based on the true anomaly ratio in the test set (the protocol used in our original evaluation). This finding is consistent with prior work showing that the F1-optimal threshold depends on a detector's attainable F1 score, rather than on the prevalence of the positive class [6]. Intuitively, setting the threshold so that the number of predicted anomalies matches the expected number of true anomalies constrains precision and recall to be equal, which need not coincide with the threshold that maximizes F1. In short, using the test anomaly ratio did not inflate our previously reported results; if anything, the original evaluation protocol was conservative. As AUROC is a threshold-independent metric, it is unaffected by the proposed threshold-selection methodology. Therefore, the AUROC results reported in the main paper remain unchanged.
Revision: We have added the full results and analysis into Table 12 in Appendix H.
[1] Han et al. ADBench. 2022.
[2] Zong et al. Deep Autoencoding Gaussian Mixture Model. ICLR 2018.
[3] Bergman & Hoshen. Classification-Based Anomaly Detection. ICLR 2020.
[4] Shenkar & Wolf. Internal Contrastive Learning for Tabular Anomaly Detection. ICLR 2022.
[5] Dai et al. Deep Noise Evaluation for Tabular Anomaly Detection. AAAI 2025.
[6] Lipton et al. Optimal Thresholding of Classifiers to Maximize F1 Measure. ECML-PKDD 2014.
W3. How does the unequal number of temperature settings across backbones affect comparisons?
Response: We thank the reviewer for raising this question. The difference arises from an API constraint, namely, Claude Opus 4.8 supports only one single temperature. In short, the unequal temperature coverage does not change our cross-backbone conclusions: 1) The success-rate trends in Table 2 hold at every tested temperature, and 2) As noted in Section 4, all detection results use MAADBench-Eval, which contains 400 (equal number of) traces per backbone at a single temperature. We elaborate on the two points below.
Task success rates (Extending Table 2). The original Table 2 averages each non-Opus backbone over its three temperatures. To allow direct comparison, we now also report results for each temperature separately (Table 8 in the Appendix):
| LLM | Temp | # Room | Easy | Medium | Hard | Extreme | Overall | Puzzle Attempted / Total | Puzzle SR |
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.8 | - | 400 | 70.0% | 63.0% | 29.0% | 30.0% | 48.0% | 805 / 1000 | 61.0% |
| Claude Sonnet 4.6 | 0.0 | 400 | 53.0% | 43.0% | 9.0% | 23.0% | 32.0% | 692 / 1000 | 42.6% |
| 0.3 | 400 | 52.0% | 31.0% | 14.0% | 12.0% | 27.3% | 679 / 1000 | 39.5% | |
| 0.6 | 400 | 52.0% | 32.0% | 10.0% | 23.0% | 29.2% | 694 / 1000 | 42.1% | |
| DeepSeek-R1 | 0.0 | 400 | 60.0% | 39.0% | 12.0% | 18.0% | 32.2% | 643 / 1000 | 38.6% |
| 0.3 | 400 | 69.0% | 38.0% | 12.0% | 22.0% | 35.2% | 692 / 1000 | 44.3% | |
| 0.6 | 400 | 71.0% | 49.0% | 13.0% | 26.0% | 39.8% | 719 / 1000 | 48.8% | |
| GPT-4.1 | 0.0 | 400 | 47.0% | 21.0% | 4.0% | 0.0% | 18.0% | 509 / 1000 | 21.4% |
| 0.3 | 400 | 46.0% | 20.0% | 2.0% | 0.0% | 17.0% | 507 / 1000 | 21.2% | |
| 0.6 | 400 | 46.0% | 16.0% | 3.0% | 0.0% | 16.2% | 505 / 1000 | 20.5% | |
| GPT-5.4 | 0.0 | 400 | 61.0% | 27.0% | 11.0% | 1.0% | 25.0% | 563 / 1000 | 30.2% |
| 0.3 | 400 | 62.0% | 29.0% | 11.0% | 0.0% | 25.5% | 566 / 1000 | 30.7% | |
| 0.6 | 400 | 57.0% | 25.0% | 9.0% | 0.0% | 22.8% | 560 / 1000 | 29.2% |
The broad cross-backbone trends remain consistent. Namely, Opus outperforms every evaluated configuration, while DeepSeek-R1 and Sonnet outperform GPT-5.4 and GPT-4.1 at all tested temperatures. While temperature setting has a higher impact for certain models (DeepSeek), it does not overturn the conclusions.
Detector training. Table 2 reports task success rates on MAADBench-Full, which covers all temperature settings. All detection results, in contrast, are computed on MAADBench-Eval, which selects a single temperature per backbone (temperature 0, or the default for Opus 4.8). Each backbone therefore contributes exactly 400 traces to MAADBench-Eval, so detector training and evaluation are balanced across backbones.
Revision: We have added the extended Table 2 (Table 8, Appendix F) into the manuscript. At the beginning of Section 4, we further stressed that the data (MAADBench-Eval) used in the actual evaluation has a balanced portion (400 traces each) for each LLM backbone.
We thank the reviewer for the positive assessment of MAADBench's bottom-up construction, deterministic labeling, and broad experimental evaluation. We address the comments as below.
W1. Failures arising from free-form inter-agent communication and coordination
Response: In our environment, agents interact through structured handoffs. Thus, failures raised from free-form negotiation and joint deliberation are not covered. We now state this explicitly in our revised paper in the Discussion section (Appendix B).
We would like to clarify, however, that the inter-agent labels do not describe errors made by a single agent in isolation. Our ignored-input cases (FM-2.5) are handoff failures. More precisely, an upstream agent produces a correct output and passes it on, but the downstream agent ignores or replaces it. We verify upstream correctness before assigning this label; if the upstream message itself is wrong, the case is labeled as FM-5 (now called Downstream Failures in the revision) instead. The failure therefore arises between agents: correct information is sent but not used by the receiver, which matches MAST’s definition of inter-agent misalignment. They are genuine communication failures that cannot arise in a single-agent system – albeit narrower than negotiation or joint-reasoning failures. We have added this clarification into Appendix F for Figure 9.
We also want to explain why we chose this scope. Structured MAS is a significant paradigm in its own right. Well-known systems and frameworks such as MetaGPT, ChatDev, CrewAI, and LangGraph illustrate how predefined workflows can coordinate multiple agents with controllable, auditable workflows.
We chose structured interaction to make task requirements and action-level checks explicit and reproducible. Extending the benchmark to free-form communication would require additional interaction specifications and validated labeling procedures, which remains an important and challenging direction for future work.
Revision: We have added a short discussion about future work to extend MAADBench to free-form communication in the Discussion section (Appendix B), and a further explanation of the ignored-input cases (FM-2.5) with other failure modes in Appendix F.