acceptodds
Under review as a conference paper at ICLR 2027

TRACE-MAS: Diagnosing and Repairing Collaboration Failures in Multi-Agent Systems

Abstract

LLM-based multi-agent systems (MAS) rely on not only individual agent behaviors but also the interactions that connect agents, including message exchanges and task dependencies. Existing MAS safeguards mainly perform agent-level anomaly detection, which becomes insufficient when failures originate from agent interactions and detected anomalies do not necessarily reveal their true sources. To address this limitation, we propose TRACE-MAS, a finer-grained framework for diagnosing and repairing MAS failures. TRACE-MAS learns normal collaborative execution patterns from benign trajectories and detects deviations in agent state changes, message transitions, and task dependency evolutions. It further identifies responsible execution elements through source-aware attribution and selects minimal interventions to restore collaboration while preserving task-critical capabilities. Experiments on five MAS benchmarks demonstrate that TRACE-MAS significantly improves root-cause localization accuracy by 60.2% and achieves consistent gains in task recovery over strong MAS safeguards. Our code is available on https://anonymous.4open.science/r/x7k2m9f3.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Multi-Agent Systems, Multi-Agent Security, Event Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_f6yWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TRACE-MAS, a monitoring framework for detecting, diagnosing, and repairing failures in multi-agent systems. It models agent states, message exchanges, and task dependencies through a structured execution graph, learns normal collaboration patterns from benign trajectories, and uses counterfactual restoration to identify failure sources and select corrective interventions. Evaluations show promising improvements in failure detection, root-cause localization, and task recovery.

Critical strengths
  • Detecting and correcting failures in multi-agent systems is an important problem for practical deployments.
  • The paper goes beyond agent-level failure detection by considering message exchanges and task dependencies and providing corrective interventions.
  • The evaluation demonstrates encouraging improvements over existing approaches.
Critical weaknesses
  • Incremental novelty. The proposed solution largely combines existing techniques, such as graph-based representation learning, anomaly detection, counterfactual attribution, and training.
  • The prevalence and practical significance of the targeted failure types are unclear. Existing work, such as Who&When (ICML 2025) and CORRECT (ICML 2026), has studied agent-level failures, but it is unclear how frequently message-level and dependency-level corruptions occur in realistic multi-agent workloads.
  • The evaluation relies on a single backbone model, limiting the generalizability of the results. Additional experiments and a more comprehensive cost analysis would strengthen the paper.
Questions for authors

Thank you for submitting your work. I enjoyed reading the paper and agree that diagnosing and repairing collaboration failures is an important research problem. The event-level perspective is interesting, and the empirical improvements are encouraging. However, I have concerns about the methodological novelty, workload representativeness, and practical applicability of the proposed framework.

  • The individual designs largely build on existing techniques. What are the key technical innovations beyond combining these methods?

  • The paper motivates its approach through message-level and dependency-level attacks. However, how frequently do these failures occur in real-world multi-agent systems compared with agent-level failures?

  • How would TRACE-MAS perform with other backbone models, especially more recent and capable LLMs?

  • What is the end-to-end overhead, including training costs, generation latency, token consumption, and repair execution?

Reviewer_szg9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TRACE-MAS, a framework for detecting, attributing, and repairing failures in MAS on the level of agents, messages, and task dependency. TRACE-MAS learns execution contracts from benign trajectories to do anomaly detection and uses counterfactual restoration to trace propagated failures back to their originating sources. It further selects minimal interventions through predictive rollouts to restore reliable collaboration while preserving task performance.

Critical strengths
  • Clear and sound problem framing. Three observations are valid and persuasive.
  • Analyses on Section 4 are well designed.
  • No LLM call required for detection and diagnosing unlike other failure attribution methods.
  • Performance gain is significant.
Critical weaknesses

The most critical issue of this paper is its writing quality. The main body of the paper only contains high-level ideas and concepts, while important details are provided in the Appendix. Reviewers are not required to read the Appendix. The main body is more like a brief table of contents for the Appendix rather than a complete research paper.

Specifically, in Section 4 of the main body, the authors just announce their results without any experimental details, such as what each baseline does, what each attack method does, and so on. Furthermore, I could not interpret the experimental results in the figures. For example, in Figure 1, what does the dashed line mean? What does the red solid line mean? Even the minimal level of readability is insufficient.

Similarly, in Section 5, how does ExecFormer work? How is Δet\Delta e_t formed? How is heterogeneous information such as agent states and messages represented as embedding vectors? And so on.

The Related Work section lacks detailed explanations of what problems existing works attempted to solve, what solutions they proposed, what their limitations are, and what the critical research gap between TRACE-MAS and them is.

Furthermore, Section 3.1 lacks discussion of existing works on agent-level, message-level, and task dependency-level attack methods. They should be addressed separately. While TRACE-MAS sounds quite relevant to works leveraging task dependencies for failure attribution [See the reference], the authors omitted discussion of them.

Section 3.2 lacks detailed analysis of existing detection and mitigation works. For now, it lacks discussion of why they have critical limitations in addressing message-level and dependency-level corruption.

The paper lacks case studies or real examples, which could improve readability and help readers understand the target problem.

Reference

  1. https://arxiv.org/abs/2609.01360
  2. https://arxiv.org/abs/2608.24361
  3. https://arxiv.org/abs/2610.01256
  4. https://arxiv.org/abs/2608.06346
  5. https://arxiv.org/abs/2607.06273
  6. https://arxiv.org/abs/2604.23581
  7. https://arxiv.org/abs/2602.11964
  8. https://arxiv.org/abs/2408.04682
Questions for authors

See Weaknesses.

Reviewer_bkvSOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LLM multi agent systems (MAS) fail not only inside agents but between them: a message can be altered, or a task dependency can be broken, while every agent looks normal. Existing MAS safeguards detect anomalous agents. This paper argues for execution element granularity, that means agent states, messages and task dependencies, and supports the argument with a controlled study: agent centric monitors drop about 22.7 points when attacks move from agents to interactions, the maximally anomalous event is the true source only 32.1% of the time, and repairing the most abnormal point loses about 30 recovery points.

TRACE MAS converts each trajectory into a typed event graph, where nodes are execution events and edges are message, dependency, and temporal links. It trains a conditional Gaussian "contract" on benign trajectories only: a relation aware context encoder summarizes what happened before an event, and the contract predicts the expected change (transition residual, built on frozen MiniLM text features with a per channel diagonal covariance). An event is flagged when its Mahalanobis distance from the prediction exceeds a quantile threshold calibrated on held out benign data, with no attack labels needed and an exchangeability based false alarm bound. Attribution then tests each candidate source by contract restoring it and measuring the hop discounted drop in downstream violations. Repair ranks minimal interventions by predicted residual violation plus task degradation plus operator cost.

Results on five benchmarks under three corruption levels: best detection F1 in every cell, averaging +15.5 points over the strongest competitor (for example level 3 PoisonRAG 98.0 vs 35.9), best root cause localization (HotPotQA Hit@1 71.1 vs 23.3, PoisonRAG 86.7 at 0.18 hops), and best recovery (PoisonRAG attack success rate 3.0%, task accuracy 93.8% at 0.54 operators). Ablations, calibration curves, sensitivity sweeps, stress tests and runtime profiling are all included.

Critical strengths
  1. The protocol against data leakage is careful. Patients, not slides, are bootstrapped 40,000 times. The bootstrap draw used during the search is separate from the one used for final verdict. The baseline is the strongest panel method on each split, chosen after the fact, which makes the comparison conservative, and the evaluation data is scored only once, by the frozen program.
  2. There is external validation. The margin holds at CMUH, a second hospital with a different scanner, and is even larger there (+0.037).
  3. The ablations are honest. Section 5.1 dismantles the paper's own headline: it shows that most of the margin comes from the enriched rule pool (learned combiners and cross lesion borrowing), and that the agent designed search adds at most +0.017 AUC over ordinary bagged greedy selection on same pool. The authors report this plainly.
  4. Missing stain handling is sensible. Rules only use stains that a patient actually has, nothing at patch level is imputed, and Section 5.2 separately measures performance on patients with complete and incomplete stain panels.
  5. Presentation is good. Writing is clear, ethics and reproducibility statements are complete, and a code release is promised.
Critical weaknesses
  1. The effect sizes are small, and the lymphoma result is not significant. The AUC gains are only +0.009 to +0.037. On DLBCL Morph the reported margin is +0.035 Cindex, and it has p = 0.088 even with a one sided test, yet the abstract claims ARISE "outperforms the strongest conventional method on every task in every cohort". Kidney lesion T is significant on neither cohort, and lesion C is significant only externally. Whether a +0.014 AUC margin is clinically meaningful is never discussed.
  2. The paper's own data shrink the "agent" contribution. Ordinary greedy selection on the agent's pool already beats the panel in almost every cell. What is new is a search procedure (an agent that edits a pool specification), not a demonstrated capability advantage of the LLM agent. As written, this is mainly a careful comparative study of prediction level fusion, and the title oversells the mechanism.
  3. The 13 cells are not 13 independent tests. Ten cells are five correlated kidney lesion scores measured twice (two cohorts, same models), and the five lesions are scored on the same biopsies. Real diversity is three organs, and organ level diversity is thin.
  4. The strongest new pool families work only on kidney. Learned combiners need multiple validation blocks, which only the kidney cohorts have, and cross task members also exist only on kidney. So the key extension is untested on two of the three organs.
  5. There is a small self contradiction on missing stains. Section 3.1 says "No prediction is imputed for the missing stain itself", but the anchored order statistic rules "fill an absent stain at the neutral rank", which is rank level imputation.
Questions for authors
  1. Do you withdraw or qualify the abstract's "every task in every cohort" claim, given DLBCL p = 0.088 and lesion T?
  2. What AUC or Cindex difference is clinically meaningful in these tasks, and do your margins cross it?
  3. You suggest a scripted proposer as future work. If such a proposer, with the same revision budget, reaches similar programs, what is left of the contribution, and can you run that comparison now?
  4. For HANCOCK, single stain experts are "scored on the patients the stain covers". Are the bootstrap comparisons against ARISE paired on identical patients for every method?
  5. Table 3 shows cross lesion members help CMUH (external) but barely help CureGN (internal). Why would borrowing predictions across different lesions be more useful externally?
Reviewer_2QW8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TRACE-MAS, a framework for detecting, attributing, and repairing failures in LLM-based multi-agent systems. It models execution as a partially ordered event graph and fits a context-conditioned Gaussian "execution contract" over embedding-space transition residuals, using benign trajectories only. Events whose Mahalanobis score exceeds a benign quantile threshold are flagged. Each candidate source is restored to its contract mean, and the one with the largest hop-discounted violation drop is selected. A contract-space rollout then picks one of three repair operators (state, message, dependency). On five benchmarks with injected L1-L3 corruptions and a Qwen2.5-7B backbone, the paper reports higher detection F1, Hit@1, and post-repair accuracy than agent-level safeguards.

Critical strengths

The paper separates where an anomaly becomes visible, where it originates, and where repair should be applied. Counterfactual restoration is a reasonable way to ask which event best explains the downstream violations, and the locality of Restore (Property 4) keeps the candidate set manageable. Labeling one initiating event per trajectory and matching ASR across L1-L3 make Hit@1 well defined and keep the cross-level comparison free of attack-strength confounds. The method is lightweight and benign-only. Attribution and rollout run in contract space without LLM calls, and the benign-quantile threshold comes with a clear false-alarm argument (Property 2).

Critical weaknesses
  1. The contract measures deviation from typical behavior, not failure, so the claims about "collaboration failures" overreach. The residual is the change in a frozen MiniLM embedding of concatenated agent text, modeled by a unimodal diagonal Gaussian given role and context. Such a score reacts to shifts in style, topic, or length. It is insensitive to wrong-but-fluent content (a wrong intermediate number in a GSM8K solver step barely moves a sentence embedding), and it penalizes legitimate diversity among valid reasoning paths. The paper never describes the content of the L1/L2 injections, so it is unclear which of these the detector is picking up. The framing also covers failures that come from ordinary capability errors (Sec. 3.2 positions the work against MAST/Who&When), but in those cases no event is anomalous with respect to the benign distribution. B.1 confirms nothing is tested outside injected corruptions. The framework is better described as an anomaly detector for injected perturbations. In addition, the three "channels" (state, message, progress) are blocks of the same text-embedding residual, so it is unclear why the dependency channel should see coordination changes. For L3, where a role is skipped or an agent goes offline, events are removed and a per-event score has nothing to score.

  2. The attribution premise is questionable. The contract conditions on the input an event actually received, so a downstream agent that faithfully responds to a corrupted message is "normal" under its own corrupted context. A symptom can have a high VV only when the corrupted context is out of distribution for the predictor, so what the score reflects is extrapolation error of the encoder rather than abnormal agent behavior. Restore then replaces cc by μc\mu_c and refreshes descendant contexts in embedding space, so the "counterfactual" is a property of the learned encoder, not of a real counterfactual run of the MAS. Also, R(c)R(c) always credits the candidate's own V(c)V(c) at full weight (since Vcres(c)=0V^{res}_c(c)=0) plus a discounted sum over descendants. This favors upstream, high-fan-out events regardless of cause, so Hit@1 may partly reflect graph position and attack placement. Finally, tampering with a message in transit has no initiating event, because the producer is benign, yet the paper claims to identify responsible "execution elements" (messages, dependencies) while candidates are only events.

Questions for authors
  1. How does Restore treat the observed transitions of descendants? If they stay at their logged values while their contexts are refreshed, restoring the true source moves μj\mu_j toward what a benign response would be and away from the corrupted response actually logged, which could raise rather than lower the symptom's score. If they are also set to their contract means, then Vres=0V^{res}=0 on the whole branch and R(c)R(c) reduces to a discounted sum of observed violations. Which is implemented? Could the authors also report a topology-only baseline (e.g., earliest violated ancestor, or highest fan-out ancestor) to show that attribution is not driven by graph position?

  2. How is repair realized without access to the pre-attack record? "Contract-consistent" state or dependency values are embedding-space means, not text or structure. If message correction regenerates the payload with the LLM from the same corrupted upstream context, it may reproduce the corruption. Does the implementation use any information from the injection (e.g., the benign snapshot)? Also, the three operators map one-to-one to L1/L2/L3, and rollout sets regenerated events to their contract means (V=0V=0 by construction), so I suspect the rollout mostly selects the channel with the largest deviation at c⋆c^\star. What does it add beyond that, and why is the result of Obs. 3 (source repair beats symptom repair) not simply implied by a single-injection testbed, where repairing a downstream event with the corrupted input still in place reproduces the corruption?

  3. What does the contract do on natural failures and on benign variability? Does the benign training set include trajectories where the MAS fails by itself? If so, the contract absorbs those errors. If not, what is the alert rate on naturally failed trajectories and on benign perturbations (paraphrases, alternative valid reasoning paths)? Results on style-matched, semantically harmful injections would also help show that the detector responds to corruption rather than embedding-level shift.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.