acceptodds
Under review as a conference paper at ICLR 2027

MeshHeal: Two-Timescale Self-Healing for Gray Failures in Decentralized LLM Agent Networks

Abstract

Decentralized LLM-based multi-agent systems coordinate through local interactions, but an agent can remain responsive while its task-solving quality persistently degrades. Such gray failures require protecting current tasks before sufficient evidence exists to alter future routing, while still allowing recovered agents to rejoin. We introduce MeshHeal, a fully decentralized self-healing framework that couples ability-matched peer review across two timescales. At the fast timescale, an adaptive hierarchy escalates uncertain or low-scoring outputs from repeated single-reviewer evaluation to committee deliberation and, when needed, correction before use. At the slow timescale, a task- and ability-conditioned peer-relative detector aggregates scores to distinguish persistent degradation from ordinary output variation, trigger mandatory committee review, and eventually exclude degraded agents from ordinary routing; recovery probes provide fresh evidence for reintegration. To faithfully evaluate routing, we introduce Model-Backed MAS Evaluation, which ties ability assignments to execution models, since prompt-based ability assignments alone can leave routing errors hidden. Across BBH, MATH, and MMLU-Pro, MeshHeal achieves 0.839 degraded-phase accuracy using 51k total model tokens per task, versus the strongest baseline Symphony's 0.807 accuracy using 115k per task. Under staggered degradation and recovery, MeshHeal isolates degraded agents, keeps them excluded from ordinary task execution until recovery, and returns them to normal routing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent system, decentralized multi-agent system, LLM-based multi-agent system, self healing, gray failure

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EWZxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents MeshHeal, a decentralized framework for handling agents that remain responsive but produce persistently lower-quality outputs. It combines immediate peer review and correction with longer-term monitoring: ability-matched reviewers check outputs before they are used, while acumulated scores relative to neighboring agents determine whether an agent receives additional scrutiny or is excluded from ordinary routing. Recovery probes allow excluded agents to rejoin after their performance improves. The paper also introduces an evaluation protocol that switches between strong and weak model backends to control agent capabilities and simulate degradation. Experiments on BBH, MATH, and MMLU-Pro compare MeshHeal with several routing and coordination baselines, supplemented by component ablations, different model pairs, staggered degradation and recovery, and networks of up to 576 agents. The reported results show competitive accuracy at lower token cost than the strongest baseline, alongside successful isolation and reintegration under the tested conditions.

Critical strengths

There are several critical strengths in this paper:

  1. The framework connects immediate error correction with longer-term agent management. MeshHeal checks and, when necessary, replaces an output before it affects the current task, while retaining evidence about the original producer for subsequent routing decisions. Recovery probes also give excluded agents a way to demonstrate improved performance and rejoin. This is a coherent system design that addresses both error propagation and the risk of permanently excluding an agent after a temporary failure.

  2. The reported accuracy–cost trade-off is promising under the tested conditions. Across the three benchmarks, MeshHeal achieves a mean degraded-phase accuracy of 0.839 with approximately 51k tokens per task, compared with Symphony’s 0.807 and 115k tokens. The experiments also cover different model pairs and staggered degradation and recovery. These results provide useful evidence that selective review and correction can maintain task performance without the cost of repeatedly generating and refining parallel solutions.

Critical weaknesses

There are some critical weaknesses in the paper that I would like to mention:

  1. The motivation and contribution are not sufficiently distinguished from existing work. The paper overstates the limitations of approaches based on historical reputation or routing feedback. RAPS already includes an on-the-fly watchdog that evaluates incoming outputs, and its case study demonstrates preventing an erroneous output from reaching the current task’s final answer. The ICML 2025 work on faulty-agent resilience also introduces an Inspector that reviews and corrects messages. Protecting the current task therefore needs a more careful comparison with existing mechanisms. MeshHeal’s potential contribution lies in combining immediate correction, persistent degradation detection, isolation, and reintegration. However, the paper does not establish when this combination is necessary or why simpler extensions of existing methods would be insufficient. Listing differences in the related work does not resolve this question, which directly affects my assessment of novelty and significance.

  2. Several important methodological choices remain insufficiently justified or specified. The main parameter values are provided, but there is little evidence supporting the particular normalization and transition rules. For example, the detector uses a scale floor of 0.10, watch and isolation thresholds of 1.2 and 2.0, and an eight-observation window without a sensitivity study or a clear calibration procedure. Comparing scores from different questions within the same task category also leaves uncertainty about whether the detector separates persistent degradation from ordinary difficulty and capability differences. The description is sufficient to implement an approximate system, but not to reproduce the reported evaluation reliably. Complete runtime prompts, main-experiment decoding settings, task sequences, and baseline adaptations are not fully specified. The absolute-threshold and CUSUM comparisons are particularly difficult to assess without their exact rules and parameter settings.

  3. The evaluation does not establish the capability assumptions on which the framework relies. Switching to a strong model when an agent is assigned a matching ability does not demonstrate that the model lacks the other ability required by the task. A strong model receiving the full question may solve more than its assigned subproblem. Consequently, the claim that each task requires at least two complementary agents is imposed by the protocol but not convincingly validated. A strong single-agent control would materially help establish whether the gains depend on capability-aware collaboration. The tested failures replace all calls from selected agents with a weak backend, while each evaluated output updates only one ability-specific detector. This does not test partial capability degradation, despite that possibility being included in the problem definition. The distinction matters because evidence is maintained per task and ability, but isolation and reintegration apply to the entire agent. It remains unclear whether successful probes on an unaffected ability could readmit an agent whose impaired ability has not recovered.

  4. The experiments do not isolate the claimed contribution, and the baseline comparisons need further clarification. Table C6 supports the fast review-and-correction components: degraded-phase accuracy falls from 0.900 to 0.865 with a single reviewer and to 0.844 without Takeover. However, both variants retain detection and rerouting. These results do not establish the additional value of the slow-timescale detector, explicit isolation, or recovery probes. Comparisons that remove those components, or add simple monitoring and exploration to an existing method, are needed to attribute the gains. The SAC baseline is described as three teams followed by filtering, refinement, and majority aggregation. The original algorithm instead relies on receiver-side scoring, self-anchored filtering, and a fault bound (F). The paper does not explain how these mechanisms are preserved in its adaptation. Similarly, it is unclear whether the RAPS implementation retains its immediate watchdog. These omissions make the reported advantages difficult to interpret. The advantage over Symphony is also setting-dependent. In the staggered recovery experiment, pooled accuracy is 0.812 versus 0.806, while fully recovered accuracy is 0.857 versus Symphony’s 0.873. Under model pair B, degraded-phase accuracy is 0.782 versus 0.786. The lower token cost is useful, but these results do not establish consistent accuracy superiority, and the small differences require paired uncertainty estimates.

Questions for authors

The following are my questions for the authors:

  1. How faithfully do the adapted baselines preserve their original mechanisms? For SAC, how are receiver-side scoring, self-anchored filtering, and the fault bound (F) implemented within the three-team setup? For RAPS, is the on-the-fly watchdog retained when processing incoming outputs? Please provide the relevant settings and adaptation details, since these mechanisms directly affect the comparison with MeshHeal’s current-task protection.

  2. What establishes the additional benefit of the slow-timescale mechanism? Table C6 retains detection and rerouting in both ablations, so it primarily tests the fast review-and-correction components. Can you provide a comparison against a version retaining peer review and Takeover but removing persistent detection and isolation, under comparable inference budgets? This would help determine how much of the reported improvement depends on the proposed two-timescale design.

  3. How are conflicting health signals across abilities handled? Evidence is maintained separately for each task type and ability, but routing status applies to the entire agent. If an agent is isolated because one ability has degraded, what prevents successful recovery probes on an unaffected ability from restoring its eligibility while the impaired ability remains unreliable? Please clarify how evidence is combined and whether reintegration requires evidence of recovery for the ability that triggered isolation.

6 comments
AuthorsOpenReview ↗

Response to Reviewer EWZx (part 1)

We thank the reviewer for the detailed review. We have uploaded a revised PDF in which all changes are marked in blue. Direct answers to Q1–Q3 are given at the end.

W1: contribution relative to RAPS and the Inspector

We agree that checking outputs within a task is not new, and the revised related work says so. Our contribution is the coupling of the two timescales: replacing an output must be decided from one or two reviews, stopping work to an agent must not, and each decision depends on the other. Approaches that make only one of the two decisions, such as RAPS or an Inspector-style variant, lose accuracy in our results.

We agree that checking outputs within a task is not new. The Inspector [1] reviews and corrects messages, and RAPS [2] runs a watchdog on every incoming output; our related-work section overstated the gap, and the revised Sec. 1 and Sec. 2 correct the description of RAPS and add the Inspector. The contribution we claim is the coupling of the two timescales, and the reviewer’s question, when the combination is necessary, has a direct answer.

The fast decision (keep or replace this output) must act on one or two reviews. The slow decision (stop routing work to this agent) must not: healthy agents receive a low score on 4.4% of their outputs (Tab. C8(b)), and a rule on an agent’s own scores isolates all four healthy agents under pair A (Tab. C7). The two decisions also depend on each other. Review produces the evidence for the slow decision, and the slow state decides who may review, whose outputs must go to the committee, and who infers the required abilities as an entry agent. A method that makes only one of the two decisions loses accuracy in a predictable way, and each existing approach corresponds to one such case in our results:

  • Within-task correction only, the Inspector’s mechanism (our Fast-only variant, Table R4): degraded agents keep receiving work and keep reviewing others. Degraded-phase accuracy on BBH drops by 0.062 while cost rises by 33%.

  • A watchdog with cross-task avoidance (RAPS): the current task is unprotected until reputation crosses its threshold, and an avoided peer is never re-evaluated. RAPS reaches 0.740 on BBH and 0.653 over the three datasets in the degraded phase (Tab. C1, Tab. 2(a)).

  • Cross-task isolation without replacement (No Takeover, Tab. C6): degraded-phase accuracy drops by 0.056.

Compared with RAPS, three design choices differ. (1) RAPS’s watchdog returns a binary judgment that updates a Beta reputation, and the broker routes around peers whose reputation crosses τrep\tau_{\mathrm{rep}} ([2], Sec. 4.4 and Alg. 1). In its case study, the flagged output stays out of the answer because a parallel peer’s correct output is routed downstream instead. On the sequential Handoff–Contribute–Finalize path shared by all methods here, there is no parallel output to fall back on, and Takeover supplies a corrected one. (2) RAPS compares a peer’s reputation with a fixed threshold; MeshHeal compares an agent with peers on the same task type and ability and scales by their normal variation, which lets one set of thresholds hold across datasets, model pairs and network sizes (Tab. C7). (3) A peer that RAPS routes around publishes nothing further, so its reputation stops changing and it cannot return; recovery probes give an isolated agent fresh, committee-reviewed evidence.

AuthorsOpenReview ↗

Response to Reviewer EWZx (part 2)

W2: thresholds, difficulty within a task type, and reproducibility

τto\tau_{\mathrm{to}} and σcom\sigma_{\mathrm{com}} follow from the rubric, and the tuned parameters are insensitive over the tested range. Comparing an agent with peers on the same task type and ability cancels out question difficulty, and at the default thresholds no healthy agent was isolated in any reported run. The revision gives the decoding settings and both comparison rules with their parameters, and code and logs will be released.

Thresholds. τto\tau_{\mathrm{to}} and σcom\sigma_{\mathrm{com}} follow from the score levels of the review rubric rather than from tuning: τto=0.40\tau_{\mathrm{to}} = 0.40 separates “concrete error” (0.25) from “cannot verify” (0.50), and σcom=0.55\sigma_{\mathrm{com}} = 0.55 sends an output to the committee exactly when one of three reviews calls it unusable and the other two call it correct (derivations in Sec. 4.2.1 and App. C.6 of the revision). α\alpha, β\beta, the window and the probe interval were tuned on MATH streams with seeds disjoint from all reported runs, then used unchanged on BBH, MMLU-Pro, both model pairs and all network sizes. Varying α∈1.0,1.5\alpha \in \\{1.0, 1.5\\} and β∈1.5,2.5\beta \in \\{1.5, 2.5\\} on the staggered stream (Table R1 in our response to Reviewer 7GRd; Tab. C9) keeps accuracy within 0.01 and detects every degraded agent in every run; the only notable change is that β=1.5\beta = 1.5 isolates a healthy agent in 3 of 12 agent-runs, each of which returns through probes. The scale floor of 0.10 keeps zz bounded when healthy agents score so consistently that their lower spread is near zero.

Difficulty and capability differences. The reference for agent ii comes from normal neighbors answering other questions of the same task type with the same ability. These questions come from the same distribution and the window averages eight of them, so question difficulty enters the reference and the agent’s mean in the same way, and the lower spread of the peers’ mean shortfalls measures how much healthy agents differ, ordinary capability differences included. An agent is flagged only when its mean falls well outside that spread on consecutive updates. Across all runs, 0 of 6,544 healthy agent-runs were isolated and 3,272 of 3,272 degraded agent-runs were detected (Tab. C11). An agent that stays far below its peers on the same work is, for routing, the agent to avoid, whatever the reason.

Reproducibility. All agents and methods use temperature 0.7, top_p 1.0, at most 2,048 output tokens and the provider’s default reasoning effort. The absolute rule isolates an agent when the mean of its own last eight scores stays below 0.75 on two consecutive updates; CUSUM sums how far each new score falls below the healthy agents’ mean, less a slack of 0.25 and floored at zero, and isolates when the sum exceeds 2.0. As for our detector, both were set on the default model pair and used unchanged on pairs A and B. The revised App. B.5 states both rules with their parameter values and App. B.4 the decoding settings and baseline adaptations; the complete runtime prompts, the task sequence of every seed, the code and the run logs will be released with the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer EWZx (part 3)

W3: capability assumptions and partial degradation

The evaluation tests whether coordination and routing are correct, not whether an MAS beats a single agent, so tasks that need two agents are part of the construction rather than a claim about the benchmarks. The paper degrades all abilities of an agent together; MeshHeal extends directly to single-ability degradation by keying the routing state and the recovery probes on the ability. A new experiment (Table R3) shows that this extension detects the impaired ability, never readmits it before recovery, and keeps the agent's other abilities in service; detection is slower than under whole-backend degradation because the signal is sparser.

Testing whether coordination and routing are correct needs tasks on which routing matters. Standard benchmarks lack this property, since one capable model can attempt every task, and assigning abilities through system prompts does not create it (Tab. 1). Model-Backed MAS Evaluation builds it from existing benchmarks: each task type is split into two complementary abilities held by disjoint sets of agents, and the backend makes an agent strong only on the abilities it holds. A task is then answered correctly only when each part reaches a healthy agent that holds the matching ability, so routing errors and gray failures become visible in accuracy. In deployed heterogeneous systems, such complementarity comes from tools, data access or specialized models that one general model cannot replace; the construction reproduces this situation with known ground truth.

That every task needs two agents is therefore part of the construction: the execution prompts restrict a Contribute step to the agent's own part (App. D.2), and the rubric scores only that part (App. D.1). For the same reason, a strong single-agent control does not bear on our question: Eq. (1) is defined over local coordination policies for a given network and ability assignment, and a single agent holding every ability removes the routing problem under test. The revised Sec. 5 opens with this purpose.

Single-ability degradation (Q3). In the paper all abilities of an agent fail and recover together. Evidence is already kept per task type and ability, but the routing state is kept per agent and follows the most recently updated detector, so a probe on an unaffected ability can readmit an agent whose impaired ability has not recovered. Because routing is already computed per ability (Sec. 4.1, App. B.5), the extension is direct: the routing state, the isolation counter and the recovery probes are keyed on the agent and the ability, so an agent isolated on ability ss is removed only from work that needs ss and probed only with such work. Reintegration on ss then requires recovered evidence on ss itself, while the agent's other abilities stay in service.

A new experiment runs both versions under this failure (App. C.6 of the revision). On the 504-task BBH stream with seeds 1 to 3, A5 degrades only on spatial and A3 only on mathematical during tasks 253 to 420; both are restored from task 421, and every other ability keeps the strong backend.

Table R3 (Tab. C14 in the revision). Single-ability degradation on BBH (seeds 1 to 3).

MetricPer-agent state (paper)Per-ability state (extension)
Detection delay, A5 / A3 (tasks)31.0 / 22.724.3 / 17.0
Impaired abilities isolated6/66/6
Readmissions before recovery, per agent-run2.00
Unaffected abilities withheld after detectionallnone
Impaired outputs accepted without committee review8%4%
Reintegration delay after restoration, A5 / A3 (tasks)n/a (readmitted before recovery)52.7 / 44.3
False isolations (healthy agent-runs)0/120/12
Accuracy, healthy phase / after task 2530.899 / 0.8940.899 / 0.898
Tokens per task after task 25347k46k

Detection is slower than under whole-backend degradation (9.3 and 14.0 tasks on BBH, Tab. C10), because only outputs that use the impaired ability carry the signal and fewer of them arrive per task. With the per-agent state of the paper, outputs on unaffected abilities also reset the isolation counter, and probes on those abilities readmit each agent about 2.0 times per run before its impaired ability recovers. The fast timescale limits the cost: a readmitted agent is watched first, its impaired outputs are reviewed and replaced, and accuracy after task 253 is 0.005 below the healthy phase. The per-ability state removes the problem: it isolates the impaired ability sooner, because unaffected outputs no longer reset its counter, never readmits it before recovery, keeps the agent's other abilities in service, and reintegrates it 52.7 and 44.3 tasks after restoration, more slowly than whole-agent reintegration (Tab. C10) for the same reason that detection is slower. No healthy agent is isolated in either version.

Reviewer_4AUXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MeshHeal, a decentralized framework for handling persistent quality degradation in LLM agents. It combines immediate peer review and output correction with longer-term degradation detection, routing isolation, and recovery probes. The paper also introduces Model-Backed MAS Evaluation, using backend selection to implement ability assignments and inject failures. Experiments on BBH, MATH, and MMLU-Pro report degraded-phase accuracy of 0.839 at 51k tokens per task, compared with Symphony's 0.807 at 115k. Additional experiments examine recovery, component ablations, alternative model pairs, and sparse networks with up to 576 agents.

Critical strengths
  1. The two-timescale design is conceptually clear. Fast review and Takeover protect the current task, while slow detection and routing isolation address persistent degradation. Recovery probes provide an explicit mechanism for reintegration.
  2. The paper provides a detailed controlled evaluation. The authors report model-pair experiments, token costs, ablations, recovery trajectories, sparse-network results, and implementation details. This makes the proposed mechanism relatively easy to inspect and reproduce within the stated experimental setup.
Critical weaknesses
  1. Limited coverage of gray failures. Although several model pairs are tested, degradation is consistently induced through backend replacement. Ability inference is guided by examples covering all evaluated task types, and the sparse graphs preserve reviewer availability. These controlled conditions leave uncertainty about whether the gains extend to less structured failures, noisy ability assignments, or insufficient qualified reviewers.
  2. Limited validation of detection reliability. The recovery study covers only MATH, while the detector comparison in Table C7 involves two degraded and four healthy agents per configuration. The paper also lacks a threshold-sensitivity analysis, and its review-discrimination metrics use injected health states rather than output correctness. Consequently, the evidence for reliably low false-isolation rates and transferable detection thresholds remains limited.
Questions for authors
  1. How does MeshHeal perform if the task-to-ability examples are removed, or if the ability labels and task-type mappings contain controlled noise?
  2. How does the system behave when all direct neighbors with a required ability are watched or isolated, when multiple neighbors fail simultaneously, or when an output requires multiple abilities?
  3. How were α = 1.2, β = 2, the eight-observation history window, and the ten-task probe interval selected?
  4. Can the authors evaluate review scores against actual output correctness using an independent judge model or human annotations, especially for Contribute outputs?
  5. Please clarify whether the prompts, context lengths, parallelism, execution order, and token accounting are fully comparable across all baselines. Token count alone may not reflect practical latency or monetary cost.
9 comments
AuthorsOpenReview ↗

Response to Reviewer 4AUX (part 1)

We thank the reviewer for the careful review. We have uploaded a revised PDF in which all changes are marked in blue. The new results are Table R2 and the detection counts under W2, and the correctness audit under Q4 (Tab. C10, C11 and C13 in the revision). W1, Q1 and Q2 are answered together because they concern the same conditions.

AuthorsOpenReview ↗

Response to Reviewer 4AUX (part 2)

W1, Q1 and Q2: coverage of gray failures

MeshHeal uses only output quality, so any fault that lowers quality while the agent stays online looks the same to it, and failures too mild or sporadic to isolate are still caught output by output. A new experiment shows that degradation of a single ability is also detected (Table R3 in our response to Reviewer EWZx). Noisy abilities do not cause false isolation: a wrong declared ability looks like degradation and is excluded, and a one-off decomposition error affects only one output. Every peer-checking method needs available reviewers; MeshHeal needs only one qualified normal neighbor, which our sparse graphs provide.

Less structured failures. Backend replacement is how we inject degradation, not what MeshHeal assumes. In deployment it comes from a misconfigured or quantized backend, context corruption or overflow, stale memory, failing tools, or faulty and non-colluding Byzantine agents (W1 in our response to Reviewer 7GRd), all of which lower output quality while the agent stays responsive. The experiments vary its strength (pair A is mild, pair B has a weaker healthy model), timing (staggered degradation and recovery) and extent (one third of the agents at once). When only one ability of an agent degrades (Table R3), the impaired abilities are still detected, more slowly because fewer outputs carry the signal, and no healthy agent is isolated. A failure that persists relative to peers is isolated at the slow timescale; one too sporadic or mild to cross the isolation threshold is handled output by output at the fast timescale, where every output is reviewed and can be replaced. Takeover already corrects such errors from healthy agents: removing it lowers healthy-phase accuracy by 0.020 (Tab. C6).

Noisy ability assignments (W1, Q1). Two kinds of noise are possible. First, an agent whose declared ability does not match what it can do stays below its peers on that ability and looks exactly like a degraded agent, so MeshHeal excludes it from that work, the right routing decision. Second, a task may be broken into the wrong abilities. Ability inference runs in the shared execution layer, and MeshHeal and all five baselines work from the same inferred pair (App. B.4). With the task-to-ability examples, a task description the deployer registers once, inference matches the documented mapping on every task (Tab. B7); removing them would change the inputs of all methods alike. A one-off decomposition error affects one output, which is still reviewed and, if low-scored, replaced. Its score moves the eight-observation average by at most one eighth, which cannot cross the isolation threshold on its own; healthy agents receive such scores on 4.4% of their outputs and were never isolated. A systematic error for a task type affects all agents on it alike, so the peer-relative statistic does not shift.

Insufficient qualified reviewers (W1, Q2). Any peer-checking method needs qualified healthy peers. Distributed fault tolerance has conditions of the same kind, such as a healthy majority, and SAC's fault bound FF plays this role in [1]. MeshHeal's condition is local and mild: each agent needs one normal neighbor with the same ability. Our sparse graphs meet it with 0.88% of the edges of the complete graph (Tab. B12) and stay within 0.010 of its accuracy (Tab. 2(b)); the revised Sec. 3.2 states it as an explicit assumption. It can be relaxed without changing the method: the distance tables that route tasks can also route a review request to the nearest qualified normal agent two or more hops away, at the cost of a few relay messages and no additional model calls.

All qualified neighbors watched or isolated. The detector marks only agents worse than their peers and isolation needs two consecutive crossings, so all same-ability neighbors can be watched or isolated only when they have really degraded. If one qualified reviewer remains, it is called twice in separate contexts; if none remains, the output is accepted without review and does not update the detector, and the base workflow runs for that output until recovery probes bring agents back (App. B.5). Several neighbors failing at once. A3 and A5 are degraded together during tasks 253 to 378 of the staggered experiment, and one third of all agents degrade at once in the scaling experiment, where all are detected (Tab. 2(b)). The reference uses only normal-state neighbors and measures normal variation as the median minus the 25th percentile, so degraded neighbors not yet detected do not shift it, provided healthy agents are the majority among the neighbors pooled per task type, as the revised Sec. 3.2 states. Outputs that need several abilities. These are supported by design: the reviewer must hold every ability the output uses, and the score updates the detector for each (Sec. 4.2.1, App. B.5). In our task mappings each output uses one ability, so the case did not arise in the experiments.

AuthorsOpenReview ↗

Response to Reviewer 4AUX (part 3)

W2: reliability of detection

On BBH and MMLU-Pro, as on MATH, both degraded agents are isolated in every run, closure holds and no healthy agent is isolated (Table R2). At the default thresholds, all 3,272 degraded agent-runs across the reported experiments are detected and none of the 6,544 healthy agent-runs is isolated.

We added the staggered degradation and recovery experiment on BBH and MMLU-Pro, three seeds each, with the same schedule as on MATH.

Table R2 (Tab. C10 in the revision). Self-healing under staggered degradation and recovery on all three datasets.

DatasetSeedsDetection delay (A5 / A3)ClosureReintegration delay (A5 / A3)False isolationAccuracy (healthy phase / after degradation)
MATH (Tab. 3)58.0 / 12.45/5, 5/546.0 / 45.00/200.833 / 0.812
BBH39.3 / 14.03/3, 3/341.3 / 40.00/120.902 / 0.893
MMLU-Pro39.7 / 14.33/3, 3/340.7 / 39.30/120.780 / 0.779

On all three datasets both degraded agents are isolated in every run, closure holds, and no healthy agent is isolated. Degradation is most severe on MATH: when the degraded agent is AutoGen’s mandatory manager, accuracy falls to 0.214 on MATH but stays above 0.52 on BBH and MMLU-Pro (Tab. C1). Degraded outputs on MATH therefore receive low scores more consistently, which gives the shortest detection delay but also leaves more low scores in the evidence window at recovery, so reintegration takes longest there. BBH and MMLU-Pro behave alike.

Because Tab. C7 covers only a few agents, we also counted detection outcomes at the default thresholds in the logs of every existing run (Tab. C11 in the revision).

ExperimentDegraded agent-runsDetectedHealthy agent-runsFalse isolations
Main experiment (3 datasets ×\times 5 seeds)3030600
Model pairs A and B (BBH, 5 seeds each)2020400
Staggered experiments (Table R2)2222440
Network scaling (96, 288, 576 agents; complete and sparse graphs; 5 seeds each)3,2003,2006,4000
Total3,2723,2726,5440

All 3,272 degraded agent-runs were detected and none of the 6,544 healthy agent-runs was isolated; treating agent-runs as independent, the 95% upper bound on the false-isolation rate is about 0.05%. Threshold sensitivity is in Table R1 (Q3), and the review scores are checked against correctness under Q4.

Q3: how α\alpha, β\beta, the window and the probe interval were chosen

All values were tuned on held-out MATH streams and then fixed, and Table R1 shows that the results hold across the tested range. The eight-observation window averages out single errors while keeping detection within about ten tasks; the probe interval trades reintegration speed against cost.

All values were tuned on MATH task streams with seeds disjoint from those of the reported experiments and then fixed; BBH and MMLU-Pro were not used for tuning (Sec. 4.2.2, App. B.6). Table R1 in our response to Reviewer 7GRd (Tab. C9) gives the sensitivity: τto\tau_{\mathrm{to}} and σcom\sigma_{\mathrm{com}} follow from the score levels of the rubric, and every tested value of α\alpha and β\beta detects all degraded agents, with false isolation appearing only when β\beta is too low. A window of eight observations averages out chance errors on single tasks while keeping detection within about ten tasks (Tab. 3(b)). The probe interval sets how quickly an isolated agent can return; each probe sends one real task to the isolated agent under committee review, so a shorter interval speeds up reintegration at a higher cost per isolated agent.

Reviewer_7GRdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MeshHeal, a self-healing framework designed for LLM-based decentralized MAS to address the phenomenon of "gray failure." Gray failure occurs when an agent remains responsive and produces syntactically valid outputs, yet its actual performance on certain tasks falls below its nominal capability. MeshHeal relies primarily on reviews from neighboring agents to detect and isolate agents experiencing gray failure, while simultaneously correcting their outputs. The framework operates across two distinct timescales: On the fast timescale, neighboring agents evaluate each intermediate output—initially through a single reviewer, which can adaptively escalate to committee review when necessary—with the capability to correct the output if required. On the slow timescale, MeshHeal assesses the degree of an agent's degradation by comparing its performance against that of its normal neighbors, accumulating evidence of degradation across tasks. Once a specific threshold is exceeded, the agent’s outputs are subjected to mandatory committee review; if a higher threshold is subsequently surpassed, the agent is isolated. Isolated agents undergo periodic "recovery probes" to verify their restoration before being reintegrated into normal routing. Experimental results demonstrate that this approach detects and isolates degraded agents, thereby enhancing the overall task performance of the MAS.

Critical strengths
  • In the designed decentralized MAS, the system first infers the set of abilities required for a given task. Each agent then assesses whether its own abilities can address any missing requirements, subsequently deciding whether to hand off, contribute to, or finalize the task. This approach offers greater flexibility, enabling the dynamic matching of suitable agents and preventing unnecessary involvement.

  • Continuous evaluation of an agent’s output by its neighbors functions as an intrinsic capability assessor—independent of any external modules—that can identify and isolate agents within the MAS that lack the necessary capabilities to handle specific types of tasks.

Critical weaknesses
  • The motivation for the "self-healing" mechanism lacks a realistic basis. For an agent operating normally, errors resulting from insufficient capability across a series of tasks are typically sporadic, discrete, and independent events. Even if an agent consistently receives low review scores over several consecutive tasks, it is more likely that the agent is simply ill-suited for that specific type of task. The paper, however, interprets this as "sustained performance degradation"—implying that a deficiency in capability occurs only over a specific, contiguous time interval—and proceeds to design mechanisms for detection, isolation, and subsequent reintegration. Yet, the paper provides no evidence that such "gray failures"—characterized by sustained degradation followed by recovery—are common in real-world LLM-based MAS. Furthermore, the degradation and recovery observed in the experiments are simulated by replacing the strong model with a weak one, and then swapping it back. The authors should either provide evidence of such degradation in real-world scenarios or evaluate the proposed method under more realistic degradation patterns.

  • The method relies on a large number of manually set thresholds. These include, on a fast timescale, the Takeover low-score threshold (τto=0.40\tau_\text{to} = 0.40) and the review-disagreement threshold (σcom=0.55\sigma_\text{com} = 0.55); and on a slower timescale, the "watched" threshold (α=1.2\alpha = 1.2), the isolation threshold (β=2.0\beta = 2.0), as well as parameters for evidence windows and recovery probe intervals. The paper merely states that these thresholds were "selected empirically," without specifying the data used for their selection or conducting any ablation or sensitivity analysis.

Questions for authors
  • The sustained degradation in the experiments is simulated by artificially replacing the agent's backend with a weak model. How do you demonstrate that this degradation phenomenon also occurs during inference in real-world LLM-based MAS?

  • Could you provide ablation studies for key thresholds—such as τto\tau_\text{to}, σcom\sigma_\text{com}, α\alpha, and β\beta—to clarify how these values were determined and to assess the sensitivity of the proposed method to them?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7GRd (part 1)

We thank the reviewer for the review. We have uploaded a revised PDF in which all changes are marked in blue. Q1 is answered under W1 and Q2 under W2.

W1 and Q1: realism of gray failures

Gray failure is a broad class of hard-to-detect faults: the agent stays online and answers fluently, but its answers get worse, and only their quality shows it. Public incident reports show such periods lasting weeks in production. MeshHeal does not assume that a failure is continuous or that the agent recovers, and it is built not to react to the occasional errors of healthy agents. Replacing the backend is how we inject the failure with known timing; to the neighbors it looks the same. A new experiment in which only one ability of an agent degrades shows that such partial failures are also detected (Table R3 in our response to Reviewer EWZx).

To its neighbors, a gray-failed agent has normal heartbeats, connectivity and output format; only the content of its outputs shows the problem (Sec. 1; [1]). This covers faults that prior work studies separately: a replaced, quantized or misconfigured backend, a corrupted or overflowing context, stale memory, a faulty tool, and the faulty and non-colluding Byzantine agents of [2, 3, 4]. Fail-stop faults are the extreme case, which a heartbeat catches. MeshHeal uses only output quality, so it does not need to know the cause. It also assumes neither one continuous failure interval nor recovery, only that an agent's recent scores can be compared with those of healthy neighbors doing the same work; the evidence window slides and every transition is reversible.

Such periods occur in production and last for weeks. In August and September 2025, three infrastructure bugs lowered the quality of Claude's answers and escaped the provider's own evaluations [5]. In March and April 2026, a lower default reasoning effort, a caching bug that discarded the agent's earlier reasoning in affected sessions, and a system-prompt change degraded Claude Code for periods of four days to five weeks, while the model API was unaffected [6]. Had Claude been one agent in an MAS during these periods, it would have been the gray failure studied here: online, protocol-compliant and fluent, with quality below its normal level for weeks and restored after a fix. Both reports note that each bug hit a different slice of traffic, so the degradation looked inconsistent to the provider but was sustained for an affected user or session, through sticky routing in the first incident and a per-session bug in the second. An agent backed by such a service is such a session. Moonshot AI likewise publishes a tool that checks whether third-party deployments match the accuracy of its reference deployment [7].

Most errors of a healthy agent are occasional, and MeshHeal is designed not to react to them. It averages the last eight peer-relative comparisons on the same task type and ability, and isolates an agent only after two consecutive threshold crossings. Healthy agents receive a low score on 4.4% of their outputs (Tab. C8(b)), yet none of the 6,544 healthy agent-runs was isolated at the default thresholds (Tab. C11). An agent ill-suited to a task type looks the same to its neighbors as a degraded one, and routing the work elsewhere is right whatever the cause. The two cases differ only in whether recovery follows, which costs one committee-reviewed probe per ten tasks for each isolated agent. Isolation in the paper applies to the whole agent. Because routing and evidence are already computed per ability (Sec. 4.1, App. B.5), limiting isolation to the affected ability is a direct extension, which we implemented (Table R3): an ill-suited ability keeps failing its probes and stays excluded, while the agent keeps its other work.

Replacing the backend lowers output quality, the only signal MeshHeal uses, and fixes the exact onset and end of a failure, like the reverted serving changes above and providers serving a different version of a model [7]. The experiments vary its strength, timing and extent. The mild degradation of pair A (Gemma-3-4B as the weak model) is detected at the default thresholds without false isolation (Tab. C7). In the staggered experiment, failures overlap and agents recover at different times, and all three self-healing criteria hold in every seed on all three datasets (Tab. 3, Tab. C10). When one third of the agents in networks of up to 576 degrade at once, all are detected (Tab. 2(b)). When only one ability of an agent degrades, both impaired abilities are detected without false isolation, though more slowly than under whole-backend degradation because fewer outputs carry the signal (Table R3). The revised Sec. 1 adds the two incident reports.

AuthorsOpenReview ↗

Response to Reviewer 7GRd (part 2)

W2 and Q2: thresholds

τto\tau_{\mathrm{to}} and σcom\sigma_{\mathrm{com}} follow directly from the score levels of the review rubric. α\alpha and β\beta were tuned on held-out MATH streams and then fixed; across the tested range, every degraded agent is still detected and accuracy changes by less than 0.01 (Table R1).

τto\tau_{\mathrm{to}} and σcom\sigma_{\mathrm{com}} were set with the judge rubric, not tuned. α\alpha, β\beta, the window and the probe interval were tuned on MATH streams with seeds disjoint from every reported experiment, then fixed. All are used unchanged on BBH and MMLU-Pro (not used for tuning), on both additional model pairs, and on networks of up to 576 agents (App. B.6).

τto\tau_{\mathrm{to}}. The rubric (App. D.1) anchors scores at 1.00 (correct), 0.50 (cannot verify), 0.25 (concrete error) and 0.00 (unusable), and requires a concrete error for any score of 0.40 or below, so τto=0.40\tau_{\mathrm{to}} = 0.40 separates "concrete error" from "cannot verify". Any value above 0.25 and below 0.50 gives the same Accept or Takeover decision except for final scores strictly between the two, such as 0.375, the mean of a 0.25 and a 0.50 committee score. In the staggered experiment (Sec. 6.2), 0.9% of reviewed outputs have such a score, in 1.6% of tasks, so moving τto\tau_{\mathrm{to}} within this range can change at most that share of final answers.

σcom\sigma_{\mathrm{com}}. The sample standard deviation of three scores in [0,1][0, 1] is at most 0.577. On the rubric levels it reaches 0.55 only for 0,1,1\\{0, 1, 1\\} and 0,0,1\\{0, 0, 1\\}, and the second already goes to the committee because its median is 0. σcom=0.55\sigma_{\mathrm{com}} = 0.55 therefore sends an output to the committee exactly when one of three reviews calls it unusable and the other two call it correct, and any value above 0.50 and at most 0.577 behaves the same. In the staggered experiment, it was the only trigger of committee review for 0.5% of outputs.

α\alpha and β\beta. We ran the full staggered MATH experiment with α∈1.0,1.5\alpha \in \\{1.0, 1.5\\} and β∈1.5,2.5\beta \in \\{1.5, 2.5\\}, one parameter at a time, on seeds 1 to 3, so every row sees the same tasks in the same order; the default row comes from the original runs with these seeds. Metrics follow Sec. 3.2 and Tab. 3 and are means over the three seeds. A false isolation is an always-healthy agent (A1, A2, A4, A6) isolated at least once in a run, out of 12 agent-runs per setting.

Table R1 (Tab. C9 in the revision). Sensitivity to α\alpha and β\beta on the staggered MATH stream (three seeds).

SettingDetection delay (A5 / A3)False isolationReintegration delay (A5 / A3)AccuracyTokens / task
Default (α=1.2\alpha = 1.2, β=2.0\beta = 2.0)7.3 / 11.30 / 1245.7 / 44.00.80949k
α=1.0\alpha = 1.07.7 / 11.70 / 1247.0 / 45.30.81049k
α=1.5\alpha = 1.57.3 / 11.00 / 1244.3 / 43.00.80849k
β=1.5\beta = 1.56.7 / 10.03 / 1246.3 / 44.70.80150k
β=2.5\beta = 2.511.0 / 15.70 / 1237.3 / 35.00.80650k

In every run both degraded agents (A5, A3) are isolated and closure holds, and accuracy varies by less than 0.01 across settings. α\alpha sets when an agent's outputs must go to the committee (the watched state) and does not appear in the isolation rule; α=1.0\alpha = 1.0 and 1.5 stay within the run-to-run variation of the default. β\beta sets how much evidence isolation needs. A degraded agent receives a low score on almost every output (mean 0.08, Tab. C8(a)), so β=2.5\beta = 2.5 delays its isolation by about four tasks. A healthy agent receives one on 4.4% of outputs after committee review (Tab. C8(b)), so at β=1.5\beta = 1.5 a few such scores close together can isolate it. This happened in 3 of 12 agent-runs, and each agent returned to normal routing through recovery probes after 34 tasks on average. Recovery applies the same thresholds in reverse, so a larger β\beta also returns a recovered agent sooner (37.3 and 35.0 tasks at β=2.5\beta = 2.5 against 45.7 and 44.0 at the default). The default gives fast detection without false isolation.

The results are insensitive because zz measures how far an agent's recent scores fall below those of healthy neighbors doing the same kind of work, in units of how much healthy agents vary, and degraded agents fall far below (0.08 against 0.89). A wide range of thresholds therefore separates the two groups, and under the milder degradation of pair A the same defaults still detect both degraded agents without a false isolation (Tab. C7). A deployment that wants different behavior sets β\beta from Table R1: lower for faster detection, higher for fewer false isolations and faster reintegration. The revision adds the derivations of τto\tau_{\mathrm{to}} and σcom\sigma_{\mathrm{com}} (Sec. 4.2.1), the selection protocol (Sec. 4.2.2, App. B.6) and this analysis (App. C.6).

AuthorsOpenReview ↗

Response to Reviewer 7GRd (references)

[1] P. Huang, C. Guo, L. Zhou, J. R. Lorch, Y. Dang, M. Chintalapati, and R. Yao. Gray failure: The Achilles’ heel of cloud-scale systems. In HotOS, 2017.

[2] J.-T. Huang, J. Zhou, T. Jin, X. Zhou, Z. Chen, W. Wang, Y. Yuan, M. Lyu, and M. Sap. On the resilience of LLM-based multi-agent collaboration with faulty agents. In ICML, 2025.

[3] H. Lee, V.-D. Yun, D. Panagou, and S. P. Karimireddy. Robust multi-agent LLMs under Byzantine faults. arXiv:2605.09076, 2026.

[4] R. Li, Z. Zhang, X. Bo, Q. Dai, C. Li, F. Wen, and X. Chen. Towards adaptive, scalable, and robust coordination of LLM agents: A dynamic ad-hoc networking perspective. arXiv:2602.08009, 2026.

[5] Anthropic. A postmortem of three recent issues. 2025. https://www.anthropic.com/research/a-postmortem-of-three-recent-issues

[6] Anthropic. An update on recent Claude Code quality reports. 2026. https://www.anthropic.com/research/april-23-postmortem

[7] Moonshot AI. Kimi Vendor Verifier. 2026. https://github.com/MoonshotAI/Kimi-Vendor-Verifier

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.