- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents MeshHeal, a decentralized framework for handling agents that remain responsive but produce persistently lower-quality outputs. It combines immediate peer review and correction with longer-term monitoring: ability-matched reviewers check outputs before they are used, while acumulated scores relative to neighboring agents determine whether an agent receives additional scrutiny or is excluded from ordinary routing. Recovery probes allow excluded agents to rejoin after their performance improves. The paper also introduces an evaluation protocol that switches between strong and weak model backends to control agent capabilities and simulate degradation. Experiments on BBH, MATH, and MMLU-Pro compare MeshHeal with several routing and coordination baselines, supplemented by component ablations, different model pairs, staggered degradation and recovery, and networks of up to 576 agents. The reported results show competitive accuracy at lower token cost than the strongest baseline, alongside successful isolation and reintegration under the tested conditions.
Critical strengths
There are several critical strengths in this paper:
-
The framework connects immediate error correction with longer-term agent management. MeshHeal checks and, when necessary, replaces an output before it affects the current task, while retaining evidence about the original producer for subsequent routing decisions. Recovery probes also give excluded agents a way to demonstrate improved performance and rejoin. This is a coherent system design that addresses both error propagation and the risk of permanently excluding an agent after a temporary failure.
-
The reported accuracy–cost trade-off is promising under the tested conditions. Across the three benchmarks, MeshHeal achieves a mean degraded-phase accuracy of 0.839 with approximately 51k tokens per task, compared with Symphony’s 0.807 and 115k tokens. The experiments also cover different model pairs and staggered degradation and recovery. These results provide useful evidence that selective review and correction can maintain task performance without the cost of repeatedly generating and refining parallel solutions.
Critical weaknesses
There are some critical weaknesses in the paper that I would like to mention:
-
The motivation and contribution are not sufficiently distinguished from existing work. The paper overstates the limitations of approaches based on historical reputation or routing feedback. RAPS already includes an on-the-fly watchdog that evaluates incoming outputs, and its case study demonstrates preventing an erroneous output from reaching the current task’s final answer. The ICML 2025 work on faulty-agent resilience also introduces an Inspector that reviews and corrects messages. Protecting the current task therefore needs a more careful comparison with existing mechanisms. MeshHeal’s potential contribution lies in combining immediate correction, persistent degradation detection, isolation, and reintegration. However, the paper does not establish when this combination is necessary or why simpler extensions of existing methods would be insufficient. Listing differences in the related work does not resolve this question, which directly affects my assessment of novelty and significance.
-
Several important methodological choices remain insufficiently justified or specified. The main parameter values are provided, but there is little evidence supporting the particular normalization and transition rules. For example, the detector uses a scale floor of 0.10, watch and isolation thresholds of 1.2 and 2.0, and an eight-observation window without a sensitivity study or a clear calibration procedure. Comparing scores from different questions within the same task category also leaves uncertainty about whether the detector separates persistent degradation from ordinary difficulty and capability differences. The description is sufficient to implement an approximate system, but not to reproduce the reported evaluation reliably. Complete runtime prompts, main-experiment decoding settings, task sequences, and baseline adaptations are not fully specified. The absolute-threshold and CUSUM comparisons are particularly difficult to assess without their exact rules and parameter settings.
-
The evaluation does not establish the capability assumptions on which the framework relies. Switching to a strong model when an agent is assigned a matching ability does not demonstrate that the model lacks the other ability required by the task. A strong model receiving the full question may solve more than its assigned subproblem. Consequently, the claim that each task requires at least two complementary agents is imposed by the protocol but not convincingly validated. A strong single-agent control would materially help establish whether the gains depend on capability-aware collaboration. The tested failures replace all calls from selected agents with a weak backend, while each evaluated output updates only one ability-specific detector. This does not test partial capability degradation, despite that possibility being included in the problem definition. The distinction matters because evidence is maintained per task and ability, but isolation and reintegration apply to the entire agent. It remains unclear whether successful probes on an unaffected ability could readmit an agent whose impaired ability has not recovered.
-
The experiments do not isolate the claimed contribution, and the baseline comparisons need further clarification. Table C6 supports the fast review-and-correction components: degraded-phase accuracy falls from 0.900 to 0.865 with a single reviewer and to 0.844 without Takeover. However, both variants retain detection and rerouting. These results do not establish the additional value of the slow-timescale detector, explicit isolation, or recovery probes. Comparisons that remove those components, or add simple monitoring and exploration to an existing method, are needed to attribute the gains. The SAC baseline is described as three teams followed by filtering, refinement, and majority aggregation. The original algorithm instead relies on receiver-side scoring, self-anchored filtering, and a fault bound (F). The paper does not explain how these mechanisms are preserved in its adaptation. Similarly, it is unclear whether the RAPS implementation retains its immediate watchdog. These omissions make the reported advantages difficult to interpret. The advantage over Symphony is also setting-dependent. In the staggered recovery experiment, pooled accuracy is 0.812 versus 0.806, while fully recovered accuracy is 0.857 versus Symphony’s 0.873. Under model pair B, degraded-phase accuracy is 0.782 versus 0.786. The lower token cost is useful, but these results do not establish consistent accuracy superiority, and the small differences require paired uncertainty estimates.
Questions for authors
The following are my questions for the authors:
-
How faithfully do the adapted baselines preserve their original mechanisms? For SAC, how are receiver-side scoring, self-anchored filtering, and the fault bound (F) implemented within the three-team setup? For RAPS, is the on-the-fly watchdog retained when processing incoming outputs? Please provide the relevant settings and adaptation details, since these mechanisms directly affect the comparison with MeshHeal’s current-task protection.
-
What establishes the additional benefit of the slow-timescale mechanism? Table C6 retains detection and rerouting in both ablations, so it primarily tests the fast review-and-correction components. Can you provide a comparison against a version retaining peer review and Takeover but removing persistent detection and isolation, under comparable inference budgets? This would help determine how much of the reported improvement depends on the proposed two-timescale design.
-
How are conflicting health signals across abilities handled? Evidence is maintained separately for each task type and ability, but routing status applies to the entire agent. If an agent is isolated because one ability has degraded, what prevents successful recovery probes on an unaffected ability from restoring its eligibility while the impaired ability remains unreliable? Please clarify how evidence is combined and whether reintegration requires evidence of recovery for the ability that triggered isolation.
6 comments
Response to Reviewer EWZx (part 2)
W2: thresholds, difficulty within a task type, and reproducibility
and follow from the rubric, and the tuned parameters are insensitive over the tested range. Comparing an agent with peers on the same task type and ability cancels out question difficulty, and at the default thresholds no healthy agent was isolated in any reported run. The revision gives the decoding settings and both comparison rules with their parameters, and code and logs will be released.
Thresholds. and follow from the score levels of the review rubric rather than from tuning: separates “concrete error” (0.25) from “cannot verify” (0.50), and sends an output to the committee exactly when one of three reviews calls it unusable and the other two call it correct (derivations in Sec. 4.2.1 and App. C.6 of the revision). , , the window and the probe interval were tuned on MATH streams with seeds disjoint from all reported runs, then used unchanged on BBH, MMLU-Pro, both model pairs and all network sizes. Varying and on the staggered stream (Table R1 in our response to Reviewer 7GRd; Tab. C9) keeps accuracy within 0.01 and detects every degraded agent in every run; the only notable change is that isolates a healthy agent in 3 of 12 agent-runs, each of which returns through probes. The scale floor of 0.10 keeps bounded when healthy agents score so consistently that their lower spread is near zero.
Difficulty and capability differences. The reference for agent comes from normal neighbors answering other questions of the same task type with the same ability. These questions come from the same distribution and the window averages eight of them, so question difficulty enters the reference and the agent’s mean in the same way, and the lower spread of the peers’ mean shortfalls measures how much healthy agents differ, ordinary capability differences included. An agent is flagged only when its mean falls well outside that spread on consecutive updates. Across all runs, 0 of 6,544 healthy agent-runs were isolated and 3,272 of 3,272 degraded agent-runs were detected (Tab. C11). An agent that stays far below its peers on the same work is, for routing, the agent to avoid, whatever the reason.
Reproducibility. All agents and methods use temperature 0.7, top_p 1.0, at most 2,048 output tokens and the provider’s default reasoning effort. The absolute rule isolates an agent when the mean of its own last eight scores stays below 0.75 on two consecutive updates; CUSUM sums how far each new score falls below the healthy agents’ mean, less a slack of 0.25 and floored at zero, and isolates when the sum exceeds 2.0. As for our detector, both were set on the default model pair and used unchanged on pairs A and B. The revised App. B.5 states both rules with their parameter values and App. B.4 the decoding settings and baseline adaptations; the complete runtime prompts, the task sequence of every seed, the code and the run logs will be released with the camera-ready version.
Response to Reviewer EWZx (part 3)
W3: capability assumptions and partial degradation
The evaluation tests whether coordination and routing are correct, not whether an MAS beats a single agent, so tasks that need two agents are part of the construction rather than a claim about the benchmarks. The paper degrades all abilities of an agent together; MeshHeal extends directly to single-ability degradation by keying the routing state and the recovery probes on the ability. A new experiment (Table R3) shows that this extension detects the impaired ability, never readmits it before recovery, and keeps the agent's other abilities in service; detection is slower than under whole-backend degradation because the signal is sparser.
Testing whether coordination and routing are correct needs tasks on which routing matters. Standard benchmarks lack this property, since one capable model can attempt every task, and assigning abilities through system prompts does not create it (Tab. 1). Model-Backed MAS Evaluation builds it from existing benchmarks: each task type is split into two complementary abilities held by disjoint sets of agents, and the backend makes an agent strong only on the abilities it holds. A task is then answered correctly only when each part reaches a healthy agent that holds the matching ability, so routing errors and gray failures become visible in accuracy. In deployed heterogeneous systems, such complementarity comes from tools, data access or specialized models that one general model cannot replace; the construction reproduces this situation with known ground truth.
That every task needs two agents is therefore part of the construction: the execution prompts restrict a Contribute step to the agent's own part (App. D.2), and the rubric scores only that part (App. D.1). For the same reason, a strong single-agent control does not bear on our question: Eq. (1) is defined over local coordination policies for a given network and ability assignment, and a single agent holding every ability removes the routing problem under test. The revised Sec. 5 opens with this purpose.
Single-ability degradation (Q3). In the paper all abilities of an agent fail and recover together. Evidence is already kept per task type and ability, but the routing state is kept per agent and follows the most recently updated detector, so a probe on an unaffected ability can readmit an agent whose impaired ability has not recovered. Because routing is already computed per ability (Sec. 4.1, App. B.5), the extension is direct: the routing state, the isolation counter and the recovery probes are keyed on the agent and the ability, so an agent isolated on ability is removed only from work that needs and probed only with such work. Reintegration on then requires recovered evidence on itself, while the agent's other abilities stay in service.
A new experiment runs both versions under this failure (App. C.6 of the revision). On the 504-task BBH stream with seeds 1 to 3, A5 degrades only on spatial and A3 only on mathematical during tasks 253 to 420; both are restored from task 421, and every other ability keeps the strong backend.
Table R3 (Tab. C14 in the revision). Single-ability degradation on BBH (seeds 1 to 3).
| Metric | Per-agent state (paper) | Per-ability state (extension) |
|---|---|---|
| Detection delay, A5 / A3 (tasks) | 31.0 / 22.7 | 24.3 / 17.0 |
| Impaired abilities isolated | 6/6 | 6/6 |
| Readmissions before recovery, per agent-run | 2.0 | 0 |
| Unaffected abilities withheld after detection | all | none |
| Impaired outputs accepted without committee review | 8% | 4% |
| Reintegration delay after restoration, A5 / A3 (tasks) | n/a (readmitted before recovery) | 52.7 / 44.3 |
| False isolations (healthy agent-runs) | 0/12 | 0/12 |
| Accuracy, healthy phase / after task 253 | 0.899 / 0.894 | 0.899 / 0.898 |
| Tokens per task after task 253 | 47k | 46k |
Detection is slower than under whole-backend degradation (9.3 and 14.0 tasks on BBH, Tab. C10), because only outputs that use the impaired ability carry the signal and fewer of them arrive per task. With the per-agent state of the paper, outputs on unaffected abilities also reset the isolation counter, and probes on those abilities readmit each agent about 2.0 times per run before its impaired ability recovers. The fast timescale limits the cost: a readmitted agent is watched first, its impaired outputs are reviewed and replaced, and accuracy after task 253 is 0.005 below the healthy phase. The per-ability state removes the problem: it isolates the impaired ability sooner, because unaffected outputs no longer reset its counter, never readmits it before recovery, keeps the agent's other abilities in service, and reintegrates it 52.7 and 44.3 tasks after restoration, more slowly than whole-agent reintegration (Tab. C10) for the same reason that detection is slower. No healthy agent is isolated in either version.
Response to Reviewer EWZx (part 1)
We thank the reviewer for the detailed review. We have uploaded a revised PDF in which all changes are marked in blue. Direct answers to Q1–Q3 are given at the end.
W1: contribution relative to RAPS and the Inspector
We agree that checking outputs within a task is not new, and the revised related work says so. Our contribution is the coupling of the two timescales: replacing an output must be decided from one or two reviews, stopping work to an agent must not, and each decision depends on the other. Approaches that make only one of the two decisions, such as RAPS or an Inspector-style variant, lose accuracy in our results.
We agree that checking outputs within a task is not new. The Inspector [1] reviews and corrects messages, and RAPS [2] runs a watchdog on every incoming output; our related-work section overstated the gap, and the revised Sec. 1 and Sec. 2 correct the description of RAPS and add the Inspector. The contribution we claim is the coupling of the two timescales, and the reviewer’s question, when the combination is necessary, has a direct answer.
The fast decision (keep or replace this output) must act on one or two reviews. The slow decision (stop routing work to this agent) must not: healthy agents receive a low score on 4.4% of their outputs (Tab. C8(b)), and a rule on an agent’s own scores isolates all four healthy agents under pair A (Tab. C7). The two decisions also depend on each other. Review produces the evidence for the slow decision, and the slow state decides who may review, whose outputs must go to the committee, and who infers the required abilities as an entry agent. A method that makes only one of the two decisions loses accuracy in a predictable way, and each existing approach corresponds to one such case in our results:
Within-task correction only, the Inspector’s mechanism (our Fast-only variant, Table R4): degraded agents keep receiving work and keep reviewing others. Degraded-phase accuracy on BBH drops by 0.062 while cost rises by 33%.
A watchdog with cross-task avoidance (RAPS): the current task is unprotected until reputation crosses its threshold, and an avoided peer is never re-evaluated. RAPS reaches 0.740 on BBH and 0.653 over the three datasets in the degraded phase (Tab. C1, Tab. 2(a)).
Cross-task isolation without replacement (No Takeover, Tab. C6): degraded-phase accuracy drops by 0.056.
Compared with RAPS, three design choices differ. (1) RAPS’s watchdog returns a binary judgment that updates a Beta reputation, and the broker routes around peers whose reputation crosses τrep ([2], Sec. 4.4 and Alg. 1). In its case study, the flagged output stays out of the answer because a parallel peer’s correct output is routed downstream instead. On the sequential Handoff–Contribute–Finalize path shared by all methods here, there is no parallel output to fall back on, and Takeover supplies a corrected one. (2) RAPS compares a peer’s reputation with a fixed threshold; MeshHeal compares an agent with peers on the same task type and ability and scales by their normal variation, which lets one set of thresholds hold across datasets, model pairs and network sizes (Tab. C7). (3) A peer that RAPS routes around publishes nothing further, so its reputation stops changing and it cannot return; recovery probes give an isolated agent fresh, committee-reviewed evidence.