- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ToCoMAS, which uses a shared task-and-tool topology to organize agent responsibilities, communication, and memory permissions. Task nodes are grouped into responsibility regions, and cross-region dependencies determine required handoffs. An online evolution process proposes structural changes and retains candidates that satisfy coherence constraints and improve reward. Experiments cover reasoning, workplace tool use, software repair, and collaborative memory workflows.
Critical strengths
The paper identifies a useful organizational problem: changing agent responsibilities can invalidate communication paths and memory permissions. Treating these changes together is a sensible design principle for evolving multi-agent systems.
The evaluation considers more than final task success. Verified node progress, accepted handoffs, and resistance to polluted context provide useful views of where coordination succeeds or fails. The reported improvements across several benchmarks and backbones are also encouraging.
Critical weaknesses
The source of the task graphs is insufficiently explained. ToCoMAS begins with an upstream-provided task graph, which supplies much of the structure needed to assign responsibility and derive handoffs. CoMemBench provides such graphs, but the paper does not clearly explain how they are obtained for BBEH, WorkBench, and SWE-Bench-Verified, or whether competing methods receive equivalent information. This is important for interpreting the gains. Several central implementation choices are also underspecified, including the whole-graph tool-grounding procedure, compatibility scoring, proposal generation, and acceptance margin.
The ablations do not isolate the proposed coherence principle. Removing a module also removes useful functionality, such as compatible task grouping, consumer-aware handoffs, or restricted memory visibility. The resulting performance drops support those components, but do not establish that coupling their updates is the reason for improvement. A more direct control would retain the same components and capabilities while comparing independent updates with coherence-constrained updates.
The evaluation protocol needs more detail before the results can be assessed reliably. The paper does not report benchmark sample counts, repeated-run uncertainty, or sensitivity to task order. Task order is particularly relevant because feedback from one task changes the organization used on later tasks. The cost analysis also pairs post-recheck progress with first-pass tokens for WorkBench, while using a separate cohort for CoMemBench. These choices make the efficiency comparison difficult to interpret. Costs and outcomes should be reported for the same attempts, including candidate evaluation and evolution overhead.
Questions for authors
How are task graphs generated for the three benchmarks without native workflow graphs, and what graph information is available to each baseline?
Can you compare coupled and independent structural updates while keeping task grouping, handoff validation, memory filtering, and search budgets otherwise comparable?
Please provide evaluated sample counts, repeated-run results, task-order controls, and cost measurements that include all evolution-related calls.
4 comments
2nd Response to Reviewer 9Yn8
The noisy-graph evolution and no-evolution conditions use identical perturbation instances, task sets, initialization, and first-execution budgets.
Table N1. CoMemBench with predicted or perturbed dependencies (%).
Fixed denotes a single execution without evolution; Round 0 and Final denote first execution and post-adaptation outcomes with online evolution. Perturbation drops 25% of dependency edges and adds up to 10% type-compatible spurious edges, retaining executable nodes and acyclicity.
Setting SR VNCR VHS
Released / fixed 10.35 ± 1.03 39.97 ± 1.25 47.29 ± 1.28
Predicted / fixed 9.24 ± 0.63 37.11 ± 0.74 39.26 ± 1.23
Predicted / Round 0 10.66 ± 1.01 37.50 ± 0.45 47.16 ± 0.97
Predicted / Final 12.59 ± 0.85 42.58 ± 0.71 46.79 ± 0.34
Perturbed / fixed 8.00 40.67 42.91
Perturbed / Round 0 10.65 46.97 49.47
ToCoMAS derives responsibility regions and interfaces from the supplied dependencies, while verified artifact delivery constrains error propagation. Evolution improves first-attempt performance under both predicted and perturbed dependencies; Final additionally captures current-task recovery. These results indicate useful adaptation under imperfect dependencies, rather than exact reconstruction of the reference graph.
Cases: local recovery under predicted dependencies. A targeted retry increased verified nodes from nine to eleven in code task v15_family_0335_0a440df98d99; configuration changes increased them from four to six in stateful task v15_family_0073_1e248b8fd711. These traces show partial progress recovery within the dependency-based organization, without reconstructing the reference graph.
Because dependencies alter eligible handoffs, VHS characterizes coordination within each graph setting; SR and canonical-node VNCR provide the primary cross-graph comparison.
Q2. Coupled versus independent structural updates
We supplement the component ablations with a decoupled structural-update control that retains structural operations, handoff validation, and memory filtering, but does not reconcile handoff interfaces and existing memory ownership after an edit. The full and decoupled configurations share the same initial organization, capabilities, validators, and search budget; only post-edit reconciliation differs. Two additional ablations remove structural operators or memory-conditioned retry. Table N2 compares their post-adaptation performance with full ToCoMAS; all rows report Final outcomes, separately from the first-attempt results in Q3.
Table N2. Full ToCoMAS and three ablations on CoMemBench (Final, %).
Configuration SR VNCR VHS
Full ToCoMAS 15.79 ± 1.24 48.72 ± 0.47 55.32 ± 0.31
Decoupled structural updates 14.14 48.00 54.52
Without structural operators 13.50 47.48 53.98
Without memory-conditioned retry 12.00 47.57 53.26
Full-model values are means ± sample standard deviations; ablations are point estimates.
Full ToCoMAS exceeds all three ablations. Decoupling tests whether interfaces and memory ownership must track structural edits; removing structural operators tests adaptation beyond retries; removing memory-conditioned retry tests memory-assisted recovery. The latter produces the largest SR reduction despite a smaller VNCR gap, consistent with critical unresolved nodes blocking terminal completion.
Execution evidence makes the effect of decoupling concrete. In the 800-task evaluation, 408 tasks exercised at least one structural-edit round. The records contain 184 cross-region handoff-drop events caused by interfaces not being rebuilt and 28 accepted structural edits.
Case: django-11299. Both configurations apply the same split, making n01 → n02 a cross-agent dependency. ToCoMAS rebuilds the interface and delivers the artifact; the decoupled variant retains the old interface, leaving n02 without its required input and exhausting the budget through subsequent missing-input failures. The dependency is unchanged, but its responsibility boundary changes: this illustrates why structural edits require corresponding handoff updates. The ablation also tests reconciliation of memory ownership.
3rd Response to Reviewer 9Yn8
Q3. Evaluation size, ordering, repeated results, and total cost
Evaluation protocol. The main evaluation covers 4,520 BBEH tasks, 690 WorkBench tasks, 500 SWE-Bench-Verified issues, and 800 CoMemBench tasks. Its four domains contain 200 tasks each; the matched structural-update evaluation covers the full set. The online experiments use a score-before-update protocol: each task's Round 0 score is fixed before its own candidate evaluations, and retained updates become available to later tasks. Final is reported separately to show additional within-task adaptation. Repeated runs use the same task set in different shuffled orders.
For each ToCoMAS task, all evolutionary candidates share one additional step budget equal to the domain-level full-graph budget; it is not renewed per candidate. Rejected candidates also consume this budget. Tasks read the available pool state at launch; with up to four concurrent tasks per domain-fold stream, they use updates committed before that snapshot. In the EvoMAS evolution comparison, evaluator pass/fail feedback replaces the default LLM judge, so terminal correctness feedback is not exclusive to ToCoMAS. EvoMAS follows the same score-before-update protocol.
Table N3. Repeated CoMemBench outcomes with Qwen3.8-27B (%).
Setting SR VNCR VHS
No evolution 10.35 ± 1.03 39.97 ± 1.25 47.29 ± 1.28
Online / Round 0 13.75 ± 1.21 42.04 ± 1.31 49.39 ± 1.73
Online / Final 15.79 ± 1.24 48.72 ± 0.47 55.32 ± 0.31
Table N4. Repeated general-task outcomes with Qwen3.8-27B (%).
Setting WorkBench SR BBEH accuracy
No evolution 73.66 50.34
Online / Round 0 78.83 ± 1.26 54.89 ± 0.99
Online / Final 80.42 ± 0.77 57.31 ± 1.08
Values with uncertainty are means ± sample standard deviations: four runs for CoMemBench online evolution, three for its non-evolving reference, and three for WorkBench and BBEH online evolution. The main table uses single runs for all methods; repeats were added in response to the review. The online configuration has higher mean first-attempt performance than the non-evolving reference, and Final shows further within-task gains. The BBEH rows compare ToCoMAS configurations on the same item set.
Online total costs include all model calls: planning, execution, coordination, memory, meta-model calls, and candidate evaluation, including calls incurred before rejection or rollback.
Clarifying “post-recheck.” This denotes offline rescoring of the original WorkBench execution trace, without additional model calls. Figure 5 therefore pairs progress and tokens from the same first execution; Tables N5–N6 additionally account for online evolution, including rejected candidates.
Table N5. WorkBench tokens per task (k = thousand).
Method / configuration Mean Median
Direct LLM Call 2.6k 2.4k
ADAS 20.7k 16.8k
OMAC 47.1k 44.1k
Peer Review 268k 209k
SMoA 313k 246k
ToCoMAS, first execution 32.0k 29.6k
ToCoMAS, online evolution 86.2k 60.1k
EvoMAS, frozen configuration 30.5k 18.0k
EvoMAS, online evolution 290k 160k
ToCoMAS's first-execution mean is close to that of frozen EvoMAS. Including evolution, its mean cost is 70.3% lower than online EvoMAS; Direct and ADAS remain cheaper.
Table N6. CoMemBench tokens per task with Qwen3.8-27B (k = thousand; M = million).
Method / configuration Mean Median
Direct LLM Call 44k 16k
ADAS 565k 436k
EvoMAS, first execution 1.31M 977k
SMoA 1.42M 858k
Peer Review 3.61M 2.27M
OMAC 8.01M 5.54M
EvoMAS, online evolution 10.7M 8M
ToCoMAS, no evolution 343k 208k
ToCoMAS, online evolution 937k 784k
The CoMemBench online totals likewise include candidate evaluation. ToCoMAS's recorded expenditure is allocated as 38.2% to Round 0, 6.1% to accepted candidates, and 55.7% to rejected or rolled-back candidates. Thus, unsuccessful search is included rather than discarded from the cost.
Targeted recovery reuses verified artifacts and re-executes affected nodes. In code task v15_family_0335_0a440df98d99, re-executing five nodes increased verified progress from nine to eleven. This limits repeated work and provides a mechanism consistent with the observed cost pattern.
Manuscript revisions. We will clarify tool grounding, region formation, and candidate acceptance; add the matched coherence control; and report evaluation sizes, scoring denominators, repeated results, and total costs.
Dear Reviewer 9Yn8,
Could you please review the authors’ rebuttals and follow up with any further comments or questions?
Many thanks for your time and efforts.
Best regards, AC
Response to Reviewer 9Yn8
We thank the reviewer for recognizing the importance of maintaining valid communication and memory access when agent responsibilities change. We clarify the graph inputs and implementation, compare full ToCoMAS with three ablations, and explain the effects of decoupling through execution evidence. We also report repeated outcomes and costs including evolutionary search.
Q1. Graph construction, baseline inputs, and implementation details
Graph access. On CoMemBench, all methods receive the same released task nodes, dependencies, artifact types, and acceptance requirements, and their artifacts are assessed by the same verifiers. ToCoMAS derives responsibility regions, agent assignments, handoff contracts, and memory permissions from this shared input. On BBEH, WorkBench, and SWE-Bench-Verified, ToCoMAS generates its task graph, while baselines follow their native protocols. These benchmarks compare complete systems; CoMemBench supplies the comparison under shared graph information.
The planner reads the request, public environment information, and fixed domain vocabularies and dependency rules, without accessing answers, tests, or evaluators. It outputs typed nodes with descriptions, roles, and upstream inputs. Validation checks vocabulary membership, acyclicity, and a single terminal node. For the controlled CoMemBench planner study, predicted dependencies are mapped onto released executable nodes; on the other three benchmarks, the predicted nodes themselves are executed. CoMemBench reference graphs incur no construction calls. All planning calls on the other benchmarks are charged to ToCoMAS.
Tool grounding and region formation. In the CoMemBench grounding pipeline, a node's declared execution contract specifies its required operations, resources, capabilities, effects, and receipts. Candidate tool bundles must cover these requirements; read-only nodes exclude side-effecting tools. We enumerate bundles in increasing cardinality and retain candidates at the smallest feasible size. A width-32 beam search then selects reference bundles in topological order, considering lexical mismatch, surplus capabilities, and provider continuity. Artifact-type incompatibility is rejected before search, and incompatible shared-resource provider assignments are pruned. Thus, “minimal” concerns each node's reference bundle, not the entire authorized tool set or a claim of exact global optimality. Execution may additionally authorize compatible alternatives. The other benchmarks bind nodes to tools through their benchmark adapters.
In the ToCoMAS experiments, node-pair affinity combines tool overlap, input-output compatibility, dependency adjacency, and lexical similarity with weights 0.35, 0.25, 0.25, and 0.15. Tool and lexical overlap use Jaccard similarity; input-output compatibility takes the larger directional Jaccard, and direct non-revision adjacency is binary. Eligibility gates first check adjacency, isolation, and role/verification compatibility. Eligible pairs are processed in descending score order, with an organization-affinity threshold of 0.45. Each proposed single-linkage region merge must satisfy capacities of four nodes and eight tools, convexity, and an acyclic quotient. They do not constitute a general cross-node side-effect conflict solver.
Proposal and acceptance. Execution diagnosis guides retries, configuration changes, memory-conditioned retries, upstream re-execution, and structural split, move, or merge. Candidate selection requires structural validity and either of two improvement criteria: its scalar reward exceeds the parent's by more than 0.01, or it completes without a crash and increases the number of verified nodes without losing any previously verified node. These are alternative acceptance criteria within the same setup, not requirements that must both hold. Unaccepted changes are rolled back.
Across all four CoMemBench domains, node recall is 85.7% and Edge F1 is 60.8%. Alignment is one-to-one, requires matching output types and similarity of at least 0.45, and considers roles, descriptions, and identifiers. Edge F1 measures aligned dependency overlap, excluding revision edges; neither metric measures execution success.
Case:
v15_family_0707_1c6ff89bea6c(mathematics). Exact graph agreement is not necessary for successful execution. In this task, the supplied graph retained nine of thirteen released dependencies, omitted four, and added one; all eight executable nodes and the task still passed. This separates reference-graph fidelity from executable organization: a different edge set can still support the required computation. Topological coherence concerns compatibility among task dependencies, responsibility assignments, handoffs, and memory permissions, rather than reproducing a particular graph shape.