- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Inherit-MAS, an approach that enhances the test-time evolution of multi-agent systems (MAS) through the inheritance of both workflows and execution results. Based on review feedback, the method retains useful nodes, implements constrained modifications to workflows, and reuses historical results whenever the request and execution context match perfectly. Experiments on WorkBench and HotpotQA demonstrate that this approach outperforms existing evolutionary multi-agent baselines while simultaneously reducing the inference overhead of operational nodes.
Critical strengths
- The problem addressed is of practical significance. It specifically considers the loss of valid components—often resulting from iterative modifications to workflows—as well as the overhead of redundant execution, directly aligning with the real-world need for optimization in multi-agent testing.
- The rules for result reuse are rigorous. Reuse decisions are based on comprehensive validation—encompassing the full request, execution context, and the integrity of recorded data—while explicitly excluding state-modifying operations, thereby establishing clear safety boundaries for reuse.
- The experimental analysis is highly transparent. It covers two distinct task categories and two operational models, distinguishes between the overheads of individual work nodes and the controller, and reports on phenomena such as candidate degradation, ranking errors, and limitations in overall cost-effectiveness.
Critical weaknesses
- The current experiments cover only office-tool tasks and multi-hop question answering; validation for the optimization of general multi-agent workflows remains insufficient. Could additional tests be included?
- The approach requires intermediate nodes to be read-only, with all write operations performed exclusively by the final execution node. Please explain how this is adapted for WorkBench.
- The distinctions between this method and existing workflow optimization and result-caching mechanisms require further clarification; it is recommended to include targeted comparisons to more clearly highlight its unique contributions.
- Several evolutionary multi-agent baselines underperform compared to "Single ReAct" on the WorkBench benchmark, for reasons that remain unclear. It is recommended to provide further clarification on whether this performance drop stems primarily from the collaboration and evolutionary mechanisms themselves, or from differences in baseline adaptation and configuration, to help better interpret the comparative results.
Questions for authors
refer weakness
2 comments
Rebuttal - Part 2
W4: Several evolving-MAS baselines underperform Single ReAct on WorkBench. Why? Mechanism or adaptation?
To understand why several multi-agent baselines underperform compared to "Single ReAct" on WorkBench, we traced every failed task for every system in the saved WorkBench logs. We find that the gap mainly comes from multi-agent collaboration.
Where the gap is. 25 of the 130 tasks require no state change, and every system solves 20 to 23 of them. The whole gap is in the 105 tasks that need a write action.
| System (GPT-4o-mini workers) | Solved of 105 | Failed | Failed: record never retrieved | Failed: record retrieved by an agent, but lost before the final action | Failed: other |
|---|---|---|---|---|---|
| Single ReAct | 34 | 71 | 54 | 0 | 17 |
| EvoAgent | 22 | 83 | 59 | 17 | 7 |
| EvoMAS | 20 | 85 | 54 | 14 | 17 |
| TacoMAS | 24 | 81 | 41 | 26 | 14 |
"Other" covers tasks whose writes need no record, and writes that used the right record with a wrong argument.
Collaboration (the evidence is found but lost between agents). Retrieval failures are similar for Single ReAct and the baselines (41 to 59 tasks): this is the shared limit of the worker model. The baselines add the fourth column. In 14 to 26 tasks, one agent retrieved the right record, but the final action did not carry it because the integrating agent issued no write or wrote a placeholder instead. Single ReAct has no handoff and loses none. These losses are as large as the gap itself: the baselines fail 10 to 14 more tasks than Single ReAct.
Evolutionary mechanism. Evolution does not lower quality, but its gain is not returned. EvoMAS's two initial candidates solve 30 tasks each, and its three mutations solve 32 to 35. Some candidates solve the task in 60 of 130 cases (46.2%, above Single ReAct), yet the label-free selection returns only 40.
Rebuttal - Part 1
W1: Only office-tool tasks and multi-hop QA; could additional tests be included?
To add more tests, we ran experiments on BIG-Bench Extra Hard (BBEH), which contains a suite of 23 general reasoning tasks with no tools, retrieval, or environment state, a different regime from both WorkBench and HotpotQA. Following the same experiment settings, the comparison against EvoMAS is shown below.
Inherit-MAS is 4.3 points more accurate than EvoMAS while using 45% fewer worker tokens and 23% fewer total tokens.
W2: Intermediate nodes are read-only and the final node performs all writes: how is this adapted for WorkBench?
The condition applies only to WorkBench, the one evaluated benchmark with mutable state. On HotpotQA, no node changes any state, so the rule is never exercised. It is a condition for a node's result to be eligible for inheritance, not a limit on what a workflow may do.
WorkBench fits it with almost no adaptation, because the benchmark already has this shape. WorkBench itself defines 13 of its 27 tools as read-only and 14 as state-changing, and its reference solution for a task is an ordered list of state-changing calls. Worker nodes are the only nodes with tools, and they may call only the read-only tools declared for them. Integrators and verifiers have no tools: they emit the proposed changes as one JSON batch in the benchmark's own write signatures, which a verifier may approve or revise. A single deterministic, prompt-free executor then validates the batch against the official tool names and argument schemas and applies the surviving calls in order. The benchmark evaluator then scores the final environment state. "Gather evidence, then commit one validated batch" is thus the native form of a WorkBench task.
W3: Distinctions from existing workflow optimization and result-caching mechanisms; targeted comparisons.
Distinctions against existing workflow optimization and result-caching mechanisms.
Targeted comparisons. We ran both comparisons on a HotpotQA subset (GPT-4o-mini workers, same retriever and settings as the paper).
The result shows that Inherit-MAS achieves the best F1 score with the fewest tokens used. Both comparisons point to the same conclusion: the savings come from coupling scoped edits with the store, not from either part alone. With our search-and-only provider prompt caching, the provider recognizes repeated prompt prefixes (21.7% of worker prompt tokens) but only discounts them, leading to marginal token savings. With our store attached to EvoMAS, only 6.1% of worker tokens are reused, compared with 34.5% for Inherit-MAS, because one EvoMAS update can change several components and few requests repeat. Therefore, our workflow and execution inheritance are distinct from existing approaches, and their combination uniquely brings performance and efficiency benefits.