- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CoMemBench, an execution-grounded benchmark for multi-agent workflows comprising 800 composite workflows across four domains, five dependency topologies, and matched information contamination challenges. It evaluates whether multi-agent systems can reliably share required information while isolating contextually inapplicable information across different workflow states, and investigates the potential trade-off between information sharing and isolation robustness.
Critical strengths
-
CoMemBench addresses an important evaluation gap between existing memory benchmarks and multi-agent collaboration benchmarks. Rather than focusing solely on whether required information is successfully transferred, it also examines whether contextually inapplicable information interferes with downstream execution. The paper formalizes this problem as task-conditioned collaborative memory boundaries and systematically evaluates it through five workflow topologies, explicit artifact handoffs, and matched clean–polluted challenges.
-
CoMemBench provides a comprehensive benchmark comprising 800 composite workflows across four domains and 800 matched isolation challenges. The use of domain-native execution environments, automated verifiers, and multi-stage quality checks strengthens the reliability of the evaluation.
Critical weaknesses
-
Limited validity of the core evaluation metrics. VHS conflates information handoff reliability with downstream task execution success, while ICS is computed only over pairs whose clean targets succeed, potentially leading to different evaluation subsets across systems. Moreover, successful task completion does not necessarily indicate that irrelevant information was properly isolated. These limitations raise concerns about whether the proposed metrics adequately capture collaborative memory boundary maintenance.
-
Insufficient evidence for topology-specific conclusions. The isolation challenges are heavily imbalanced across topologies (476/800 for Reuse versus only 17/800 for Bounded Revision), with strong dependencies between topology types and contamination operators. Without controlled comparisons accounting for contamination type and task difficulty, it remains unclear to what extent the observed performance differences can be attributed to workflow topology itself.
-
Extremely low end-to-end success rates limit the benchmark's effectiveness. In Table 3, all evaluated configurations achieve 0% workflow success on Stateful Tool and Formal Mathematics, while the highest success rates on Repository Code and Offline Retrieval are only 8% and 18%, respectively. Consequently, the evaluation relies heavily on partial node-level progress. Without a more detailed analysis of end-to-end failures, it remains unclear whether the benchmark can effectively distinguish systems in terms of complete multi-agent workflow execution.
Questions for authors
-
How do the authors distinguish memory handoff failures from downstream execution failures in VHS, and ensure that ICS scores are comparable across systems with different eligible subsets?
-
Can the authors provide controlled comparisons to disentangle the effects of workflow topology, contamination type, and task difficulty?
-
Can the authors provide a more detailed analysis of the low end-to-end success rates and explain how the benchmark distinguishes complete workflow execution capabilities across systems?
3 comments
2nd Response to Reviewer DNGC
2. What can the topology-stratified results establish?
Each challenge records a local target-topology witness identifying where the consumer boundary lies and an operator identifying why the artifact is inapplicable. Their joint distribution is:
Table D3. Target topology by pollution operator.
| Target topology | Sibling crosswire | Incompatible payload | Unverified result | Completed as pending | Wrong entity | Stale version | Total |
|---|---|---|---|---|---|---|---|
| Chain | 0 | 4 | 58 | 0 | 61 | 0 | 123 |
| Split–join | 93 | 0 | 0 | 6 | 0 | 0 | 99 |
| Reuse | 0 | 15 | 315 | 0 | 146 | 0 | 476 |
| Overlap | 0 | 11 | 0 | 0 | 74 | 0 | 85 |
| Bounded revision | 0 | 0 | 1 | 0 | 0 | 16 | 17 |
We select each operator according to the target consumer's actual dependencies and artifact state, rather than allocating equal numbers across topologies. A sibling crosswire requires distinguishable branches, so all 93 instances target split–join boundaries. A stale-version challenge instead offers an earlier tabular artifact after an upstream update has superseded it; all 16 such instances target bounded revisions. The remaining bounded-revision challenge involves an unverified result. A workflow may contain several topology motifs; this table records the local consumer boundary tested by each challenge.
Topology locates the challenged consumer boundary; the operator identifies why the artifact is inapplicable. Their joint distribution makes these operating conditions explicit: wrong-entity challenges span Chain, Reuse, and Overlap (61, 146, and 74 corpus instances), while unverified-result challenges occur mainly at Chain and Reuse (58 and 315). Table 5 therefore compares systems under the realized boundary conditions, and the operator breakdown explains part of each topology stratum's composition. These strata characterize observed behavior rather than an operator- and task-controlled causal effect of topology; ICS remains conditioned on each system's clean-successful targets.
3rn Response to Reviewer DNGC
3. Can the benchmark distinguish complete execution despite low Table 3 SR?
Table 3's SR requires a native-verifier-accepted terminal outcome for the complete workflow. Table D4 separately locates execution stages in the fixed-executor memory comparison; its percentages do not stand in for the framework-specific traces in Table 3:
Table D4. Node-execution stages in controlled memory runs.
| Domain | Required-node occurrences not executed | Executed-node occurrences verified |
|---|---|---|
| Stateful Tool | 6.1% | 88.2% |
| Repository Code | 23.4% | 90.1% |
| Offline Retrieval | 20.0% | 86.3% |
| Formal Mathematics | 51.8% | 25.5% |
In these controlled-memory runs, Mathematics loses progress both before execution and at verification. Stateful Tool attempts most required nodes and verifies most attempts, yet terminal success still requires every necessary operation and the final native acceptance condition. Thus node progress and workflow success reveal different stages; D4 describes the fixed-executor experiment, not every Table 3 system.
We also tested complete workflow execution with GPT-6.1 Sol at medium reasoning through Codex sessions on 50 supplied workflows per domain. It received the supplied workflow and domain-native tools, subject to node-level and workflow-level action limits and a serialized-observation limit. These were separate solver sessions, not a rerun of the Table 3 Codex baseline; no polluted variants or ICS were evaluated. Native evaluators produced the following outcomes:
Table D5. Bounded non-Gold workflow execution with GPT-6.1 Sol.
| Domain | SR | VNCR | VHS |
|---|---|---|---|
| Stateful Tool | 24.0% | 87.60% | 89.73% |
| Repository Code | 82.0% | 98.31% | 99.03% |
| Offline Retrieval | 30.0% | 50.44% | 42.97% |
| Formal Mathematics | 70.0% | 94.25% | 96.26% |
The bounded Sol runs achieve terminal success in all four domains, ranging from 24.0% in Stateful Tool to 82.0% in Repository Code. This establishes differentiated whole-workflow completion by a non-Gold solver under the stated setting. Gold replay separately passes domain-native checks for all 800 released workflows. SR distinguishes completed objectives; node and handoff results explain progress within workflows that do not reach that objective.
Manuscript revision. We will define VHS as downstream success under a valid required handoff and make the clean-success condition for ICS explicit. We will place eligible supports and uncertainty intervals alongside conditional and sparse-stratum results, distinguish the Table 4 node-stage analysis from Table 3 system results, and describe topology/operator strata as observed boundary conditions rather than isolated topology effects.
Response to Reviewer DNGC
We thank the reviewer for the questions about metric interpretation, topology-stratified results, and low terminal success. We address each with the corresponding execution evidence.
1. What do VHS and ICS measure, and how are failures distinguished?
Verified Handoff Success (VHS) measures whether a consumer completes its task with a valid required handoff: mandatory predecessors verify, their declared artifacts arrive, remain applicable, and are used, and the consumer passes its native verifier. It therefore reflects both handoff validity and local execution capability, rather than delivery alone. Verified Node Completion Rate (VNCR) and terminal Success Rate (SR) locate progress at the node and whole-workflow levels.
To examine local execution after a valid handoff, we restored trusted predecessor outputs and the required environment state, then let the worker execute target nodes that had previously failed or not run. These are target-node observations, not complete workflows; the same native target verifiers judged the new outputs:
Table D1. Target-node success after restoring trusted predecessors.
With trusted prerequisites, Stateful targets pass and 67.0% of Retrieval targets recover, while Code and Mathematics reach 25.0% and 1.6%. This domain contrast identifies local solving as one contributor to VHS even when the required inputs are valid. For Stateful Tool, restoring prerequisites also restores the application state needed by the target.
Isolation Challenge Success (ICS) asks whether a clean-successful consumer also passes its native verifier under the matched inapplicable artifact. This clean-success condition focuses each configuration's score on consumers it demonstrably reaches; Tables G3 and G4 report the corresponding eligible supports. Table D2 separately compares benign and polluted additions within two memory configurations:
Table D2. Benign-addition success and ICS on matched targets.
Full Context falls from 83.5% with a benign addition to 64.9% with polluted content, an 18.6-point difference; Local-only scores 91.4% in both conditions. D2 tests content sensitivity within these mechanisms, while the main system comparison applies the same clean-success eligibility and native-verifier criterion to each configuration's reached targets. It is not a common-target re-scoring of Table 3.