acceptodds
Under review as a conference paper at ICLR 2027

CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies

Abstract

Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated. We call this task-conditioned scope of information a collaborative memory boundary. Workflow topology determines which intermediate artifacts are applicable to which downstream workers and when they cease to be valid, thereby providing a structural stress dimension for sharing and isolation. Existing memory benchmarks primarily evaluate retention and retrieval, whereas multi-agent benchmarks emphasize coordination and end-to-end completion, leaving topology-conditioned memory boundaries largely unmeasured. We introduce COMEMBENCH, an executiongrounded benchmark for collaborative memory sharing and isolation across multiagent workflow topologies. It constructs 800 composite workflows across four domains from source-grounded dependency graphs, with node-local specifications, verifiable artifact handoffs, native evaluators, and matched isolation challenges.COMEMBENCH measures workflow completion, verified node progress, requiredhandoff reliability, isolation robustness, and token cost. Experiments reveal a sharing–isolation trade-off: broader context improves information availability but can weaken isolation, while system rankings shift across topologies and artifact violations. COMEMBENCH is available at https://huggingface.co/datasets/anonymoussubmission-333/anonymous-submission.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Agent Memory, Collaborative Memory Boundaries, Workflow Topology, Memory Sharing, Memory Isolation, Multi-Agent Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_DNGCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoMemBench, an execution-grounded benchmark for multi-agent workflows comprising 800 composite workflows across four domains, five dependency topologies, and matched information contamination challenges. It evaluates whether multi-agent systems can reliably share required information while isolating contextually inapplicable information across different workflow states, and investigates the potential trade-off between information sharing and isolation robustness.

Critical strengths
  1. CoMemBench addresses an important evaluation gap between existing memory benchmarks and multi-agent collaboration benchmarks. Rather than focusing solely on whether required information is successfully transferred, it also examines whether contextually inapplicable information interferes with downstream execution. The paper formalizes this problem as task-conditioned collaborative memory boundaries and systematically evaluates it through five workflow topologies, explicit artifact handoffs, and matched clean–polluted challenges.

  2. CoMemBench provides a comprehensive benchmark comprising 800 composite workflows across four domains and 800 matched isolation challenges. The use of domain-native execution environments, automated verifiers, and multi-stage quality checks strengthens the reliability of the evaluation.

Critical weaknesses
  1. Limited validity of the core evaluation metrics. VHS conflates information handoff reliability with downstream task execution success, while ICS is computed only over pairs whose clean targets succeed, potentially leading to different evaluation subsets across systems. Moreover, successful task completion does not necessarily indicate that irrelevant information was properly isolated. These limitations raise concerns about whether the proposed metrics adequately capture collaborative memory boundary maintenance.

  2. Insufficient evidence for topology-specific conclusions. The isolation challenges are heavily imbalanced across topologies (476/800 for Reuse versus only 17/800 for Bounded Revision), with strong dependencies between topology types and contamination operators. Without controlled comparisons accounting for contamination type and task difficulty, it remains unclear to what extent the observed performance differences can be attributed to workflow topology itself.

  3. Extremely low end-to-end success rates limit the benchmark's effectiveness. In Table 3, all evaluated configurations achieve 0% workflow success on Stateful Tool and Formal Mathematics, while the highest success rates on Repository Code and Offline Retrieval are only 8% and 18%, respectively. Consequently, the evaluation relies heavily on partial node-level progress. Without a more detailed analysis of end-to-end failures, it remains unclear whether the benchmark can effectively distinguish systems in terms of complete multi-agent workflow execution.

Questions for authors
  1. How do the authors distinguish memory handoff failures from downstream execution failures in VHS, and ensure that ICS scores are comparable across systems with different eligible subsets?

  2. Can the authors provide controlled comparisons to disentangle the effects of workflow topology, contamination type, and task difficulty?

  3. Can the authors provide a more detailed analysis of the low end-to-end success rates and explain how the benchmark distinguishes complete workflow execution capabilities across systems?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DNGC

We thank the reviewer for the questions about metric interpretation, topology-stratified results, and low terminal success. We address each with the corresponding execution evidence.

1. What do VHS and ICS measure, and how are failures distinguished?

Verified Handoff Success (VHS) measures whether a consumer completes its task with a valid required handoff: mandatory predecessors verify, their declared artifacts arrive, remain applicable, and are used, and the consumer passes its native verifier. It therefore reflects both handoff validity and local execution capability, rather than delivery alone. Verified Node Completion Rate (VNCR) and terminal Success Rate (SR) locate progress at the node and whole-workflow levels.

To examine local execution after a valid handoff, we restored trusted predecessor outputs and the required environment state, then let the worker execute target nodes that had previously failed or not run. These are target-node observations, not complete workflows; the same native target verifiers judged the new outputs:

Table D1. Target-node success after restoring trusted predecessors.

DomainPass rate after restoring trusted predecessors
Stateful Tool100.0%
Repository Code25.0%
Offline Retrieval67.0%
Formal Mathematics1.6%

With trusted prerequisites, Stateful targets pass and 67.0% of Retrieval targets recover, while Code and Mathematics reach 25.0% and 1.6%. This domain contrast identifies local solving as one contributor to VHS even when the required inputs are valid. For Stateful Tool, restoring prerequisites also restores the application state needed by the target.

Isolation Challenge Success (ICS) asks whether a clean-successful consumer also passes its native verifier under the matched inapplicable artifact. This clean-success condition focuses each configuration's score on consumers it demonstrably reaches; Tables G3 and G4 report the corresponding eligible supports. Table D2 separately compares benign and polluted additions within two memory configurations:

Table D2. Benign-addition success and ICS on matched targets.

Memory mechanismMatched target pairsBenign-addition target successICS on these matched targets
Local-only28091.4%91.4%
Full Context38883.5%64.9%

Full Context falls from 83.5% with a benign addition to 64.9% with polluted content, an 18.6-point difference; Local-only scores 91.4% in both conditions. D2 tests content sensitivity within these mechanisms, while the main system comparison applies the same clean-success eligibility and native-verifier criterion to each configuration's reached targets. It is not a common-target re-scoring of Table 3.

AuthorsOpenReview ↗

2nd Response to Reviewer DNGC

2. What can the topology-stratified results establish?

Each challenge records a local target-topology witness identifying where the consumer boundary lies and an operator identifying why the artifact is inapplicable. Their joint distribution is:

Table D3. Target topology by pollution operator.

Target topologySibling crosswireIncompatible payloadUnverified resultCompleted as pendingWrong entityStale versionTotal
Chain04580610123
Split–join930060099
Reuse01531501460476
Overlap0110074085
Bounded revision001001617

We select each operator according to the target consumer's actual dependencies and artifact state, rather than allocating equal numbers across topologies. A sibling crosswire requires distinguishable branches, so all 93 instances target split–join boundaries. A stale-version challenge instead offers an earlier tabular artifact after an upstream update has superseded it; all 16 such instances target bounded revisions. The remaining bounded-revision challenge involves an unverified result. A workflow may contain several topology motifs; this table records the local consumer boundary tested by each challenge.

Topology locates the challenged consumer boundary; the operator identifies why the artifact is inapplicable. Their joint distribution makes these operating conditions explicit: wrong-entity challenges span Chain, Reuse, and Overlap (61, 146, and 74 corpus instances), while unverified-result challenges occur mainly at Chain and Reuse (58 and 315). Table 5 therefore compares systems under the realized boundary conditions, and the operator breakdown explains part of each topology stratum's composition. These strata characterize observed behavior rather than an operator- and task-controlled causal effect of topology; ICS remains conditioned on each system's clean-successful targets.

AuthorsOpenReview ↗

3rn Response to Reviewer DNGC

3. Can the benchmark distinguish complete execution despite low Table 3 SR?

Table 3's SR requires a native-verifier-accepted terminal outcome for the complete workflow. Table D4 separately locates execution stages in the fixed-executor memory comparison; its percentages do not stand in for the framework-specific traces in Table 3:

Table D4. Node-execution stages in controlled memory runs.

DomainRequired-node occurrences not executedExecuted-node occurrences verified
Stateful Tool6.1%88.2%
Repository Code23.4%90.1%
Offline Retrieval20.0%86.3%
Formal Mathematics51.8%25.5%

In these controlled-memory runs, Mathematics loses progress both before execution and at verification. Stateful Tool attempts most required nodes and verifies most attempts, yet terminal success still requires every necessary operation and the final native acceptance condition. Thus node progress and workflow success reveal different stages; D4 describes the fixed-executor experiment, not every Table 3 system.

We also tested complete workflow execution with GPT-6.1 Sol at medium reasoning through Codex sessions on 50 supplied workflows per domain. It received the supplied workflow and domain-native tools, subject to node-level and workflow-level action limits and a serialized-observation limit. These were separate solver sessions, not a rerun of the Table 3 Codex baseline; no polluted variants or ICS were evaluated. Native evaluators produced the following outcomes:

Table D5. Bounded non-Gold workflow execution with GPT-6.1 Sol.

DomainSRVNCRVHS
Stateful Tool24.0%87.60%89.73%
Repository Code82.0%98.31%99.03%
Offline Retrieval30.0%50.44%42.97%
Formal Mathematics70.0%94.25%96.26%

The bounded Sol runs achieve terminal success in all four domains, ranging from 24.0% in Stateful Tool to 82.0% in Repository Code. This establishes differentiated whole-workflow completion by a non-Gold solver under the stated setting. Gold replay separately passes domain-native checks for all 800 released workflows. SR distinguishes completed objectives; node and handoff results explain progress within workflows that do not reach that objective.

Manuscript revision. We will define VHS as downstream success under a valid required handoff and make the clean-success condition for ICS explicit. We will place eligible supports and uncertainty intervals alongside conditional and sparse-stratum results, distinguish the Table 4 node-stage analysis from Table 3 system results, and describe topology/operator strata as observed boundary conditions rather than isolated topology effects.

Reviewer_fabuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CoMemBench, a benchmark for sharing and isolation in multi-agent workflows. A required artifact should reach its consumer. An inapplicable artifact should not change the consumer's behavior.

Contributions:

  1. 800 workflows from four executable sources, with typed handoff contracts.
  2. Matched clean and polluted pairs that differ in one artifact.
  3. Metrics for sharing (VHS) and isolation (ICS), tested on ten systems.
Critical strengths
  1. A clear and testable problem. End-to-end success cannot separate a missing handoff from a bad record. Matched pairs separate the two.

  2. Grounded evaluation. All domains use native evaluators, such as unit tests and Lean compilation, not LLM grading. Node-level metrics still give signal when workflow success is zero.

Critical weaknesses
  1. The main findings rest on unreported sample sizes. Some strata have only 6 or 16 cases in the whole corpus. They are still ranked without confidence intervals.

  2. It is unclear what ICS measures. The polluted artifact seems to go straight into the target's input, so no memory policy can block it. A clean rerun and a harmless extra artifact are needed as controls.

Questions for authors
  1. How many workflows were run per domain?
  2. How often does a clean target fail on a plain rerun?
  3. Were the requests written by people, from templates, or by an LLM?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fabu

We thank the reviewer for focusing on evaluation support, the location of the isolation intervention, and the controls needed to interpret target failures. We address each point with the corresponding execution evidence.

1. How many workflows were evaluated, and how should rare strata be read?

Tables 3 and 4 evaluate 200 workflows in each of four domains, 800 in total. Gold replay independently passes the domain-native execution checks for all 800 workflows. Each workflow has one matched isolation challenge. ICS includes a pair only when its clean target succeeds, so eligible support can be smaller than the evaluated workflow count and varies by configuration.

The 800 matched challenges have the following domain and target-topology composition:

Table F1. Challenge operators by domain and target topology.

OperatorCountDomains (count)Target topologies (count)
Sibling crosswire93Stateful 15; Code 41; Math 37Split–join 93
Incompatible payload30Stateful 14; Math 16Chain 4; Reuse 15; Overlap 11
Unverified result374Stateful 70; Code 125; Retrieval 97; Math 82Chain 58; Reuse 315; Bounded revision 1
Completed as pending6Stateful 6Split–join 6
Wrong entity281Stateful 79; Code 34; Retrieval 103; Math 65Chain 61; Reuse 146; Overlap 74
Stale version16Stateful 16Bounded revision 16

These challenge types follow conditions that actually occur in the workflows rather than equal quotas. Completed-as-pending requires a completed side effect, so its six instances target Stateful split–join boundaries; stale-version requires a real supersession, so its sixteen instances target Stateful bounded revisions. These counts establish coverage of the corresponding boundary conditions, not a basis for stable system rankings within the rare categories. ICS separately conditions on each configuration's clean-successful matched targets.

2. Where does the artifact enter, and what does ICS measure?

Clean and polluted variants keep the query, dependency graph, initial state, tools, and other inputs fixed. The polluted variant adds one inapplicable artifact at a selected consumer's incoming-artifact boundary, without revealing the challenge label. The two comparisons use this boundary differently. In Table 3, the harness presents the candidate alongside legitimate predecessor inputs before invoking the worker: this tests consumer robustness to inapplicable task input, not the system's memory-filtering policy. In Table 4, the candidate enters the memory policy's candidate pool, while required predecessor handoffs remain available; the policy determines whether the additional candidate appears in the worker's rendered context. We will clarify in Section 3.4 that the benchmark-side incoming-artifact boundary is not necessarily the context ultimately seen by the worker. The same domain-native verifier checks the target output in both variants. ICS reports polluted-target success among matched targets that succeed when clean; it is a conditional execution outcome, not by itself a causal estimate of harm from delivered content.

We checked actual exposure in the target's rendered context and retrieval records for distinguishable wrong-entity and stale-version candidates. The table reports both delivery and detectable appearance of candidate content in the target output; each percentage uses all scored trials in that row.

Table F2. Candidate exposure and output-content match (%).

Memory mechanismTrialsDeliveredDelivered and output-matched
Local-only1630.00.0
Full Context22494.621.4
Shared Flat RAG21276.923.1
Mem02070.00.0
A-Mem20886.517.3
PlugMem1750.00.0

F2 detects delivery at different rates for Full Context, Shared Flat RAG, and A-Mem; no delivery was detected in the tested Local-only, Mem0, or PlugMem traces. This probe covers distinguishable wrong-entity and stale-version candidates, not all six operators. Output matching records detectable candidate-content propagation separately from the native-verifier success criterion used for ICS.

AuthorsOpenReview ↗

2nd Response to Reviewer fabu

3. How often does an originally successful target fail on rerun?

We reran 210 originally successful targets in their recorded contexts and checked each result with the native target verifier. The same targets were also tested with a benign added artifact and with the original pollution candidate:

Table F3. Native verification of the same 210 targets under fixed-context conditions.

ConditionVerified targetsRate (%)
Original successful context210/210100.0
Clean rerun198/21094.3
Benign artifact198/21094.3
Original pollution candidate202/21096.2

Plain reruns fail on 12/210 originally successful targets (5.7%), and benign additions yield the same aggregate rate. The original candidate verifies on 202/210 targets, so it does not lower aggregate success in this fixed-context cohort. F3 quantifies target-level repeatability in recorded contexts; Table 3 separately measures system-level execution under direct-input challenges.

We separately compared benign and polluted content on matched clean-successful targets within each memory mechanism, using the same native target verifier in both arms:

Table F4. Matched benign–polluted target verification by memory mechanism.

MechanismMatched pairsBenign (%)Polluted ICS (%)Difference (pp)
Local-only28091.491.40.0
Full Context38883.564.918.6

Within Full Context's matched targets, native-verifier success falls from 83.5% with benign content to 64.9% with polluted content; Local-only remains at 91.4% in both arms. F4 thus identifies content sensitivity in the tested Full Context path, whereas F3 measures repeatability in a separate fixed-context cohort. Equal Local-only rates do not establish resistance after confirmed exposure.

4. How were the requests written?

The requests are constructed, rather than transcribed from user logs or generated as free-standing LLM scenarios. We first pin the source task and its executable evidence. For Toolathlon-GYM, the units are source-authored operations on named resources, linked by state changes and evaluator predicates; for SWE-bench Verified, they are issue reproduction, localization, patching, and regression checks tied to a repository commit and tests; for BrowseComp+, they are entity and evidence constraints over a fixed corpus; and for formal mathematics, they are Lean declarations linked by actually used constants. We extract a connected, dependency-closed subgraph with one terminal objective before writing its request. Each retained edge has a source witness, such as a consumed resource, issue–test relation, required evidence, or Lean constant.

A structured draft then carries the selected objective, entities, resources, deliverables, constraints, and acceptance conditions into a natural-language query. Researchers revise that draft to make the request complete without revealing an execution plan; LLM assistance is used for wording, not to invent task dependencies or acceptance criteria. Node instructions are checked against the authorized local view and declared incoming artifacts, while the public query is checked for leaked Gold answers, patches, relevance labels, or hidden challenge fields. All 800 workflows underwent human review against four concrete questions: Is the request natural and complete? Can each node run from its local information? Is each dependency necessary for its consumer? Does native evaluator success entail the requested outcome? Rejected cases are repaired and recertified or excluded; Gold execution provides the separate end-to-end executability check.

Manuscript revision. We will state evaluated workflow counts and ICS eligibility next to the results, provide eligible supports and uncertainty intervals for sparse operator and topology strata, and avoid ranking systems from those rare cells. We will also distinguish direct-input from memory-mediated challenges and present the plain-rerun and benign-addition controls beside the isolation interpretation; the construction section will make human review and the source-grounded request-writing process explicit.

Reviewer_V4naOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

CoMemBench is an execution-grounded benchmark for what the authors call collaborative memory boundaries: a downstream agent must receive the artifacts it requires (sharing) while remaining unaffected by plausible but inapplicable artifacts (isolation), where applicability depends on task dependencies, verification state, and version.

Critical strengths
  • Framing memory as a two-sided, task- and state-conditioned boundary — rather than retention/retrieval (LoCoMo, LongMemEval) or principal/privacy scoping (GateMem, PAC-Bench) — is a real conceptual contribution,
  • Official SWE-bench harness, Lean compilation, exact-span retrieval verification, MCP state digests, pinned revisions, leakage audits, and separation of infrastructure from method failures. This is far more rigorous infrastructure than most MAS benchmarks
  • The VNCR–VHS gap and the SR–VNCR gap are informative; the controlled memory comparison (Table 4) with a fixed executor is the cleanest experiment in the paper.
Critical weaknesses
  • ICS is reported without denominators, and many cells are almost certainly single-digit.
  • No system solves any Stateful Tool or Formal Mathematics workflow; best SR is 8% (code) and 18% (retrieval).
  • A polluted target can fail for LLM stochasticity unrelated to the injected artifact.
Questions for authors

What is clean–clean replicate agreement on target success, per domain? Where is the polluted artifact injected for (a) framework baselines and (b) external memory stores? What SR does a single strong agent achieve on the undecomposed source tasks under your harness, and what is gold-plan replay success per domain? Why does GPT-6-Luna trail Qwen3.8-27B so broadly, and what adapter constraints apply to Codex and Hermes?

3 comments
AuthorsOpenReview ↗

Response to Reviewer V4na

We thank the reviewer for recognizing the execution-grounded design and for asking how to interpret isolation scores, task solvability, and runtime differences. We respond by separating ordinary rerun variation from content sensitivity, tracing candidate exposure, and distinguishing reference replay from non-Gold solving.

1. Can target failures be attributed to the challenge rather than ordinary rerun variation?

We re-executed previously successful targets under their recorded context, then added either a benign artifact or a pollution candidate and applied the same native target verifier. The first five conditions use the same 210 targets; both distinguishable-mutation conditions were evaluated on 129 targets: Table V1. Fixed-context target verification.

ConditionVerified targetsRate (%)
Original successful context210/210100.0
Clean rerun198/21094.3
Benign artifact198/21094.3
Original pollution candidate202/21096.2
Candidate plus status206/21098.1
Mutated candidate (129 targets)125/12996.9
Mutation plus status (same 129 targets)129/129100.0

A plain rerun and a benign addition each succeed on 198/210 targets (94.3%); the original pollution candidate succeeds on 202/210 (96.2%). Thus the 12 plain-rerun failures quantify repeat-execution variation, while the original candidate does not lower aggregate success in this fixed-context cohort. The status-labeled and distinguishable-mutation conditions probe candidate representation; the mutation conditions were tested on 129 targets and do not enter the plain-rerun failure rate calculated from the 210 targets. Table 3 separately tests system-level execution under direct-input challenges.

Within each memory mechanism, we also compared benign and polluted content on matched clean-successful targets using the native target verifier. Full Context falls from 83.5% to 64.9% across 388 pairs, an 18.6-point difference; Local-only remains at 91.4% in both arms across 280 pairs. This paired comparison identifies content sensitivity in the tested Full Context path, while the fixed-context rerun above measures repeatability in a separate cohort.

CS denominators in the system comparison

For each system and backbone, the ICS denominator is the number of matched target observations whose clean target succeeded; the numerator counts those whose polluted target also succeeded. Table 3 pools across four domains with 200 evaluated workflows each, so eligible support for a system/backbone configuration is bounded by 800. This support counts clean-successful challenge targets, not repeated executions of a pair. Table V2 presents the scores and supports for the systems shown, with backbones in columns.

Table V2. Manuscript Table 3 ICS and eligible matched-target supports.

Each cell shows successful targets/eligible pairs (ICS, %).

SystemQwen3.8-27BDeepSeek-V4-FlashGPT-6-Luna
Hermes0/78 (0.0)0/78 (0.0)0/114 (0.0)
Deep Agents64/168 (38.1)0/32 (0.0)0/8 (0.0)
CrewAI32/248 (12.9)8/132 (6.1)32/240 (13.3)
AutoGen46/230 (20.0)20/128 (15.6)24/204 (11.8)
Peer Review36/188 (19.1)24/140 (17.1)0/46 (0.0)
SMoA28/276 (10.1)29/195 (14.9)33/219 (15.1)
OMAC36/208 (17.3)16/362 (4.4)8/180 (4.4)
ADAS8/108 (7.4)28/196 (14.3)12/168 (7.1)
EvoMAS60/224 (26.8)16/164 (9.8)20/284 (7.0)
AuthorsOpenReview ↗

2nd Response to Reviewer V4na

2. Where is the candidate introduced, and can memory affect its exposure?

Each challenge selects one target consumer. Its clean and polluted variants share the query, dependency graph, initial environment, tools, and all other inputs; the polluted variant adds one inapplicable artifact at that consumer's incoming-artifact boundary. (a) In the framework comparison (Table 3), the candidate is presented directly in the target worker's task input. (b) In the controlled memory comparison (Table 4), it enters the extra-memory candidate path, and the mechanism determines whether the worker sees it. Valid required predecessor handoffs remain available in both settings. The same domain-native verifier checks the consumer's output in both variants.

For wrong-entity and stale-version challenges, we searched each target's rendered context and retrieval records for the candidate ID or distinguishable content, then checked whether that content appeared in the target output. Both percentages below use all scored trials of the corresponding mechanism as their denominator.

Table V3. Candidate exposure and detectable output match (%).

MechanismDeliveredDelivered and output-matched
Local-only0.00.0
Full Context94.621.4
Shared Flat RAG76.923.1
Mem00.00.0
A-Mem86.517.3
PlugMem0.00.0
Provenance filter77.014.8

V3 detects delivery at different rates for Full Context, Shared Flat RAG, and A-Mem; no delivery was detected in the tested Local-only, Mem0, or PlugMem traces. This probe covers distinguishable wrong-entity and stale-version candidates, not all six operators. Output matching records detectable propagation of candidate content, separately from the native-verifier success criterion used for ICS.

3. Can a strong agent solve undecomposed tasks?

We now directly test the reviewer's undecomposed-task setting. GPT-6.1 Sol at medium reasoning executes 50 tasks per domain from the task request without a supplied graph, under the same action and observation limits and native terminal evaluation used for the supplied-workflow Sol runs. Neither condition receives private Gold answers or runs polluted variants. Separately, Gold replay passes 200/200 workflows per domain and all 800 underwent human review. The results are:

Table V4. Gold replay and bounded non-Gold workflow solving.

DomainGold replaySol, no graphSol, supplied workflow
Stateful Tool200/2006/50 (12%)12/50 (24%)
Repository Code200/20034/50 (68%)41/50 (82%)
Offline Retrieval200/20012/50 (24%)15/50 (30%)
Formal Mathematics200/20031/50 (62%)35/50 (70%)

Without the graph, Sol completes 83/200 tasks (41.5%), including 34/50 Code and 31/50 Mathematics tasks; with the supplied workflow, it completes 103/200 (51.5%). All four domain rates are higher with the supplied workflow under this Sol setting. The no-graph condition directly answers how often a strong single agent completes the task without the benchmark decomposition, while Gold replay separately certifies the full corpus.

AuthorsOpenReview ↗

3rn Response to Reviewer V4na

4. How should we interpret GPT-6-Luna, Codex, and Hermes?

Luna's gap is not a uniform backbone decline. Across ten systems, the median Luna–Qwen VNCR difference is -11.6 points in Code and -10.6 in Retrieval, but only -0.8 in Stateful and -1.8 in Mathematics. Within Code, Peer Review falls from 49.7% to 3.1%, while ADAS rises from 17.0% to 26.7% and SMoA from 44.5% to 49.2%. We therefore interpret the reported outcomes by backbone–runtime–domain configuration, not as a single model-capability ordering or an unverified serving-failure diagnosis.

Codex and Hermes are evaluated as runtimes, not replaced by single model calls. Codex CLI 0.152.1 executes through codex exec --json with native event capture; Hermes 0.21.0 uses AIAgent.chat and delegate_task with sandbox tools. As with other baselines, each receives a node-local objective, authorized actions, declared incoming artifacts, and a native acceptance condition; the predefined dependency workflow is not replaced by an independently generated decomposition. These are the evaluated runtime–backbone configurations in Table 3. The separate Sol diagnostic uses Codex solver sessions with its own explicitly stated action and observation limits and is not a rerun of the Table 3 Codex baseline.

Together, the matched controls distinguish repeatability from content sensitivity, and the non-Gold executions establish substantial task-solving headroom. We use the system and runtime comparisons to characterize execution under the benchmark's common task interface, while keeping target-level verification and actual candidate exposure analytically distinct.

Manuscript revision. We will separate the two intervention paths and the fixed-context versus matched-content controls in the protocol and results, state ICS eligibility and the supporting counts and uncertainty intervals with the scores, and present Gold replay, supplied-workflow Sol execution, and the completed no-graph Sol comparison as distinct evaluations. We will also make the Codex and Hermes runtime constraints easier to locate.

Reviewer_GqmuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoMemBench evaluates whether multi-agent workflows both deliver required upstream information and resist irrelevant, stale, unverified, or otherwise inapplicable artifacts. It constructs 800 workflow families across stateful tool use, repository code change, offline retrieval, and formal mathematics, with one matched isolation challenge per clean workflow. Source-grounded dependency graphs specify typed, scoped, version-aware handoffs, while native execution evaluators verify node and terminal outcomes. Five topology motifs and six pollution operators characterize the tested boundaries. The evaluation distinguishes workflow success, verified node completion, verified handoff success, conditional isolation success, and token cost. Results compare ten systems across three backbones and six memory mechanisms under a fixed executor. The findings suggest that additional shared context can improve task progress while reducing isolation robustness, and that observed rankings vary across domains and boundary conditions.

Critical strengths
  1. The benchmark operationalizes both sides of collaborative information use. Required handoffs test whether appropriate evidence reaches a consumer, while a matched intervention introduces exactly one inapplicable artifact and checks downstream behavior. Holding the query, graph, environment, tools, and other inputs fixed within a pair gives a much more targeted test than terminal task success alone. This distinction is consequential because a system can fail by withholding necessary information or by exposing information that should not affect the consumer.

  2. The tasks have substantive execution and certification support across four domains. The benchmark combines application-state evaluators, repository tests, document and exact-span checks, and Lean compilation. Its construction protocol includes source locks, dependency witnesses, replay of known-good traces, rejection of targeted negative mutations, and human checks of local executability and evaluator alignment. Family-level and source-group split controls further strengthen the dataset design. These features make the benchmark more useful for reproducible agent evaluation than a collection judged only from final text.

  3. The controlled memory comparison helps separate memory policy from the surrounding multi-agent executor. Worker prompts, tools, handoffs, budgets, and evaluation remain fixed while the experiment varies additional cross-worker exposure and memory mechanisms. For example, full context raises verified node completion in stateful tasks from 74.6% to 86.9%, while the reported conditional isolation score falls from 97.1% to 72.4%. The results also include token accounting and show that no tested memory mechanism dominates all outcomes. This provides a useful empirical starting point for studying the trade-off between availability and isolation, subject to the comparability concern below.

Critical weaknesses
  1. The evaluation does not establish how much of the challenge requires memory selection beyond explicit artifact-contract validation. Consumers receive handoff contracts, and artifact records include type, entity scope, version, verification state, and provenance. Several pollution operators directly violate these fields or requirements. The evaluated baselines do not include a simple contract-aware filtering policy, leaving unclear how much improvement is available through deterministic validation before involving an agent's memory decisions. Such a control would help distinguish challenges solved by checking explicit validity metadata from those requiring semantic judgments about contextual applicability. This matters for understanding what capability the benchmark actually measures.

  2. Extremely low clean completion limits the diagnosis of memory-specific failure. None of the tested system/backbone configurations completes a stateful-tool or formal-mathematics workflow, and even the controlled memory comparison has no successful full mathematics workflow. Verified node and handoff metrics are useful responses to this difficulty, but they do not fully establish whether failures are driven by information sharing, local task competence, or the demands of the imposed workflow decomposition. Certification with a known-good execution establishes solvability, rather than showing that evaluated workers can reliably execute each node when supplied with all required correct artifacts. An independent worker-level control supplying trusted handoffs to every required node, including nodes never reached in ordinary runs, would help determine whether the benchmark's difficulty is concentrated in the collaborative memory boundaries it aims to test.

  3. The influence of the benchmark-imposed workflow decomposition is not sufficiently separated from system capability. Tasks are realized through predefined node-local specifications and declared handoff contracts, while the evaluated systems differ substantially in how they normally delegate, schedule, and organize intermediate work. Although each system makes its own execution decisions, the paper does not establish whether the benchmark's decomposition and adapter interface affect some architectures more than others. Consequently, system rankings may partly reflect compatibility with the prescribed task representation rather than collaborative memory capability. A comparison with each system's native execution of the same underlying task, or an analysis using alternative valid decompositions, would help establish the scope of these comparisons.

Questions for authors
  1. What are the evaluated workflow counts and eligible clean-target counts for each system, domain, topology, and pollution type?
  2. For each pollution operator, can invalidity be determined from the consumer-visible contract and artifact metadata alone, or does it require interpreting the payload?
  3. Were the same evaluated workers tested independently on all required nodes with trusted, correct predecessor artifacts, including nodes not reached during ordinary runs? What do the existing traces show about the contribution of local task failures versus incorrect handoffs or inappropriate artifact use, especially in domains with no completed workflows?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Gqmu

We thank the reviewer for recognizing the benchmark's two-sided evaluation of required sharing and isolation, and for asking which capabilities the reported scores reflect. We address the four questions with an explicit rule-based control, trusted-predecessor execution, a comparison under common workflow requirements, and the corresponding score supports.

1. Which challenges can explicit rules detect?

We agree that the role of explicit rules should be tested directly. We applied two deterministic admission policies before a candidate reaches its consumer. Provenance parses the producer from the artifact ID and checks whether it is a mandatory direct predecessor. Contract additionally checks the record's declared semantic type, verification state, and version; neither policy receives the hidden pollution label. Table G1 reports candidate rejection rates by policy and operator.

Table G1. Deterministic candidate rejection by operator.

OperatorProvenanceContractMetadata alone?
Sibling crosswire100.0%100.0%Yes: graph + producer
Wrong entity38.7%37.9%If entity binding is trusted
Stale version8.0%8.3%If supersession is known
Incompatible payload5.9%0.0%No: validate payload
Completed as pending0.0%0.0%Partial: state/receipt needed
Unverified resultPublication gatePublication gateYes: trusted receipt
Overall29.7%26.8%—

Provenance and Contract were tested on 381 and 369 candidate records, respectively; their overall rates are not a paired comparison.

Both policies reject every tested sibling crosswire; lower rejection for the other operators reflects the narrower fields they actually check. In particular, the tested version predicate compares artifact-ID suffixes rather than a complete supersession record, and declared producer type alone does not validate an individual payload.

The metadata question has a conditional answer: sibling routing is decidable from trusted graph and producer metadata; verification from a trusted receipt; wrong entity when a trusted entity binding is available; and stale version when the current supersession relation is available. Incompatible payload requires validating its actual contents against the consumer's protocol, while completed-as-pending requires current operation state or a completion receipt. The tested Contract policy does not check entity scope, full supersession, payload conformance, or completed-operation state. Table G1 therefore measures these implemented admission rules, not an upper bound on deterministic validation. Its two overall rates are not a paired comparison. In this rule-control path, unverified-result candidates are handled at the publication gate rather than counted as downstream-filter rejections; this does not specify how the separate Table 4 memory path renders its challenge candidate. CoMemBench therefore tests whether a system maintains the boundary appropriate to the task's current state, not merely whether an artifact has an allowed producer. Candidate rejection and downstream ICS are distinct measurements: Tables 3 and 4 report ICS, whereas Table G1 does not estimate the ICS of a rule-filtered executor.

AuthorsOpenReview ↗

2nd Response to Reviewer Gqmu

2. How much difficulty is local to a node?

Gold replay confirms executable solutions, while Table G2 examines local task execution under trusted prerequisites. We stratify targets by their original status. For targets originally unexecuted or failed, we restore trusted predecessor outputs and necessary environment state, run the tested worker under its node budget, and apply the domain-native verifier. The previously verified column provides a reference category; the recovery analysis below uses the first two groups.

The table reports target-node acceptance under these trusted prerequisites: Table G2. Trusted-input success by original target status.

DomainUnexecutedExecuted, failedPreviously verified
Stateful Tool99/99 (100%)16/16 (100%)491/491 (100%)
Repository Code0/4 (0%)4/12 (33.3%)71/71 (100%)
Offline Retrieval198/301 (65.8%)76/108 (70.4%)119/119 (100%)
Formal Mathematics4/199 (2.0%)0/47 (0%)16/16 (100%)

The recovery of originally unsuccessful targets reveals different task demands. In Stateful Tool, 115/115 targets pass once the required artifacts and application state are established, indicating that the tested local operations can be completed under those conditions; full workflow execution must still maintain the state and handoffs across nodes. In Retrieval, 274/409 (67.0%) pass, showing both the value of available upstream evidence and residual evidence-selection work at the consumer. In Code, 4/16 (25.0%) pass: a correct predecessor does not itself localize, repair, and validate a repository issue. In Formal Mathematics, 4/246 (1.6%) pass despite Gold predecessor proofs, making local proof construction a substantial part of the difficulty for this worker.

We also asked a stronger solver to execute complete supplied workflows without Gold answers. Across 50 workflows per domain, GPT-6.1 Sol with medium reasoning used separate Codex solver sessions on supplied workflows, with bounded action and observation budgets; this was not the Table 3 Codex baseline and did not include polluted runs. It completes 24% of Stateful, 82% of Code, 30% of Retrieval, and 70% of Mathematics workflows. Code and Mathematics are particularly informative: the Qwen trusted-input control finds their local targets difficult, yet the stronger solver completes many entire workflows. These node-level and whole-workflow results show that CoMemBench tests information-boundary maintenance alongside domain-native solving, whose difficulty differs markedly by domain.

3. Why prescribe a workflow for every system?

Our evaluation target is execution on a specified task-dependency workflow: delivering required information to its consumer and handling inapplicable artifacts at that boundary. We hold the query, node objectives, dependencies, authorized resources, acceptance conditions, and intervention target fixed so systems face the same task and information-boundary requirements.

This common task specification does not replace each system's collaboration mechanism. Deep Agents retains its supervisor–subagent loop; CrewAI its Agent/Task/Crew orchestration; AutoGen its group chat; Peer Review its creation, review, and revision; SMoA its role layers; OMAC its multi-round collaboration; and ADAS and EvoMAS their selected, frozen organizations. Each system performs the node work through its evaluated runtime, while the native evaluator applies the same task acceptance criteria. The fixed dependency and handoff requirements make Table 3 a comparison of execution on the same specified workflow; Table 4 additionally fixes the execution organization to compare memory mechanisms.

The experiments then address two complementary levels. Table 3 presents the inapplicable candidate directly in the consumer's task input, testing system-level execution robustness rather than isolating an internal memory-filtering policy. Table 4 holds the supervisor–worker executor, workers, prompts, graph, tools, budgets, and retry policy fixed and varies memory exposure. Under that control, Full Context raises Stateful VNCR from 74.6% to 86.9% while ICS falls from 97.1% to 72.4%; in Retrieval, ICS changes from 60.9% to 80.6%, an observed difference in conditional target success that alone does not establish stronger isolation. A separate matched benign–polluted comparison shows an 18.6-point decrease under Full Context, identifying content sensitivity on its tested targets.

A separate Sol/medium input comparison in Table V4 tests this task representation directly: without a supplied graph Sol completes 83/200 tasks (41.5%), versus 103/200 (51.5%) with the workflow provided. This probes the effect of the supplied task structure for one strong solver without changing the interpretation of the multi-system Table 3 comparison.

AuthorsOpenReview ↗

3rd Response to Reviewer Gqmu

4. How many observations support the scores?

Tables 3 and 4 evaluate 200 workflows per domain, 800 in total, with one released matched challenge per workflow. For ICS, a matched pair is eligible when its clean target succeeds; the reported outcome is whether the corresponding polluted target also succeeds. Table G3 reports this eligible support separately for each memory mechanism and domain (at most 200 per cell), whereas Table G4 pools it across the four domains for each system/backbone configuration (at most 800 per cell). The tables concern different experiments, and their supports do not imply repeated execution of a pair.

Table G3. Domain-specific ICS supports for manuscript Table 4.

MechanismStatefulCodeRetrievalMathTotal
Local-only1701899219470
Full Context18512813919471
Shared Flat RAG18718012420511
Mem01751849242493
A-Mem1821471284461
PlugMem180927012354

Table G4. Manuscript Table 3 ICS (%), with eligible support in parentheses.

SystemQwen3.8-27BDeepSeek-V4-FlashGPT-6-Luna
Codex0.0 (62)0.0 (12)0.0 (4)
Hermes0.0 (78)0.0 (78)0.0 (114)
Deep Agents38.1 (168)0.0 (32)0.0 (8)
CrewAI12.9 (248)6.1 (132)13.3 (240)
AutoGen20.0 (230)15.6 (128)11.8 (204)
Peer Review19.1 (188)17.1 (140)0.0 (46)
SMoA10.1 (276)14.9 (195)15.1 (219)
OMAC17.3 (208)4.4 (362)4.4 (180)
ADAS7.4 (108)14.3 (196)7.1 (168)
EvoMAS26.8 (224)9.8 (164)7.0 (284)

These counts make the comparisons interpretable. For instance, A-Mem's Mathematics ICS rests on four eligible observations, while its Stateful ICS has 182; the two percentages should not be read with the same precision. The corpus's topology/operator totals describe challenge coverage, whereas the supports above describe which clean-successful targets entered a method's ICS calculation. We keep corpus size, actual workflow execution, and eligible target-pair observations distinct.

Manuscript revision. We will clarify that VHS includes downstream task execution, specify ICS eligibility and its observation unit, and make the corresponding eligible supports and uncertainty intervals accessible with the reported scores. We will also state the tested rule predicates and candidate-set sizes, and frame system comparisons as execution on the same specified workflows.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers

We thank the four reviewers for recognizing the central value of evaluating both required information sharing and isolation of inapplicable artifacts in multi-agent workflows. They also highlighted the benchmark's domain-native verification and matched clean–polluted design; Gqmu and V4na specifically noted the value of the fixed-executor memory comparison and the diagnostic distinction between workflow, node, and handoff outcomes. We have organized the supplementary evidence around the questions raised across the reviews: whether the workflows can be executed, how the isolation intervention affects behavior, and how conditional and topology-stratified scores should be interpreted.

Executability and low workflow success. All 800 workflows completed human review and passed domain-native Gold execution at construction and supplementary replay (V4na, Table V4). We also ran GPT-6.1 Sol at medium reasoning on 50 supplied workflows per domain in separate Codex solver sessions with bounded action and observation budgets. This was not the Table 3 Codex baseline and did not include polluted runs or ICS. Its non-Gold terminal success rates are 24% in Stateful Tool, 82% in Repository Code, 30% in Offline Retrieval, and 70% in Formal Mathematics (V4na, Table V4; DNGC, Table D5). On the same stated Sol setting without a supplied graph, success is 12%, 68%, 24%, and 62% across the four domains, respectively (V4na, Table V4). Trusted-predecessor execution further shows how much of a target remains difficult after its prerequisites are established (Gqmu, Table G2), while node-stage traces locate losses in reaching and verifying required nodes (DNGC, Table D4). Together, these results distinguish task executability, local solving, and complete workflow execution.

Sharing, isolation, and content sensitivity. Manuscript Table 3 presents the inapplicable candidate directly in the consumer's task input; Table 4 fixes the executor and lets the memory mechanism determine additional exposure. In Stateful Tool, Full Context raises VNCR from 74.6% to 86.9% while ICS falls from 97.1% to 72.4%, showing that verified progress and conditional isolation need not rise together. Supplementary controls locate distinct mechanisms: a plain rerun verifies 198/210 originally successful targets (V4na, Table V1), whereas matched benign versus polluted content reduces Full Context target success from 83.5% to 64.9% (fabu, Table F4). Rule checks and distinguishable-content tracing further show which candidates are rejected and when their content is detectably exposed (Gqmu, Table G1; fabu, Table F2).

Common task requirements and score interpretation. Systems retain their own runtimes while executing the same specified dependencies and target boundaries; Table 4 additionally holds execution organization fixed to compare memory mechanisms (Gqmu, §3). VHS measures downstream success under a valid required handoff, and ICS measures polluted-target success conditional on clean-target success; their interpretation is supported by trusted-input results and eligible supports (DNGC, Table D1; Gqmu, Tables G3–G4). The corpus distributions locate which operator is tested at each local topology boundary (fabu, Table F1; DNGC, Table D3), without treating these strata as an independent causal effect of topology.

Revision in the manuscript. We will make the VHS and ICS conditions, the two intervention paths, and the interpretation of topology/operator strata explicit. We will place eligible supports and uncertainty intervals with the conditional results, identify sparse strata without ranking them, and make the rule-control settings and runtime constraints easy to locate. Gold replay, supplied-workflow Sol execution, and the completed no-graph Sol comparison will be presented as distinct evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.