acceptodds
Under review as a conference paper at ICLR 2027

CollabMemBench: Benchmarking Shared Memory under Asynchronous Multi-Agent Work

Abstract

Asynchronous multi-agent systems need shared memory: decisions, evidence, and updated constraints from one agent must remain available to others beyond private histories. During concurrent work, this memory must preserve provenance and causal dependencies, distinguish parallel branches, and identify records superseded by later updates. Existing memory benchmarks focus on recall and retrieval, while multi-agent benchmarks measure overall team performance; neither isolates whether a shared-memory backend provides valid evidence during overlapping work. We introduce CollabMemBench, built from timestamped GitHub events collected through GH Archive from public open-source repositories. It organizes issues, pull requests, reviews, commits, and releases into controlled Sequential, Parallel, and Entangled workflows with non-blocking event release and distinct private agent histories. Holding the agent team, model, tools, schedule, and resource budgets fixed, we compare replaceable memory backends on answer quality, complete-case success, evidence sufficiency, and token and time costs. Across six external memory systems and a long-context baseline, even the best-performing memory system reaches only 47.6% Macro QA Score, 20.3% Complete Case Rate, and 37.4% Evidence Sufficiency Rate. Performance varies across workflow structures, with recurring failures to recover prerequisite evidence, separate branches, and reject superseded records. These results show that relevance-oriented retrieval alone does not reliably support shared memory for asynchronous multi-agent coordination. CollabMemBench lays the foundation for developing shared-memory systems that enable asynchronous agent teams to coordinate reliably as their work evolves. Code and benchmark artifacts are available at https://anonymous.4open.science/r/CollabMemBench-475D.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking, Shared memory, Asynchronous workflows, Multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wXx2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CollabMemBench evaluates shared-memory backends for agents working on overlapping tasks. It constructs 300 Sequential, Parallel, and Entangled workflows from public GitHub histories, with reconstructed schedules and separate agent histories. Questions inserted during execution test whether memory supplies current, correctly scoped information and its supporting evidence. The paper compares six external memory systems and a long-context baseline, measuring answer quality, complete-case correctness, evidence sufficiency, and resource costs. GraphRAG leads the reported main comparison with 47.6% Macro QA Score and 20.3% Complete Case Rate. Oracle-evidence, causal-depth, and serialization experiments provide additional diagnostics.

Critical strengths

Evaluating retrieved evidence separately from the final answer is a useful choice. An agent can produce a correct answer without recovering the records needed to justify it, and this distinction matters when decisions depend on versions, branches, and earlier validations. The paper also goes beyond a leaderboard by including oracle evidence, matched serial delivery, and causal-depth experiments. The differing effects of serialization across memory systems and the competitiveness of BM25 are informative findings.

Critical weaknesses
  1. The evidence annotations and scoring need stronger validation. Sections 3.2–3.4 rely on judgments about dependencies, supersession, and what constitutes a complete sufficient evidence chain. Structural and temporal checks are useful, but do not establish that these semantic judgments are correct. The paper does not report human agreement on the annotations or evaluation, and Section 5 acknowledges this gap. The 0/1/2 answer-scoring rubric and the procedure for matching summaries or graph-derived outputs to reference evidence also need more detail. Since evidence validity is central to the benchmark, it is important to establish that these scores measure missing or incorrect information rather than a mismatch with one expected representation.

  2. A particularly relevant baseline is missing from the reported comparison. Section 4.1 says that the evaluation includes a Versioned Event Log preserving provenance, scope, dependencies, and supersession before lexical retrieval. I could not find its results in either the main tables or the appendix. This comparison matters: it would show how much of the observed difficulty can be addressed by straightforward record management, rather than more elaborate memory systems. The experimental reporting also needs reconciliation. Appendix B.3 identifies Mem0 at 49.0% as the best real-memory result, whereas Table 2 reports GraphRAG at 47.6% and Mem0 at 46.5%. This may be an editing issue, but the paper should make clear which results belong to the final evaluation.

Questions for authors
  1. Annotation and scoring clarification: Who annotated the dependencies, supersession relations, and reference evidence chains, and how were they independently checked? Please also specify the answer-scoring procedure and how alternative sufficient evidence or compressed summaries are handled.

  2. Missing comparison: What are the Versioned Event Log results under the same budgets? Are its dependency and supersession relations inferred from agent-visible inputs, or supplied from annotations? This distinction is important for interpreting the comparison.

  3. Experimental scope clarification: How many cases and repetitions support the serialization experiment? Appendix A refers to “four five-case pilot comparisons,” while Figure 5 presents six systems. Please clarify the evaluated subsets and the variability of the reported changes.

1 comment
AuthorsOpenReview ↗

Annotation and Scoring, Baseline Results, and Experimental Scope

Response to W1 and Q1: Thank you for your feedback. Candidate questions are proposed by LLMs and manually screened, replaced, or rewritten. We collect three independently generated LLM responses per question and use LLM assistance to draft the reference answers and evidence annotations. Two human annotators independently verify the annotations, including dependency and supersession relations, and review the final reference answers and evidence chains. Disagreements are resolved through discussion. The annotations distinguish necessary core evidence, comprising the most directly relevant records that must be included, from acceptable evidence, defining the set of relevant records that may be cited. Different combinations of supporting evidence are permitted within this set, provided that the core evidence is retained.

We assess both conclusion correctness and evidence validity, checking whether the required core evidence is present, all citations fall within the acceptable evidence set, and supporting claims are neither fabricated nor contradictory. The 0/1/2 scoring rubric is:

ScoreCondition
2Conclusion matches the reference answer; required core evidence is present; all citations are within the acceptable set; and supporting claims are not fabricated.
1Conclusion matches but evidence has minor imprecision or missing core support, or the conclusion is partially correct, without triggering the invalid-evidence conditions below.
0Conclusion is wrong, contradictory, irrelevant, or empty; or evidence is fabricated or includes citations outside the acceptable set.

We apply the same scoring rubric to compressed summaries and graph-derived outputs. Wording and format may differ from the reference, but full credit requires a correct conclusion and preservation of the required core evidence, including event IDs and dependency relations. Missing or invalid evidence is handled according to the criteria above.

Following the human-validation practice in GateMem [1], we compared the LLM evaluator’s final 0/1/2 scores with those independently assigned by two human annotators across all 300 cases. We used quadratic-weighted Cohen’s κ [2] to measure agreement in final answer scoring:

ComparisonQuadratic-weighted Cohen’s κ
LLM evaluator vs. Human 10.9820
LLM evaluator vs. Human 20.9338

Response to W2 and Q2: We have added Versioned Event Log (VEL), an append-only baseline preserving provenance, scope, dependencies, and supersession before lexical retrieval. Results under the same budgets are:

SettingMQSCCRESRCRPDCRTokens/qTime (s/q)
Sequential33.79.049.783.054.06,49261.8
Parallel25.28.020.077.80.08,19468.1
Entangled16.86.09.769.40.08,02563.5
Overall25.27.726.576.718.07,57064.5

VEL achieves an Overall MQS of 25.2%, CCR of 7.7%, and ESR of 26.5%, compared with 41.8%, 10.3%, and 35.3% for BM25, respectively. Its Sequential ESR of 49.7% is second only to GraphRAG's 60.9%, but falls to 20.0% and 9.7% under Parallel and Entangled workflows. These results suggest that VEL provides useful evidence support in Sequential workflows, but basic version tracking alone remains insufficient for consistently correct answers and complete-case success across settings.

VEL's dependency and supersession relations are inferred from agent-visible inputs (timestamps, artifact/revision IDs, workflow scope, and explicit update/status links in the event metadata) rather than supplied from evaluator-only annotations; the evaluator-only manifest is used only for scoring.

And we have corrected the appendix’s experimental analysis to reflect the final results reported in the main tables and checked the rest of the manuscript for consistency.

Response to Q3: We have updated the manuscript to clarify the experimental scope. The phrase “four five-case pilot comparisons” refers to an early 20-case pilot and was inadvertently retained after the experiment was expanded. The final comparison in Figure 5 covers all 100 Parallel cases across six memory backends. Each backend is evaluated on the same 100 cases, once under the original Parallel delivery and once under serialized delivery, changing only the delivery order while preserving the facts, questions, and reference answers. Figure 5 reports aggregate metrics over these matched cases and the corresponding Serial–Parallel differences.

References

[1] Ren et al. GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents. arXiv, 2026.

[2] Fleiss and Cohen. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability. Educational and Psychological Measurement, 1973.

Reviewer_Qiv2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

CollabMemBench evaluates shared memory for asynchronous multi-agent systems using workflows reconstructed from GitHub histories. It tests sequential, parallel, and entangled workflows and compares several memory systems using answer, evidence, and efficiency metrics.

Critical strengths

The benchmark targets an important gap: shared memory under asynchronous multi-agent interaction rather than standard memory recall. Evaluating both answers and supporting evidence is also useful.

Critical weaknesses

The asynchronous setting is partly synthetic. Concurrency and replay timing are reconstructed rather than taken directly from naturally occurring asynchronous execution.

The benchmark is also limited to software-development workflows, where provenance, branches, and supersession are unusually explicit. It is unclear how well the conclusions transfer to less structured domains.

Finally, several metrics depend on annotated dependency and evidence relations, but stronger human-agreement analysis is missing.

Questions for authors

1.How sensitive are the results to the reconstructed concurrency schedule?

2.What human agreement or annotation-quality analysis was performed?

  1. Do the main findings hold outside software-development tasks?
1 comment
AuthorsOpenReview ↗

Schedule Sensitivity, Evaluation Reliability, and Broader Relevance

Response to Q1: We have conducted a parallel-versus-serial diagnostic, reported in the main text and appendix. We serialize all 100 Parallel cases, changing only the delivery order while preserving their facts, questions, and reference answers. Results are below; Δ = Serial − Parallel in percentage points (pp).

Memory backendMQS: Parallel → SerialΔ MQSΔ CCR
BM2556.5 → 76.5+20.0+40.0
Dense RAG53.2 → 59.0+5.8+16.0
Mem055.6 → 59.6+4.0+1.0
GraphRAG50.8 → 39.8−11.0−6.0
Rolling Summary43.7 → 38.4−5.3−13.0
G-Memory41.0 → 41.00.0+10.0

The results show that sensitivity to delivery order varies substantially across memory backends, both in magnitude and in direction. Some backends benefit from serialization, whereas others degrade. BM25 and GraphRAG provide two contrasting examples. For BM25, serialization increases MQS by 20.0 pp and CCR by 40.0 pp, consistent with reduced competition among interleaved records during lexical retrieval. GraphRAG instead loses 11.0 MQS points and 6.0 CCR points. The main reason is that changes in arrival order alter its graph construction and the evidence exposed to the agent: reducing interleaving alone may not improve how a graph-based memory system organizes and retrieves evidence.

We include asynchronous workflows in our benchmark primarily because progressing on different subtasks concurrently while exchanging intermediate results is a common pattern in real-world collaboration. MultiAgentBench [1] evaluates coordination across diverse multi-agent tasks, while ParaCook [2] studies parallel and asynchronous collaborative planning and reports difficulties with parallel actions and coordination. These studies motivate evaluating shared memory under overlapping work. Our reconstruction preserves causal order, and the serialization diagnostic measures how delivery order affects evidence recovery and downstream decisions.

Response to Q2: Candidate questions are generated by LLMs and then manually screened, replaced, or rewritten. For each question, we collect three independently generated LLM responses and use LLM assistance to draft the reference answer and identify its required core evidence and allowed supporting records. To assess annotation quality, two human annotators independently verified the evidence annotations and reviewed the final reference answers and evidence chains. The checks covered both the required core evidence and the allowed supporting records, with disagreements resolved through discussion.

Across all 300 cases, we compared LLM-assigned final 0/1/2 scores with two independent human ratings, following the human-validation practice in GateMem [3]. We report quadratic-weighted Cohen’s κ [4]:

ComparisonQuadratic-weighted Cohen’s κ
LLM evaluator vs. Human 10.9820
LLM evaluator vs. Human 20.9338

Response to Q3: The coordination requirements we study recur beyond software development: Similar coordination requirements arise in scientific discovery, transportation, and embodied collaboration:

Scientific discovery. Co-Scientist [5] uses asynchronous agents for hypothesis generation, review, and refinement, with persistent context memory and feedback across iterations. This motivates retaining the links between each hypothesis, its supporting evidence, and the critiques informing subsequent work.

Transportation and dynamic scheduling. REALM-Bench [6] includes ride-sharing with traffic delays, road closures, and cancellations; its example workflow connects traffic monitoring, route planning, and vehicle dispatch agents. Here, downstream decisions must incorporate updated constraints and distinguish revised routes from obsolete plans.

Embodied collaboration. ParaCook [2] studies agents cooperating on cooking tasks with sequential dependencies, parallel actions, asynchronous waiting, and shared-resource contention. This requires coordinating prerequisite completion and current resource availability as different tasks progress.

We use software histories because their timestamps, provenance, and linked artifacts make cross-agent dependencies and evolving information observable and auditable.

References

[1] Zhu et al. MultiAgentBench: Evaluating the Collaboration and Competition of LLM Agents. ACL, 2025.

[2] Zhang et al. ParaCook: On Time-Efficient Planning for Multi-Agent Systems. Findings of ACL, 2026.

[3] Ren et al. GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents. arXiv, 2026.

[4] Fleiss and Cohen. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability. Educational and Psychological Measurement, 1973.

[5] Gottweis et al. Accelerating scientific discovery with Co-Scientist. Nature, 2026.

[6] Geng et al. REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Tasks. KDD, 2026.

Reviewer_G7q3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces CollabMemBench, a benchmark for evaluating shared-memory backends in asynchronous multi-agent workflows. It constructs 300 cases from public GitHub histories. Agents maintain private histories and use shared memory to recover collaborators’ decisions and subsequent updates. The evaluation measures answer quality, complete-case success, evidence sufficiency, and resource costs. The paper argues that relevance-oriented retrieval alone is insufficient for maintaining coherent evidence across evolving collaborative tasks.

Critical strengths

The paper addresses a meaningful challenge: retrieving relevant information does not necessarily recover evidence that is correctly scoped and sufficient for a downstream decision. Evaluating replaceable memory backends is a useful contribution.

Critical weaknesses
  1. The reliability of the ground truth is insufficiently validated. The paper does not clearly explain how the reference answers and evidence chains were constructed or independently checked. It is also unclear whether the evaluator accepts alternative evidence that sufficiently supports an answer.
  2. Important baselines are missing or unclear. Section 4.1 introduces a Versioned Event Log baseline, but none of the tables reports its results. This comparison is essential because simple version tracking may address some of the failures highlighted by the paper. The Long Context baseline also needs clarification. The main text states that it directly uses interaction history, whereas the appendix describes it as retrieving only a few records. I will update my score based on the authors’ response
Questions for authors

What are the results of the Versioned Event Log baseline? What information does Long Context actually receive, and how is its history merged and truncated?

1 comment
AuthorsOpenReview ↗

Clarifications on Evidence-Chain Construction and Baseline Evaluation

Response to W1: Thank you for your feedback. We have added details on the construction and independent verification of the reference answers and evidence chains. LLMs first generate candidate questions, which are manually screened, replaced, or rewritten. We then collect three independently generated LLM responses and use LLM-assisted pre-annotation to identify two types of evidence: necessary core evidence, comprising the most directly relevant records that must be included, and acceptable evidence, defining the set of relevant records that may be cited. The evaluator allows different combinations of supporting evidence within this set, provided that the core evidence is retained. Two human annotators independently verify these annotations. Overall, this process combines LLM-assisted construction with independent human verification to establish the final reference answers and evidence chains.

Following the human-validation practice in GateMem [1], we compared the LLM evaluator’s final 0/1/2 scores with those independently assigned by two human annotators across all 300 cases. We measured agreement using quadratic-weighted Cohen’s κ [2], which corrects for chance agreement and penalizes larger rating discrepancies more heavily. The results are:

ComparisonQuadratic-weighted Cohen’s κ
LLM evaluator vs. Human 10.9820
LLM evaluator vs. Human 20.9338

These results indicate strong agreement between LLM and human judgments in final answer scoring.

Response to W2 and Q1: Thank you for the feedback. We have added Versioned Event Log (VEL), an append-only baseline preserving provenance, scope, dependencies, and supersession before lexical retrieval. Results under the same budgets are:

SettingMQSCCRESRCRPDCRTokens/qTime (s/q)
Sequential33.79.049.783.054.06,49261.8
Parallel25.28.020.077.80.08,19468.1
Entangled16.86.09.769.40.08,02563.5
Overall25.27.726.576.718.07,57064.5

VEL achieves an Overall MQS of 25.2%, CCR of 7.7%, and ESR of 26.5%, compared with 41.8%, 10.3%, and 35.3% for BM25, respectively. Its Sequential ESR of 49.7% is second only to GraphRAG's 60.9%, but falls to 20.0% and 9.7% under Parallel and Entangled workflows. These results suggest that VEL provides useful evidence support in Sequential workflows, but basic version tracking alone remains insufficient for consistently correct answers and complete-case success across settings.

Long Context receives the full interaction history in chronological order, without retrieval or summarization. We merge records by release time and apply FIFO truncation, dropping the oldest records when the history exceeds our default context budget of 128k tokens. We have added these details in the main text.

References

[1] Ren et al. GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents. arXiv, 2026. [2] Fleiss and Cohen. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability. Educational and Psychological Measurement, 1973.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.