CollabMemBench: Benchmarking Shared Memory under Asynchronous Multi-Agent Work

Asynchronous multi-agent systems need shared memory: decisions, evidence, and updated constraints from one agent must remain available to others beyond private histories. During concurrent work, this memory must preserve provenance and causal dependencies, distinguish parallel branches, and identify records superseded by later updates. Existing memory benchmarks focus on recall and retrieval, while multi-agent benchmarks measure overall team performance; neither isolates whether a shared-memory backend provides valid evidence during overlapping work. We introduce CollabMemBench, built from timestamped GitHub events collected through GH Archive from public open-source repositories. It organizes issues, pull requests, reviews, commits, and releases into controlled Sequential, Parallel, and Entangled workflows with non-blocking event release and distinct private agent histories. Holding the agent team, model, tools, schedule, and resource budgets fixed, we compare replaceable memory backends on answer quality, complete-case success, evidence sufficiency, and token and time costs. Across six external memory systems and a long-context baseline, even the best-performing memory system reaches only 47.6% Macro QA Score, 20.3% Complete Case Rate, and 37.4% Evidence Sufficiency Rate. Performance varies across workflow structures, with recurring failures to recover prerequisite evidence, separate branches, and reject superseded records. These results show that relevance-oriented retrieval alone does not reliably support shared memory for asynchronous multi-agent coordination. CollabMemBench lays the foundation for developing shared-memory systems that enable asynchronous agent teams to coordinate reliably as their work evolves. Code and benchmark artifacts are available at https://anonymous.4open.science/r/CollabMemBench-475D.