acceptodds
Under review as a conference paper at ICLR 2027

Tapestry: Enabling Continuous Collaboration Across Episodes in Embodied Multi-Agent Systems

17 views

Abstract

Embodied multi-agent collaboration is inherently continuous: task execution and intervening environmental changes can reshape the world in which subsequent tasks are performed. Effective collaboration requires agents to accumulate, share, and reuse distributed historical information across episodes, rather than repeatedly starting from scratch. Yet existing embodied multi-agent benchmarks largely isolate episodes, while memory-centric benchmarks primarily study long-horizon historical reasoning within individual embodied tasks, leaving interdependent cross-episode collaboration underexplored. To address this, we introduce the Tapestry Benchmark, to our knowledge the first controlled 3D benchmark for continuous embodied multi-agent collaboration across interdependent episodes. It organizes tasks into continuous streams in evolving environments and provides a unified testbed for studying how embodied agents accumulate, share, and reuse distributed experience under evolving environments and cross-episode continuity dependencies, where earlier task completion, environmental changes, and distributed multi-agent experience jointly determine the context of subsequent tasks. To this end, we propose Tapestry Reasoning, which views the evolving environment as a tapestry woven from distributed observations, actions, state transitions, and execution outcomes across agents and episodes. Tapestry Weaving (TW) interweaves fragmented historical evidence into coherent cross-episode histories, while Tapestry Evidence Grounding (TEG) retrieves and grounds relevant historical evidence to interpret current instructions, infer implicit task-relevant information such as object identities, states, locations, and configurations, and translate these into executable decisions. Evaluations on the benchmark demonstrate that Tapestry Reasoning consistently outperforms existing methods under matched backbone and execution settings across diverse LLM backbones. Extensive experiments further demonstrate its robustness under historical and execution interference and validate the increasing difficulty introduced by the proposed continuity dependencies.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-agent system, Agent memory, Embodied intelligence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pM7QOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of continuous collaboration in embodied multi-agent systems. It introduces the Tapestry Benchmark to evaluate the ability of embodied multi-agent systems to accumulate, share, and reuse historical information across multiple continuous and interdependent episodes. The benchmark constructs cross-episode dependencies among entities, states, and configurations through evolving environments, and further incorporates robustness challenges including instance ambiguity, historical evidence interference, and infeasible constraints. The authors also propose Tapestry Reasoning, which consists of Tapestry Weaving and Tapestry Evidence Grounding. The former organizes historical information across agents and episodes, while the latter retrieves and leverages relevant evidence to perform history grounding for current tasks.

Critical strengths
  1. This paper investigates a continuous multi-agent embodied collaboration problem that is clearly meaningful yet remains underexplored, and provides a relatively comprehensive benchmark for its evaluation.
  2. The experimental analysis is comprehensive, with in-depth diagnostics of historical reasoning and multi-agent collaboration.
Critical weaknesses
  1. Fig. 2 could be redrawn or further refined, as the current presentation appears somewhat cluttered. In addition, some of the text is too small, and the font styles are not fully consistent. Other figures and tables could also benefit from further optimization.
  2. If possible, I suggest including a few simple but more informative baselines, such as a Long-Context baseline that directly feeds the entire interaction history into the model, and a Flat Retrieval baseline based on BM25 or embedding similarity under the same history and context budget.
Questions for authors
  1. The experiments on 3DSPMR, FindingDory, and MEMENTO in the appendix are important for understanding the effectiveness of the proposed method. In the final version, I suggest moving these results to the main paper.
Reviewer_4frcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Tapestry, a benchmark and reasoning system for embodied multi-agent tasks that span multiple episodes. The benchmark contains 160 ten-episode streams across 15 Habitat scenes. Object states and layouts evolve over time, and later instructions may refer to earlier events or information observed by another agent. To construct the streams, the authors execute tasks with an embodied policy, correct unmet requirements, and introduce scene-valid changes between episodes. Tapestry Reasoning uses Tapestry Weaving (TW) to organize distributed observations and actions into versioned entity and configuration histories, and Tapestry Evidence Grounding (TEG) to retrieve evidence for resolving history-dependent instructions. The main evaluation covers 300 episodes and includes comparisons with PARTNR and adapted memory methods, component ablations, and analyses of grounding, communication, and robustness.

Critical strengths
  1. A useful benchmark for history-dependent multi-agent tasks. The 160 task streams retain object identities, states, and layouts across ten episodes, allowing later instructions to depend on earlier activity (Section 3 and Appendix D). The evaluator checks temporal dependencies and shared-argument constraints as well as final object placement. Human review and simulator checks make the construction more credible, and the annotations allow researchers to examine different kinds of historical grounding errors.

  2. Good evidence for the proposed memory and grounding components. Table 3 shows that TW and TEG improve DeepSeek-V4 task completion from 44.47% to 55.99% and success rate from 8.33% to 19.00%, relative to removing both components. Replacing TW with a semantic summary also performs worse. The retrieval and binding analyses (Figure 3; Tables 13-14) are useful for understanding both the gains and the remaining failures.

Critical weaknesses
  1. Historical sharing and online coordination are not separated in the No-share experiment. In Appendix C.2.2, removing cross-agent information affects both the current episode's coordination and the reuse of teammates' history. Table 11 shows lower task performance, but it also shows more waiting: Wait actions account for 43.42% of executed actions without sharing, compared with 21.93% in the full system. Some of the performance loss may therefore come from weaker coordination during execution. The other ablations do not show how much of the gain is specifically due to using another agent's experience from earlier episodes, which is an important part of the paper's motivation.

  2. The adapted memory baseline results need more detail. Appendix C.1 compares 3DSPMR, FindingDory, and MEMENTO under the same collaboration pipeline, which is a sensible way to control for differences in execution. It is less clear which memory variants were implemented and whether their retrieval and context budgets were comparable; Appendix E also omits baseline planner prompts. Those choices matter for interpreting Table 9. For DeepSeek-V4, the success-rate difference between TW-TEG and MEMENTO is 2.33 percentage points, with no paired uncertainty estimate for that comparison. The internal ablations are helpful, but the advantage over existing memory methods is harder to judge from the information provided.

Questions for authors
  1. In the No-share experiment, could the authors distinguish the effect of removing archived teammate history from the effect of removing current-episode communication? If a separate ablation is unavailable, please clarify exactly what information is lost in this setting.

  2. How many of the history-dependent tasks require evidence that only another agent observed in an earlier episode? A breakdown of grounding accuracy on those tasks would help establish when cross-agent memory is necessary.

  3. Which memory variants were used for 3DSPMR, FindingDory, and MEMENTO in Table 9? How were their retrieval and context budgets matched to TW-TEG?

  4. All adapted memory baselines have zero impossible-task detection in Table 9. Did they receive the same instructions and output interface for reporting infeasible tasks? Please clarify how these scores were obtained.

  5. How stable is the 2.33-point DeepSeek-V4 success-rate difference from MEMENTO across task streams? A paired uncertainty estimate that accounts for episodes within the same stream would help interpret the result.

Reviewer_kNM6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tapestry introduces 160 continuous task streams comprising 1,600 embodied episodes. Tapestry Weaving organizes distributed histories, and Tapestry Evidence Grounding resolves historical references into executable goals. The main evaluation uses 300 episodes sampled as complete streams.

Critical strengths
  • The benchmark combines persistent environments, historical references, and distributed agent experience in a useful evaluation setting. Versioned histories and evidence-grounded interpretation directly address these requirements.

  • Under DeepSeek-V4, task completion rises from 31.39% to 55.99% and success from 6.33% to 19.00% over PARTNR. Component ablations, adapted memory baselines, sharing controls, and prompt/subset checks provide substantial supporting evidence.

Critical weaknesses
  • Construction-time corrections and interventions are explicit, but their relationship to evaluation-time state progression needs clarification. Construction-side historical bindings do not by themselves establish that the evaluated agents observed every required antecedent. This affects the interpretation and reproducibility of cross-episode reasoning.

  • Continuity categories also vary in horizon, representation, and constraints, so performance differences cannot cleanly isolate historical dependence. Absolute success remains low, and the gain over the strongest adapted memory baseline is smaller than over PARTNR: 19.00% versus 16.67% success with DeepSeek-V4.

Overall assessment: Weak acceptance, borderline leaning accept. The benchmark and controlled memory results narrowly outweigh the concerns. The state-progression and evidence-availability questions are consequential; an adverse answer would lower this assessment.

Questions for authors
  • Clarification: Are later initial states fixed from construction or derived from each evaluated policy? What construction policy is used, and how are corrections and interventions made observable to evaluated agents?

  • Evidence: Can an agent-visible evidence audit establish solvability of historical references, and can matched-complexity results separate continuity demands from task difficulty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.