acceptodds
Withdrawn from ICLR 2027

SYNAPSE-Bench: Structured Multi-Agent Coordination under Multimodal Partial Observability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied agents are increasingly expected to operate in multi-agent environments, where effective collaboration requires not only perception and control, but also communication and shared reasoning under partial observability. However, how state representation affects heterogeneous collaboration under distributed observations remains insufficiently understood. In this work, we introduce SYNAPSE-Bench, a new benchmark for evaluating high-level decentralized coordination in heterogeneous multi-agent systems. SYNAPSE-Bench features teams of agents with different embodiments, including humanoid and quadruped robots, operating in indoor environments with both egocentric and exocentric visual inputs. Agents interact over multiple timesteps, generating both high-level actions from embodiment-conditioned action spaces and communication messages to accomplish shared tasks under decentralized execution. A key design principle of SYNAPSE-Bench is using structured state representations to study coordination under partial observability. We incorporate scene graphs as an explicit intermediate representation of objects and their relation, and formulate an evaluation setting that compares unstructured histories, local scene graphs, and fused scene graphs as decision interfaces. To support systematic study, we provide GRAPHSTITCH as a reference approach for aligning and fusing scene graphs across agents. We conduct extensive zero-shot evaluations and controlled ablations over communication and representation design. Our experimental results show that local structured representations improve task success for several VLM backbones, while graph fusion yields model-dependent gains and regressions. SYNAPSE-Bench provides a common evaluation framework for studying how state representation, communication, and model assignment shape embodied multi-agent coordination.

Primary area:
datasets and benchmarks
Keywords:
Heterogeneous Multi-Agent Collaboration, Embodied Intelligence, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m4RZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SYNAPSE-Bench, a benchmark for high-level coordination between a humanoid and a quadruped under partial observability. It contains 461 demonstration-derived episodes with complementary manipulation and transport roles. Experiments use a fixed 100-episode subset to compare unstructured observations, local scene graphs, and fused scene graphs across VLM backbones, communication settings, and model pairings. GRAPHSTITCH is provided as a reference graph-fusion method. Performance is reported through Reference-Trajectory Pass Rate (RTPR) and Normalized Decision-Attempt Agreement (NDA).

Critical strengths
  1. SYNAPSE-Bench provides a concrete evaluation setting for coordination under asymmetric capabilities and distributed observations, with explicit action, communication, and scene-graph interfaces. The documented task dependencies and representation settings make it possible to study how agents combine local knowledge and coordinate complementary roles. This is a meaningful benchmark contribution because it provides a common experimental setting for investigating the interaction between state representation and multi-agent decision-making.

  2. The evaluation across multiple VLM backbones yields informative findings about the use of structured information. In particular, the model-dependent gains and regressions from graph fusion show that additional shared information does not uniformly improve performance under the evaluated protocol. The human-score correlation and threshold analyses also make limitations of the evaluation more transparent. These results contribute useful empirical evidence about how models respond to structured interfaces, supporting the benchmark's value as an experimental testbed.

Critical weaknesses
  1. The task and evaluation design leave uncertainty about what coordination abilities the benchmark scores actually measure. Although the paper describes closed-loop execution, RTPR uses judge-scored reference decision pairs, without clearly explaining how valid alternative strategies affect subsequent observations or receive credit. The human-score correlations do not establish independent task completion, and averaging action and communication scores leaves unclear whether invalid actions are separately rejected. All episodes also follow fixed manipulation-transport roles, while local-task and dependency fields may supply part of the planning information. The paper acknowledges its restricted scope, but the exact inputs and model performance across dependency structures remain unclear. These issues affect the interpretation of the benchmark's primary outcome: whether scores reflect successful coordination under partial observability and how much reasoning the agents must perform themselves.

  2. The claimed benefits of structured communication are not fully supported by controlled comparisons and efficiency measurements. Scene graphs provide simulator-derived, human-verified information that unstructured agents must infer from images, while fusion additionally supplies information from another agent. The free-form versus oracle comparison also does not isolate the value of delivering messages to teammates, and communication costs for messages and shared graphs are not reported. These comparisons show how models respond to the supplied interfaces, but leave the specific benefits of representation structure, information exchange, and communication efficiency unresolved. Matched-information controls, a message-delivery ablation that avoids directly penalizing missing outputs, and transmission-cost reporting would help substantiate the paper's central motivation of studying effective, communication-efficient structured coordination.

Questions for authors
  1. Please clarify whether predicted actions generate new environment observations or whether accepted decisions advance through recorded reference observations. How are valid alternative actions and successful non-reference strategies handled? Is action validity or final goal satisfaction checked independently of the averaged action/communication score at the benchmark's high-level interface?
  2. Can the authors provide exact input examples for each representation setting, distinguishing simulator annotations from inferred information? If feasible, a matched-information graph/prose comparison or a baseline supplying both local graphs with source frames identified would help distinguish the respective effects.
  3. Could the authors mask incoming messages only in the recipient's history while preserving the required outputs, judge interface, and graph-sharing protocol? Please report action-only performance on comparable decision contexts, or independently verified task success where available, so changes in communication scores are not the sole evidence. Please also account for graph exchange in the communication costs.
  4. Which local-task, dependency, belief, and teammate-model fields are supplied, and are they derived from observations, simulator state, or reference trajectories? Per-family performance with evaluation sample counts, if available, would help distinguish direct relays from door-dependent episodes.
Reviewer_ZRriOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SYNAPSE-Bench, a benchmark for high-level decentralized coordination between a humanoid (Unitree G1) and a quadruped (Unitree Go2) in Isaac Sim indoor scenes. Each episode is a manipulation–transport relay: the humanoid loads an object (mostly a medical kit), the quadruped delivers it, and humanoid door operations create four dependency families (direct relay 291, access 41, egress 112, two-door 17; 461 episodes in total, 3–10 paired steps, 5 environments). Trajectories come from human teleoperation. Action and message labels were drafted by GPT-4o and verified by humans. Per-frame scene graphs are derived from simulator 3D bounding boxes and checked by humans.

At each step, each VLM agent outputs a high-level action and a message. A GPT-4o-mini judge scores both against the reference annotation. RTPR counts an episode as passed if every paired step clears a gate (λ=0.3). NDA is the fraction of decision attempts whose mean score is at least ε=0.75. The paper compares three input conditions (unstructured, local scene graph, scene graph fused by the proposed GRAPHSTITCH reference method), free-form vs. oracle communication, and homogeneous vs. heterogeneous VLM pairings. Nine VLMs are evaluated on a 100-episode subset. Main findings: local scene graphs raise RTPR for all models, fusion helps some models and hurts others, oracle messages raise NDA but sharply lower RTPR, and heterogeneous pairings are sensitive to which models are paired.

Critical strengths
  1. A relevant and under-covered evaluation setting, backed by a sizable, richly annotated release. No existing benchmark in Table 1 combines capability-asymmetric humanoid–quadruped teams, decentralized execution with explicit messages, egocentric and exocentric views, and per-frame scene graphs. The release contains 3,746 agent-timestep samples, 74,520 RGB frames, 37,260 local scene graphs, and per-step dependency annotations. The dependency-family analysis (Table 5, Fig. 6: more waits and requests in longer, door-gated episodes; low H–Q node overlap with Jaccard about 0.14–0.26) shows that the episodes contain actual cross-agent dependencies. This is the main reason the work could be valuable to the community.
  2. Candid analysis of its own limits. The appendix reports judge–human correlation (r≈0.82, ρ≈0.85), threshold sweeps showing RTPR collapsing at λ=0.40, and an episode-clustered bootstrap showing that the fused-vs-local NDA difference for Gemma4 is not significant. It also states plainly that the benchmark does not show that communication is necessary. This honesty helps readers, although it also undercuts some headline claims (see Weaknesses).
Critical weaknesses
  1. The protocol is teacher-forced matching against one reference trajectory, scored by an LLM judge. That does not measure the closed-loop coordination the paper claims. Agents do not change a simulated environment. Their free-form text actions and messages are scored by GPT-4o-mini against the annotated reference action and message (App. E). The rollout continues only if both agents score above λ, and "history is advanced only after a successful paired decision" (App. A.4). Observations are pre-recorded frames from the human demonstration. As a result:

    • Any valid strategy that departs from the reference is penalized. The authors acknowledge that "another valid strategy may differ from it" (L1001–1002).
    • Communication quality is reduced to similarity with GPT-4o-drafted reference wording.
    • Errors can never propagate or be recovered from.

    The oracle-communication result supports this concern. Giving agents privileged ground-truth information lowers RTPR on every backbone (e.g., Eagle2.5 50.4 → 16.0) while NDA rises sharply (43.98 → 58.47). This looks like a scoring artifact, not a finding about "communication alignment." Several claims therefore lack support from the evaluation: that the benchmark evaluates "closed decision loop" coordination (Fig. 2), "coordination emerges via repeated interaction," and the conclusion that "evaluating them in isolation underestimates their combined impact, with the strongest gains when both are present" (L477–478). No experiment crosses communication with representation, and there is no no-communication condition, even though Fig. 1 is built around it.

  2. The main empirical conclusions do not hold up statistically, and the representation comparison is confounded.

    • Small sample, no uncertainty. Results use 100 episodes, mostly with no reported variance. Some values do not fit a single run over 100 episodes (e.g., RTPR 26.7). The authors' own bootstrap shows the fused-vs-local difference is within noise, and several "gains" are tiny (InternVL +0.4 RTPR).
    • Metrics disagree. The statement "local scene graphs improve RTPR for every evaluated model" ignores that NDA decreases under local graphs for 5 of 9 models (e.g., GPT-5.4 39.10 → 21.68, Gemma4 37.60 → 27.24). It is not clear which metric reflects coordination quality.
    • Ground-truth graphs confound the comparison. The "structured" conditions use ground-truth scene graphs built from simulator bounding boxes. Any gain therefore mixes "structured format" with "privileged, perfect perception" (object identity, door state, relations the VLM may not perceive from 4 frames). An unstructured-text control with the same information content, or graphs predicted by a perception model, is needed.
    • Bandwidth claim not tested. The communication-efficiency argument that motivates scene graphs is never measured (no bytes or tokens exchanged). The fused graph is computed centrally and broadcast to both agents, which conflicts with the "decentralized" framing.
    • Fusion quality never measured. GRAPHSTITCH's fusion accuracy is not compared against the available ground-truth global graph.
Questions for authors
  1. Are agent actions ever executed in Isaac Sim during evaluation, or is every step scored against the reference annotation over pre-recorded observations? If the latter, how would an agent following a different but valid plan be scored?
  2. In the oracle-communication condition, what exactly is "oracle," and why does it reduce RTPR so much? Is the judge scoring the agent's own message against a reference that differs in form from the oracle signal?
  3. What is the RTPR/NDA under a no-communication condition, and under the 2×2 crossing of {no, free-form} communication × {unstructured, local graph}?
  4. How many runs and seeds back each number in Tables 2–4, and what are the confidence intervals? How is RTPR 26.7 obtained on 100 episodes?
  5. Can you separate structure from information content? For example: (a) unstructured text that lists the same objects and relations, or (b) scene graphs predicted from RGB rather than ground truth.
  6. What is GRAPHSTITCH's node-alignment and relation accuracy against the ground-truth global scene graph?
  7. Will the remaining 361 episodes be released as a public test split, and how do you prevent judge-specific overfitting (e.g., by fixing a judge version)?
Reviewer_YQGaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The submission introduces synapse-bench, a benchmark for coordination between two different robots, a humanoid (unitree g1) and a quadruped (go2), when each sees only part of the scene. Episodes are recorded by human teleoperation in isaac sim across five indoor layouts. Each episode is a short relay: the humanoid picks up an object and opens doors if needed, and the quadruped carries the object to the goal. The release has 461 episodes in four task families, with per-step text actions, messages, rgb frames and ground truth scene graphs. The problem is framed as a Dec-POMDP with a natural language message channel. Three inputs are compared: raw visual-text history, a local scene graph per agent, and a fused scene graph built with graphstitch, a constraint based alignment method. Nine vlms are evaluated zero-shot on 100 episodes using two judge-based metrics (rtpr and nda). There are also ablations on free-form vs oracle messages and on same vs mixed model pairs. The main claim is that local scene graphs improve pass rate for every model, and fusion helps some models but not others.

Critical strengths
  1. The question is reasonable for current vlm agents: does a compact, explicit state help two differently built robots work together?

The humanoid + quadruped pairing creates a real split in roles. The dependency labels (which agent enables which action, and when) are useful on their own. The data is human checked, the reported counts are consistent, and the authors plan to release data, prompts and code.

  1. The appendix is honest about its limits.

It includes threshold sweeps, a bootstrap interval, a judge vs human agreement study, and a limitations section that admits single-agent or no-communication solutions are not ruled out. Many models are tested, both proprietary and open weight. The reviewer appreciates this, though several of these analyses weaken the main claims more than they support them.

Critical weaknesses

W1. The problem setting is not new, and the benchmark covers one short task template.

-- Coordination under partial observability, with different agents and limited communication, is a long-standing topic, and the paper covers it in about one sentence (sec 6). Relevant published work includes centralized training with decentralized execution [1, 2, 3], learned and bandwidth limited communication [4, 5, 6, 7, 8], heterogeneous teams [9, 10, 11], and visual multi-agent embodied tasks with explicit or emergent communication [12, 13, 14, 15, 16]. Some of these, e.g. [8] and [11], already treat “different agents need compact messages” as the main problem. The abstract and intro do not position the work against this literature.

--the bandwidth argument for scene graphs is stated but never measured. No message sizes, serialization costs or communication budgets are reported. Learned image codecs reach a few kilobytes per image at good quality (e.g. hific, glc), and collaborative perception work (when2com, v2vnet, where2comm) already studies what to share under bandwidth limits. A compressed image or shared feature baseline at a matched budget is missing.

-- The scene graphs come from simulator ground truth with manual checks (appendix a.5) and are not predicted by the agents. In the fused setting, one graph built from both agents' graphs is given to both (sec 3.2), outside the message channel. So the agents never build or send a graph themselves, which is the process the intro motivates. The fused setting also gives each agent a pooled view of the scene, which weakens the decentralized framing for that setting.

-- The data is 461 episodes from five layouts, all variants of one relay structure (load, optionally open a door, transport), 3 to 10 steps, with no train/val/test split and evaluation on 100 episodes. The appendix says it does not claim these tasks are impossible for one capable robot. Since all evaluation is zero-shot prompting with no learning baseline, the benchmark shows how prompted vlms score on these episodes, not what could be learned from them.

W2. Success is scored as agreement with one recorded demonstration, and the main result reverses under a stricter threshold.

-- The text says actions are executed and parsed into primitives. But the appendix reads as if the rollout follows the recorded episode: a judge compares each step's text to one reference trajectory, and the episode moves on only if both agents pass a score threshold. If so, success means matching one human demonstration, and an alternative valid strategy would be scored as failure. Task success checked in the simulator is not reported.

---In the authors' own sweep (Gemma4 only), results hold from 0.30 to 0.35. At 0.40, local and fused pass rates fall to 11.6% and 17.6%, well below the unstructured one, which reverses the main claim for that model. On nda, adding structure lowers the score for several models, including gpt-5.4. The bootstrap interval for Gemma4's fused vs local nda gap includes zero.

-- Oracle messages, presented as an upper bound, lower rtpr but raise nda for every model tested. The two metrics disagree on whether better information helps. In the reviewer's view that points to a problem with the protocol or metrics, not a finding about communication. The content of the oracle message is not described.

--The main comparisons are point estimates with no uncertainty, though the appendix gives five-run and bootstrap results for one model. Several rtpr gains are under one point on 100 episodes. Judge vs human agreement is checked for one model only. Models from the same provider create the annotations, act as the judge, and are among the evaluated systems, so an independent judge check would be needed to rule out bias.


References

[1] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (maddpg). NeurIPS 2017.

[2] Counterfactual Multi-Agent Policy Gradients (coma). AAAI 2018.

[3] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. ICML 2018.

[4] Learning to Communicate with Deep Multi-Agent Reinforcement Learning (dial/rial). NeurIPS 2016.

[5] Learning Multiagent Communication with Backpropagation (commnet). NeurIPS 2016.

[6] TarMAC: Targeted Multi-Agent Communication. ICML 2019.

[7] Learning to Schedule Communication in Multi-agent Reinforcement Learning (schednet). ICLR 2019.

[8] Learning Efficient Multi-agent Communication: An Information Bottleneck Approach (imac). ICML 2020.

[9] ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. ICML 2020.

[10] Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning (hatrpo/happo). ICLR 2022.

[11] Learning Efficient Diverse Communication for Cooperative Heterogeneous Teaming (hetnet). AAMAS 2022.

[12] Two Body Problem: Collaborative Visual Task Completion. CVPR 2019.

[13] A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks. ECCV 2020.

[14] Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration. ICLR 2021.

[15] Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents. ICCV 2021.

[16] Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots. ICLR 2024.

[17] High-Fidelity Generative Image Compression (hific). NeurIPS 2020.

[18] Generative Latent Coding for Ultra-Low Bitrate Image Compression (glc). CVPR 2024.

[19] When2com: Multi-Agent Perception via Communication Graph Grouping. CVPR 2020.

[20] V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction. ECCV 2020.

[21] Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps. NeurIPS 2022.

Questions for authors
  1. Sec 2.3 describes actions as executed with the environment updated each step, and sec 4.1 says a rollout advances when both agents meet “environment-defined criteria”. The same section describes a judge that scores actions and messages against reference annotations. How are each agent’s next observations produced during evaluation, and what are the environment-defined criteria? In the reviewer’s reading of appendix a.4 and b, the rollout steps through the recorded decision pairs, and progress is decided by the judge’s score. If so, how is task completion separated from agreement with the reference when a valid strategy differs from it? If the reviewer has misread this, a short clarification would help. Task success checked in isaac sim, with predicted actions executed, would give stronger evidence that the gains in table 2 go beyond matching the reference.

  2. The main claim (sec 5.1) is that local scene graphs improve rtpr for every model. rtpr depends on a gate of λ = 0.3, which sec 4.1 sets without justification. In the same table 2, nda decreases with structure for several models, including gpt-5.4. In table 3, oracle messages, described in sec 4.2 as an approximate upper bound, lower rtpr while raising nda. How was the 0.30 gate chosen, and which of the two metrics should readers trust when they disagree? The appendix threshold analysis is helpful here. It shows the Gemma4 ranking changing at a gate of 0.40 (fig 8). Do the main comparisons hold across models over a reasonable range of thresholds, with uncertainty reported?

  3. The intro motivates scene graphs as a compact alternative to sending raw visual data under bandwidth limits (sec 1, 3.1). In sec 3.2, the fused graph is provided to all agents. Is sharing the graph counted toward any communication budget in the protocol? Are there measurements of per-step message size, or a comparison with compressed images or shared features at a matched budget? Relatedly, appendix a.5 indicates the graphs are built from simulator ground truth. How would results change with graphs predicted from each agent’s own observations? This would help separate the communication-efficiency claim from the benefit of giving vlms structured input.

Convincing answers to these, especially 1 and 2, could meaningully change the reviewer’s assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.