- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SYNAPSE-Bench, a benchmark for high-level coordination between a humanoid and a quadruped under partial observability. It contains 461 demonstration-derived episodes with complementary manipulation and transport roles. Experiments use a fixed 100-episode subset to compare unstructured observations, local scene graphs, and fused scene graphs across VLM backbones, communication settings, and model pairings. GRAPHSTITCH is provided as a reference graph-fusion method. Performance is reported through Reference-Trajectory Pass Rate (RTPR) and Normalized Decision-Attempt Agreement (NDA).
Critical strengths
-
SYNAPSE-Bench provides a concrete evaluation setting for coordination under asymmetric capabilities and distributed observations, with explicit action, communication, and scene-graph interfaces. The documented task dependencies and representation settings make it possible to study how agents combine local knowledge and coordinate complementary roles. This is a meaningful benchmark contribution because it provides a common experimental setting for investigating the interaction between state representation and multi-agent decision-making.
-
The evaluation across multiple VLM backbones yields informative findings about the use of structured information. In particular, the model-dependent gains and regressions from graph fusion show that additional shared information does not uniformly improve performance under the evaluated protocol. The human-score correlation and threshold analyses also make limitations of the evaluation more transparent. These results contribute useful empirical evidence about how models respond to structured interfaces, supporting the benchmark's value as an experimental testbed.
Critical weaknesses
-
The task and evaluation design leave uncertainty about what coordination abilities the benchmark scores actually measure. Although the paper describes closed-loop execution, RTPR uses judge-scored reference decision pairs, without clearly explaining how valid alternative strategies affect subsequent observations or receive credit. The human-score correlations do not establish independent task completion, and averaging action and communication scores leaves unclear whether invalid actions are separately rejected. All episodes also follow fixed manipulation-transport roles, while local-task and dependency fields may supply part of the planning information. The paper acknowledges its restricted scope, but the exact inputs and model performance across dependency structures remain unclear. These issues affect the interpretation of the benchmark's primary outcome: whether scores reflect successful coordination under partial observability and how much reasoning the agents must perform themselves.
-
The claimed benefits of structured communication are not fully supported by controlled comparisons and efficiency measurements. Scene graphs provide simulator-derived, human-verified information that unstructured agents must infer from images, while fusion additionally supplies information from another agent. The free-form versus oracle comparison also does not isolate the value of delivering messages to teammates, and communication costs for messages and shared graphs are not reported. These comparisons show how models respond to the supplied interfaces, but leave the specific benefits of representation structure, information exchange, and communication efficiency unresolved. Matched-information controls, a message-delivery ablation that avoids directly penalizing missing outputs, and transmission-cost reporting would help substantiate the paper's central motivation of studying effective, communication-efficient structured coordination.
Questions for authors
- Please clarify whether predicted actions generate new environment observations or whether accepted decisions advance through recorded reference observations. How are valid alternative actions and successful non-reference strategies handled? Is action validity or final goal satisfaction checked independently of the averaged action/communication score at the benchmark's high-level interface?
- Can the authors provide exact input examples for each representation setting, distinguishing simulator annotations from inferred information? If feasible, a matched-information graph/prose comparison or a baseline supplying both local graphs with source frames identified would help distinguish the respective effects.
- Could the authors mask incoming messages only in the recipient's history while preserving the required outputs, judge interface, and graph-sharing protocol? Please report action-only performance on comparable decision contexts, or independently verified task success where available, so changes in communication scores are not the sole evidence. Please also account for graph exchange in the communication costs.
- Which local-task, dependency, belief, and teammate-model fields are supplied, and are they derived from observations, simulator state, or reference trajectories? Per-family performance with evaluation sample counts, if available, would help distinguish direct relays from door-dependent episodes.