acceptodds
Under review as a conference paper at ICLR 2027

Visual Grounding and Deliberation Help Multi-Robot Teams Make High-Stakes Decisions

Abstract

Humans have developed communication skills to disambiguate and coordinate during high-stakes collaborative tasks. Whether teams of robotic systems can do the same remains unclear, especially when no single robot has full access to the information necessary to make the correct decision. With recent developments in Vision-Language Model (VLM)-powered generalist robotic systems, such scenarios are becoming prevalent and have raised serious safety concerns. To address the gap in understanding such systems, we study this problem with a benchmark across three household environments modeled as Dec-POMDPs with communication, in which a team of robots must jointly decide whether a proposed action should Proceed or Hold. We find that unstructured communication is unreliable: team accuracy can fall to chance level, and common changes to prompting, communication topology, team size, and interaction rounds do not consistently resolve these failures. Further, an idealized full-information diagnostic, in which every robot receives all observations and local context, raises accuracy to 100% in some settings, indicating that handling distributed evidence is a major source of error. To address this, we introduce a structured deliberation protocol, called HUDDLE, that requires robots to explicitly share local context, ground visual claims in observations, and assess the pooled evidence before deciding. HUDDLE substantially improves decision accuracy, reaching up to 96% in our best setting. These findings highlight communication skills as the key bottleneck for reliable multi-robot decision-making and provide a foundation for designing safer, more collaborative autonomous systems in partially observable environments.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-robot systems, embodied AI, vision-language models, multi-agent communication, hazard-aware decision making

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X12pOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how robot teams make safety-related decisions when the required information is distributed across their visual observations and private contextual knowledge. The authors introduce three household environments, analyze failures of ordinary deliberation, and propose HUDDLE, a staged protocol for sharing and evaluating evidence before reaching a decision. The evaluation shows improvements across several models and communication baselines.

Critical strengths

I find the distinction between individual reasoning ability and collective information handling useful. The full-information experiment provides a clear motivation for studying communication failures rather than treating all errors as limitations of the underlying models. The proposed protocol follows naturally from the failure analysis. Separating evidence sharing from assessment and decision-making is a reasonable response to the observed tendency to omit or dismiss relevant information. The evaluation includes multiple models and several straightforward alternatives, including changes to prompting, topology, discussion length, and aggregation. Reporting HUDDLE with and without the safeguard is also helpful.

Critical weaknesses

My main concern is that the experiments do not fully isolate what makes HUDDLE effective. The protocol changes communication structure, voting timing, assessment requirements, and action execution together. This is particularly important in the bedroom setting: the baseline can immediately consume its inspection budget, whereas HUDDLE delays movement and introduces action announcement and review. The improvement therefore cannot be attributed entirely to better deliberation. A baseline with the same action-control mechanism would make the comparison more convincing. The benchmark is also small, with 12 instances per environment. Repeated trials help establish consistency on these instances, but provide limited evidence about generalization to different scenes or evidence dependencies. It is unclear how much the protocol was developed around the evaluated instances. Finally, I would like a clearer account of the safety trade-off. Overall accuracy does not distinguish an unsafe PROCEED decision from an unnecessary HOLD. This distinction matters for interpreting both the safeguard and the broader safety motivation.

Questions for authors
  1. How does ordinary deliberation perform when given the same action announcement/review mechanism and a comparable inference budget?
  2. Were any instances or evidence configurations held out during protocol development?
  3. Can the authors report incorrect PROCEED and incorrect HOLD rates separately, with and without the safeguard?
Reviewer_XAQdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work studies finds that unstructured communication between VLM agents is unreliable for joint decision making. Based on their investigation, they propose a structured deliberation protocol called HUDDLE.

Critical strengths

HUDDLE a simple framework that seems to work reasonably well to improve performance of a team of agents.

Critical weaknesses

Overall, I think the idea is reasonable but not particularly significant. The investigation could be deeper to reveal more insights and to justify the design.

  • The failure analysis should be deepened in that the causes behind the observed failure modes are not identified or explained. For example, can the authors give some explanation as to why in the bedroom, information was never obtained, or in the kitchen, why the information was never shared? Also, are the failure modes shown in the Fig 3 and 4 totaled across the different model types? Are there differences across models?

  • The "benchmark" is rather small. It comprises three tasks and a total of 12 instances. I think it is useful as a study but I'm not sure it is appropriate to term it as a benchmark for evaluating communication.

  • The connection to Dec-POMDPs does not appear deep. Dec-POMDPs are a specific theoretical structure along with solvers and theoretical results. Here, none of this prior work is used, leveraged, or even deeply discussed.

  • Could be authors provide additional motivation for HUDDLE. The experiment in 4.2 shows that a single model can solve the problem with full-information. If a single agent had to perform the acquisition and make a decision, how well would it perform?

  • The "standard inventions" are poorly explained in the main text. What is topology? These are important because they help to support whether a complex system like HUDDLE is necessary.

  • Analysis of the framework could be more thorough. What happens if you break down the protocol further and analyze performance if steps are missing/ablated?

Questions for authors

please see weaknesses above.

Reviewer_7kYSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether teams of VLM-driven robots can make safe decisions from distributed images and private textual information. It introduces 36 instances across kitchen, living-room, and bedroom environments, with the bedroom additionally requiring coordinated camera inspections under a movement budget. Under open deliberation, five models achieve average accuracies of approximately 75, 61, and 49 percent across the three environments (Figure 3). A full-information diagnostic, communication ablations, and transcript analysis suggest that acquiring, sharing, and using evidence are substantial bottlenecks. The proposed HUDDLE protocol structures grounding, information sharing, assessment, and discussion, coordinates camera movements, and optionally applies a conservative final safeguard. HUDDLE with the safeguard improves mean accuracy across all evaluated model–environment pairs, with gains ranging from 3 to 42 percentage points (Figure 7), accompanied by reductions in several measured failure categories.

Critical strengths
  • The benchmark connects communication with active information gathering - The bedroom environment provides a meaningful extension beyond exchanging a fixed collection of facts. Robots must use information obtained by teammates to determine what to inspect next, so an early movement can prevent the team from obtaining decisive evidence. The dependency chain and trajectory analysis make this coordination problem concrete. Appendix B.2 documents early movements under open deliberation and later, better-informed movements under HUDDLE. This is consequential because it demonstrates a setting in which communication affects the evidence available for the final decision, giving the protocol’s action-coordination component a clear motivation.
  • The diagnostic analysis makes the improvements more informative - The paper combines complementary evidence: the living-room full-information experiment substantially improves accuracy, conventional communication changes do not provide a consistent solution, and HUDDLE’s gains coincide with reductions in targeted failure categories. For example, Figure 7b reports that the measured “never shared” rate falls from 0.62 to 0.05 in the kitchen and “never obtained” falls from 0.88 to 0.37 in the bedroom under HUDDLE with the safeguard. Human validation provides additional support for several information-flow labels, although the annotated subset is small and “wrong reasoning” is less reliable. Together, these analyses support a useful account of the observed failures without establishing the causal contribution of each protocol stage.
Critical weaknesses
  • The additional value of HUDDLE over existing structured communication is insufficiently established - [1] (especially Section 6.4) evaluates an Exchange-then-Decide protocol: agents exchange relevant facts and challenge the leading option, then summarize evidence and uncertainty before voting. This establishes a directly relevant precedent for staged evidence sharing. HUDDLE adds a more detailed assessment procedure and coordination of camera movements, which may be valuable. However, comparison with open deliberation does not isolate their benefit beyond this established approach. Appendix C also documents different movement-execution rules, while comparable inference costs are not established. An adapted structured baseline with comparable action rules and inference resources, together with a targeted ablation of dependency planning or movement coordination, would clarify what specifically accounts for the improvement. This matters because the current evidence supports the complete intervention more strongly than its distinctive design choices.
  • The evaluation leaves uncertainty about generalization and whether every environment requires the intended evidence integration - HUDDLE is motivated by failures on the reported benchmark, but the paper does not describe an independent development and evaluation split. Its performance on the 36 instances therefore provides limited evidence for the broader task-agnostic claim (L416-419). This does not demonstrate overfitting, but it leaves transfer to unfamiliar evidence structures untested. There is also a specific concern about the living-room construction. Appendix D and Table 6 associate all HOLD cases with child presence and all PROCEED cases with child absence. Figure 1 assigns child-presence information to Robot 3 in the illustrated example. If this allocation holds throughout the environment, the recipient could predict the labels from that contextual fact alone. The label correlation is documented, while the complete per-instance information allocation needs clarification. Consequently, higher living-room accuracy does not by itself establish better integration of distributed visual evidence. Held-out variations with counterbalanced contextual facts would help resolve both concerns.

References

[1] Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs, ICML'26

Questions for authors
  • Could you compare HUDDLE with an adapted Exchange-then-Decide protocol from [1], using comparable inference resources and camera-action rules? Please report total model calls, token usage, and permitted movement opportunities so that the comparison can be interpreted.
  • Which benchmark instances and model runs informed protocol and prompt development? Were any evaluation instances withheld from this process? Results on held-out variations with different evidence dependencies would materially strengthen the task-agnostic claim.
  • Across all living-room instances, which robot receives the child-presence information? Given the correlation between this fact and the labels in Table 6, what prevents a local-context-only decision rule from solving this environment?
  • In the bedroom, how much improvement remains when the dependency-planning field or announce-and-review movement mechanism is removed while other conditions are held fixed? A targeted comparison would help distinguish the benefit of structured discussion from that of movement coordination.

References

[1] Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs, ICML'26

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions and new results (to all reviewers)

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF with changes highlighted in blue. Several reviewers asked for the same experiments, so we summarize them here (G1–G6) and refer to them in the individual replies. Values marked "Gemini / Sonnet" are Gemini 3.5 Flash-Lite / Sonnet 4.6. Solved is the fraction of runs in which the team reaches the correct decision with all required evidence.

G1. New hospital environment (Sec. 3.1, 5.2, App. E.4). We added a fourth, more complex environment that played no role in designing HUDDLE. Robots must decide whether a patient can safely enter an MRI scanner, which requires four safety checks (implant, oxygen cylinder, infusion pump, skin patch). Robots navigate between rooms, have different access rights, and encounter decoy records. It has easy and hard versions (floor plan in the reply below). The table below shows the accuracy gain (in percentage points) of HUDDLE + Safeguard over Open Deliberation (OD) for each model in the hospital:

Sonnet 4.6GPT-5.6 lunaGemini 3.5 FLQwen3.8 27bGrok 4.3GPT-6 Astra*Gemini 4 Argon*
+4+8+8+17+0+8+17

The two new frontier models (Astra and Argon) use the hard version, following our earlier proposal; the other models use the easy version. As explained in Sec. 5.3, we evaluate each model on the version where communication causes an accuracy drop.

G2. When does a protocol help? Central Controller (Sec. 5.3, App. A). A Central Controller is a single model that receives all robots' views and private facts and chooses all camera moves, so evidence must still be gathered, but nothing is communicated. Its gap to OD measures the cost of communication. The gap vanishes when a task is too easy for a model or too hard (even the Central Controller fails); HUDDLE targets the gap in between, so we evaluate each model in the hospital version where this gap exists. The table below shows the fraction of runs solved in the hospital under each method:

ModelODHUDDLE + SGCentral Controller
Gemini 4 Argon0.710.961.00
GPT-6 Astra0.750.880.83
Sonnet 4.60.460.580.75
GPT-5.6 luna0.170.420.58
Qwen3.8 27b0.420.620.67
Gemini 3.5 FL0.420.670.67

G3. Stage ablation (App. B.1). In the bedroom, we removed one component at a time with the number of rounds fixed: Grounding & Share (describe observations, share private facts), Assess (identify missing evidence and plan how to obtain it), and announce-and-review (announce camera moves and wait for review). The table below shows the accuracy and solved rate of each variant (Gemini / Sonnet):

VariantAccuracySolved
HUDDLE + SG (full)0.70 / 0.850.42 / 0.58
w/o Grounding & Share0.50 / 0.790.08 / 0.25
w/o Assess0.67 / 0.620.17 / 0.21
w/o announce-and-review0.75 / 0.670.33 / 0.38
OD + announce-and-review0.50 / 0.670.04 / 0.08

Removing any component reduces the solved rate for both models, indicating that the gains do not come from any single component. The most important component differs by model: Grounding & Share (information pooling) for Gemini and Assess (planning) for Sonnet.

G4. Inference cost (App. B.3, Table 3). In the kitchen and living room, HUDDLE uses fewer model calls than OD (24–27 vs. 30), with comparable tokens, and reach much higher performance (please refer to Tab3 of revised PDF) . The table below shows the model calls, tokens, and accuracy of each method in the bedroom, where HUDDLE is most expensive (Gemini / Sonnet):

MethodCallsTokens in/out (Gemini)Accuracy
OD, 5–20 rounds18–6354–313k / 21–43k0.46–0.50 / 0.43–0.58
OD, 17 rounds + announce54362k / 55k0.50 / 0.67
HUDDLE + SG54344k / 66k0.70 / 0.85

G5. Exchange-then-Decide (ETD) baseline (App. B.2). We adapted ETD (Li et al., 2026), also with HUDDLE's camera rule. The table below shows the accuracy range across three models (Sonnet 4.6, GPT-5.6 luna, Gemini 3.5 Flash-Lite):

ODETDETD + announceHUDDLE + SG
Kitchen0.71–0.780.54–0.71n/a0.77–0.96
Living room0.55–0.650.67–0.75n/a0.83–0.95
Bedroom0.43–0.490.54–0.580.58–0.670.62–0.85

ETD stays below HUDDLE almost everywhere.

G6. New living-room instances. We added 8 instances, all with a child present (4 HOLD, 4 PROCEED). The table below shows each model's accuracy on these new instances, from OD to HUDDLE + Safeguard:

Sonnet 4.6GPT-5.6 lunaGemini 3.5 FLGrok 4.3Qwen3.8 27b
0.50 → 0.880.56 → 0.690.50 → 0.940.56 → 0.750.31 → 0.69
AuthorsOpenReview ↗

G1 (continued). Hospital layout

The floor plan below shows the hospital environment. All three robots start in the west hallway and must move between rooms to gather evidence.

                                                          ┌──────────────────┐
 ┌──────────────────┐                                     │ SCANNER (Zone IV)│
 │ WARD 4B          │   ┌────────────┐       ┌────────────┤ R3 only          │
 │ bed 12: chart    │   │ RECORDS    │       │ CONTROL    │ magnet           │
 │ trolley: transfer│   │ patient    │       │ Zone III   │ red line 200 G   │
 │ sheet, O2 tag,   │   │ folders    │       │ console:   │ yellow line 100 G│
 │ pump label       │   │ R2 badge   │       │ scan mode,   spots P1-P4      │
 │                  │   │            │       │ notice       on 2 walls       │
 │                  │   │            │       │            │                  │
┌┴────────────   ───┴───┴──   ───────┴───────┴──   ───────┴────────┬─────────┘
│  hall west               hall middle         hall east           │
│  R1 R2 R3 start here                                             │
└────────────────────────┬──   ───────┬──────┬──   ─────────┬──────┘
                         │ ARCHIVE    │      │ EQUIPMENT    │
                         │ X-rays,    │      │ O2 swap log, │
                         │ floor plan │      │ O2 register, │
                         │ of spots   │      │ binder of    │
                         └────────────┘      │ cond. sheets │
                                             └──────────────┘

The table below shows the evidence in each room and which robots can use it:

RoomEvidence thereWho can use it
Wardbed chart (implant card), transfer sheet, O2 cylinder tag, pump labelall
Recordspatient folders (surgery notes)R2 only (badge)
ArchiveX-rays, floor plan of parking spotsall
Equipment baycylinder swap log and register, condition-sheet binderall; sheets need hands (R1, R2)
Control (Zone III)console: field strength, scan mode, suite noticeall
Scanner (Zone IV)gauss lines, parking spots P1–P4R3 only, through the control room

Each action (entering or leaving a room, moving to the next hallway segment, inspecting an item, or opening a sheet) costs one move. R1 has 6 moves, R2 has 7, and R3 has 3 (4 in the easy version).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.