- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how robot teams make safety-related decisions when the required information is distributed across their visual observations and private contextual knowledge. The authors introduce three household environments, analyze failures of ordinary deliberation, and propose HUDDLE, a staged protocol for sharing and evaluating evidence before reaching a decision. The evaluation shows improvements across several models and communication baselines.
Critical strengths
I find the distinction between individual reasoning ability and collective information handling useful. The full-information experiment provides a clear motivation for studying communication failures rather than treating all errors as limitations of the underlying models. The proposed protocol follows naturally from the failure analysis. Separating evidence sharing from assessment and decision-making is a reasonable response to the observed tendency to omit or dismiss relevant information. The evaluation includes multiple models and several straightforward alternatives, including changes to prompting, topology, discussion length, and aggregation. Reporting HUDDLE with and without the safeguard is also helpful.
Critical weaknesses
My main concern is that the experiments do not fully isolate what makes HUDDLE effective. The protocol changes communication structure, voting timing, assessment requirements, and action execution together. This is particularly important in the bedroom setting: the baseline can immediately consume its inspection budget, whereas HUDDLE delays movement and introduces action announcement and review. The improvement therefore cannot be attributed entirely to better deliberation. A baseline with the same action-control mechanism would make the comparison more convincing. The benchmark is also small, with 12 instances per environment. Repeated trials help establish consistency on these instances, but provide limited evidence about generalization to different scenes or evidence dependencies. It is unclear how much the protocol was developed around the evaluated instances. Finally, I would like a clearer account of the safety trade-off. Overall accuracy does not distinguish an unsafe PROCEED decision from an unnecessary HOLD. This distinction matters for interpreting both the safeguard and the broader safety motivation.
Questions for authors
- How does ordinary deliberation perform when given the same action announcement/review mechanism and a comparable inference budget?
- Were any instances or evidence configurations held out during protocol development?
- Can the authors report incorrect PROCEED and incorrect HOLD rates separately, with and without the safeguard?
Summary of revisions and new results (to all reviewers)
We thank all reviewers for their constructive feedback. We have uploaded a revised PDF with changes highlighted in blue. Several reviewers asked for the same experiments, so we summarize them here (G1–G6) and refer to them in the individual replies. Values marked "Gemini / Sonnet" are Gemini 3.5 Flash-Lite / Sonnet 4.6. Solved is the fraction of runs in which the team reaches the correct decision with all required evidence.
G1. New hospital environment (Sec. 3.1, 5.2, App. E.4). We added a fourth, more complex environment that played no role in designing HUDDLE. Robots must decide whether a patient can safely enter an MRI scanner, which requires four safety checks (implant, oxygen cylinder, infusion pump, skin patch). Robots navigate between rooms, have different access rights, and encounter decoy records. It has easy and hard versions (floor plan in the reply below). The table below shows the accuracy gain (in percentage points) of HUDDLE + Safeguard over Open Deliberation (OD) for each model in the hospital:
The two new frontier models (Astra and Argon) use the hard version, following our earlier proposal; the other models use the easy version. As explained in Sec. 5.3, we evaluate each model on the version where communication causes an accuracy drop.
G2. When does a protocol help? Central Controller (Sec. 5.3, App. A). A Central Controller is a single model that receives all robots' views and private facts and chooses all camera moves, so evidence must still be gathered, but nothing is communicated. Its gap to OD measures the cost of communication. The gap vanishes when a task is too easy for a model or too hard (even the Central Controller fails); HUDDLE targets the gap in between, so we evaluate each model in the hospital version where this gap exists. The table below shows the fraction of runs solved in the hospital under each method:
G3. Stage ablation (App. B.1). In the bedroom, we removed one component at a time with the number of rounds fixed: Grounding & Share (describe observations, share private facts), Assess (identify missing evidence and plan how to obtain it), and announce-and-review (announce camera moves and wait for review). The table below shows the accuracy and solved rate of each variant (Gemini / Sonnet):
Removing any component reduces the solved rate for both models, indicating that the gains do not come from any single component. The most important component differs by model: Grounding & Share (information pooling) for Gemini and Assess (planning) for Sonnet.
G4. Inference cost (App. B.3, Table 3). In the kitchen and living room, HUDDLE uses fewer model calls than OD (24–27 vs. 30), with comparable tokens, and reach much higher performance (please refer to Tab3 of revised PDF) . The table below shows the model calls, tokens, and accuracy of each method in the bedroom, where HUDDLE is most expensive (Gemini / Sonnet):
G5. Exchange-then-Decide (ETD) baseline (App. B.2). We adapted ETD (Li et al., 2026), also with HUDDLE's camera rule. The table below shows the accuracy range across three models (Sonnet 4.6, GPT-5.6 luna, Gemini 3.5 Flash-Lite):
ETD stays below HUDDLE almost everywhere.
G6. New living-room instances. We added 8 instances, all with a child present (4 HOLD, 4 PROCEED). The table below shows each model's accuracy on these new instances, from OD to HUDDLE + Safeguard: