- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a controlled benchmark for embodied multi-agent collaboration under within-episode task revisions. Building on Habitat 3.0 / PARTNR, 50 evaluation tasks carry preconstructed revision recipes (ADD / DELETE / MODIFY / REORDER, ≤3 progress-triggered events each) that synchronously update the natural-language instruction and the executable evaluator while preserving physical continuity and valid evaluation histories. Three conditions share the same initial state and intended final requirements: ORIGINAL-STATIC, FINAL-UPFRONT, and ONLINE-REVISED. Across nine LLM backbones under the no-negotiation protocol, mean SR falls 73.11% → 46.33%; crucially, 78.8% of this loss (21.11 pp) persists under FINAL-UPFRONT, leaving a 5.67 pp online gap that appears on every backbone and concentrates in object-state and spatial-adjacency tasks. Using explicit allocation dialogue as a coordination probe on four backbones, broader reallocation (Full) does not beat limited repair (Partial), and two experience-based Dynamic configurations beat both (+4.38 pp over Full) with lower action-planning token usage. The paper concludes that negotiation benefits are context-dependent—governed by compatibility of new requirements with existing allocations and goal dependencies—rather than by overall task difficulty.
Critical strengths
S1. The matched-disclosure design is the paper’s core contribution and it is genuinely clean. Holding the intended final specification fixed while varying only when requirements are disclosed (upfront vs. online) is exactly the controlled contrast needed to separate final-task expansion from online adaptation. The 78.8% attribution is a striking, policy-relevant finding: most of the “cost of revisions” is not the revision process at all.
S2. Careful executable-evaluator engineering. Revisions jointly update instruction and evaluation specification; the six-step history-migration operator (Appendix A.2) preserves first-satisfaction indices, remaps constraint references, prevents backdating, and handles terminal-stage deletion by searching surviving predecessors. Structural checks over 422 specification versions, independent semantic review by two LLMs plus human review, and manual execution validation under both successive-revision and upfront-final conditions (A.4) is well above benchmark-construction norms.
S3. Broad backbone coverage with honest aggregation. Nine backbones (open and proprietary) with model-equal macro-averages; the online gap’s uniformity across all nine backbones is itself evidence of robustness. Effort accounting separates simulator steps, planner calls, and action-planning tokens, and the finding that online revision raises simulator steps relatively more than planner calls (Table 23) is a nice, non-obvious observation.
S4. Honest limitation-scoping throughout. The paper repeatedly flags its own confounds: subgroup comparisons are “descriptive and do not isolate” constraint types (L282–283); protocol comparisons bundle negotiation rules with execution prompts (E.4); budget-limited episodes are scored under intermediate specifications (B.4); two agents with fixed capability profiles in simulation (Conclusion). This discipline makes the main claims easy to trust at their stated scope.
Critical weaknesses
W1. Two-agent, two-role setting limits external validity of RQ3. All negotiation conclusions derive from one team configuration (simulated human + Spot) with a single capability axis (human-only state skills). The paper’s own framing—allocation compatibility with existing commitments—predicts that dynamics change qualitatively with 3+ agents, duplicate capabilities, or multiple capability axes. This is acknowledged, but Section 5’s title-level claims (“when negotiation helps”) outrun what one team topology can establish.
W2. Narrow revision operator space and trajectory-dependent arrival timing. Four operators, ≤3 events per task, and no operator that revokes completed physical work (e.g., “return the object to its original location”), which is a common real-world revision class with the strongest allocation-repair demands. Moreover, because realized injection times depend on each model’s execution trajectory (Eq. 16), the FINAL-UPFRONT vs. ONLINE-REVISED contrast averages over model-dependent arrival stages: the 5.67 pp online gap is a protocol-level effect, not a per-revision causal estimate. The per-revision decomposition machinery (Eq. 26–27) exists but is not used to report gap-vs-arrival-stage structure.
W3. No variance estimates anywhere in the main results. Two runs per model-condition (four for DeepSeek Never/Full), ~100 episodes per cell (SR granularity = 1 pp), and headline numbers (21.11 / 5.67 / 4.38 pp) reported without SEs, CIs, or paired tests. Given LLM planning stochasticity, conclusions such as “Dynamic-skip and Dynamic-max both achieve 62.50%” cannot be assessed for significance. The three-run protocol used in contemporaneous embodied benchmarks is feasible here.
W4. Negotiation comparisons are protocol bundles, not scope ablations. E.4 concedes that each mode bundles negotiation rules with execution prompts (“performance differences cannot be attributed solely to allocation scope”). Full vs. Partial differ not only in reallocation scope but also in reviewer constraints (frozen-entry R1/R2 rules, blocking-error lists). A scope-only ablation—identical prompts, varying only the frozen set—would isolate the claimed mechanism (“plan stability”) behind Partial’s advantage.
W5. Dynamic’s advantage is difficult to attribute. Experience banks are backbone-specific (1.3–1.4k records), collected under approximately balanced mode schedules; retrieval depth, five field weights, penalty weights (ωt, ωn), and resolution rules are fixed a priori but no sensitivity analysis is provided. Mode-selection overhead (embedding + retrieval) is excluded from reported token costs yet is a real cost of Dynamic. Without these ablations, the “experience-based selection works” conclusion rests on a single configuration choice set.
W6. Incomplete engagement with recent work on hierarchical decomposition and communication-conditioned embodied agents. The paper’s central object—allocating heterogeneous-capability work under evolving requirements, and deciding when explicit dialogue pays off—is adjacent to two recent lines that go uncited. First, hierarchical decomposition of embodied reasoning, in which spatial-affordance reasoning is distributed across modules of different competence and the allocation of “who resolves what” is itself the object of study (A0, Xu et al., 2025) — this mirrors, at the intra-system level, the allocation questions posed here at the inter-agent level, and would sharpen the paper’s claim to novelty for “capability-constrained work allocation.” Second, systematic frameworks for versatile LLM-driven embodied agents that explicitly condition communication and collaboration on task context and discuss when such overhead is warranted (P³, Zhou et al., 2025) — directly relevant to RQ3’s framing, beyond the single dialogue-diagnosis citation (Dongre & Hakkani-Tür, 2026) currently carrying that argument. Third, A1 (Zhang et al., CVPR 2026) treats adaptive action-generation truncation, relevant to the paper’s planner-call and token-effort accounting in Table 4. Engaging these lines would position the benchmark as the controlled counterpart to design-level proposals rather than appearing to start the conversation from PARTNR.
Missing references (should be added) Rongtao Xu, Jian Zhang, Minghao Guo, Youpeng Wen, Haoting Yang, Min Lin, Jianzheng Huang, Zhe Li, Kaidong Zhang, Liqiong Wang, Yuxuan Kuang, Meng Cao, Feng Zheng, Xiaodan Liang. A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation. arXiv:2504.12636, 2025. Zhou et al. P 3: Toward Versatile Embodied Agents. 2025. Kaidong Zhang, Jian Zhang, Rongtao Xu, Yu Sun, Youpeng Wen, et al. A1: Adaptive Truncated Vision-Language-Action Model from Affordance to Action. CVPR 2026. Yukun Song, Changwei Wang, Xingtian Pei, Shibiao Xu, …, Rongtao Xu, et al. DialogueVPR: Towards Conversational Visual Place Recognition. arXiv:2607.14115, 2026. (Tangential, but its dialogue-driven interactive reasoning paradigm is methodologically adjacent to the paper’s use of dialogue as a coordination probe.)
Questions for authors
Can you report run-level SEs or bootstrap CIs for Table 2 and Table 4, and per-backbone paired tests for the B−C gap? Is the Dynamic-skip vs. Dynamic-max equivalence distinguishable from noise? Would a scope-only ablation (same execution prompts for Full and Partial, frozen set toggled) reproduce the Full ≤ Partial ordering? What fraction of episodes are budget-limited in each condition, and how much of the B−C gap survives when restricted to episodes where all scheduled events were applied before termination (removing the intermediate-specification scoring asymmetry)? How sensitive are Dynamic’s gains to ωt, ωn, and retrieval depth k, and how often do the two agents’ recommendations disagree? Any preliminary evidence with a third agent or with duplicated capabilities that would indicate whether the stratum-level conclusions (Table 5) generalize beyond the current capability axis?
6 comments
Response to Reviewer cA45 on Revision Arrival, Budget Exhaustion, and Scoring
W2 / Q4 | Can revisions affect completed work? How do revision arrival stage and budget exhaustion affect the online gap?
The evaluation set contains 91 prespecified revision events comprising 122 atomic operations, 60 of which are MODIFY operations. Table 12 in the appendix illustrates how a revision can require physical rework. The original task requires the basketball and the carrying case to end up next to each other. The first revision also requires the carrying case to be clean. If the case has already been placed, the team may need to retrieve it, clean it, and place it back next to the basketball. Thank you for suggesting the example of returning an object to its original location. One could cancel the original placement requirement and make the object’s initial location the new goal. This would allow us to test when the original assignee should perform the rework and when transferring it to a teammate improves completion of the task as a whole.
All models use the same prespecified progress thresholds for a given task. The physical state and remaining budget when a revision takes effect depend on the execution trajectory. Equations (26)–(27) in Appendix B.4 distinguish the immediate change in percent complete (PC) caused by a specification update from subsequent execution progress under a fixed specification. Following your suggestion, we grouped Online-Revised (C) episodes on the 46 tasks with prespecified revisions by the PC recorded under the preceding specification immediately before the first revision took effect. We matched each C episode to Final-Upfront (B) results for the same model and task.
| First revision arrival | C episodes | Distinct tasks | Matched B SR (%) | C SR (%) | B−C in pp [95% CI] |
|---|---|---|---|---|---|
| Early (PC < 50%) | 352 | 14 | 49.69 | 46.48 | 3.21 [−2.97, 9.21] |
| Later (PC ≥ 50%) | 782 | 35 | 69.37 | 65.81 | 3.56 [−0.40, 7.31] |
| No revision triggered | 154 | 36 | 45.33 | 0.00 | 45.33 [31.13, 62.36] |
Each group weights the 14 models equally. A task can appear in more than one group because its trigger history can differ across episodes. The B−C gap was 0.35 pp larger in the later group than in the early group (later minus early, 95% CI [−6.90, 7.65], based on 20,000 paired cluster bootstrap resamples by task ID).
The budget analysis covered all 50 tasks and included 1,400 episodes each for Original-Static (A), B, and C. Based on the recorded simulator-step count, 10.36% of A episodes (145), 18.36% of B episodes (257), and 23.86% of C episodes (334) reached the 20,000-step budget. Of the 334 C episodes that reached the budget, 159 ended with no revisions pending and 175 ended with revisions pending. The latter accounted for 12.50% of all C episodes. A further 104 C episodes ended with revisions pending before reaching the budget. In total, 279 C episodes (19.93%) ended before all scheduled revisions took effect. Of these, 154 triggered no revisions and 125 triggered some of their scheduled revisions.
Thank you for suggesting that we test whether the advantage of upfront disclosure persists when both conditions are scored against the complete final specification. An episode with revisions still pending cannot succeed, so SR applies the same success criterion to B and C; scoring under an intermediate specification affects only PC. Retaining only the C episodes in which all revisions took effect would remove 279 C episodes, all of them unsuccessful, without any corresponding removal from B, leaving the two conditions unbalanced. We therefore restricted the comparison to the 46 tasks with prespecified revisions, a property fixed before execution, and retained all runs on these tasks. On this subset, SR was 61.80% under B and 55.67% under C, a B−C gap of 6.13 pp (95% CI [2.72, 9.39]).
Response to Reviewer cA45 on Uncertainty Estimates and Allocation Scope
W3 / Q1, Q2 | How uncertain are the main performance gaps? Are the two Dynamic configurations equivalent?
Thank you for suggesting that we quantify uncertainty from task sampling and variation across repeated runs separately. Your question about the identical mean SR of 62.50% prompted us to estimate the paired difference between the two rules on the same tasks and report its confidence interval.
We used all 50 tasks and the original repeated runs from Tables 2 and 4. For each model, task, and condition or protocol, we first averaged over repeated runs. We then computed SR differences between conditions or protocols, weighting tasks and models equally. The 95% confidence intervals in Table R1 come from 20,000 paired cluster bootstrap resamples by task ID. Each resample keeps all models and conditions for a given task together.
Table R1. Mean SR differences and task-paired 95% confidence intervals, in percentage points (pp). A = Original-Static, B = Final-Upfront, and C = Online-Revised.
| Comparison | Models | Mean SR difference (pp) | Task-paired 95% CI (pp) |
|---|---|---|---|
| A − B | 14 | 16.14 | [10.79, 21.86] |
| B − C | 14 | 5.36 | [2.21, 8.43] |
| Dynamic-skip − Full | 4 | 4.38 | [−1.50, 10.25] |
| Dynamic-max − Full | 4 | 4.38 | [−0.50, 9.38] |
| Dynamic-skip − Dynamic-max | 4 | 0.00 | [−4.50, 4.75] |
Under Never, final-task expansion was associated with a 16.14 pp decline in mean SR, and online disclosure was associated with a further 5.36 pp decline. Both task-paired intervals lie above zero. At your suggestion, we also examined the online gap for each model. The B−C difference was positive for all 14 models and ranged from 2 to 10 pp. Across the SR entries in Tables 2 and 4, run-level standard errors ranged from 0 to 6.00 pp.
Both Dynamic configurations achieved higher SR than Full on each of the four models, with a mean gain of 4.38 pp for each configuration. Dynamic-skip and Dynamic-max both achieved a mean SR of 62.50%, but their ordering varied by model. Dynamic-max led by 6 pp on Gemma-4-31B, and Dynamic-skip led by 6 pp on GPT-OSS-20B. The mean difference between the configurations was 0.00 pp (Dynamic-skip minus Dynamic-max, 95% CI [−4.50, 4.75]). We could not distinguish their overall SRs on these data and therefore do not rank the two rules on this basis. Across the four negotiation models, run-level standard errors ranged from 0 to 2.00 pp for Dynamic-skip and from 0 to 3.00 pp for Dynamic-max.
W4 / Q3 | Does changing only the scope of reallocation reproduce Partial’s advantage over Full?
Thank you for drawing this distinction. We agree that varying only which assignments are frozen would isolate the contribution of reallocation scope. The existing experiments compare the two protocols as implemented. Full and Partial use the same proposal–review procedure. Partial also specifies which existing assignments are frozen by default, when R1 or R2 permits their release, and how existing assignments carry over into execution. In Table 5, Partial’s mean SR was 1.13 pp higher than Full’s, and its mean action-planning token use was 16.3% lower. Partial used fewer planning calls and fewer action-planning tokens on all four models. Three of the four also achieved higher SR and PC under Partial.
The control you suggest can use the empty frozen-set rule already present in the Partial template. Both arms would use the same prior plan, action history, proposal and review prompts, execution prompts, and budget. One arm would receive the frozen set specified by Partial and the other an empty set. The empty set permits reconsideration of all known unfinished work while preserving information about prior assignments. Alongside final SR, the comparison could measure how many assignments unaffected by the revision were changed and how often objects had to be moved again after those changes.
Appendix E.4 explains that the comparisons are between complete protocol configurations. Table 5 reports Partial’s advantage over Full in task completion and planning effort at that level. In response to your suggestion, we have added the design of this control to Appendix E.3 so that the effect of the frozen set can be tested directly on the benchmark.
Response to Reviewer cA45 on Team Configuration and the Single-Agent Control
Thank you for your valuable and well-articulated comments. We provide our point-by-point replies below, and we hope these responses adequately address your questions. We would welcome further discussion on how to strengthen the paper.
W1 / Q6 | What do the two-agent experiments tell us about when negotiation helps? Is there evidence from three-agent teams or teams with overlapping skills?
Section 5 examines which task structures benefit more from negotiation when the simulated human and Spot have fixed skill sets. On tasks with object-state requirements, all four negotiation configurations yield positive mean success-rate (SR) gains over Never (Table 6). On tasks with spatial-adjacency constraints, Full and Partial improve mean SR over Never by 5.25 and 7.63 percentage points (pp), respectively. On tasks without those constraints, their mean SRs fall below Never by 1.98 and 1.58 pp, respectively. These strata are defined by the intended final task specification. Determining whether a particular revision invalidates an existing allocation also requires knowing who holds the object when the revision arrives and who is responsible for the relevant operation.
We added a single-agent control using the same 50 tasks and 14 models. We retained the LLM-controlled simulated human, disabled Spot’s planner, and used the same revision recipes, evaluator, and simulator-step budget. From Final-Upfront to Online-Revised, the single agent’s mean SR fell by 2.00 pp (task-paired 95% CI [−0.57, 4.57]). The corresponding decline for the heterogeneous two-agent team was 5.36 pp in Table 2 of the revised paper. The online gap was larger in the two-agent setting for 11 of the 14 models.
We agree that team composition matters. Your example of overlapping skills is particularly instructive. If Spot could clean an object it was carrying, adding a cleaning requirement would no longer require its teammate to take over that operation. The benefit of negotiation would then warrant a fresh comparison. Our negotiation experiments focus on the simulated human–Spot team. We have yet to evaluate teams of three or more agents or teams in which agents share these state-changing skills. We appreciate this concrete suggestion. In future work, we would like to use the same tasks and revision recipes to examine two changes separately: adding a teammate and changing how skills are distributed across teammates. For each change, we would test which tasks continue to benefit from negotiation.