acceptodds
Under review as a conference paper at ICLR 2027

When Does Negotiation Help? Benchmarking Embodied Multi-Agent Collaboration under Successive Task Revisions

Abstract

Successive task revisions require embodied teams to accommodate new requirements during execution, and performance losses may reflect both final-task expansion and online adaptation. Existing evaluations lack upfront and online conditions with matched final requirements, making these sources difficult to distinguish. We introduce a controlled benchmark of 50 evaluation tasks for LLM-driven heterogeneous teams, synchronizing task and evaluator revisions during continuous physical execution and providing both conditions with matched intended final requirements. Model-equal averages across nine backbones show that 78.8% of the success-rate loss from original to online-revised tasks is already present when final requirements are provided upfront, while online introduction is associated with an additional loss of 5.67%. This online gap occurs on every backbone and is larger in object-state and spatial-adjacency tasks; tasks with temporal constraints exhibit the largest expansion-related loss but no larger online gap. Using explicit allocation dialogue as a coordination probe across four backbones, we find negotiation gains similarly concentrated in the former two groups; broader reallocation does not necessarily improve completion. Two Dynamic configurations that select negotiation modes based on experience achieve higher success rates than full negotiation on all four backbones, each with a model-equal mean gain of 4.38 pp and lower mean action-planning token usage. These findings show that negotiation benefits are context-dependent and cannot be explained by overall task difficulty alone; the compatibility of new requirements with existing allocations and goal dependencies offers a perspective for interpreting these conditional benefits. Code and data are available at https://www.negobench.top/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Multi-Agent Collaboration, Successive Task Revisions, Negotiation, Task Allocation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_cA45OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a controlled benchmark for embodied multi-agent collaboration under within-episode task revisions. Building on Habitat 3.0 / PARTNR, 50 evaluation tasks carry preconstructed revision recipes (ADD / DELETE / MODIFY / REORDER, ≤3 progress-triggered events each) that synchronously update the natural-language instruction and the executable evaluator while preserving physical continuity and valid evaluation histories. Three conditions share the same initial state and intended final requirements: ORIGINAL-STATIC, FINAL-UPFRONT, and ONLINE-REVISED. Across nine LLM backbones under the no-negotiation protocol, mean SR falls 73.11% → 46.33%; crucially, 78.8% of this loss (21.11 pp) persists under FINAL-UPFRONT, leaving a 5.67 pp online gap that appears on every backbone and concentrates in object-state and spatial-adjacency tasks. Using explicit allocation dialogue as a coordination probe on four backbones, broader reallocation (Full) does not beat limited repair (Partial), and two experience-based Dynamic configurations beat both (+4.38 pp over Full) with lower action-planning token usage. The paper concludes that negotiation benefits are context-dependent—governed by compatibility of new requirements with existing allocations and goal dependencies—rather than by overall task difficulty.

Critical strengths

S1. The matched-disclosure design is the paper’s core contribution and it is genuinely clean. Holding the intended final specification fixed while varying only when requirements are disclosed (upfront vs. online) is exactly the controlled contrast needed to separate final-task expansion from online adaptation. The 78.8% attribution is a striking, policy-relevant finding: most of the “cost of revisions” is not the revision process at all.

S2. Careful executable-evaluator engineering. Revisions jointly update instruction and evaluation specification; the six-step history-migration operator (Appendix A.2) preserves first-satisfaction indices, remaps constraint references, prevents backdating, and handles terminal-stage deletion by searching surviving predecessors. Structural checks over 422 specification versions, independent semantic review by two LLMs plus human review, and manual execution validation under both successive-revision and upfront-final conditions (A.4) is well above benchmark-construction norms.

S3. Broad backbone coverage with honest aggregation. Nine backbones (open and proprietary) with model-equal macro-averages; the online gap’s uniformity across all nine backbones is itself evidence of robustness. Effort accounting separates simulator steps, planner calls, and action-planning tokens, and the finding that online revision raises simulator steps relatively more than planner calls (Table 23) is a nice, non-obvious observation.

S4. Honest limitation-scoping throughout. The paper repeatedly flags its own confounds: subgroup comparisons are “descriptive and do not isolate” constraint types (L282–283); protocol comparisons bundle negotiation rules with execution prompts (E.4); budget-limited episodes are scored under intermediate specifications (B.4); two agents with fixed capability profiles in simulation (Conclusion). This discipline makes the main claims easy to trust at their stated scope.

Critical weaknesses

W1. Two-agent, two-role setting limits external validity of RQ3. All negotiation conclusions derive from one team configuration (simulated human + Spot) with a single capability axis (human-only state skills). The paper’s own framing—allocation compatibility with existing commitments—predicts that dynamics change qualitatively with 3+ agents, duplicate capabilities, or multiple capability axes. This is acknowledged, but Section 5’s title-level claims (“when negotiation helps”) outrun what one team topology can establish.

W2. Narrow revision operator space and trajectory-dependent arrival timing. Four operators, ≤3 events per task, and no operator that revokes completed physical work (e.g., “return the object to its original location”), which is a common real-world revision class with the strongest allocation-repair demands. Moreover, because realized injection times depend on each model’s execution trajectory (Eq. 16), the FINAL-UPFRONT vs. ONLINE-REVISED contrast averages over model-dependent arrival stages: the 5.67 pp online gap is a protocol-level effect, not a per-revision causal estimate. The per-revision decomposition machinery (Eq. 26–27) exists but is not used to report gap-vs-arrival-stage structure.

W3. No variance estimates anywhere in the main results. Two runs per model-condition (four for DeepSeek Never/Full), ~100 episodes per cell (SR granularity = 1 pp), and headline numbers (21.11 / 5.67 / 4.38 pp) reported without SEs, CIs, or paired tests. Given LLM planning stochasticity, conclusions such as “Dynamic-skip and Dynamic-max both achieve 62.50%” cannot be assessed for significance. The three-run protocol used in contemporaneous embodied benchmarks is feasible here.

W4. Negotiation comparisons are protocol bundles, not scope ablations. E.4 concedes that each mode bundles negotiation rules with execution prompts (“performance differences cannot be attributed solely to allocation scope”). Full vs. Partial differ not only in reallocation scope but also in reviewer constraints (frozen-entry R1/R2 rules, blocking-error lists). A scope-only ablation—identical prompts, varying only the frozen set—would isolate the claimed mechanism (“plan stability”) behind Partial’s advantage.

W5. Dynamic’s advantage is difficult to attribute. Experience banks are backbone-specific (1.3–1.4k records), collected under approximately balanced mode schedules; retrieval depth, five field weights, penalty weights (ωt, ωn), and resolution rules are fixed a priori but no sensitivity analysis is provided. Mode-selection overhead (embedding + retrieval) is excluded from reported token costs yet is a real cost of Dynamic. Without these ablations, the “experience-based selection works” conclusion rests on a single configuration choice set.

W6. Incomplete engagement with recent work on hierarchical decomposition and communication-conditioned embodied agents. The paper’s central object—allocating heterogeneous-capability work under evolving requirements, and deciding when explicit dialogue pays off—is adjacent to two recent lines that go uncited. First, hierarchical decomposition of embodied reasoning, in which spatial-affordance reasoning is distributed across modules of different competence and the allocation of “who resolves what” is itself the object of study (A0, Xu et al., 2025) — this mirrors, at the intra-system level, the allocation questions posed here at the inter-agent level, and would sharpen the paper’s claim to novelty for “capability-constrained work allocation.” Second, systematic frameworks for versatile LLM-driven embodied agents that explicitly condition communication and collaboration on task context and discuss when such overhead is warranted (P³, Zhou et al., 2025) — directly relevant to RQ3’s framing, beyond the single dialogue-diagnosis citation (Dongre & Hakkani-Tür, 2026) currently carrying that argument. Third, A1 (Zhang et al., CVPR 2026) treats adaptive action-generation truncation, relevant to the paper’s planner-call and token-effort accounting in Table 4. Engaging these lines would position the benchmark as the controlled counterpart to design-level proposals rather than appearing to start the conversation from PARTNR.

Missing references (should be added) Rongtao Xu, Jian Zhang, Minghao Guo, Youpeng Wen, Haoting Yang, Min Lin, Jianzheng Huang, Zhe Li, Kaidong Zhang, Liqiong Wang, Yuxuan Kuang, Meng Cao, Feng Zheng, Xiaodan Liang. A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation. arXiv:2504.12636, 2025. Zhou et al. P 3: Toward Versatile Embodied Agents. 2025. Kaidong Zhang, Jian Zhang, Rongtao Xu, Yu Sun, Youpeng Wen, et al. A1: Adaptive Truncated Vision-Language-Action Model from Affordance to Action. CVPR 2026. Yukun Song, Changwei Wang, Xingtian Pei, Shibiao Xu, …, Rongtao Xu, et al. DialogueVPR: Towards Conversational Visual Place Recognition. arXiv:2607.14115, 2026. (Tangential, but its dialogue-driven interactive reasoning paradigm is methodologically adjacent to the paper’s use of dialogue as a coordination probe.)

Questions for authors

Can you report run-level SEs or bootstrap CIs for Table 2 and Table 4, and per-backbone paired tests for the B−C gap? Is the Dynamic-skip vs. Dynamic-max equivalence distinguishable from noise? Would a scope-only ablation (same execution prompts for Full and Partial, frozen set toggled) reproduce the Full ≤ Partial ordering? What fraction of episodes are budget-limited in each condition, and how much of the B−C gap survives when restricted to episodes where all scheduled events were applied before termination (removing the intermediate-specification scoring asymmetry)? How sensitive are Dynamic’s gains to ωt, ωn, and retrieval depth k, and how often do the two agents’ recommendations disagree? Any preliminary evidence with a third agent or with duplicated capabilities that would indicate whether the stratum-level conclusions (Table 5) generalize beyond the current capability axis?

6 comments
AuthorsOpenReview ↗

Response to Reviewer cA45 on Team Configuration and the Single-Agent Control

Thank you for your valuable and well-articulated comments. We provide our point-by-point replies below, and we hope these responses adequately address your questions. We would welcome further discussion on how to strengthen the paper.

W1 / Q6 | What do the two-agent experiments tell us about when negotiation helps? Is there evidence from three-agent teams or teams with overlapping skills?

Section 5 examines which task structures benefit more from negotiation when the simulated human and Spot have fixed skill sets. On tasks with object-state requirements, all four negotiation configurations yield positive mean success-rate (SR) gains over Never (Table 6). On tasks with spatial-adjacency constraints, Full and Partial improve mean SR over Never by 5.25 and 7.63 percentage points (pp), respectively. On tasks without those constraints, their mean SRs fall below Never by 1.98 and 1.58 pp, respectively. These strata are defined by the intended final task specification. Determining whether a particular revision invalidates an existing allocation also requires knowing who holds the object when the revision arrives and who is responsible for the relevant operation.

We added a single-agent control using the same 50 tasks and 14 models. We retained the LLM-controlled simulated human, disabled Spot’s planner, and used the same revision recipes, evaluator, and simulator-step budget. From Final-Upfront to Online-Revised, the single agent’s mean SR fell by 2.00 pp (task-paired 95% CI [−0.57, 4.57]). The corresponding decline for the heterogeneous two-agent team was 5.36 pp in Table 2 of the revised paper. The online gap was larger in the two-agent setting for 11 of the 14 models.

We agree that team composition matters. Your example of overlapping skills is particularly instructive. If Spot could clean an object it was carrying, adding a cleaning requirement would no longer require its teammate to take over that operation. The benefit of negotiation would then warrant a fresh comparison. Our negotiation experiments focus on the simulated human–Spot team. We have yet to evaluate teams of three or more agents or teams in which agents share these state-changing skills. We appreciate this concrete suggestion. In future work, we would like to use the same tasks and revision recipes to examine two changes separately: adding a teammate and changing how skills are distributed across teammates. For each change, we would test which tasks continue to benefit from negotiation.

AuthorsOpenReview ↗

Response to Reviewer cA45 on Revision Arrival, Budget Exhaustion, and Scoring

W2 / Q4 | Can revisions affect completed work? How do revision arrival stage and budget exhaustion affect the online gap?

The evaluation set contains 91 prespecified revision events comprising 122 atomic operations, 60 of which are MODIFY operations. Table 12 in the appendix illustrates how a revision can require physical rework. The original task requires the basketball and the carrying case to end up next to each other. The first revision also requires the carrying case to be clean. If the case has already been placed, the team may need to retrieve it, clean it, and place it back next to the basketball. Thank you for suggesting the example of returning an object to its original location. One could cancel the original placement requirement and make the object’s initial location the new goal. This would allow us to test when the original assignee should perform the rework and when transferring it to a teammate improves completion of the task as a whole.

All models use the same prespecified progress thresholds for a given task. The physical state and remaining budget when a revision takes effect depend on the execution trajectory. Equations (26)–(27) in Appendix B.4 distinguish the immediate change in percent complete (PC) caused by a specification update from subsequent execution progress under a fixed specification. Following your suggestion, we grouped Online-Revised (C) episodes on the 46 tasks with prespecified revisions by the PC recorded under the preceding specification immediately before the first revision took effect. We matched each C episode to Final-Upfront (B) results for the same model and task.

First revision arrivalC episodesDistinct tasksMatched B SR (%)C SR (%)B−C in pp [95% CI]
Early (PC < 50%)3521449.6946.483.21 [−2.97, 9.21]
Later (PC ≥ 50%)7823569.3765.813.56 [−0.40, 7.31]
No revision triggered1543645.330.0045.33 [31.13, 62.36]

Each group weights the 14 models equally. A task can appear in more than one group because its trigger history can differ across episodes. The B−C gap was 0.35 pp larger in the later group than in the early group (later minus early, 95% CI [−6.90, 7.65], based on 20,000 paired cluster bootstrap resamples by task ID).

The budget analysis covered all 50 tasks and included 1,400 episodes each for Original-Static (A), B, and C. Based on the recorded simulator-step count, 10.36% of A episodes (145), 18.36% of B episodes (257), and 23.86% of C episodes (334) reached the 20,000-step budget. Of the 334 C episodes that reached the budget, 159 ended with no revisions pending and 175 ended with revisions pending. The latter accounted for 12.50% of all C episodes. A further 104 C episodes ended with revisions pending before reaching the budget. In total, 279 C episodes (19.93%) ended before all scheduled revisions took effect. Of these, 154 triggered no revisions and 125 triggered some of their scheduled revisions.

Thank you for suggesting that we test whether the advantage of upfront disclosure persists when both conditions are scored against the complete final specification. An episode with revisions still pending cannot succeed, so SR applies the same success criterion to B and C; scoring under an intermediate specification affects only PC. Retaining only the C episodes in which all revisions took effect would remove 279 C episodes, all of them unsuccessful, without any corresponding removal from B, leaving the two conditions unbalanced. We therefore restricted the comparison to the 46 tasks with prespecified revisions, a property fixed before execution, and retained all runs on these tasks. On this subset, SR was 61.80% under B and 55.67% under C, a B−C gap of 6.13 pp (95% CI [2.72, 9.39]).

AuthorsOpenReview ↗

Response to Reviewer cA45 on Uncertainty Estimates and Allocation Scope

W3 / Q1, Q2 | How uncertain are the main performance gaps? Are the two Dynamic configurations equivalent?

Thank you for suggesting that we quantify uncertainty from task sampling and variation across repeated runs separately. Your question about the identical mean SR of 62.50% prompted us to estimate the paired difference between the two rules on the same tasks and report its confidence interval.

We used all 50 tasks and the original repeated runs from Tables 2 and 4. For each model, task, and condition or protocol, we first averaged over repeated runs. We then computed SR differences between conditions or protocols, weighting tasks and models equally. The 95% confidence intervals in Table R1 come from 20,000 paired cluster bootstrap resamples by task ID. Each resample keeps all models and conditions for a given task together.

Table R1. Mean SR differences and task-paired 95% confidence intervals, in percentage points (pp). A = Original-Static, B = Final-Upfront, and C = Online-Revised.

ComparisonModelsMean SR difference (pp)Task-paired 95% CI (pp)
A − B1416.14[10.79, 21.86]
B − C145.36[2.21, 8.43]
Dynamic-skip − Full44.38[−1.50, 10.25]
Dynamic-max − Full44.38[−0.50, 9.38]
Dynamic-skip − Dynamic-max40.00[−4.50, 4.75]

Under Never, final-task expansion was associated with a 16.14 pp decline in mean SR, and online disclosure was associated with a further 5.36 pp decline. Both task-paired intervals lie above zero. At your suggestion, we also examined the online gap for each model. The B−C difference was positive for all 14 models and ranged from 2 to 10 pp. Across the SR entries in Tables 2 and 4, run-level standard errors ranged from 0 to 6.00 pp.

Both Dynamic configurations achieved higher SR than Full on each of the four models, with a mean gain of 4.38 pp for each configuration. Dynamic-skip and Dynamic-max both achieved a mean SR of 62.50%, but their ordering varied by model. Dynamic-max led by 6 pp on Gemma-4-31B, and Dynamic-skip led by 6 pp on GPT-OSS-20B. The mean difference between the configurations was 0.00 pp (Dynamic-skip minus Dynamic-max, 95% CI [−4.50, 4.75]). We could not distinguish their overall SRs on these data and therefore do not rank the two rules on this basis. Across the four negotiation models, run-level standard errors ranged from 0 to 2.00 pp for Dynamic-skip and from 0 to 3.00 pp for Dynamic-max.

W4 / Q3 | Does changing only the scope of reallocation reproduce Partial’s advantage over Full?

Thank you for drawing this distinction. We agree that varying only which assignments are frozen would isolate the contribution of reallocation scope. The existing experiments compare the two protocols as implemented. Full and Partial use the same proposal–review procedure. Partial also specifies which existing assignments are frozen by default, when R1 or R2 permits their release, and how existing assignments carry over into execution. In Table 5, Partial’s mean SR was 1.13 pp higher than Full’s, and its mean action-planning token use was 16.3% lower. Partial used fewer planning calls and fewer action-planning tokens on all four models. Three of the four also achieved higher SR and PC under Partial.

The control you suggest can use the empty frozen-set rule already present in the Partial template. Both arms would use the same prior plan, action history, proposal and review prompts, execution prompts, and budget. One arm would receive the frozen set specified by Partial and the other an empty set. The empty set permits reconsideration of all known unfinished work while preserving information about prior assignments. Alongside final SR, the comparison could measure how many assignments unaffected by the revision were changed and how often objects had to be moved again after those changes.

Appendix E.4 explains that the comparisons are between complete protocol configurations. Table 5 reports Partial’s advantage over Full in task completion and planning effort at that level. In response to your suggestion, we have added the design of this control to Appendix E.3 so that the effect of the frozen set can be tested directly on the benchmark.

Reviewer_XXFDOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper constructs an embodied multi-agent benchmark for online task revision and uses three conditions to attempt to separate “final-task expansion” from “online adaptation burden”, with negotiation as a coordination probe. The problem is clearly motivated, the benchmark design is complete, and the history-aware evaluator is solid, but the core decomposition lacks theoretical grounding, the multi-agent factor is not isolated, and the title does not match the weight of the contributions.

Critical strengths
  1. The problem is meaningful. existing embodied benchmarks mostly assume fixed tasks, whereas this paper treats "online revision" as a manipulated variable.
  2. The history-aware evaluator is well designed: proposition additions/deletions, constraint migration, non-backdating, and terminal supplementation are formally described and validated.
  3. The data and pipeline are complete: 50 evaluation + 100 experience tasks, four revision operators, progress-triggered triggering, dual LLM semantic checks + human review.
Critical weaknesses
  1. What is the basis for the decomposition? Can it rule out other coupling factors?
  2. If "embodied multi-agent" is removed, do Contributions 1/2 still hold? The keywords of Contribution 1 are features of an "online revision benchmark", not features of multi-agent. Contribution 2 is model-level diagnosis, which can also be done in a single-agent or non-embodied online revision setting. Only Contribution 3 actually involves multi-agent allocation dialogue. If multi-agent is central, why do Contributions 1/2 not treat coordination demand, capability heterogeneity, and communication as experimental variables? The paper has no single-agent control, no homogeneous vs. heterogeneous capability comparison, no communication-disabled baseline, and no coordination-specific metrics.
  3. Why does negotiation cover only 4 backbones? Does the title match the content? The title is "When Does Negotiation Help?", but the main diagnosis in Section 4 uses 9 backbones + NEVER, and negotiation is covered only in Section 5 with 4 backbones; the contribution list also places negotiation third.
  4. Why SR and PC? Are coordination metrics needed?
  5. Is a single-agent control needed? How is the capability difference isolated? If a single agent also yields a similar A→B→C decomposition, then the residual burden is not specific to multi-agent coordination.
  6. What is the impact of the difference between the simulated human and a real human on the generalizability of the negotiation conclusions?
Questions for authors
  1. What is the basis for the decomposition? Can it rule out other coupling factors?
  2. If "embodied multi-agent" is removed, do Contributions 1/2 still hold?
  3. Is a single-agent control needed? How is the capability difference isolated? If a single agent also yields a similar Original-Static/Final-Upfront/Online-Revised decomposition, then the residual burden is not specific to multi-agent coordination.
6 comments
AuthorsOpenReview ↗

Response to Reviewer XXFD on the Three-Condition Decomposition and Single-Agent Control

We sincerely appreciate your insightful and detailed comments. Below, we present our point-by-point responses and hope that our clarifications address the issues you raised. We would welcome further discussion on how to strengthen the paper.

W1 / Q1 | What is the basis for the three-condition decomposition? Can it rule out other factors?

The decomposition uses Final-Upfront as a directly measured intermediate condition. Original-Static and Final-Upfront both provide all requirements at the start of the episode, so comparing them measures the effect of changing the task content under upfront disclosure. Final-Upfront and Online-Revised share the same intended final requirements. Comparing them measures the difference between providing those requirements before execution and introducing them progressively during execution.

Let A, B, and C denote Original-Static, Final-Upfront, and Online-Revised, respectively. A−B measures how much the success rate (SR) drops when moving from the original task to the final task, even when the team knows all requirements upfront. B and C use the same tasks, initial physical states, and intended final requirements. They also hold fixed the model, agent skills, observation interfaces, Never protocol, and simulator-step budget. B−C measures the SR gap between upfront and online disclosure under these conditions. The total loss can therefore be written as SR(A) − SR(C) = (SR(A) − SR(B)) + (SR(B) − SR(C)).

The two differences correspond to the change in task specification and the change in disclosure protocol, respectively. In Table 2, the equally weighted mean SRs across 14 models are 80.71%, 64.57%, and 59.21% for A, B, and C. Of the total decline of 21.50 percentage points (pp), 16.14 pp, or 75.1%, is already present when the final requirements are given upfront. The remaining 5.36 pp is associated with online disclosure, and this gap is positive for all 14 models.

The B−C comparison fixes task composition and the intended final specification while preserving the interaction between disclosure timing and the state reached during execution. The same new requirement may call for very different actions depending on whether it arrives before an object has been moved or after it has been placed. In C, execution progress triggers revisions. If an episode ends with revisions pending, the evaluator scores it against the specification active at termination. B−C therefore measures the overall gap between the two disclosure protocols under the given budget.

W2, W5 / Q2, Q3 | Do the first two contributions depend on multiple agents? What does a single-agent control establish?

Thank you for raising the question of a single-agent control. We conducted this experiment on the same 50 tasks with the same 14 models. The single agent achieved mean SRs of 83.29%, 72.29%, and 70.29% under Original-Static (A), Final-Upfront (B), and Online-Revised (C), respectively. Even when one agent performed the entire task, mean SR was 2.00 pp lower when requirements were introduced online than when they were given upfront. For comparison, the corresponding drop for the heterogeneous two-agent team in Table 2 of the revised paper was 5.36 pp, giving a difference of 3.36 pp.

Single-agent control. We retained the LLM-controlled simulated human from the original team and disabled Spot's planner. The human agent retained its original embodiment, initial state, and tool set and could independently navigate, transport objects, and perform operations such as cleaning and filling. We used the same tasks, original and final specifications, and revision recipes as in the two-agent experiment, with the same budget of 20,000 simulator steps per episode. Each model ran once per task per condition, yielding 700 episodes per condition and 2,100 across the three conditions.

As Table R1 in the next comment shows, single-agent mean SR fell by 11.00 pp from A to B and by a further 2.00 pp from B to C. The task-paired 95% confidence interval for the latter gap was [−0.57, 4.57] pp. The three-condition comparison and history-aware evaluation thus also apply to the single-agent setting you describe. In that setting, too, most of the observed mean SR decline from the original to the online-revised task was already present when the expanded requirements were given upfront.

AuthorsOpenReview ↗

Response to Reviewer XXFD on Single-Agent Results and the Comparison with Two-Agent Teams

Table R1. Single-agent task completion and execution effort. SR and percent complete (PC) are expressed as percentages. SR differences are in percentage points. The last three columns give per-episode means over successful and unsuccessful episodes.

ModelSR (A → B → C)PC (A → B → C)SR A−BSR B−CAction-planning tokens (A → B → C)Simulator steps (A → B → C)Planning calls (A → B → C)
GPT-4o mini68.00 → 36.00 → 36.0086.11 → 75.53 → 76.1332.000.0086,582 → 127,463 → 151,6933,165.42 → 4,925.48 → 4,946.6220.40 → 28.58 → 31.22
DeepSeek-v4-flash92.00 → 90.00 → 86.0096.53 → 97.21 → 97.042.004.0043,426 → 88,111 → 82,9802,577.62 → 4,324.56 → 4,461.3812.26 → 20.32 → 19.34
GPT-OSS-20B82.00 → 58.00 → 64.0093.29 → 85.21 → 85.2024.00−6.0072,211 → 140,344 → 153,5133,126.34 → 5,621.66 → 5,525.1216.76 → 27.34 → 29.24
Gemma-4-31B96.00 → 84.00 → 86.0098.53 → 96.77 → 96.9112.00−2.0040,253 → 84,068 → 98,3532,510.94 → 4,209.30 → 4,421.3811.18 → 18.62 → 20.68
Qwen3-30B-A3B76.00 → 44.00 → 36.0090.59 → 77.31 → 78.6132.008.00110,916 → 223,427 → 224,8663,333.04 → 6,102.50 → 4,858.1821.36 → 38.36 → 37.76
Devstral-2-123B90.00 → 76.00 → 74.0094.64 → 90.29 → 91.1414.002.0075,144 → 142,541 → 146,8422,776.04 → 4,536.28 → 4,748.9217.52 → 27.72 → 28.14
Llama-3.3-70B82.00 → 68.00 → 64.0092.93 → 89.84 → 89.3214.004.0080,201 → 149,839 → 146,2993,134.54 → 4,676.24 → 4,214.8817.52 → 27.92 → 27.56
GPT-OSS-120B94.00 → 86.00 → 80.0097.53 → 95.99 → 92.488.006.0048,430 → 122,691 → 124,4482,790.68 → 4,956.88 → 5,044.4213.04 → 25.34 → 25.90
GPT-4.1 nano12.00 → 2.00 → 4.0030.44 → 23.24 → 21.1410.00−2.00232,551 → 231,566 → 250,7997,349.64 → 9,394.84 → 8,334.9841.06 → 41.32 → 44.02
Claude Sonnet 5.596.00 → 90.00 → 90.0098.53 → 97.71 → 96.496.000.0042,204 → 89,502 → 87,0862,399.58 → 4,204.28 → 4,177.789.18 → 16.34 → 15.60
GPT-6-sol94.00 → 92.00 → 92.0097.87 → 98.33 → 98.412.000.0026,901 → 67,641 → 67,6402,387.98 → 4,640.72 → 4,549.069.16 → 18.36 → 18.22
GPT-6.1-sol94.00 → 96.00 → 86.0097.28 → 99.49 → 95.76−2.0010.0016,246 → 42,204 → 42,0902,368.78 → 4,075.58 → 4,251.405.96 → 12.94 → 12.66
GPT-6.1-terra94.00 → 96.00 → 94.0097.78 → 99.21 → 98.23−2.002.0018,566 → 42,214 → 47,0212,586.14 → 4,295.58 → 4,578.386.56 → 13.22 → 13.94
GPT-6-astra96.00 → 94.00 → 92.0098.53 → 98.66 → 97.742.002.0016,532 → 43,831 → 42,4782,437.36 → 4,203.18 → 4,198.046.06 → 13.12 → 12.64
Equal-weight mean (14 models)83.29 → 72.29 → 70.2990.76 → 87.48 → 86.7611.002.0065,012 → 113,960 → 119,0083,067.44 → 5,011.93 → 4,879.3214.86 → 23.54 → 24.07

From B to C, the single agent's mean action-planning token usage increased by 4.43%, planning calls increased by 2.25%, and simulator steps decreased by 2.65%. Online disclosure was therefore accompanied by more planning calls and higher action-planning token usage even when one agent performed all the work.

W2, W5 / Q2, Q3 | Single-agent and two-agent results on the same models and tasks

Table R2 compares each single-agent result with the corresponding two-agent result under Never in Table 2 of the revised paper. Both settings use the same 50 tasks. With equal weight given to each of the 14 models, the B−C gap was 3.36 pp larger for the two-agent team. The gap was larger in the two-agent setting for 11 models, the same for Qwen3-30B-A3B, and larger in the single-agent setting for DeepSeek-v4-flash and GPT-6.1-sol.

Table R2. Online SR gaps for the single agent and the two-agent team. The two-agent column uses the per-model results from Table 2 of the revised paper. All values are in percentage points.

ModelSingle-agent B−CTwo-agent B−C (Table 2)Two-agent gap minus single-agent gap
GPT-4o mini0.003.003.00
DeepSeek-v4-flash4.002.00−2.00
GPT-OSS-20B−6.003.009.00
Gemma-4-31B−2.004.006.00
Qwen3-30B-A3B8.008.000.00
Devstral-2-123B2.0010.008.00
Llama-3.3-70B4.008.004.00
GPT-OSS-120B6.0010.004.00
GPT-4.1 nano−2.003.005.00
Claude Sonnet 5.50.004.004.00
GPT-6-sol0.002.002.00
GPT-6.1-sol10.005.00−5.00
GPT-6.1-terra2.008.006.00
GPT-6-astra2.005.003.00
Equal-weight mean (14 models)2.005.363.36
AuthorsOpenReview ↗

Response to Reviewer XXFD on Revision Triggering and Negotiation Across Models

W2, W5 / Q2, Q3 | Episodes in which all revisions took effect

We also examined episodes in which all scheduled revisions took effect. In the runs used for this analysis, 109 of 700 single-agent online episodes (15.57%) and 279 of 1,400 two-agent online episodes (19.93%) ended with revisions pending. In 502 model–task combinations, all revisions took effect in the single-agent run and in both two-agent runs. On this common subset, B−C was −2.89 pp for the single agent and 0.59 pp for the team. The difference was 3.48 pp, with a task-paired 95% confidence interval of [−0.75, 7.69] pp. The point estimate on this subset, selected by realized execution progress, had the same sign as the full-sample estimate.

Multi-agent allocation and negotiation. The new control shows that an individual agent can also incur an online adaptation burden. On the same tasks, most models exhibited a larger online gap in the heterogeneous team. Section 5 directly varies a team-level factor by testing whether negotiation helps when a revision affects existing assignments. If Spot is carrying an object that must now be cleaned, for example, only the simulated human can clean it, so the agents must coordinate cleaning with subsequent transport. Under Never, both agents receive the updated instruction and continue planning independently. Full raised SR over Never by 5.25 pp on tasks with spatial-adjacency constraints and lowered it by 1.98 pp on tasks without them (Table 6). Never SR was slightly higher in the adjacency group (59.03%) than in the group without adjacency (56.65%). Explicit allocation dialogue thus yielded larger mean gains when the team had to maintain existing spatial relations between objects.

W3 | Why does the main negotiation study cover four models, and how does the title reflect the contributions?

Sections 4 and 5 use complementary comparisons. Section 4 fixes the Never protocol and expands the three-condition evaluation from 9 to 14 models to examine the SR gap between upfront and online disclosure. Section 5 fixes the Online-Revised condition and compares all five protocols on four models from distinct families: DeepSeek-v4-flash, Gemma-4-31B, GPT-OSS-20B, and Devstral-2-123B. The full comparison requires a separate experience bank for each model. The collection schedule in Appendix C.1 specifies 852 offline episodes per model on 100 tasks disjoint from the evaluation set. Each model's two Dynamic configurations share its bank.

To test whether the Section 5 findings also hold for frontier models, we evaluated GPT-6.1-sol and GPT-6-astra. Each model ran all five protocols twice on the same 50 Online-Revised tasks. The table below reports results for the two additional models. The Section 5 means refer to the original four.

ProtocolGPT-6.1-sol SRGPT-6.1-sol PCGPT-6-astra SRGPT-6-astra PC
Never82.00%95.64%86.00%96.52%
Full79.00%95.13%88.00%97.18%
Partial82.00%95.45%88.00%97.51%
Dynamic-skip85.00%96.53%91.00%97.84%
Dynamic-max88.00%97.10%91.00%98.08%

Switching from Never to Full lowered SR by 3 pp for GPT-6.1-sol and raised it by 2 pp for GPT-6-astra. Both Dynamic configurations achieved higher SR and PC than Full on both models. With the two models weighted equally, Dynamic-skip and Dynamic-max improved SR over Full by 4.50 and 6.00 pp, respectively, and reduced mean simulator steps by 14.51% and 17.25%, respectively. In the main four-model experiment, both Dynamic configurations also achieved higher SR than Full on every model, with an equal-weight mean gain of 4.38 pp for each configuration.

The task strata directly address the title question, When Does Negotiation Help? Consider object-state requirements. In Section 4, the online SR gap across 14 models was 6.09 pp on tasks with these requirements and 2.76 pp on tasks without them. Section 5 then tested whether explicit allocation dialogue helps teams handle such tasks. In the main four-model experiment, Full increased SR over Never by 3.03 pp on tasks with object-state requirements and decreased it by 7.93 pp on tasks without them. The two additional models showed the same pattern on average. With object-state requirements, Full and Partial improved mean SR over Never by 1.92 and 3.85 pp, respectively. Without them, both lowered mean SR by 9.09 pp.

The contributions follow this experimental sequence. The benchmark provides replayable revisions and continuous execution. Section 4 uses them to identify task structures with larger online performance gaps. Section 5 tests whether changing the allocation protocol improves completion on those tasks. The subtitle identifies the benchmark and its setting, and the main title states the question these experiments jointly address.

Reviewer_VQQCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies embodied multi-agent collaboration under successive task revisions, with the goal of separating performance degradation caused by increased final-task difficulty from the additional burden of adapting to requirements introduced during execution.

The main contribution is a controlled benchmark built on PARTNR with 50 evaluation tasks and three matched conditions: Original-Static, Final-Upfront, and Online-Revised. The comparison between Original-Static and Final-Upfront is intended to measure the effect of final-task expansion, while the comparison between Final-Upfront and Online-Revised measures the additional online adaptation burden. The benchmark synchronously updates natural-language instructions and executable evaluators while preserving physical state and evaluation history. The paper further uses several negotiation protocols, including fixed Full/Partial negotiation and experience-based Dynamic selection, to study when explicit coordination helps under task revisions.

For validation, the authors evaluate nine LLM backbones on the three benchmark conditions and analyze performance by temporal, object-state, and spatial-adjacency task structure. They report that most of the original-to-online success-rate degradation is already present under Final-Upfront, with an additional 5.67 percentage-point gap associated with online revision. They further evaluate negotiation strategies on four backbones and find that negotiation benefits depend on task structure, while the Dynamic variants outperform Full negotiation on average with lower reported action-planning cost.

Critical strengths

1. The paper identifies a meaningful evaluation gap.\textbf{1. The paper identifies a meaningful evaluation gap.} I find the distinction between final-task difficulty and online adaptation burden useful. If an agent performs worse after several requirements are added during execution, comparing only against the original task does not tell us whether the degradation comes from the harder final task or from having to adapt after actions have already been taken. The three-condition design provides a simple and interpretable way to study this distinction.

Critical weaknesses

1. Mismatch in Realized Final Tasks. \textbf{1. Mismatch in Realized Final Tasks. } The central comparison between Final-Upfront and Online-Revised does not always isolate online adaptation cleanly. In Online-Revised, task revisions are progress-triggered, and revisions whose thresholds are not reached are never applied. If an episode reaches the execution budget with pending revisions, it is evaluated under the currently active intermediate task specification, whereas Final-Upfront is always evaluated against the complete final specification. The paper itself notes that the two conditions therefore match intended final requirements but do not necessarily have identical realized terminal specifications.

This is consequential because the paper’s main claim relies on interpreting the Final-Upfront vs. Online-Revised performance gap as the additional cost of online adaptation. If the two conditions are sometimes evaluated on different realized tasks, part of this gap may reflect differences in the task being scored rather than adaptation itself.

Questions for authors

How often do Online-Revised episodes terminate before all scheduled revisions are triggered, and does the reported Final-Upfront vs. Online-Revised gap remain when restricting the comparison to episodes that realize the complete final task specification?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VQQC: Pending revisions and aligned success criteria

We are grateful for your careful review and helpful suggestions. Below, we provide our point-by-point responses, which we hope address your concerns. We would welcome further discussion that could refine the work.

W1 / Q1 | How many episodes terminate before all revisions are triggered, and how does the success-rate gap change when restricting to tasks with prespecified revisions?

In the revised 14-model evaluation, Online-Revised (C) contains 1,400 episodes. Of these, 279 terminate before all prespecified revisions are triggered, representing 19.93% of all episodes and 21.66% of the 1,288 episodes with prespecified revisions.

The main results use the model-equal aggregation reported in Table 2. Across all 50 tasks, Final-Upfront (B) and Online-Revised (C) achieve success rates of 64.57% and 59.21%, respectively, a gap of 5.36 percentage points. When the comparison is restricted to the 46 tasks with prespecified revisions, B and C achieve success rates of 61.80% and 55.67%, respectively, and the gap increases to 6.13 percentage points.

Comparison scopeB success rateC success rateB−C (percentage points)
All 50 tasks, 14 models64.57%59.21%5.36
46 tasks with prespecified revisions, all runs retained61.80%55.67%6.13

Scoring and revision triggering

As you note, B is scored against the complete final specification GK\mathcal{G}_K from the beginning of an episode. Automatic success termination for C requires all KK revision events to have been triggered and the current task to be successful. If an episode terminates with only j<Kj<K events triggered, it is scored against the specification active at termination, Gj\mathcal{G}_j (Section 3.3 and Appendix B.2–B.4).

All 279 of these episodes are unsuccessful. A revision is triggered when completion of the current specification reaches its threshold, and the threshold is no greater than 1. Thus, once the current specification is fully satisfied, the next revision takes effect at that action boundary. If an episode terminates with revisions still pending, the team has not completed the active specification Gj\mathcal{G}_j. Scoring the same episode against the complete final specification GK\mathcal{G}_K therefore also yields failure. The success criteria for B and C are consequently aligned, and rescoring under GK\mathcal{G}_K does not change the success status of any episode.

If we retained only episodes in which all revisions were triggered, all excluded episodes would be failures on the C side, making the two conditions imbalanced. We therefore compare all runs on the 46 tasks with prespecified revisions. Four of the 50 tasks have no scheduled revisions, and the corresponding 112 C episodes satisfy j=K=0j=K=0.

AuthorsOpenReview ↗

Response to Reviewer VQQC: Success rate and execution effort on tasks with revisions

Success rate and execution effort on tasks with prespecified revisions

On the 46 tasks with prespecified revisions, B and C each contain 1,288 episodes (92 per model per condition), and the task-paired bootstrap 95% confidence interval for the gap is [2.72, 9.39] percentage points. Per-model results are shown below.

ModelB SRC SRB−C (percentage points)Tokens (B → C)Steps (B → C)Planning calls (B → C)
GPT-4o mini25.00%21.74%3.26321,773 → 344,65213,417.24 → 14,082.7933.08 → 34.20
DeepSeek-v4-flash59.78%52.17%7.61300,435 → 318,0118,437.49 → 9,602.5127.45 → 28.59
GPT-OSS-20B54.35%51.09%3.26351,017 → 357,5219,990.13 → 10,655.3031.19 → 31.73
Gemma-4-31B63.04%58.70%4.35240,966 → 255,2999,071.51 → 9,221.6223.98 → 24.14
Qwen3-30B-A3B34.78%26.09%8.70410,316 → 430,20512,732.93 → 13,871.1835.90 → 37.30
Devstral-2-123B58.70%47.83%10.87294,699 → 288,1528,829.85 → 9,623.3427.38 → 26.84
Llama-3.3-70B58.70%50.00%8.70259,490 → 285,9337,101.62 → 8,726.9624.26 → 26.64
GPT-OSS-120B78.26%67.39%10.87228,768 → 264,3146,471.60 → 8,252.3522.30 → 25.95
GPT-4.1 nano2.17%0.00%2.17514,221 → 455,57911,328.89 → 12,743.6344.33 → 41.38
Claude Sonnet 5.585.87%81.52%4.35162,491 → 201,8474,835.25 → 5,944.1415.18 → 17.42
GPT-6-sol81.52%79.35%2.17156,496 → 142,8046,556.89 → 6,288.8220.13 → 18.53
GPT-6.1-sol85.87%80.43%5.43105,202 → 120,5965,507.50 → 6,531.9214.41 → 16.18
GPT-6.1-terra86.96%78.26%8.70103,534 → 129,6075,664.45 → 7,116.4914.59 → 16.93
GPT-6-astra90.22%84.78%5.4392,524 → 117,7494,809.93 → 6,842.0913.03 → 16.34
14-model mean61.80%55.67%6.13252,995 → 265,1628,196.81 → 9,250.2224.80 → 25.87

Effort values in the table are episode means and include successful and unsuccessful episodes. Tokens are the combined input and output tokens for action planning by both agents. Steps are simulator steps.

Under the model-equal mean, C has a 6.13-percentage-point lower success rate than B, while mean Tokens, Steps, and Planning calls increase by 4.81%, 12.85%, and 4.31%, respectively. B−C is positive for all 14 models on these 46 tasks, and simulator steps increase for 13 models. Thus, when the intended final requirements are the same, introducing them online lowers task success and increases average execution and planning effort.

Dear Reviewer VQQC,

Could you please review the authors’ rebuttals and follow up with any further comments or questions?

Many thanks for your time and efforts.

Best regards, AC

Reviewer_c8WFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies what happens when a task evolves over time during collaboration between embodied agents. The authors introduce a benchmark derived from PARTNR by injecting revision events that modify task requirements during execution. They empirically evaluate nine LLMs on the constructed benchmark and find that performance degrades when task requirements are introduced online compared with when the final task requirements are explicitly provided upfront.

Critical strengths

The work studies a realistic and practically important setting: people frequently change their minds or introduce new requirements while a task is already being carried out. Instead of resetting the environment, the benchmark requires agents to adapt while preserving the consequences of their previous actions. Relatedly, the benchmark enables us to study how changing requirements can alter the appropriate division of labor between agents.

Critical weaknesses

The experimental results are weak at times. For example, it does not fully establish why negotiation helps or fails. Although the paper reports performance across different task structures and negotiation strategies, many of these comparisons are descriptive, hard to follow, and based on relatively small task subsets. This makes it difficult to determine whether the observed gains are truly caused by revision-induced coordination problems or by other differences in task difficulty and composition. Additionally, the models evaluated are out-of-date and do not include the strongest contemporary frontier models. This limits how confidently the results can be generalized to the capabilities and coordination behavior of current state-of-the-art agents (or track how this capability has evolved over time).

Questions for authors
  1. Does coordination ability improve naturally as the general capabilities of LLMs improve, or will agents need to be explicitly optimized or trained for effective coordination?
  2. Can you systematically identify what determines whether negotiation helps or hurts on an individual revision, and are the differences you observe across task categories statistically significant?
4 comments
AuthorsOpenReview ↗

Response to Reviewer c8WF on Model Coverage and Coordination Ability

We appreciate your insightful and detailed comments. Below, we present our point-by-point responses and hope our clarifications address the issues you raised. We would welcome further discussion to improve the work.

W2 / Q1. Model coverage, general capability, and coordination

To address your concern about model coverage, we added Claude Sonnet 5.5 and four GPT-6-series models, expanding the three-condition evaluation from 9 to 14 models on the same 50 tasks. Under Never, both agents receive task updates and plan independently. The five added models achieved a mean success rate of 94.4% on the original tasks, 87.2% when the final requirements were specified upfront, and 82.4% when those requirements were introduced during execution. Even for teams exceeding 90% success on the original tasks, the timing of disclosure still matters.

Table 2. Three-condition evaluation under Never. A = Original-Static, B = Final-Upfront, and C = Online-Revised. SR is success rate and PC is percent complete. The last three columns report SR differences in percentage points (pp). The final five model rows are new.

ModelSR (%, A → B → C)PC (%, A → B → C)A−BA−CB−C
GPT-4o mini72.00 → 31.00 → 28.0087.31 → 76.67 → 72.1741.0044.003.00
DeepSeek-v4-flash82.00 → 63.00 → 61.0092.84 → 90.06 → 89.3719.0021.002.00
GPT-OSS-20B81.00 → 58.00 → 55.0090.48 → 87.06 → 83.1723.0026.003.00
Gemma-4-31B85.00 → 66.00 → 62.0094.21 → 92.43 → 90.5319.0023.004.00
Qwen3-30B-A3B67.00 → 39.00 → 31.0083.59 → 74.89 → 74.4628.0036.008.00
Devstral-2-123B82.00 → 62.00 → 52.0092.99 → 89.41 → 85.5820.0030.0010.00
Llama-3.3-70B81.00 → 62.00 → 54.0092.26 → 89.97 → 88.0519.0027.008.00
GPT-OSS-120B92.00 → 80.00 → 70.0097.41 → 94.19 → 91.1912.0022.0010.00
GPT-4.1 nano16.00 → 7.00 → 4.0036.53 → 29.74 → 27.279.0012.003.00
Claude Sonnet 5.592.00 → 87.00 → 83.0097.71 → 97.56 → 96.485.009.004.00
GPT-6-sol93.00 → 83.00 → 81.0097.83 → 96.59 → 95.7710.0012.002.00
GPT-6.1-sol96.00 → 87.00 → 82.0098.70 → 97.60 → 95.649.0014.005.00
GPT-6.1-terra95.00 → 88.00 → 80.0098.10 → 98.01 → 95.467.0015.008.00
GPT-6-astra96.00 → 91.00 → 86.0098.91 → 98.58 → 96.525.0010.005.00
Mean (14 models, equally weighted)80.71 → 64.57 → 59.2189.92 → 86.63 → 84.4016.1421.505.36

The five-protocol comparison across four models addresses your coordination question more directly. DeepSeek-v4-flash and Devstral-2-123B both achieved 82% SR on the original tasks. Yet on the same online-revised tasks, switching from Never to Full raised SR by 0.5 pp for DeepSeek and lowered it by 4.0 pp for Devstral (Appendix D.2, Table 25). Thus, even identical original-task performance can accompany opposite gains from explicit negotiation. We also demonstrate a way to improve coordination while keeping model parameters fixed. Using experience collected for each model on 100 tasks disjoint from the evaluation set, both Dynamic configurations outperformed Full on each of the four models, with a mean gain of 4.38 pp for each configuration.

AuthorsOpenReview ↗

Response to Reviewer c8WF on When Negotiation Helps and How Gains Vary Across Task Categories

W1 / Q2. When does an individual revision call for negotiation, and how robust are the differences across task categories?

Episode 615 in Appendix A.2 (Table 12) illustrates why the same revision can create different coordination demands. The team must place a basketball and then a carrying case on a chest of drawers, leaving them next to each other. The first revision adds a requirement to clean the case. If Spot is responsible for the case, the new operation exceeds its capabilities: the human agent must take part, and the final placement must still satisfy adjacency. If the human agent already owns the assignment, it may simply add cleaning to its existing plan. The relevant question at the moment of revision is whether the new requirement invalidates an assignment or creates a dependency with other work. The episode-level results are consistent with this interpretation. On tasks with object-state requirements, all four negotiation configurations improved mean SR over Never. Of the 16 model–configuration combinations, 13 improved and three tied their respective baselines (Section 5.2, Appendix D.2).

Baseline difficulty alone cannot explain where negotiation helps. Under Never, the 18 tasks with spatial adjacency had higher SR than the 32 tasks without it (59.03% and 56.65%, respectively). Switching to Full raised SR by 5.25 pp in the adjacency group and lowered it by 1.98 pp in the group without adjacency. Temporal tasks had the lowest Never baseline (41.35%), yet Full lowered SR by a further 2.35 pp. To make these comparisons easier to follow, revised Table 6 places task counts, baseline SR, and protocol gains side by side. New Table 5 shows how switching from Full to Partial changes both completion and effort for each model.

For your question about category differences, Eq. (2) compares the gain from the same protocol between tasks with and without a given feature. For Full, the gain was 7.2 pp larger for tasks with spatial adjacency and 11.0 pp larger for tasks with object-state requirements than for tasks without the corresponding feature. To check whether one model drove these contrasts, we used the per-model results in Appendix D.2 to repeat the analysis for all four protocols, leaving out each model in turn. All 16 leave-one-model-out estimates were positive for adjacency, and 15 of 16 were positive for object state. The sole exception was Dynamic-max after omitting Devstral (approximately −0.2 pp).

Thank you for raising the question of statistical significance. For the main disclosure result in Section 4, the online gap (B−C) was positive for all 14 models. Treating the model-level signs as independent and equally likely to be positive or negative under the null gives a two-sided sign-test p ≈ 0.00012. The category contrasts in Section 5 concern a different unit of analysis. Each stratum contains 11–39 tasks, and a task can contribute to more than one feature analysis, limiting the precision of task-level inference. We therefore report the category differences as effect sizes and show explicitly how their direction changes under model omission.

Dear Reviewer c8WF,

Could you please review the authors’ rebuttals and follow up with any further comments or questions?

Many thanks for your time and efforts.

Best regards, AC

AuthorsOpenReview ↗

Follow-Up to Reviewer c8WF on Model Capability and Negotiation Gains

Thank you again for your constructive review. We especially appreciate your recognition of the practical relevance of our setting. When task requirements change during execution, the benchmark preserves the physical state produced by the agents’ earlier actions. The agents must adapt from that state, allowing us to study how changing requirements can alter the appropriate division of labor between them. Your questions about whether model capability and task difficulty can explain negotiation gains prompted us to evaluate more models and reexamine the results across task categories.

As the discussion period draws to a close, we would like to briefly follow up on our earlier response.

  • We added Claude Sonnet 5.5 and four models from the GPT-6 series, expanding the three-condition evaluation from 9 to 14 models. Under the protocol without explicit negotiation (Never), the five additional models achieved a mean success rate of 94.4% on the original tasks. Even for these models, the mean success rate was 4.8 percentage points lower when the final requirements were introduced progressively during execution than when they were provided upfront. All 14 models showed a difference in this direction (two-sided sign test across models, p ≈ 0.00012).

  • Models with the same original-task performance can benefit differently from negotiation. DeepSeek-v4-flash and Devstral-2-123B both achieved an 82% success rate on the original tasks. On the online-revised tasks, switching from Never to full negotiation (Full) raised the success rate by 0.5 percentage points for DeepSeek-v4-flash and lowered it by 4.0 percentage points for Devstral-2-123B. With model parameters held fixed, both strategies that select a negotiation mode based on prior experience achieved higher success rates than always using Full on each of the four models tested.

  • Tasks with spatial-adjacency constraints already had a slightly higher baseline success rate under Never than tasks without those constraints. Even so, the gain from switching to Full was 7.2 percentage points larger in the adjacency group. When we repeated the comparison across the four negotiation configurations, leaving out one model at a time, the difference in gains between tasks with and without the relevant feature was positive in all 16 estimates for spatial adjacency and in 15 of 16 for object-state requirements. Episode 615 (Appendix A.2, Table 12) illustrates how the need for negotiation can depend on whether a revision disrupts an existing assignment or creates a dependency between teammates’ tasks.

  • To make these comparisons easier to follow, we reorganized Table 6 to present task counts, baseline success rates, and gains under each protocol side by side. We also added Table 5 to show how switching from Full to partial negotiation (Partial) changes task completion and planning effort for each model.

We hope these additional experiments and clarifications address your questions about when negotiation helps and whether model capability or baseline task difficulty can explain its benefits. If any concerns remain or any part of our response needs further clarification, we would welcome your feedback and would be happy to respond before the discussion period ends.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment

Dear Reviewers,

As the AC–reviewer discussion phase is about to begin, please highlight any disagreements or inconsistencies across the reviews, as well as any concerns that remain unresolved after the author rebuttal.

Please share your perspectives on these points and engage with one another to help us reach a well-supported assessment of the paper.

Thank you for your time and contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.