acceptodds
Under review as a conference paper at ICLR 2027

FWCollab: Benchmarking Structured Multi-Agent Collaboration with Executable Dependency Graphs

Abstract

Terminal success does not describe how a team establishes, maintains, and recovers the intermediate conditions required by its task. We introduce FWCollab, a symbolic two-agent benchmark that operationalizes these cross-role dependencies as private, benchmark-authored executable graphs bound to authoritative environment state. A deterministic, replay-authenticated evaluator measures progress, handoffs, condition regressions, and structural blockers without a language-model judge. The construction library contains 72 Core tasks; evaluation uses a selected 24-task Core subset plus information, bounded-synchronization, and parallel-join tracks, totaling 52 tasks and 936 episodes across six models. Under the tested budgets, intermediate dependency satisfaction and environment completion can differ substantially; these differences characterize execution rather than establish general collaboration quality. A separate paired communication study is analyzed within its own fixed-budget protocol, with interface diagnostics and sensitivity to how controller releases are counted. Conformance and alternative-execution checks validate implementation against the declared specification, not human construct validity. A repair audit identifies historical credit for already-completed stages; requiring new progress removes clear evidence of either selector’s superiority. FWCollab provides a reproducible instrument for examining specified cross-role dependencies, with explicit boundaries between executable measurement, downstream utility, and capability comparison.

Primary area:
datasets and benchmarks
Keywords:
Multi-agent collaboration, Collaboration benchmarks, Executable dependency graphs, Process-aware evaluation, Large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B4ruOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FWCOLLAB is a benchmark for evaluating structured two-agent collaboration through task-required cross-role dependencies, rather than terminal task success alone. It represents dependencies as benchmark-authored executable directed acyclic graphs (DAGs) bound to authoritative environment states, distinguishing persistent enabling dependencies (ENABLE) from conditions that must remain active during another agent's progress (MAINTAIN). A deterministic, replay-authenticated evaluator measures intermediate progress, handoffs, regressions, and structural blockers without relying on an LLM judge.

The benchmark contains a construction library of 72 Core tasks and an evaluation suite of 52 tasks across four tracks: 24 selected Core tasks, Information, Sync-v2, and Join. The authors evaluate six models over 936 episodes and conduct a separate fixed-budget communication study. They also audit interface reliability, evaluator conformance, and the sensitivity of repair results to alternative credit criteria.

The results demonstrate that intermediate dependency satisfaction can differ substantially from terminal environment success. For example, Qwen3.5-27B achieves 55.6% dependency success but only 2.8% environment success on the Information track. Conformance tests and deterministic replay provide evidence that the evaluator implements its declared specification. Overall, the main contribution is a reproducible instrument for measuring specified cross-role dependencies, rather than a new collaboration algorithm or evidence of universally superior collaborative capability.

Critical strengths
  1. Explicit and executable modeling of cross-role dependencies. The benchmark goes beyond terminal success and communication frequency by binding dependency conditions to authoritative environment states. Its distinction between persistent enabling conditions and continuously maintained support allows it to identify trajectories that eventually succeed but violate intermediate coordination requirements. This provides a useful, more fine-grained instrument for analyzing collaborative execution.

  2. Strong traceability and implementation-level verification. The evaluator reconstructs environment states through deterministic replay and links reported metrics to concrete execution events. Appendix D reports that all 555 applicable conformance cases pass, while the paper explicitly distinguishes implementation conformance from construct validity. This emphasis on auditing evaluation rules, trajectories, and derived results improves reproducibility and interpretability.

Critical weaknesses
  1. The independent methodological novelty relative to existing structured evaluation frameworks remains insufficiently established. The paper acknowledges that DAG-based evaluation, intermediate-state checking, and execution-trace diagnosis have appeared in prior work. Its main contribution is to combine these elements with explicit cross-role dependency semantics. However, the experiments primarily establish that the system follows its own specification and that the resulting metrics capture different aspects of execution. They provide less direct evidence of which distinctive and consequential evaluation findings arise specifically from the proposed formalization, compared with a straightforward extension of existing evaluation frameworks. Consequently, the evidence for engineering completeness is stronger than the evidence for independent methodological novelty.

  2. The relationship between the proposed metrics and meaningful collaborative capability is not fully established by the reported results. The paper demonstrates differences among terminal success, DAG completion, process violations, and repair usefulness, but these metrics are jointly determined by the predefined graph structures and evaluation rules. Showing that the metrics disagree establishes that they measure different quantities; it does not, by itself, establish that the new metrics more accurately characterize meaningful collaboration quality or reliably distinguish effective collaboration strategies. The operational definitions and implementation correctness are supported in detail, but the incremental evaluation value over conventional task-outcome metrics remains less clearly demonstrated.

Questions for authors
  1. Independent methodological contribution: Could the authors provide a concrete example in which the proposed cross-role dependency formalization reveals an important execution phenomenon that would be difficult to capture using existing DAG-based evaluation or trace-diagnosis frameworks? Ideally, the authors would identify which specific design choice enables this finding, beyond the combination of existing components.

  2. Interpretive validity of the metrics: How do the authors distinguish success as defined by the evaluator's rules from collaborative performance that is meaningful beyond those rules, particularly for DAG completion, violation rates, and repair usefulness? Is there evidence independent of the current metric definitions that validates their interpretive value when comparing collaboration strategies?

These questions address the central methodological contribution and the interpretation of the empirical findings, respectively, and could affect the overall assessment of novelty and empirical significance.

6 comments
AuthorsOpenReview ↗

Response to W1: Comparison with a straightforward CRAB-style extension

We thank the reviewer. We agree with the reviewer's summary of the contribution: a reproducible instrument for measuring specified cross-role dependencies, not a new collaboration algorithm and not evidence of general collaborative ability. We will revise the abstract and introduction to state the contribution in these terms. We answer each point in a separate comment.

W1. "… They provide less direct evidence of which distinctive and consequential evaluation findings arise specifically from the proposed formalization, compared with a straightforward extension of existing evaluation frameworks."

Short answer. We built that straightforward extension and scored the same 1,224 trajectories with it. The findings below hold under our evaluator and disappear under the extension; the clearest is 32 of 130 apparently perfect Join episodes.

For our tasks, the most direct extension of existing structured evaluation is a CRAB-style graph over the same nodes: predecessor-gated and scored by monotone completion, with no MAINTAIN volatility, role ownership, or information ordering. We re-scored all 1,224 Core/Information/Join trajectories with both the extension and our evaluator:

DataPerfect successes under the extension…flagged by our evaluatorFailures…with a mechanism label
Standard-52 Core4918 (37%)383250 (65%)
Standard-52 Join13032 (25%)14—
Standard-52 Information7020961 (wrong controller first)
Phase II Core (reasoning=low)1477 (5%)2853
  • Join: the 32 flagged episodes reach the exits with 100% completion but break a post-join MAINTAIN condition and then recover. The extension scores all 130 as perfect. This is the only Join measure that separates GPT-5.5 from the two mini models (Q1).
  • Core: besides the counts above, 46 groups of episodes (171 episodes) that the extension cannot tell apart are scored differently by our evaluator.
  • Information: the 61 failures begin with a wrong controller activated before the partner's fact arrives, which the extension cannot separate from inaction.
  • With reasoning enabled (Phase II), violations are rare and the two evaluators mostly agree.

Trace-diagnosis frameworks such as CoCoBench and TraceElephant are built to explain failures. The Join cases above end in success, so there is no failure to start from. Finding them requires knowing, for each task, which role must keep which condition active and until when. Q1 names the design choices responsible. We agree that the submission's evidence was stronger on engineering completeness than on what the formalization adds; this ablation is that missing evidence, and the revision will report it in the main text.

AuthorsOpenReview ↗

Response to W2: Interventions on the dependency, and where the evidence stops

W2. "Showing that the metrics disagree establishes that they measure different quantities; it does not, by itself, establish that the new metrics more accurately characterize meaningful collaboration quality …"

Short answer. We agree. Our evidence comes from outside the metric definitions: interventions on the dependency itself, with geometry fixed. We also state where this evidence stops.

Two interventions change only the dependency and keep the geometry fixed:

Intervention (geometry fixed)ModelSuccessProcess measure
Remove only the message channel (Information-12)GPT-5.5 (reasoning=low)100% → 33.3% (−66.7 pp [41.7, 91.7])DAG completion −58.3 pp; clean handoff 100% → 0%
Wire each gate to the traveler's own controller instead of the partner's (matched multi-stage pairs, 20 episodes per arm)GPT-5.5 (reasoning=none)55% → 45%closed-gate walk-ins 5 → 0

In both interventions, the process measure moves the way the manipulated dependency predicts. Only the first also moves success.

Where the evidence stops.

  • We tested whether violation features measured mid-episode predict final success beyond completion and AUC. They add little (ΔAUROC +0.011 [0.000, 0.027] in Standard-52 Core, none in Phase II), so we make no predictive-validity claim.
  • In the matched pairs, the dependency toggle does not change success (second row).
  • Repair usefulness did not survive the criterion correction, and we withdraw that claim.

Q2 takes the three metrics one at a time.

AuthorsOpenReview ↗

Response to Q1: Post-join MAINTAIN breaks in successful Join episodes

Q1. "Could the authors provide a concrete example in which the proposed cross-role dependency formalization reveals an important execution phenomenon that would be difficult to capture using existing DAG-based evaluation or trace-diagnosis frameworks? … identify which specific design choice enables this finding …"

Short answer. In successful Join episodes, a role releases a condition its partner still depends on and then restores it. Two design choices detect this: volatile MAINTAIN predicates, and a regression ledger keyed to the dependent role's pending crossing.

Phenomenon. In Join-8, 130 episodes reach both exits with 100% DAG completion, so terminal success and any monotone checkpoint graph score them identically. Thirty-two of them (24.6%) release a role-held MAINTAIN condition after the join while the partner still depends on it, and then restore it. For the GPT models, all such cases are confined to the three maintenance-bearing tasks. The resulting measure separates models whose success and completion are tied:

TrackModelSRDAG completionCross-role metric
Join-8GPT-5.5100%100%violation-free 100%
Join-8GPT-5-mini95.8%99.5%violation-free 62.5%
Join-8GPT-5.4-mini100%100%violation-free 66.7%
Core (Standard-52)GPT-5.4-mini12.5%31.3%6.4 violations / 100 rounds
Core (Standard-52)Qwen3.5-27B6.9%31.2%19.5 violations / 100 rounds

Design choices. Two choices produce the Join finding, and neither exists in predecessor-gated milestone evaluators:

  1. MAINTAIN predicates are volatile. They are recomputed every round, so a condition can be achieved and later lost; a monotone checkpoint, once reached, stays reached.
  2. The regression ledger is keyed to the dependent role's pending crossing. A release is recorded only while the partner still needs the condition; releasing it after the partner has crossed is normal and is not counted.

A third choice. In Information, 61 failures begin with a wrong controller activated before the partner's fact arrives. Detecting this requires information ordering. W1 gives the full ablation.

Reviewer_5ztaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces FWCollab, a two-agent collaboration benchmark based on executable dependency DAGs. Instead of only looking at the final task success rate, FWCollab tracks cross-agent dependencies to evaluate task progress, handoffs, condition regressions, and failure stages. The evaluation uses deterministic trajectory replay without relying on an LLM Judge. The authors evaluate six models on 52 tasks across four collaboration settings, with 936 episodes in total. The results show that final task success and intermediate collaboration metrics can be quite different.

Critical strengths

S1. More detailed and reproducible evaluation of multi-agent collaboration.

I like that FWCollab goes beyond final task success by using executable dependency DAGs to track intermediate progress, agent handoffs, and coordination failures. It also uses environment states and deterministic trajectory replay instead of an LLM Judge, making the evaluation more consistent and reproducible.

S2. Detailed experiments and good discussion of limitations.

The authors test multiple models on different collaboration tasks and show how final success rates can differ from process-level metrics. I also like the trajectory consistency checks, communication experiments, and failure repair analysis. The discussion of time budgets, interface reliability, and evaluation criteria makes the results more convincing.

Critical weaknesses

W1. Limited novelty and unclear practical benefits of the new metrics.

DAG-based evaluation has been explored before. While this paper adds more explicit cross-agent dependencies, I would like to see stronger evidence that the new metrics better reflect collaboration quality. Also, Full-DAG does not clearly outperform Local-event in Failure Repair after correcting the evaluation criteria, and a controlled DAG ablation is missing.

W2. Limitations in benchmark design and experimental settings.

The tasks are mostly symbolic two-agent environments, and only 24 of the 72 Core tasks are tested with all six models. Some tasks have very low or near-perfect success rates, and the Information Track has a tight time budget. Also, 9 of 24 Communication Study tasks cannot be completed within the given budget. I think these issues limit the benchmark's ability to distinguish models and generalize to real-world settings.

Questions for authors

Q1. Could the authors provide more evidence that the proposed DAG-based metrics actually reflect meaningful collaboration quality? For example, a small human evaluation or a more controlled DAG ablation could help show their benefits over simpler process-level metrics.

Q2. Could the authors evaluate the Information Track with a larger time budget (e.g., 45 or 60 rounds)? This would help verify whether the gap between Dependency Success and final task success still exists.

5 comments
AuthorsOpenReview ↗

Response to W1: The novelty is the cross-role layer, isolated by a controlled ablation

We thank the reviewer for the careful reading and the two concrete requests. The controlled DAG ablation could be run on our existing trajectories and is reported in W1 and Q1. The 45/60-round Information run is in Q2. We answer each point in a separate comment.

W1. "DAG-based process evaluation has already been explored in previous work. … it lacks independent validation showing that the new metrics better reflect collaboration quality. Also, the Failure Repair experiments do not show a significant advantage of Full-DAG over Local-event … and a controlled DAG ablation is still missing."

Short answer. We agree that DAG-based evaluation is not new and that repair is a null result. What is new is the cross-role layer on top of the graph. The controlled ablation shows that this layer changes the verdict on 32 of 130 apparently perfect Join episodes, and it separates models that success and completion cannot.

Section 2 credits CRAB's predecessor-gated checkpoint graph, and we do not claim it. Our claim is narrower: the cross-role semantics on top of the graph. These record who supplies each condition, who needs it, whether it must stay active while the partner moves (MAINTAIN), and in which order information must arrive.

Practical benefit. The cross-role metrics separate models that success and DAG completion cannot:

TrackModelSRDAG completionCross-role metric
Join-8GPT-5.5100%100%violation-free 100%
Join-8GPT-5-mini95.8%99.5%violation-free 62.5%
Join-8GPT-5.4-mini100%100%violation-free 66.7%
Core (Standard-52)GPT-5.4-mini12.5%31.3%6.4 violations / 100 rounds
Core (Standard-52)Qwen3.5-27B6.9%31.2%19.5 violations / 100 rounds

For the GPT models in Join, every gap between success and violation-free execution is a MAINTAIN condition released after the join while the partner still depends on it, and then restored. A monotone checkpoint graph scores these episodes as perfect.

Controlled ablation. The ablation the reviewer asks for isolates exactly this layer. We re-scored all 1,224 Core/Information/Join trajectories with a CRAB-style version of the same graph: same nodes, predecessor-gated, monotone completion, and no MAINTAIN volatility, role ownership, or information ordering. Q1 gives the full table. For example, 32 of the 130 Join episodes that this graph scores as perfect break a post-join MAINTAIN condition.

Repair. After the criterion correction, Full-DAG is not significantly better than Local-event. We will remove repair from the contributions and keep it only as an appendix sensitivity analysis.

AuthorsOpenReview ↗

Response to W2: Representativeness of Diagnostic-24 and the infeasible tasks

W2. "… only 24 out of 72 Core tasks are evaluated with the six models. Some tasks have very low or near-perfect success rates, and the Information Track has a tight time budget. Also, 9 out of 24 tasks in the separate Communication Study are impossible to complete within the given budget."

Short answer. On existing runs, Diagnostic-24 behaves like the full set of 72 tasks. The nine infeasible communication tasks dilute SR contrasts but do not change their sign. Process metrics still separate models on the floor and ceiling tracks, and Q2 answers the Information budget.

Only 24 of 72 Core tasks with six models. Diagnostic-24 was selected structurally, before any outcomes were seen, and shows no difficulty bias (|d| < 0.4). It is also behaviorally representative. Existing Full-72 runs (earlier protocol, one seed, budgets scaled to each task's witness length) let us compare it with the other 48 tasks:

ModelDiagnostic-24 SR / completion (witness-scaled budgets)Other 48, difficulty-reweightedFull-72 solved, witness-scaled budgetsFull-72 solved, 100 rounds
GPT-5.5 (reasoning=low)100 / 100%98.3 / 99.8%71/7239/72
Gemini-3.7-Flash100 / 100%100 / 100%72/7229/72
GPT-5-mini12.5 / 33.8%8.5 / 28.4%6/723/72

Diagnostic-24 is not harder than the other tasks; for GPT-5-mini it is slightly easier. We will add these Full-72 results to the appendix.

9 of 24 tasks infeasible in the communication study. These nine are provably infeasible at 80 rounds even with all coordination removed: with every non-wall cell passable, one agent's shortest path alone is longer than 80 moves. Of the other 15 tasks, 10 are certified feasible (an evaluator-valid success within 80 rounds exists) and 5 are unresolved. The submission reports the certified subset in the appendix. This study uses a fixed 80-round budget rather than the witness-scaled budgets above, so its success rates are not comparable with the first table:

Condition (80 rounds)SR, all 24 tasksSR, 10 certified tasks
GPT-5.5 self-play38.9%93.3%
GPT-5.5 No-Comm40.3%96.7%
Gemini-3.7-Flash self-play31.9%76.7%
Gemini-3.7-Flash No-Comm23.6%56.7%

The nine infeasible tasks score 0 in every condition. They shrink SR contrasts without changing their sign, and the process metrics remain defined on them. We will place the certified-subset numbers next to every communication-study result, not only in the appendix. The communication study is secondary to the six-model benchmark.

Very low or near-perfect success. When Core SR is near floor, DAG completion still separates models (GPT-5.5 58.9% vs. Qwen3.5-9B 13.8%). When Join SR is near ceiling, violation-free execution separates them (W1). Sync-v2 is a positive control: it checks that bounded synchronization is measurable, not that it separates frontier models.

Tight Information budget. See Q2.

Symbolic, two agents. This is a deliberate trade-off for inspectable, executable semantics, and Section 9 states it as a scope limit.

AuthorsOpenReview ↗

Response to Q1: Controlled DAG ablation and interventions on the dependency

Q1. "Could the authors provide more evidence that the proposed DAG-based metrics actually reflect meaningful collaboration quality? For example, a small human evaluation or a more controlled DAG ablation …"

Short answer. We provide the controlled DAG ablation the reviewer suggests (a) and, in addition, direct manipulations of the dependency itself (b).

(a) Controlled DAG ablation against a simpler process metric. We scored all 1,224 trajectories with both the CRAB-style monotone graph described in W1 and the full evaluator:

DataPerfect successes under the monotone graph…flagged by the full evaluatorFailures…with a mechanism label
Standard-52 Core4918 (37%)383250 (65%)
Standard-52 Join13032 (25%)14—
Standard-52 Information7020961 (wrong controller first)
Phase II Core (reasoning=low)1477 (5%)2853
  • Flagged means coordination violations or harmful regressions; for Join, these are the post-join MAINTAIN breaks in W1. A mechanism label names what went wrong at the blocking stage, such as a premature crossing attempt or an early support release.
  • In Information, the label is a wrong controller activated before the partner's fact arrives. Without information ordering, this cannot be told apart from not acting.
  • Some episodes look identical under the monotone graph (same task, outcome, completion, and AUC) but are scored differently by the full evaluator: 46 groups in Core (171 episodes), 12 in Information (118), and 4 in Join (44).

With reasoning enabled (Phase II, last row), violations are rare and the two evaluators mostly agree. The cross-role layer adds the most where coordination actually breaks down.

(b) Controlled manipulations of the dependency itself, with geometry fixed.

  • Removing only the message channel on Information-12 lowers the SR of GPT-5.5 (reasoning=low) by 66.7 pp [41.7, 91.7] and DAG completion by 58.3 pp, and clean information handoff falls from 100% to 0%.
  • In geometry-matched multi-stage pairs, we toggle only whether each gate is controlled by the partner or by the traveler itself. Travelers walked into a closed gate 5 times in 20 episodes when the partner controlled it, and never in 20 episodes when they controlled it themselves.
  • The same toggle does not lower success (55% vs. 45%). We report this as a negative result: the metric tracks the dependency, but for GPT-5.5 without reasoning the dependency is not what makes these routes hard.
Reviewer_XYd8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

FWCollab is a symbolic two-agent benchmark where two LLM agents (Fire and Water) must cooperate. Each task has a hidden dependency graph, checked directly against the game state, of steps one agent must finish before the other can go on. There are 72 main tasks plus three small tracks for sharing information, timing and joining. FWCollab provides a reproducible instrument for examining specified cross-role dependencies, with explicit boundaries between executables.

Critical strengths
  1. Teamwork is an important quality that needs to be measured and improved and this paper is an effort to formalize the definition of teamwork. The rules for when a dependency exists are clear, and every score traces back to real changes in the game state.

  2. Its great that scoring is cheap, repeatable and doesn’t depend on an LLM judge since score comes from fixed rules applied to the game state, so the same episode always gets the same score. Many other agent benchmarks rely on an LLM to grade behavior, which can be biased, inconsistent or change between versions.

Critical weaknesses
  1. There is no evidence that the graph helps diagnose failures. Just showing that the scores differ from success doesn’t show that they are useful. In the corrected repair audit, the full-graph method’s gains have confidence intervals that include zero. Also there is no human check that the “first unfinished step” is actually a real failure point.

  2. It seems this paper mixes teamwork with navigation and budget. Graph depth almost exactly tracks the minimum path length (correlation 0.994). A third of open-weight model moves are blocked by walls. The information track’s 30-round limit leaves almost no room for exploration/error.

  3. The language of the paper is super hard to follow and needs a major rewrite to better explain the problem, settings and details of the benchmark tasks.

Questions for authors
  1. Teamwork vs. path length: Can you run one or two models on the 16-task matched-pair suite and report how scores differ between the version with the dependency and the version without it? This would directly show whether the scores respond to teamwork rather than path length.

  2. Single-controller baseline: Can you run the single-controller setup on the 24 diagnostic tasks for at least one model? If one controller does much better, the low scores reflect coordination. If it does about as badly, they mostly reflect navigation and planning.

  3. Why does GPT-5.5 do better with 80 rounds and reasoning “low” (38.89%) than with 200 rounds and reasoning “none” (29.2%)? Is graph completion calculated the same way in both studies (85.48% vs. 58.9%)?

8 comments
AuthorsOpenReview ↗

Response to W1: Repair is a null result; failures stop at cross-role stages

We thank the reviewer for the direct questions. For this response we ran two matched-pair suites (Q1), a larger-budget Information rerun (W2), and a single-controller baseline (Q2); the other tables come from analyses of existing trajectories. We answer each point in a separate comment.

W1. "There is no evidence that the graph helps diagnose failures. … In the corrected repair audit, the full-graph method's gains have confidence intervals that include zero. Also there is no human check that the 'first unfinished step' is actually a real failure point."

Short answer. We agree on repair. On diagnosis, the first unfinished step almost always sits at a cross-role stage and in 45–87% of failures carries a recorded coordination event; an intervention shows that the metrics track the dependency, and an ablation shows that they add information beyond success.

The corrected repair audit is a null result. We no longer present repair as evidence of diagnostic value, and the revision keeps it only as an appendix sensitivity analysis.

On whether the first unfinished step is a real failure point, we checked what the evaluator records at that step in every Standard-52 Core failure:

ModelCore failuresStopped at a cross-role controller stageCoordination violation at that stage
GPT-5.5 (reasoning=none)5145 (88%)23 (45%)
GPT-5-mini6254 (87%)30 (48%)
GPT-5.4-mini6362 (98%)55 (87%)
Qwen3.5-9B6969 (100%)56 (81%)
Qwen3.5-27B6764 (96%)48 (72%)
Qwen3.5-122B7170 (99%)38 (54%)

Almost all failures stop at a stage where one role must act for the other, not on the final path to the exits. In 45–87% of them, the evaluator records a specific coordination event at that stage, such as a premature crossing attempt or an early support release. This localizes failures but does not by itself establish their cause.

Two further pieces of evidence bear on what the graph measures:

  • Causal check. On Information-12, removing only the message channel lowers the SR of GPT-5.5 (reasoning=low) by 66.7 pp [41.7, 91.7] and DAG completion by 58.3 pp, and clean information handoff falls from 100% to 0%. The geometry is identical in both arms.
  • Ablation. We re-scored all 1,224 Core/Information/Join trajectories with a monotone checkpoint graph over the same nodes (full table in our reply to Reviewer 5zta, Q1). For example, 32 of 130 Join episodes that reach the exits with 100% completion break a post-join MAINTAIN condition; the monotone graph scores all 130 as perfect.
AuthorsOpenReview ↗

Response to W2: Separating navigation, budget, and coordination

W2. "It seems this paper mixes teamwork with navigation and budget. Graph depth almost exactly tracks the minimum path length (correlation 0.994). A third of open-weight model moves are blocked by walls. The information track's 30-round limit leaves almost no room for exploration/error."

Short answer. Partly agreed. Navigation errors are large for small and non-reasoning models, and the revision will report them separately. Coordination errors can be separated from them, and for Qwen3.5-122B they outnumber wall collisions. For GPT-5.5, path length does not lower success once the dependency is held fixed, and the 30-round budget lowers environment success without creating the dependency failures.

(a) Path length. Depth and the movement lower bound co-vary in the natural curriculum (ρ = 0.994). This is why we built the matched-pair suites, in which the dependency is toggled while geometry and the movement lower bound are held identical (Q1). In those pairs, lengthening every stage by 16 moves per agent does not lower GPT-5.5's success (80→80%, 20→60%); it only adds rounds roughly equal to the added lower bound. Toggling the dependency does not lower success either. We therefore agree with the reviewer that, for this model, much of the difficulty is multi-stage spatial execution rather than the dependency itself.

(b) Blocked moves. A clarification: the 33,419 Qwen events are moves blocked by the environment, not necessarily by walls. We split every blocked move in Standard-52 Core by cause:

ModelWall collisionsPremature crossing attemptsPremature crossings, share
GPT-5.5 (reasoning=none)2,4811214.6%
GPT-5-mini3,69278917.6%
GPT-5.4-mini4,12568614.3%
Qwen3.5-9B9,7721,40912.6%
Qwen3.5-27B6,7832,58627.6%
Qwen3.5-122B3,7835,37858.7%

A premature crossing attempt is a move into a closed door or an inactive platform. Every closed-door block we can map to a stage, and almost every inactive-platform block, comes from that stage's traveler before its partner's support holds, so these are coordination errors, not navigation errors. Walls dominate for every model except Qwen3.5-122B, where premature crossings outnumber them. With reasoning=low, GPT-5.5's wall share of non-WAIT moves falls from 16.2% to 1.3%. The revision will report both rates beside each result.

(c) Information budget. In an earlier run, GPT-5.5 with reasoning=low solved 36/36 Information episodes within 30 rounds, so the budget itself is feasible. We also reran reasoning=none at 45 and 60 rounds for this response (details in our reply to Reviewer 5zta, Q2):

BudgetEnvironment successDependency successFailures: wrong controller first
30 rounds (submission)3/3612/3620
45 rounds8/3613/3619
60 rounds8/3612/3623

Environment success rises, almost entirely from episodes in which the information had already arrived. Activating a wrong controller before the partner's fact arrives persists: by construction, recovering from a wrong first guess takes at least 47 rounds. The 30-round budget therefore lowers environment success somewhat, but it does not create the dependency failures.

(d) Collaboration is structurally required. In all 72 tasks, a team in which either agent always WAITs cannot succeed. Every controller stage has a proven cross-agent no-bypass cut, and 68/72 tasks require assistance in both directions.

(e) Where stage time goes. With reasoning=low, 86–88% of stage time is the movement lower bound and support gaps are at most 1%. In Standard-52, support gaps take 0.3–17% of stage time, while the supporter's excess approach (10–46%) and traveler slack (13–48%) dominate. We agree that, for non-reasoning and small models, much of the inefficiency is navigation and planning; the benchmark measures these components separately rather than folding them into one success bit.

AuthorsOpenReview ↗

Response to W3: Planned rewrite and a metric-to-question table

W3. "The language of the paper is super hard to follow and needs a major rewrite to better explain the problem, settings and details of the benchmark tasks."

Short answer. Agreed. The concrete revisions are below.

The revision will:

  • rewrite the abstract, introduction, and conclusion to state each claim with its evidence, and gather the caveats into one limitations paragraph;
  • add a worked example that follows one task from start to finish: what each agent sees, what it must do for the other, and how the evaluator scores it;
  • fix the role labels in Figure 1;
  • move the repair audit to the appendix;
  • add a table mapping each metric to the question it answers. A preview:
MetricQuestion it answers
Environment SRDid both agents reach their exits within the budget?
DAG completionHow far through the required cross-role dependencies did the team get?
Progress AUCHow early in the episode was that progress made?
Clean handoffWhen one role supplied a condition, did the partner use it without breaking a dependency? In Information: was the fact delivered before the correct controller was activated, with no wrong controller first?
Coordination violations / 100 roundsHow often did a condition rise before its prerequisites (e.g., crossing before support holds), or a held condition get released while the partner still needed it?
Failure stageWhere did a failed episode stop, and which role's step was pending?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.