- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FWCOLLAB is a benchmark for evaluating structured two-agent collaboration through task-required cross-role dependencies, rather than terminal task success alone. It represents dependencies as benchmark-authored executable directed acyclic graphs (DAGs) bound to authoritative environment states, distinguishing persistent enabling dependencies (ENABLE) from conditions that must remain active during another agent's progress (MAINTAIN). A deterministic, replay-authenticated evaluator measures intermediate progress, handoffs, regressions, and structural blockers without relying on an LLM judge.
The benchmark contains a construction library of 72 Core tasks and an evaluation suite of 52 tasks across four tracks: 24 selected Core tasks, Information, Sync-v2, and Join. The authors evaluate six models over 936 episodes and conduct a separate fixed-budget communication study. They also audit interface reliability, evaluator conformance, and the sensitivity of repair results to alternative credit criteria.
The results demonstrate that intermediate dependency satisfaction can differ substantially from terminal environment success. For example, Qwen3.5-27B achieves 55.6% dependency success but only 2.8% environment success on the Information track. Conformance tests and deterministic replay provide evidence that the evaluator implements its declared specification. Overall, the main contribution is a reproducible instrument for measuring specified cross-role dependencies, rather than a new collaboration algorithm or evidence of universally superior collaborative capability.
Critical strengths
-
Explicit and executable modeling of cross-role dependencies. The benchmark goes beyond terminal success and communication frequency by binding dependency conditions to authoritative environment states. Its distinction between persistent enabling conditions and continuously maintained support allows it to identify trajectories that eventually succeed but violate intermediate coordination requirements. This provides a useful, more fine-grained instrument for analyzing collaborative execution.
-
Strong traceability and implementation-level verification. The evaluator reconstructs environment states through deterministic replay and links reported metrics to concrete execution events. Appendix D reports that all 555 applicable conformance cases pass, while the paper explicitly distinguishes implementation conformance from construct validity. This emphasis on auditing evaluation rules, trajectories, and derived results improves reproducibility and interpretability.
Critical weaknesses
-
The independent methodological novelty relative to existing structured evaluation frameworks remains insufficiently established. The paper acknowledges that DAG-based evaluation, intermediate-state checking, and execution-trace diagnosis have appeared in prior work. Its main contribution is to combine these elements with explicit cross-role dependency semantics. However, the experiments primarily establish that the system follows its own specification and that the resulting metrics capture different aspects of execution. They provide less direct evidence of which distinctive and consequential evaluation findings arise specifically from the proposed formalization, compared with a straightforward extension of existing evaluation frameworks. Consequently, the evidence for engineering completeness is stronger than the evidence for independent methodological novelty.
-
The relationship between the proposed metrics and meaningful collaborative capability is not fully established by the reported results. The paper demonstrates differences among terminal success, DAG completion, process violations, and repair usefulness, but these metrics are jointly determined by the predefined graph structures and evaluation rules. Showing that the metrics disagree establishes that they measure different quantities; it does not, by itself, establish that the new metrics more accurately characterize meaningful collaboration quality or reliably distinguish effective collaboration strategies. The operational definitions and implementation correctness are supported in detail, but the incremental evaluation value over conventional task-outcome metrics remains less clearly demonstrated.
Questions for authors
-
Independent methodological contribution: Could the authors provide a concrete example in which the proposed cross-role dependency formalization reveals an important execution phenomenon that would be difficult to capture using existing DAG-based evaluation or trace-diagnosis frameworks? Ideally, the authors would identify which specific design choice enables this finding, beyond the combination of existing components.
-
Interpretive validity of the metrics: How do the authors distinguish success as defined by the evaluator's rules from collaborative performance that is meaningful beyond those rules, particularly for DAG completion, violation rates, and repair usefulness? Is there evidence independent of the current metric definitions that validates their interpretive value when comparing collaboration strategies?
These questions address the central methodological contribution and the interpretation of the empirical findings, respectively, and could affect the overall assessment of novelty and empirical significance.
6 comments
Response to W2: Interventions on the dependency, and where the evidence stops
W2. "Showing that the metrics disagree establishes that they measure different quantities; it does not, by itself, establish that the new metrics more accurately characterize meaningful collaboration quality …"
Short answer. We agree. Our evidence comes from outside the metric definitions: interventions on the dependency itself, with geometry fixed. We also state where this evidence stops.
Two interventions change only the dependency and keep the geometry fixed:
| Intervention (geometry fixed) | Model | Success | Process measure |
|---|---|---|---|
| Remove only the message channel (Information-12) | GPT-5.5 (reasoning=low) | 100% → 33.3% (−66.7 pp [41.7, 91.7]) | DAG completion −58.3 pp; clean handoff 100% → 0% |
| Wire each gate to the traveler's own controller instead of the partner's (matched multi-stage pairs, 20 episodes per arm) | GPT-5.5 (reasoning=none) | 55% → 45% | closed-gate walk-ins 5 → 0 |
In both interventions, the process measure moves the way the manipulated dependency predicts. Only the first also moves success.
Where the evidence stops.
- We tested whether violation features measured mid-episode predict final success beyond completion and AUC. They add little (ΔAUROC +0.011 [0.000, 0.027] in Standard-52 Core, none in Phase II), so we make no predictive-validity claim.
- In the matched pairs, the dependency toggle does not change success (second row).
- Repair usefulness did not survive the criterion correction, and we withdraw that claim.
Q2 takes the three metrics one at a time.
Response to Q1: Post-join MAINTAIN breaks in successful Join episodes
Q1. "Could the authors provide a concrete example in which the proposed cross-role dependency formalization reveals an important execution phenomenon that would be difficult to capture using existing DAG-based evaluation or trace-diagnosis frameworks? … identify which specific design choice enables this finding …"
Short answer. In successful Join episodes, a role releases a condition its partner still depends on and then restores it. Two design choices detect this: volatile MAINTAIN predicates, and a regression ledger keyed to the dependent role's pending crossing.
Phenomenon. In Join-8, 130 episodes reach both exits with 100% DAG completion, so terminal success and any monotone checkpoint graph score them identically. Thirty-two of them (24.6%) release a role-held MAINTAIN condition after the join while the partner still depends on it, and then restore it. For the GPT models, all such cases are confined to the three maintenance-bearing tasks. The resulting measure separates models whose success and completion are tied:
| Track | Model | SR | DAG completion | Cross-role metric |
|---|---|---|---|---|
| Join-8 | GPT-5.5 | 100% | 100% | violation-free 100% |
| Join-8 | GPT-5-mini | 95.8% | 99.5% | violation-free 62.5% |
| Join-8 | GPT-5.4-mini | 100% | 100% | violation-free 66.7% |
| Core (Standard-52) | GPT-5.4-mini | 12.5% | 31.3% | 6.4 violations / 100 rounds |
| Core (Standard-52) | Qwen3.5-27B | 6.9% | 31.2% | 19.5 violations / 100 rounds |
Design choices. Two choices produce the Join finding, and neither exists in predecessor-gated milestone evaluators:
- MAINTAIN predicates are volatile. They are recomputed every round, so a condition can be achieved and later lost; a monotone checkpoint, once reached, stays reached.
- The regression ledger is keyed to the dependent role's pending crossing. A release is recorded only while the partner still needs the condition; releasing it after the partner has crossed is normal and is not counted.
A third choice. In Information, 61 failures begin with a wrong controller activated before the partner's fact arrives. Detecting this requires information ordering. W1 gives the full ablation.
Response to W1: Comparison with a straightforward CRAB-style extension
We thank the reviewer. We agree with the reviewer's summary of the contribution: a reproducible instrument for measuring specified cross-role dependencies, not a new collaboration algorithm and not evidence of general collaborative ability. We will revise the abstract and introduction to state the contribution in these terms. We answer each point in a separate comment.
Short answer. We built that straightforward extension and scored the same 1,224 trajectories with it. The findings below hold under our evaluator and disappear under the extension; the clearest is 32 of 130 apparently perfect Join episodes.
For our tasks, the most direct extension of existing structured evaluation is a CRAB-style graph over the same nodes: predecessor-gated and scored by monotone completion, with no MAINTAIN volatility, role ownership, or information ordering. We re-scored all 1,224 Core/Information/Join trajectories with both the extension and our evaluator:
Trace-diagnosis frameworks such as CoCoBench and TraceElephant are built to explain failures. The Join cases above end in success, so there is no failure to start from. Finding them requires knowing, for each task, which role must keep which condition active and until when. Q1 names the design choices responsible. We agree that the submission's evidence was stronger on engineering completeness than on what the formalization adds; this ablation is that missing evidence, and the revision will report it in the main text.