acceptodds
Under review as a conference paper at ICLR 2027

AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering

Abstract

Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window. These findings position dependency dynamics as a useful lens for understanding and interpreting collaboration behavior in asynchronous multi-agent software engineering. Resources are available at https://anonymous.4open.science/r/AsynCodeBench-2640/.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent, SWE Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_7PxSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes AsyncCodeBench with 19 tasks from 16 repositories, and curate dependency relations. This paper proposes two new metrics to evaluate how fast the function can be achieved with different collaboration modes or how well the agent collaborates. The paper evaluates 10 models and 5 collaboration modes, revealing the hopping window in multi-agent coding.

Critical strengths
  • Important problem, and the finding highlights a meaningful gap: final task correctness alone cannot distinguish coding failures from asynchronous coordination failures.
  • Executable, dependency-aware evaluation. The benchmark uses repository-level tests and explicit producer–consumer dependency checkers, offering better interpretability.
  • Useful protocol comparisons. Comparing serial execution, private asynchronous worktrees, and manager-assisted variants provides a promising framework for studying how coordination mechanisms affect software engineering outcomes.
Critical weaknesses
  • Theoretical soundness of ADPR. A failed dependency check may reflect a coding error rather than a coordination failure. The paper fails to build ADPR to measures the phenomenon it intends to capture under this term.
  • Limitations of DRS. DRS measures the first successful resolution using run-specific normalized checkpoints, rather than wall-clock time or a fixed computational budget, making comparisons peed across protocols unfair.
  • Limited benchmark scale and test-takers. The benchmark is still scale-limited with limited number of dependencies. And many widely-used commercial models are not evaluated.
  • Insufficiently validated trajectory analysis. The reported hopping window phenomenon may depend on checkpoint design and normalization, and the paper needs stronger robustness analyses to support this interpretation.
Questions for authors
  1. How do you distinguish coordination failures from individual implementation failures, and can you validate ADPR against manually annotated failure cases?
  2. What criteria were used to select the 19 tasks and dependency division, and how sensitive are the conclusions to task selection and aggregation across tasks with different numbers of dependency edges?
  3. How many independent runs were conducted per configuration, and can the authors report uncertainty estimates alongside end-to-end latency and coordination overhead?
  4. Why do you define DRS like this, which may be unfair across protocols? Can you show the wall-clock time or token budget?
  5. Does the hopping window still persist with different settings (dependency division strategy) or collaboration modes?
7 comments
AuthorsOpenReview ↗

Thanks(1/4)

We appreciate reviewer's valuable suggestions and recognization of this important problem. Following reviewer's concern, we believe we have solved all the concerns.

Q1: The validity of ADPR and checked the results manually.

This concern is related to the effectiveness of the proposed metric ADPR. As reviewer mentioned, more information is under this metric. ADPR evaluates collaboration effectiveness through executable cross-component dependency checks, which assess both upstream prerequisite functionality (U) and downstream dependent behavior (D). Since these U/D checks underpin ADPR, analyzing their outcomes allows us to further investigate whether differences in ADPR reflect coordination-related failures. We therefore examined 104 matched dependencies from two strong models, DeepSeek and Qwen3.8, comparing Single and Async-Manager.

Table 4.1. The U/D statistical result of strong models

MetricSingleAsync-Manager
Upstream Pass89/104 (85.6%)88/104 (84.6%)
Downstream Pass87/104 (83.7%)78/104 (75.0%)
U Pass, D Fail2/104 (1.9%)12/104 (11.5%)

The results show that upstream correctness remains nearly unchanged, while downstream performance declines under Async-Manager. In particular, U-pass/D-fail cases increase from 2 to 12, suggesting that collaboration introduces additional challenges in correctly utilizing upstream functionality, beyond upstream implementation failures.

To further validate this observation, we randomly selected five U-pass/D-fail task cases involving DeepSeek and Qwen3.8 and manually inspected the agents' execution trajectories and generated code diffs. All five cases exhibited coordination-related failures. A recurring pattern was that downstream agents incorrectly invoked or incorporated upstream functionality, produced incorrect implementations, and failed to correct them in subsequent iterations.

These findings provide both quantitative and execution-level evidence supporting ADPR as a meaningful measure of cross-component collaboration effectiveness, with U/D analysis revealing how coordination failures contribute to unresolved dependencies.

Q2: DRS Definition and Computational Budget

We thank the reviewer for this question. The motivation for DRS is to capture dependency-resolution dynamics that final ADPR cannot reveal. Rather than measuring execution speed, DRS tracks when cross-agent dependencies first become satisfied during integration. Since different protocols generate different numbers of integration checkpoints, we normalize the first-resolution step by each run's checkpoint count to characterize relative resolution progress. This distinction is meaningful: as shown in Section 5.3, Qwen3.6-27B and Qwen3.8-27B achieve similar ADPR under Async-Manager (63.16% vs. 64.91%) but different DRS values (0.641 vs. 0.719), revealing differences in their resolution trajectories that final coverage alone would miss.

Regarding computational budget, we apply the same limit of 100 model calls per agent, rather than a fixed token cap, to accommodate differences in token consumption across models. We further report actual token usage for all 50 configurations in Tables 10 and 11 and explicitly analyze quality–cost trade-offs in Section 5.5. Our Pareto analysis shows that greater token consumption does not necessarily improve dependency resolution, with 5 of 10 models exhibiting different frontiers under ADPR and Test Pass Rate. Thus, DRS characterizes relative dependency-resolution progress, while our token-based analysis separately accounts for computational efficiency across protocols.

AuthorsOpenReview ↗

Thanks(2/4)

Q3: Task Selection and Aggregation Sensitivity

We thank the reviewer for this question. Our task selection and dependency construction follow two key criteria: (1) tasks must contain genuine upstream–downstream dependencies and naturally parallelizable implementation responsibilities, enabling meaningful decomposition across multiple agents; and (2) tasks must originate from established public software engineering benchmarks or pull requests (PRs) in mature open-source projects, ensuring realistic development scenarios. We further conducted human reviews to verify the validity of identified dependencies and the feasibility of parallel task decomposition. The corresponding records are available in the Huaman_Review_Record directory of our anonymous repository. We will clarify these criteria and validation procedures in the revised manuscript.

To examine sensitivity to task selection and aggregation across different dependency counts, we conducted task-macro versus edge-micro ADPR comparisons, together with leave-one-task-out (LOTO) and leave-one-repository-out (LORO) analyses.

Table 4.2. The result of Aggregation Sensitivity. (Δ = Async-Manager − Single.)

ModelMacro Δ (pp)Micro Δ (pp)LOTO ReversalsLORO Reversals
Qwen3.5-9B−11.4−11.50/190/16
Qwen3.6-27B+26.3+21.20/190/16
Qwen3.8-27B−12.3−11.50/190/16
DeepSeek-V4−5.3−5.80/190/16

Across all 40 collaborative-versus-Single comparisons, task-macro and edge-micro aggregation preserve the same performance directions. Moreover, none of the ten Async-Manager-versus-Single comparisons reverses direction under any task or repository exclusion. These results further support our central finding that the benefits of multi-agent collaboration are model-dependent, with the main Async-Manager comparisons remaining directionally consistent across task selections and aggregation methods.

Q4: Benchmark scale

Thank reviewer for this question. First we want to explain the reason why the scale is low, that's because the standard of selecting tasks in our benchmark is strict. And we have to consider the following aspects:

  • SWE tasks require explicit cross-script dependencies—meaning upstream and downstream tasks—as well as tasks that can be processed in parallel, so they can genuinely be distributed to multi-agent systems for completion.

  • Meanwhile, the benchmark is required to source suitable tasks from already public, mature benchmarks or pull requests (PRs) of large open-source projects.

This precisely highlights the novelty of our benchmark and its ability to capture multi-agent collaboration issues that existing works fail to reflect.

However, we agree with reviewer's suggestion and we **expanded our benchmark **with 5 more tasks in different languages. So the whole benchmark has 24 tasks and 4 languages. The details of new tasks are as follows:

Table 4.3. The details of new tasks

New TaskRepository Languages
TVM-20409Python/C++
TVM-18421Python/C++
TVM-20543Python/C++
Jekyll-8771Ruby
Terraform-35543Go

And the results are as follows:

Table 4.4. The results of new tasks

ModelProtocolTest Pass ↑ADPR ↑DRS ↓Tokens
Qwen3.5-9BSingle56.7%20.0%—5.25M
Qwen3.5-9BSerial49.1%20.0%0.8675.78M
Qwen3.5-9BAsync-Private49.6%0.0%1.0007.84M
Qwen3.5-9BAsync-Manager-RO48.4%0.0%1.00012.35M
Qwen3.5-9BAsync-Manager49.2%20.0%0.86713.77M
Qwen3.5-27BSingle61.6%40.0%—5.50M
Qwen3.5-27BSerial43.5%0.0%1.0004.68M
Qwen3.5-27BAsync-Private49.6%0.0%1.0006.00M
Qwen3.5-27BAsync-Manager-RO52.0%20.0%0.95010.08M
Qwen3.5-27BAsync-Manager56.0%40.0%0.79412.32M
Qwen3.6-27BSingle85.0%60.0%—5.21M
Qwen3.6-27BSerial73.2%40.0%0.8005.80M
Qwen3.6-27BAsync-Private71.8%20.0%0.8507.32M
Qwen3.6-27BAsync-Manager-RO66.4%40.0%0.82710.52M
Qwen3.6-27BAsync-Manager70.2%60.0%0.72810.99M

And the results are consistent with our findings.

  • Test Pass Rate is high does not mean ADPR is high.
  • Single agent may outperform than the multi-agent system.
AuthorsOpenReview ↗

Thanks (3/4)

Q5: Commercial Model Evaluation

We thank the reviewer for this suggestion. To broaden our model coverage, we additionally evaluated Haiku 5.5 and Muse Spark 1.3, covering all 19 benchmark tasks and five execution protocols, resulting in 190 completed model–protocol–task evaluations.

Table 4.5. The commercial model results

ModelProtocolTest Pass ↑ADPR ↑DRS ↓Tokens (M)
Haiku 5.5Single75.0%70.2%—2.80
Serial60.8%56.1%0.8003.37
Async-Private50.6%38.6%0.7972.48
Async-Manager-RO49.5%33.3%0.8893.42
Async-Manager88.6%77.2%0.7586.75
Muse Spark 1.3Single78.9%78.9%—2.63
Serial73.2%70.2%0.7616.18
Async-Private60.3%55.4%0.7116.36
Async-Manager-RO63.6%59.6%0.8019.91
Async-Manager91.7%70.2%0.73311.08

These additional experiments reinforce our main findings. First, multi-agent collaboration does not universally outperform Single: Async-Manager improves ADPR for Haiku 5.5 (70.2% → 77.2%) but reduces it for Muse Spark 1.3 (78.9% → 70.2%). Second, writable manager coordination substantially recovers dependency resolution compared with Async-Private and Async-Manager-RO, with ADPR gains of +38.6/+43.9 pp for Haiku and +14.8/+10.6 pp for Muse, respectively.

Importantly, the results further demonstrate the distinction between task-level correctness and dependency-level collaboration. For Muse Spark 1.3, Async-Manager achieves the highest Test Pass Rate (91.7%) but lower ADPR than Single (70.2% vs. 78.9%). Based on the observed token costs and DRS differences, these findings further support our conclusion that collaboration effectiveness depends on both model capability and execution protocol, and cannot be fully characterized by Test Pass Rate alone.

Q6: Independent Runs, Uncertainty, and Coordination Overhead (Token and Wall-clock result)

We thank the reviewer for this suggestion. The original study included one experimental round per configuration. To assess variability, we conducted two additional rounds for three Qwen models under Single and Async-Manager, obtaining three observations for each of these six configurations.

Table 4.6. Multi-run Results

ModelProtocolTest Pass (95% CI)ADPR (95% CI)
Qwen3.5-9BSingle31.9% [15.3, 48.4]10.7% [−2.6, 24.0]
Async-Manager43.6% [27.1, 60.1]8.4% [−13.5, 30.4]
Qwen3.6-27BSingle55.3% [46.6, 64.1]38.3% [33.8, 42.8]
Async-Manager70.4% [56.7, 84.1]55.3% [38.2, 72.3]
Qwen3.8-27BSingle91.8% [87.6, 96.1]76.6% [74.1, 79.1]
Async-Manager84.7% [75.6, 93.8]64.3% [53.4, 75.3]

Despite run-to-run variability, the results remain consistent with our central finding that collaboration benefits are model-dependent. Async-Manager achieves higher ADPR than Single in all three rounds for Qwen3.6-27B, whereas Single consistently performs better for Qwen3.8-27B.

We further examined execution latency and coordination overhead across the five protocols. The observed protocol wall-clock times are summarized below.

Table 4.7. The wall-clock results

ProtocolMean Time (min)Median Time (min)
Single29.016.9
Serial86.341.4
Async-Private45.725.8
Async-Manager-RO87.949.5
Async-Manager102.167.6

Notably, Async-Private reduces average execution time compared with Serial (45.7 vs. 86.3 minutes), illustrating the efficiency benefits of parallel specialist execution. Manager-based protocols introduce additional coordination overhead, reflecting the cost of feedback, integration, and repair.

We also analyzed token consumption, as reported in Tables 10 and 11.

Table 4.8. The Token Usage

ModelSingle TokensAsync-Private TokensAsync-Manager Tokens
DeepSeek-V42.54M6.92M13.65M
Qwen3.6-27B4.96M8.18M14.49M
Qwen3.8-27B3.45M7.09M13.73M

Additionally, the Async-Manager logs record 819 manager intervention events, including 145 accepted repairs, providing direct evidence of coordination activity.

These results extend our quality–cost analysis in Section 5.5, showing that asynchronous execution can reduce elapsed time, while additional coordination incurs computational overhead without guaranteeing better dependency resolution.

Reviewer_Ub8DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AsynCodeBench evaluates multi-agent coding through explicitly specified dependencies. It contains 19 tasks, including 15 from Commit0 and 4 from Apache TVM PRs. Before model evaluation, the authors fix each task's division into 2 to 4 specialists and its dependency graph, yielding 52 edges overall. Each edge has upstream, downstream and integrated checkers built from pytest selectors. ADPR measures final dependency satisfaction. DRS measures the first successful integrated checkpoint, assigns T+1 to an unresolved edge, and normalizes by T+1; the hopping-window plots separately use k/T. Ten open-weight models are evaluated under Single, Serial, Async-Private and two manager protocols. The paper reports that collaboration benefits 6 of 10 models but harms the strongest ones, that Test Pass Rate and ADPR can disagree, and that generational gains in Qwen's standalone dependency resolution exceed those in Async-Manager. It also studies concentrated phases of first closure, including an aggregate window of 0.50<k/T≤0.750.50 < k/T \le 0.75.

Critical strengths
  1. Explicit dependencies and executable checks provide a useful view of integration beyond final task outcomes. Tracking their satisfaction at successive checkpoints is a reusable evaluation approach. Protocols share task packages, the harness, environments and evaluation checkers. Role permissions differ by design: Single owns the full task, specialists have scoped writable regions, and the writable manager has separate intervention constraints. These distinctions make the protocol definition important to interpreting the comparisons.

  2. The reporting includes complete configuration tables, token costs and a normalization check in Table 8. I could recover the main averages from Tables 10 and 11. The negative result for the strongest models is useful: Qwen3.8-27B and DeepSeek-V4 both achieve higher ADPR under Single than under any multi-agent protocol.

Critical weaknesses
  1. ADPR's interpretation as a coordination measure needs stronger controls. High-test/low-ADPR outcomes occur under Single as well: 22.1%, compared with 23.2 to 27.4% for multi-agent protocols, with no low-test/high-ADPR cases in Table 5. This could partly reflect all-or-nothing edge checks versus partial credit in Test Pass Rate, rather than coordination-specific failure. Appendix B explicitly gives managers access to dependency graphs and checker diagnostics, but does not fully specify equivalent access for Single. A protocol-by-protocol visibility matrix and feedback-matched ablation would resolve this uncertainty. Finally, the upstream and downstream checks in Sec. 3.2 could distinguish a composition failure from failure within one component. An aggregate breakdown of those outcomes, including Single, and an oracle-upstream control would make the coordination interpretation more convincing.

  2. The strength of the quantitative and temporal conclusions is limited by the task set and incomplete uncertainty analysis. Results cover 19 tasks and 52 edges, without reported variability across independent runs. A one-edge improvement in a three-edge task changes task-averaged ADPR by about 1.75 points, comparable to Async-Manager's 63.16% to 64.91% change from Qwen3.6 to Qwen3.8; this equivalence does not establish that only one actual edge changed. Table 8 already examines checkpoint normalization: 50.0% to 73.9% of peak-region events arise at k=2,T=3, while raw k=2 closures remain concentrated. A schedule-conditioned null model and full T distributions would help establish how much concentration exceeds that induced by integration opportunities, and whether the window was specified before inspecting results should be clarified. Across the 40 multi-agent configurations in Tables 10 and 11, I obtain ADPR–DRS Pearson r of approximately -0.985. This high correlation does not eliminate DRS's information: Sec. 5.3 shows similar ADPR with DRS 0.641 versus 0.719. A conditional analysis controlling for final coverage would clarify its additional value.

Questions for authors
  1. Please provide a protocol-by-protocol account of access to dependency graphs, checker definitions and outputs, and public tests. How does Async-Manager perform without checker feedback, or when Single receives equivalent feedback?

  2. For every protocol, including Single, can failed edges be separated into (U pass, D pass, I fail), (U pass, D fail) and (U fail)? An oracle-upstream experiment would be another useful control. Evidence of substantially more composition failures in multi-agent runs than in Single would strengthen the central interpretation.

  3. To what extent do checker selectors overlap with evaluation items? Does any run achieve 100% Test Pass Rate while retaining ADPR below 1?

  4. Could the main comparisons in Fig. 2a, 3 and 4a be repeated at least 3 times and reported with confidence intervals?

  5. What is the distribution of T for each protocol? Beyond Table 8's normalization analysis, does closure concentration exceed a null model conditioned on each run's integration schedule, and was the hopping window selected in advance?

5 comments
AuthorsOpenReview ↗

Thanks(1/4)

First of all, we really appreciate that reviewer recovered our benchmark, which is a great encouragement to us. And reviewer's valuable comments make our work more rogirous.

Q1: The account of protocol access and the feedback ablation.

We thank for this question and this question is more about the key difference between all the protocols. To better demonstrate the difference of the access between all the protocols. We provide the table to show the details.

Table 3.1. The details of protocol access

ProtocolDependency GraphChecker DefinitionsChecker FeedbackPublic Tests
SingleFull graphNo explicit mappingNoYes
SerialAssigned dependencies + previous handoffsPrevious checker mappings available to later agentsYes, from previous agentsYes
Async-PrivateAssigned dependencies onlyNo explicit mappingNoYes
Async-Manager-ROFull graph (Manager); assigned dependencies (Specialists)No explicit mappingNo automatic feedbackYes
Async-ManagerFull graph (Manager); assigned dependencies (Specialists)Manager receives checker mappingsYes, to ManagerYes
Manager-NoFeedbackFull graph (Manager); assigned dependencies (Specialists)No explicit mappingNoYes

And then we conduct the ablation study of Dependency Checker feedback. We evaluated Async-Manager-NoFeedback on three Qwen models across all 19 tasks, preserving the same collaboration structure, task decomposition, and model-call budgets.

Table 3.2. The results of feedback ablation

ModelNoFeedback ADPR ↑Manager ADPR ↑Δ ADPR ↑NoFeedback Test Pass ↑Manager Test Pass ↑
Qwen3.5-9B4.6%2.8%−1.8 pp43.2%36.9%
Qwen3.6-27B43.0%63.2%+20.2 pp59.7%76.7%
Qwen3.8-27B62.3%64.9%+2.6 pp79.3%86.7%

The results show that the benefit of structured checker feedback is model-dependent. Qwen3.6-27B benefits substantially, with a 20.2 pp ADPR improvement (95% task-bootstrap CI: [5.26, 36.84] pp) and a 17.0 pp increase in Test Pass Rate. In contrast, Qwen3.8-27B shows only a 2.6 pp ADPR improvement, while Qwen3.5-9B exhibits no clear benefit. Notably, Qwen3.8-27B remains below its Single ADPR of 77.2% even with feedback. These results suggest that checker feedback can facilitate dependency resolution, but providing additional coordination information alone does not guarantee effective multi-agent collaboration. And this ablation study makes our experiments more rigorous.

AuthorsOpenReview ↗

Thanks(2/4)

Q2: The analysis of UDI results and the further interpretation of multi-agent system failure.

We thank the reviewer for this suggestion. To distinguish individual implementation limitations from multi-agent collaboration difficulties, we further analyzed the Upstream (U), Downstream (D), and Integration (I) checker outcomes across all five protocols.

1. Quantitative Analysis of Collaboration Failures

We first examined 104 matched dependencies from two strong Single-agent models, DeepSeek and Qwen3.8, comparing Single and Async-Manager.

Table 3.3. Analysis of strong models of collaboration Failures

MetricSingleAsync-Manager
Upstream Pass89/104 (85.6%)88/104 (84.6%)
Downstream Pass87/104 (83.7%)78/104 (75.0%)
U Pass, D Fail2/104 (1.9%)12/104 (11.5%)

Upstream satisfaction remains nearly unchanged (85.6% vs. 84.6%), whereas downstream satisfaction decreases from 83.7% to 75.0%. More importantly, cases where upstream functionality passes but downstream functionality fails increase from 2 to 12. This suggests that the degradation under multi-agent execution cannot be explained solely by insufficient upstream implementation capability.

2. Manual Validation of Coordination Failures

To further examine these failures, we manually reviewed five selected U-pass/D-fail cases involving DeepSeek and Qwen3.8. We inspected the agents' execution trajectories and generated code diffs, tracing upstream implementations, downstream usage, and integration behavior.

All five inspected cases exhibited coordination-related difficulties, including incorrect downstream use of upstream functionality, incompatible implementations, and unsuccessful integration repairs. These observations provide execution-level evidence that successful upstream implementation does not necessarily translate into effective multi-agent collaboration.

3. Complete U/D/I Failure Breakdown

Following the reviewer's suggestion, we also classified failed dependencies across all ten models, 19 tasks, and five protocols.

Table 3.4. U/D/I Failure Breakdown results

ProtocolFailed EdgesU FailU Pass, D FailU/D Pass, I Fail
Single317244 (77.0%)73 (23.0%)0
Serial355298 (83.9%)57 (16.1%)0
Async-Private390311 (79.7%)79 (20.3%)0
Async-Manager-RO369273 (74.0%)96 (26.0%)0
Async-Manager296227 (76.7%)68 (23.0%)1 (0.3%)

The high proportion of upstream failures partly reflects our evaluation of models with varying coding capabilities, including weaker models. More importantly, the U/D/I checker design underlying ADPR enables a finer-grained analysis of collaboration failures. As demonstrated by DeepSeek-V4 and Qwen3.8-27B, our metrics can reveal how strong individual coding capability may fail to translate into effective multi-agent collaboration.

We also examined the limited occurrence of U/D-pass, I-fail cases by analyzing the checker structure:

Table 3.5. The checker structure

Integrated Checker StructureDependenciesIndependent I Failure Observable
I = U ∪ D10No
I ⊂ U ∪ D31No
I ⊃ U ∪ D2Yes
Mixed overlap with I-only selectors9Yes
Total5211/52 (21.2%)

For 41 of the 52 dependency contracts, satisfying both U and D guarantees that I also passes. Therefore, the low frequency of independently observable integration failures is largely explained by the checker structure, rather than indicating an absence of coordination difficulties.

Q3: The extent of checker selectors overlap with evaluation items

Thank reviewer for the question. All 443 dependency-checker test items are included in the 4,874-item final evaluation suite (100% overlap). This is consistent with our benchmark design: ADPR reuses existing executable tests to evaluate predefined cross-component dependency satisfaction, providing a dependency-level perspective rather than introducing a separate test suite.

Across all 950 evaluated runs, 78 achieved exactly 100% Test Pass Rate, and all 78 also achieved ADPR = 1. No run exhibited 100% Test Pass Rate with ADPR below 1, consistent with the fact that all dependency-checker items are included in the final evaluation suite.

AuthorsOpenReview ↗

Thanks(3/4)

Q4: Multi-seed run, the confidence interval and Statistical Uncertainty

Thank reviewer for this suggestion. Due to time and computational resource constraints, we selected three representative Qwen models (Qwen3.5-9B, Qwen3.6-27B, and Qwen3.8-27B) and conducted two additional runs under both Single and Async-Manager, resulting in three runs per configuration across the original 19 tasks. We report the mean and 95% confidence intervals for Test Pass Rate, ADPR, and normalized DRS.

Table 3.6. Multi-run evaluation results (mean [95% CI]).

ModelProtocolTest Pass (%) ↑ADPR (%) ↑Normalized DRS ↓
Qwen3.5-9BSingle31.87 [15.33, 48.41]10.70 [−2.55, 23.95]—
Async-Manager43.62 [27.12, 60.11]8.42 [−13.51, 30.36]0.955 [0.830, 1.079]
Qwen3.6-27BSingle55.33 [46.55, 64.11]38.01 [32.98, 43.04]—
Async-Manager70.35 [56.65, 84.06]54.39 [35.39, 73.39]0.689 [0.569, 0.809]
Qwen3.8-27BSingle91.83 [87.53, 96.12]76.61 [74.10, 79.12]—
Async-Manager84.69 [75.63, 93.75]63.45 [53.15, 73.75]0.719 [0.702, 0.737]

The additional runs are consistent with the main findings of our original paper. Qwen3.6-27B maintains higher ADPR under Async-Manager than Single, whereas Qwen3.8-27B consistently achieves higher ADPR under Single across all three runs. The three-run averages for Qwen3.5-9B also retain the original direction of the ADPR comparison. These results reinforce our central observation that stronger standalone coding capability does not necessarily translate into better multi-agent collaboration performance, and that the benefits of collaboration depend on the model and execution protocol.

Q5: Distribution of Checkpoint T and the statistical analysis of hopping window

We thank the reviewer for this suggestion. We re-analyzed all 760 collaborative execution trajectories (190 per protocol across the four protocols), examined the distribution of checkpoint counts, and employed a schedule-conditioned null model to test whether dependency closure exhibits clustering beyond random expectations. The results are as follows:

Table 3.7. Checkpoint Distribution

ProtocolMean TMedian TRange TMost Frequent T
Serial3.9642–54 (71/190)
Async-Private4.0543–54 (80/190)
Async-Manager-RO5.6463–106 (42/190)
Async-Manager6.0763–126 (45/190)

Based on the table, we can find that manager protocols exhibit longer and more variable checkpoint trajectories, motivating schedule-conditioned analysis.

Given the differences in checkpoint distributions, we further examined whether closure concentration exceeds what would be expected from each run's integration schedule. We constructed a null model that preserves the actual checkpoint schedule and the number of eventually resolved dependencies, while independently randomizing their first-closure positions across eligible workspace-changing checkpoints. Using 20,000 Monte Carlo permutations, we compared the observed closure burstiness against the null expectation.

Table 3.8. The distribution of Checkpoint

ProtocolEligible RunsObserved BurstinessNull MeanDifferenceAdjusted p
Serial410.65450.3984+0.25610.00020
Async-Private220.65760.4048+0.25270.00055
Async-Manager-RO320.89270.3405+0.55220.00020
Async-Manager560.72500.2724+0.45260.00020

All four protocols exhibit significantly greater closure burstiness than their schedule-conditioned null expectations, indicating that dependency resolution tends to occur in concentrated bursts rather than being uniformly distributed across integration opportunities.

And we clarify that the interval (0.50, 0.75] was selected descriptively after examining the aggregate trajectories, rather than specified in advance. Our notion of hopping refers to concentrated dependency-closure events, without assuming that they occur at a fixed normalized execution stage. For example, in Figure 5, the hopping window of Qwen3-Coder-Next is not (0.50, 0.75].

Reviewer_U4s4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AsynCodeBench, a benchmark for evaluating cross-component collaboration in asynchronous multi-agent software engineering. It comprises 19 tasks from 16 real-world software repositories and defines dependency graphs and executable checkers for 52 directed dependencies. The authors propose the Asynchronous Dependency Pass Rate (ADPR), which measures whether dependencies are satisfied in the final integrated workspace, and the Dependency Resolution Step (DRS), which tracks when each dependency is first satisfied during execution. They evaluate multiple models under five execution protocols. The results show that task-level test outcomes can diverge from dependency-level scores, and that improvements in coding performance do not necessarily lead to better dependency resolution. The benchmark aims to provide a dependency-level perspective that complements task-completion metrics for multi-agent coding.

Critical strengths
  1. Practically relevant problem. The paper studies cross-component dependencies in asynchronous multi-agent coding, providing a useful complement to task-level evaluation and exposing integration defects that aggregate test pass rates may obscure.

  2. Concrete and verifiable evaluation design. The benchmark uses tasks from real software repositories, with human-reviewed dependency graphs and predefined executable checkers that make dependency satisfaction explicit and repeatedly testable.

  3. Informative empirical analysis. The paper compares multiple models and execution protocols within a consistent evaluation framework. Its analysis of dependency-resolution trajectories and resource costs provides insights into intermediate progress and performance trade-offs beyond final task scores.

Critical weaknesses
  1. The evaluation lacks representative multi-agent baselines. Most comparisons are between execution protocols designed by the authors, with no evaluation of existing systems such as ChatDev [1] or systems built with AutoGen [2]. This limits the external validity of the results. It also leaves unclear whether the benchmark can meaningfully distinguish collaboration patterns across established multi-agent architectures.
  2. The effect of agent count needs a controlled analysis. The experiments do not systematically vary the number of agents to examine its effect on task performance, dependency resolution, or the claimed collaboration capability. Changing agent count also changes task decomposition, communication overhead, and coordination complexity, making these effects difficult to disentangle. Experiments on the same tasks under comparable resource budgets are needed to support the paper's conclusions about collaboration capability and how it scales.
  3. The metrics' contribution beyond prior failure analysis and attribution work is unclear, and their diagnostic value needs stronger validation. Who&When [3], ECHO [4], and Why Do Multi-Agent LLM Systems Fail? [5] have already examined failure localization, error propagation, and problems in system-level interactions. The proposed dependency graphs and executable checkers provide information about whether and when dependencies are satisfied. However, the current evidence does not show that these metrics can reliably distinguish an individual agent's implementation limitations from coordination failures across agents. This leaves the central claim of separating collaboration capability from coding capability unsupported. The software engineering setting offers an opportunity to test that claim directly through executable validation. Targeted experiments would provide stronger diagnostic evidence than aggregating test outcomes into dependency-level scores alone.

[1] Qian C, Liu W, Liu H, et al. Chatdev: Communicative agents for software development[C]//Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers). 2024: 15174-15186.

[2] Wu Q, Bansal G, Zhang J, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation[J]. arXiv preprint arXiv:2308.08155, 2023.

[3] Zhang S, Yin M, Zhang J, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems[J]. arXiv preprint arXiv:2505.00212, 2025.

[4] Banerjee A, Nair A, Borogovac T. Where did it all go wrong? a hierarchical look into multi-agent error attribution[J]. arXiv preprint arXiv:2510.04886, 2025.

[5] Cemri M, Pan M Z, Yang S, et al. Why do multi-agent llm systems fail?[J]. Advances in Neural Information Processing Systems, 2026, 38.

Questions for authors

See Weaknesses

7 comments
AuthorsOpenReview ↗

Thanks(1/2)

We thank the reviewer for bringing new perspectives to our work. Following the suggestions has made our contributions much clearer.

Q1: More established multi-agent architecture baselines.

Thank reviewer for this helpful suggestion. Due to the time and computational limitation, we added AutoGen as an external multi-agent baseline with Planner–Coder–Reviewer roles, evaluating three Qwen models across all 19 tasks using the same evaluators and dependency checkers. Both AutoGen and our native multi-agent protocols use a 100-model-call limit per agent.

Table 2.1. The result of AutoGen baseline

ModelTest Pass Rate ↑ADPR ↑Normalized DRS ↓Mean Tokens
Qwen3.5-9B23.2%11.2%0.9396.49M
Qwen3.6-27B43.8%35.1%0.7805.80M
Qwen3.8-27B23.9%17.5%0.9506.16M

Evaluation Results. The results further illustrate that Test Pass Rate does not fully capture dependency resolution. For example, Qwen3.8-27B achieves a 71.2% Test Pass Rate on Cachetools but 0% ADPR; this run produced no code changes, showing that substantial test passage can coexist with unresolved target dependencies. Under the evaluated AutoGen configuration, Qwen3.8-27B also underperforms Qwen3.6-27B. Inspection of the recorded tool traces reveals repeated repository inspection in several Qwen3.8 runs. On Cachetools, TinyDB, and wcwidth, the coder exhausted its 100-call model budget without producing code changes. This findings reveal deeper phenomenone of multi-agent system.

Dependency Trajectory Analysis. ADPR and DRS further reveal behaviors hidden by final task metrics. In Python-RSA, satisfied dependencies regress from 1 to 0 before recovering to 3, while in Cachetools, four dependencies close at a single checkpoint. We also observe the hopping phenomenon in AutoGen: among 13 runs that resolved at least two dependencies, 9 (69.2%) exhibited multiple first closures at the same checkpoint. These observations are consistent with the concentrated dependency-resolution dynamics reported in our original experiments.

These results demonstrate that AsynCodeBench generalizes to an established multi-agent framework and captures dependency-level collaboration behaviors that conventional Test Pass Rate alone cannot reveal.

Q2: Effect of Agent Count on Collaboration Performance

We thank the reviewer for this suggestion. To investigate the effect of agent count, we conducted an additional matched-task comparison between four and two specialists, each with one persistent Async-Manager, using Qwen3.6-27B and Qwen3.8-27B on the same five tasks. Both configurations preserve the original implementation units, dependency definitions, and evaluation checkers, with the same model-call budget limits for each specialist and manager.

Table 2.2. The Agent Count Result

ModelSpecialistsADPR ↑Task Success ↑Total Tokens (M)
Qwen3.6-27B453.3%1/596.96
233.3%1/571.86
Qwen3.8-27B480.0%4/561.67
246.7%2/557.70
Overall466.7%5/10158.62
240.0%3/10129.56

Reducing the number of specialists decreased total token consumption by 18.3%, but also reduced ADPR from 66.7% to 40.0%, while successful task completions decreased from five to three. These results show that fewer agents do not necessarily lead to better task performance or dependency resolution.

To further investigate the influence of task decomposition, we reconstructed the ownership of all 30 evaluated dependencies. Under four specialists, all dependencies were cross-agent. Reducing the team to two specialists internalized 10 dependencies, while 20 remained cross-agent.

Table 2.3. The Agent Count Detailed Results

Dependency Structure (4 → 2)ADPR with 4 SpecialistsADPR with 2 Specialists
Remained cross-agent (20 dependencies)65.0%35.0%
Became within-agent (10 dependencies)70.0%50.0%

Notably, reducing cross-agent coordination requirements did not automatically improve dependency resolution. Even among dependencies that became internal to a single specialist, resolution decreased from 70.0% to 50.0%, while dependencies that remained cross-agent declined from 65.0% to 35.0%.

This finding further highlights the motivation behind AsynCodeBench: collaboration effectiveness cannot be inferred simply from team size or the apparent complexity of task decomposition. Instead, our benchmark provides executable dependency-level measurements that quantify whether models can successfully satisfy the functional dependencies required for collaborative software development. For example, Qwen3.6-27B maintains the same final task success (1/5) under both team configurations, while ADPR reveals a substantial decrease in dependency resolution (53.3% to 33.3%).

AuthorsOpenReview ↗

Thanks (2/2)

Q3: A lack of evidence demonstrating that proposed metrics can effectively distinguish between the implementation limitations of individual agents and the collaborative issues of multi-agent systems.

We thank reviewer for this comment. All the studies reviewer's mentioned, including Who&When, ECHO, and Why Do Multi-Agent LLM Systems Fail?, primarily focus on identifying and attributing failures in multi-agent systems. In contrast, our work aims to evaluate collaboration capability through executable software dependencies, rather than merely identifying which agent fails. ADPR builds on Upstream (U), Downstream (D), and Integration (I) checkers to measure whether dependencies are successfully satisfied during collaborative coding.

To investigate this, in our work, we explicitly designed Single, Serial, and Async-Private protocols. To better demonstrate our method can provide more unseen insights, we analyze the dependencies successful resolution. Among 147 dependencies successfully resolved under Single, Serial preserves 97 (66.0%), while Async-Private preserves only 74 (50.3%), a difference of 15.6 percentage points (task-cluster bootstrap 95% CI: [6.0, 26.1]).

The Single results establish that the underlying model can solve these dependencies in a standalone setting. However, this capability is not fully preserved when the work is distributed across multiple agents. Moreover, Serial outperforms Async-Private despite using the same model and task decomposition, suggesting that access to upstream implementations and coordination during execution play important roles in successful dependency resolution. This illustrates how ADPR captures the extent to which coding capability translates into effective multi-agent collaboration, rather than measuring coding performance alone.

And we will add these valuable insights to our paper.

I sincerely appreciate the authors’ thoughtful response, and I recognize the effort involved in developing a benchmark. I think the paper’s goal of defining collaboration capability is valuable, and I understand how this differs from attributing failures along MAS trajectories. Still, I find the current definition somewhat simple and coarse-grained.

I would like to ask an open-ended question: In future work, how do the authors envision defining “collaboration capability” for different MAS architectures, such as sequential and hierarchical systems?

I look forward to hearing your thoughts on this question and will update my rating to 3 in my next review.

Reviewer_RbLyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces a benchmark for evaluating multi-agent coding systems while focusing on how well the interfile dependencies are handled by asynchronous agents. The experiments are done on one shared framework and evaluated on 10 models on 19 tasks derived from Commit0 and Apache TVM. Four research questions show observations:

  1. Multi-agent is not always better than single agent.
  2. The models better at single-agent coding, such as by scaling up Qwen 3.5, might have ADPR metric dropped instead.
  3. There is a middle "hopping window" where the dependencies are resolved in most of them.
  4. Metric matters a lot, in terms of which models' qualities are the best.
Critical strengths
  1. The novelty lies in systematically evaluating dependency resolution in a multiagent coding environment and also proposing metrics.
  2. The experiment part is relatively transparent and detailed, providing negative results honestly.
Critical weaknesses
  1. The experiment scale is relatively low, with 19 tasks and 52 dependencies, compared to the realistic coding problem settings.
  2. Dependency graphs are designed as static and only validated/evaluated for correctness. In real-world coding problems, because of environmental requirements or function updates, the dependency graph can evolve, which is usually the case and complicates the problem significantly.
  3. the existence of the hopping window is only subjectively checked, and statistical significance would be helpful in confirming its existence. Also, analysis of why this was the case would be useful
Questions for authors

n/a

5 comments
AuthorsOpenReview ↗

Thanks (1/2)

We thank the reviewer for your valuable suggestions, which have helped make our work much clearer.

Q1: The experiment scale is relatively low, with 19 tasks and 52 dependencies, compared to the realistic coding problem settings.

Thank reviewer for this comment. First we want to explain the reason why the scale is low, that's because the standard of selecting tasks in our benchmark is strict. And we have to consider the following aspects:

  • SWE tasks require explicit cross-script dependencies—meaning upstream and downstream tasks—as well as tasks that can be processed in parallel, so they can genuinely be distributed to multi-agent systems for completion.

  • Meanwhile, the benchmark is required to source suitable tasks from already public, mature benchmarks or pull requests (PRs) of large open-source projects.

This precisely highlights the novelty of our benchmark and its ability to capture multi-agent collaboration issues that existing works fail to reflect.

However, we agree with reviewer's suggestion and we expanded our benchmark with 5 more tasks in different languages. So the whole benchmark has 24 tasks and 4 languages. The details of new tasks are as follows:

Table 1.1. The details of new tasks

New TaskRepository Languages
TVM-20409Python/C++
TVM-18421Python/C++
TVM-20543Python/C++
Jekyll-8771Ruby
Terraform-35543Go

And the results are as follows:

Table 1.2. The results of new tasks

ModelProtocolTest Pass ↑ADPR ↑DRS ↓Tokens
Qwen3.5-9BSingle56.7%20.0%—5.25M
Qwen3.5-9BSerial49.1%20.0%0.8675.78M
Qwen3.5-9BAsync-Private49.6%0.0%1.0007.84M
Qwen3.5-9BAsync-Manager-RO48.4%0.0%1.00012.35M
Qwen3.5-9BAsync-Manager49.2%20.0%0.86713.77M
Qwen3.5-27BSingle61.6%40.0%—5.50M
Qwen3.5-27BSerial43.5%0.0%1.0004.68M
Qwen3.5-27BAsync-Private49.6%0.0%1.0006.00M
Qwen3.5-27BAsync-Manager-RO52.0%20.0%0.95010.08M
Qwen3.5-27BAsync-Manager56.0%40.0%0.79412.32M
Qwen3.6-27BSingle85.0%60.0%—5.21M
Qwen3.6-27BSerial73.2%40.0%0.8005.80M
Qwen3.6-27BAsync-Private71.8%20.0%0.8507.32M
Qwen3.6-27BAsync-Manager-RO66.4%40.0%0.82710.52M
Qwen3.6-27BAsync-Manager70.2%60.0%0.72810.99M

And the results are consistent with our findings.

  • Test Pass Rate is high does not mean ADPR is high.
  • Single agent may outperform than the multi-agent system.

Q2: The dependency graph interpretation.

We appreciate reviewer's question. This is a good point to clarify our contribution.

First, regarding our research objective. Our benchmark does not focus on the dependency graph during the complete SWE lifecycle. Instead, we are evaluating whether, when a coding task has already included explicit cross-script dependencies, a multi-agent system could solve, collaborate, and integrate the functions successfully.

And then we want to explain why our dependency graph is static.

  1. The dependency is from the real task requirements. According to Appendix A.1, our benchmark tasks need to include producer-consumer relationships that can be verified through code, interfaces, shared state, or tests. So the dependency graph is not designed manually, but is identified from the real task.
  2. The fixed graph serves as a consistent evaluation specification. If the dependency graph used for evaluation also changes when the same task faces different models or protocols, it becomes very difficult to determine whether differences in ADPR stem from the agent's collaborative performance or from changes in the evaluation target itself. We provide a real task in our benchmark as an example to illustrate. To implement the Cachetools task, the agent has to finish the func.py script before implementing Key Construction in the keys.py script successfully and using it correctly. This dependency doesn't arise simply because we assign the task to two agents. Even with a single agent, func.py still relies on func.py to function properly.

This also remind us to emphasize the contribution of the metrics we proposed:

We are not making the entire coding process static. Even if the graph is fixed, dependencies can still go through:

0 → 0 → 1

indicating that after multiple rounds of implementation and integration, the dependency is eventually resolved. They can also go through:

0 → 1 → 0

indicating that a previously satisfied dependency subsequently experiences a regression. This precisely demonstrates that what remains fixed is the definition of the dependency relationships, rather than the dependency satisfaction states. DRS investigates the time of first satisfaction, whereas ADPR investigates the final satisfaction outcome. And our proposed metrics are able to check the dynamics.

AuthorsOpenReview ↗

Thanks(2/2)

Q3: the existence of the hopping window is only subjectively checked, and statistical significance would be helpful in confirming its existence. Also, analysis of why this was the case would be useful.

We appreciate this comment and we present the statistical significance.

To establish the statistical significance of the hopping phenomenon, our evaluation encompasses 10 models, 19 tasks, and 4 collaboration protocols, yielding a total of 760 execution runs, 2,080 dependency instances, and 510 first closure events.

We implemented a Monte Carlo randomization test. Specifically, while holding the number of checkpoints and successfully resolved dependencies constant for each run, we randomly permuted the execution positions at which these dependencies achieved their first closure, iterating the simulation 10,000 times.

We then contrasted the proportion of first closures falling within the top 25% most concentrated execution intervals of the empirical data versus the randomized null model, thereby evaluating the statistical significance of the observed clustering.

Table 1.3. The statistical significance results

ProtocolFirst ClosuresObserved ConcentrationNull Expectedp-valueBH-adjusted q
Serial13378.2%37.1%0.000100.00013
Async-Private8366.3%37.9%0.000100.00013
Async-Manager-RO11276.8%37.5%0.000100.00013
Async-Manager18247.8%36.5%0.000300.00030

As shown by the results, the dependency first closure concentration across all four protocols is significantly higher than random expectations (BH-adjusted q<0.001q < 0.001). For instance, under the Serial protocol, 78.2% of the first closure events are concentrated within the most dense 25% execution interval, whereas the expected proportion under random baselines is only 37.1%. Async-Private and Async-Manager-RO exhibit similar significant concentration phenomena.

These findings indicate that the hopping phenomenon is not merely an artifact of subjective curve observation. Controlling for the number of checkpoints and first closure counts per run, dependencies' first resolutions still exhibit significant episodic clustering rather than a uniform random distribution. Furthermore, this concentration phase is not fixed within a specific interval.

We believe this result will improve the significance of our benchmark.

Understanding the Hopping Phenomenon. Our trajectory analysis provides further insight into this behavior. Rather than resolving dependencies independently and sequentially, multi-agent collaboration involves interdependent implementation units whose functionality must be jointly satisfied. As observed in the dependency trajectories, several dependencies can remain unresolved across successive integration checkpoints before becoming satisfied together during a later integration stage. This produces the characteristic plateau-and-burst pattern, where dependency resolution progresses through concentrated stages rather than uniformly across checkpoints.

An explanation is that changes to shared upstream functionality, downstream interfaces, or their integration can enable multiple dependencies to pass their checks together. This highlights the coupled nature of collaborative software development: progress on individual implementation units does not necessarily translate into immediate dependency resolution, while successful integration can resolve multiple dependencies at once.

AuthorsOpenReview ↗

Thank You for Your Feedback & Final Follow-up

Dear Reviewer RbLy,

Thank you very much for your time and valuable suggestions. As mentioned in our previous response, we have addressed all of your comments and questions and updated our paper accordingly.

With only a few hours remaining in the author-reviewer discussion phase, we would be more than happy to answer any further questions or continue the discussion if you have any additional thoughts. Thank you again for your time and help!

Best regards,

The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary for first round discussion (2/2)

  • Execution latency and coordination overhead. We analyzed wall-clock execution time and token consumption across all five protocols, showing that asynchronous execution reduces latency compared with Serial, while manager-based coordination introduces additional computational costs without guaranteeing better dependency resolution. (Reviewer 7PxS)

Thank reviewers again for all constructive comments. Please let us know if having any further questions or concerns.

[1] Wu Q, Bansal G, Zhang J, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation[J]. arXiv preprint arXiv:2308.08155, 2023.

AuthorsOpenReview ↗

Summary for first round discussion (1/2)

We thank all reviewers for their valuable and constructive feedback, as well as their recognition of the importance of our work. We believe that we have responsed all the concerns and provided more detailed and statistical analysis and evidence to reinforce the innovation and contribution of AsynCodeBench.

Here is a summary of the questions and our corresponding responses in this round of the rebuttal:

  • The scale of AsynCodeBench. We have expanded the benchmark by adding 5 additional tasks and extended its overall language support to 4 languages. We tested these 5 new tasks and obtained consistent results. (Reviewer RbLy, 7PxS)
  • The Intepretation of Dependency graph. We have provided a detailed explanation of why our dependency graph needs to be static, while also clarifying that the dependency satisfaction states remain dynamic during the agent's execution process. (Reviewer RbLy)
  • Statistical validation and robustness of the Hopping Window. We conducted Monte Carlo significance tests and schedule-conditioned analyses across 760 trajectories, confirming significant dependency-closure concentration across all four protocols. We also analyzed checkpoint distributions and execution trajectories, showing that hopping is not restricted to a fixed normalized logical checkpoint interval. (Reviewer RbLy, Ub8D, 7PxS)
  • More mature multi-agent system framework baseline. We evaluated AutoGen's [1] planner–coder–reviewer workflow on our benchmark, obtaining insights consistent with our findings and observing the same hopping window phenomenon.(Reviewer U4s4)
  • The effect of agent count. We compared 2 versus 4 specialists and analyzed dependency decomposition, showing that fewer agents reduce computational costs but do not necessarily improve dependency resolution, highlighting the value of AsynCodeBench in measuring collaboration capability. (Reviewer U4s4)
  • Distinguishing coding capability from collaboration capability. We compared dependency resolution across Single, Serial, and Async-Private protocols, showing that individual coding capability does not necessarily translate into successful multi-agent collaboration, which AsynCodeBench can effectively quantify through ADPR. (Reviewer U4s4)
  • Protocol access and feedback ablation. We clarified the information access of each protocol and conducted a checker-feedback ablation, showing that structured feedback can improve dependency resolution, but its benefits remain model-dependent. (Reviewer Ub8D)
  • U/D/I failure analysis and collaboration capability. We analyzed U/D/I outcomes across all five protocols and manually reviewed five failure cases, showing how AsynCodeBench identifies collaboration difficulties even when upstream implementation remains successful. (Reviewer Ub8D)
  • Overlap between dependency checkers and evaluation tests. We confirmed that all dependency-checker tests are included in the final evaluation suite, and all 78 runs achieving 100% Test Pass Rate also achieved ADPR = 1, validating the consistency of our evaluation design. (Reviewer Ub8D)
  • Multi-run evaluation and statistical uncertainty. We conducted two additional runs for three Qwen models and reported 95% confidence intervals, confirming that the observed model-dependent collaboration trends remain consistent across experimental rounds. (Reviewer Ub8D, 7PxS)
  • ADPR–DRS correlation and complementary information. We conducted a coverage-conditioned analysis of matched dependencies, showing that models with similar ADPR can exhibit significantly different dependency-resolution trajectories, confirming that DRS captures complementary temporal information. (Reviewer Ub8D)
  • ADPR validity and manual verification. We analyzed 104 matched dependencies and manually reviewed five failure cases, providing quantitative and execution-level evidence that ADPR captures collaboration difficulties beyond individual implementation limitations. (Reviewer 7PxS)
  • DRS definition and computational budget. We clarified that DRS measures dependency-resolution progress across logical checkpoints rather than execution speed, and explained our model-call budgets and token-based quality–cost analysis. (Reviewer 7PxS)
  • Task selection and aggregation sensitivity. We clarified our task selection criteria and human validation process, and conducted task-macro, edge-micro, and leave-one-out analyses, confirming that our main findings remain robust to task selection and aggregation methods. (Reviewer 7PxS)
  • Expanded commercial model evaluation. We evaluated two additional API models, Haiku 5.5 and Muse Spark 1.3, across all five protocols and 19 tasks, confirming the model-dependent benefits of collaboration and the distinction between Test Pass Rate and dependency resolution. (Reviewer 7PxS)
Area_Chair_HHXuOpenReview ↗

Please engage in the discussion

Dear Reviewers,

As the ddl of author-reviewer discussion is approaching, could you please read the authors' responses and update your reviews if necessary?

As you know, we have only four reviewers in total. So each of your assessments may have a crucial impact on the final decision. Please be active. Thanks!

Regards,

Your AC.

AuthorsOpenReview ↗

Revised Paper Uploaded

Dear Reviewers,

We have uploaded the latest revised version of our paper. For your convenience, all newly added content has been highlighted in red. Thank you very much for your valuable comments and suggestions! Below is a summary of the main revisions we have made:

In the Main Text:

  1. Added an analysis of the hopping window.

In the Appendix:

  1. Added details on Task Selection Criteria and Aggregation Sensitivity.
  2. Added experimental results for Protocol Visibility Specification and Checker-Feedback Ablation.
  3. Added experimental results for Multi-Language Benchmark Expansion.
  4. Added experimental results and analysis for Evaluation of an External Multi-Agent Framework: AutoGen.
  5. Added experimental results for Evaluation of Commercial Models.
  6. Added experimental results for Diagnostic Decomposition of Dependency Failures via U/D/I Checkers.
  7. Added Statistical Significance and Schedule-Conditioned Diagnostics for the hopping window.
  8. Added an analysis on Distinguishing Individual Coding Capability from Collaboration Loss.
  9. Added results on Multi-Run Stability and Statistical Uncertainty.

As the deadline for the author-reviewer discussion phase approaches, we would be more than happy to engage in further discussions to make our work even stronger and contribute to our community together. Thank you for your time!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.