- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes AsyncCodeBench with 19 tasks from 16 repositories, and curate dependency relations. This paper proposes two new metrics to evaluate how fast the function can be achieved with different collaboration modes or how well the agent collaborates. The paper evaluates 10 models and 5 collaboration modes, revealing the hopping window in multi-agent coding.
Critical strengths
- Important problem, and the finding highlights a meaningful gap: final task correctness alone cannot distinguish coding failures from asynchronous coordination failures.
- Executable, dependency-aware evaluation. The benchmark uses repository-level tests and explicit producer–consumer dependency checkers, offering better interpretability.
- Useful protocol comparisons. Comparing serial execution, private asynchronous worktrees, and manager-assisted variants provides a promising framework for studying how coordination mechanisms affect software engineering outcomes.
Critical weaknesses
- Theoretical soundness of ADPR. A failed dependency check may reflect a coding error rather than a coordination failure. The paper fails to build ADPR to measures the phenomenon it intends to capture under this term.
- Limitations of DRS. DRS measures the first successful resolution using run-specific normalized checkpoints, rather than wall-clock time or a fixed computational budget, making comparisons peed across protocols unfair.
- Limited benchmark scale and test-takers. The benchmark is still scale-limited with limited number of dependencies. And many widely-used commercial models are not evaluated.
- Insufficiently validated trajectory analysis. The reported hopping window phenomenon may depend on checkpoint design and normalization, and the paper needs stronger robustness analyses to support this interpretation.
Questions for authors
- How do you distinguish coordination failures from individual implementation failures, and can you validate ADPR against manually annotated failure cases?
- What criteria were used to select the 19 tasks and dependency division, and how sensitive are the conclusions to task selection and aggregation across tasks with different numbers of dependency edges?
- How many independent runs were conducted per configuration, and can the authors report uncertainty estimates alongside end-to-end latency and coordination overhead?
- Why do you define DRS like this, which may be unfair across protocols? Can you show the wall-clock time or token budget?
- Does the hopping window still persist with different settings (dependency division strategy) or collaboration modes?
7 comments
Thanks(2/4)
Q3: Task Selection and Aggregation Sensitivity
We thank the reviewer for this question. Our task selection and dependency construction follow two key criteria: (1) tasks must contain genuine upstream–downstream dependencies and naturally parallelizable implementation responsibilities, enabling meaningful decomposition across multiple agents; and (2) tasks must originate from established public software engineering benchmarks or pull requests (PRs) in mature open-source projects, ensuring realistic development scenarios. We further conducted human reviews to verify the validity of identified dependencies and the feasibility of parallel task decomposition. The corresponding records are available in the Huaman_Review_Record directory of our anonymous repository. We will clarify these criteria and validation procedures in the revised manuscript.
To examine sensitivity to task selection and aggregation across different dependency counts, we conducted task-macro versus edge-micro ADPR comparisons, together with leave-one-task-out (LOTO) and leave-one-repository-out (LORO) analyses.
Table 4.2. The result of Aggregation Sensitivity. (Δ = Async-Manager − Single.)
| Model | Macro Δ (pp) | Micro Δ (pp) | LOTO Reversals | LORO Reversals |
|---|---|---|---|---|
| Qwen3.5-9B | −11.4 | −11.5 | 0/19 | 0/16 |
| Qwen3.6-27B | +26.3 | +21.2 | 0/19 | 0/16 |
| Qwen3.8-27B | −12.3 | −11.5 | 0/19 | 0/16 |
| DeepSeek-V4 | −5.3 | −5.8 | 0/19 | 0/16 |
Across all 40 collaborative-versus-Single comparisons, task-macro and edge-micro aggregation preserve the same performance directions. Moreover, none of the ten Async-Manager-versus-Single comparisons reverses direction under any task or repository exclusion. These results further support our central finding that the benefits of multi-agent collaboration are model-dependent, with the main Async-Manager comparisons remaining directionally consistent across task selections and aggregation methods.
Q4: Benchmark scale
Thank reviewer for this question. First we want to explain the reason why the scale is low, that's because the standard of selecting tasks in our benchmark is strict. And we have to consider the following aspects:
-
SWE tasks require explicit cross-script dependencies—meaning upstream and downstream tasks—as well as tasks that can be processed in parallel, so they can genuinely be distributed to multi-agent systems for completion.
-
Meanwhile, the benchmark is required to source suitable tasks from already public, mature benchmarks or pull requests (PRs) of large open-source projects.
This precisely highlights the novelty of our benchmark and its ability to capture multi-agent collaboration issues that existing works fail to reflect.
However, we agree with reviewer's suggestion and we **expanded our benchmark **with 5 more tasks in different languages. So the whole benchmark has 24 tasks and 4 languages. The details of new tasks are as follows:
Table 4.3. The details of new tasks
| New Task | Repository Languages |
|---|---|
| TVM-20409 | Python/C++ |
| TVM-18421 | Python/C++ |
| TVM-20543 | Python/C++ |
| Jekyll-8771 | Ruby |
| Terraform-35543 | Go |
And the results are as follows:
Table 4.4. The results of new tasks
| Model | Protocol | Test Pass ↑ | ADPR ↑ | DRS ↓ | Tokens |
|---|---|---|---|---|---|
| Qwen3.5-9B | Single | 56.7% | 20.0% | — | 5.25M |
| Qwen3.5-9B | Serial | 49.1% | 20.0% | 0.867 | 5.78M |
| Qwen3.5-9B | Async-Private | 49.6% | 0.0% | 1.000 | 7.84M |
| Qwen3.5-9B | Async-Manager-RO | 48.4% | 0.0% | 1.000 | 12.35M |
| Qwen3.5-9B | Async-Manager | 49.2% | 20.0% | 0.867 | 13.77M |
| Qwen3.5-27B | Single | 61.6% | 40.0% | — | 5.50M |
| Qwen3.5-27B | Serial | 43.5% | 0.0% | 1.000 | 4.68M |
| Qwen3.5-27B | Async-Private | 49.6% | 0.0% | 1.000 | 6.00M |
| Qwen3.5-27B | Async-Manager-RO | 52.0% | 20.0% | 0.950 | 10.08M |
| Qwen3.5-27B | Async-Manager | 56.0% | 40.0% | 0.794 | 12.32M |
| Qwen3.6-27B | Single | 85.0% | 60.0% | — | 5.21M |
| Qwen3.6-27B | Serial | 73.2% | 40.0% | 0.800 | 5.80M |
| Qwen3.6-27B | Async-Private | 71.8% | 20.0% | 0.850 | 7.32M |
| Qwen3.6-27B | Async-Manager-RO | 66.4% | 40.0% | 0.827 | 10.52M |
| Qwen3.6-27B | Async-Manager | 70.2% | 60.0% | 0.728 | 10.99M |
And the results are consistent with our findings.
- Test Pass Rate is high does not mean ADPR is high.
- Single agent may outperform than the multi-agent system.
Thanks (3/4)
Q5: Commercial Model Evaluation
We thank the reviewer for this suggestion. To broaden our model coverage, we additionally evaluated Haiku 5.5 and Muse Spark 1.3, covering all 19 benchmark tasks and five execution protocols, resulting in 190 completed model–protocol–task evaluations.
Table 4.5. The commercial model results
| Model | Protocol | Test Pass ↑ | ADPR ↑ | DRS ↓ | Tokens (M) |
|---|---|---|---|---|---|
| Haiku 5.5 | Single | 75.0% | 70.2% | — | 2.80 |
| Serial | 60.8% | 56.1% | 0.800 | 3.37 | |
| Async-Private | 50.6% | 38.6% | 0.797 | 2.48 | |
| Async-Manager-RO | 49.5% | 33.3% | 0.889 | 3.42 | |
| Async-Manager | 88.6% | 77.2% | 0.758 | 6.75 | |
| Muse Spark 1.3 | Single | 78.9% | 78.9% | — | 2.63 |
| Serial | 73.2% | 70.2% | 0.761 | 6.18 | |
| Async-Private | 60.3% | 55.4% | 0.711 | 6.36 | |
| Async-Manager-RO | 63.6% | 59.6% | 0.801 | 9.91 | |
| Async-Manager | 91.7% | 70.2% | 0.733 | 11.08 |
These additional experiments reinforce our main findings. First, multi-agent collaboration does not universally outperform Single: Async-Manager improves ADPR for Haiku 5.5 (70.2% → 77.2%) but reduces it for Muse Spark 1.3 (78.9% → 70.2%). Second, writable manager coordination substantially recovers dependency resolution compared with Async-Private and Async-Manager-RO, with ADPR gains of +38.6/+43.9 pp for Haiku and +14.8/+10.6 pp for Muse, respectively.
Importantly, the results further demonstrate the distinction between task-level correctness and dependency-level collaboration. For Muse Spark 1.3, Async-Manager achieves the highest Test Pass Rate (91.7%) but lower ADPR than Single (70.2% vs. 78.9%). Based on the observed token costs and DRS differences, these findings further support our conclusion that collaboration effectiveness depends on both model capability and execution protocol, and cannot be fully characterized by Test Pass Rate alone.
Q6: Independent Runs, Uncertainty, and Coordination Overhead (Token and Wall-clock result)
We thank the reviewer for this suggestion. The original study included one experimental round per configuration. To assess variability, we conducted two additional rounds for three Qwen models under Single and Async-Manager, obtaining three observations for each of these six configurations.
Table 4.6. Multi-run Results
| Model | Protocol | Test Pass (95% CI) | ADPR (95% CI) |
|---|---|---|---|
| Qwen3.5-9B | Single | 31.9% [15.3, 48.4] | 10.7% [−2.6, 24.0] |
| Async-Manager | 43.6% [27.1, 60.1] | 8.4% [−13.5, 30.4] | |
| Qwen3.6-27B | Single | 55.3% [46.6, 64.1] | 38.3% [33.8, 42.8] |
| Async-Manager | 70.4% [56.7, 84.1] | 55.3% [38.2, 72.3] | |
| Qwen3.8-27B | Single | 91.8% [87.6, 96.1] | 76.6% [74.1, 79.1] |
| Async-Manager | 84.7% [75.6, 93.8] | 64.3% [53.4, 75.3] |
Despite run-to-run variability, the results remain consistent with our central finding that collaboration benefits are model-dependent. Async-Manager achieves higher ADPR than Single in all three rounds for Qwen3.6-27B, whereas Single consistently performs better for Qwen3.8-27B.
We further examined execution latency and coordination overhead across the five protocols. The observed protocol wall-clock times are summarized below.
Table 4.7. The wall-clock results
| Protocol | Mean Time (min) | Median Time (min) |
|---|---|---|
| Single | 29.0 | 16.9 |
| Serial | 86.3 | 41.4 |
| Async-Private | 45.7 | 25.8 |
| Async-Manager-RO | 87.9 | 49.5 |
| Async-Manager | 102.1 | 67.6 |
Notably, Async-Private reduces average execution time compared with Serial (45.7 vs. 86.3 minutes), illustrating the efficiency benefits of parallel specialist execution. Manager-based protocols introduce additional coordination overhead, reflecting the cost of feedback, integration, and repair.
We also analyzed token consumption, as reported in Tables 10 and 11.
Table 4.8. The Token Usage
| Model | Single Tokens | Async-Private Tokens | Async-Manager Tokens |
|---|---|---|---|
| DeepSeek-V4 | 2.54M | 6.92M | 13.65M |
| Qwen3.6-27B | 4.96M | 8.18M | 14.49M |
| Qwen3.8-27B | 3.45M | 7.09M | 13.73M |
Additionally, the Async-Manager logs record 819 manager intervention events, including 145 accepted repairs, providing direct evidence of coordination activity.
These results extend our quality–cost analysis in Section 5.5, showing that asynchronous execution can reduce elapsed time, while additional coordination incurs computational overhead without guaranteeing better dependency resolution.
Thanks(1/4)
We appreciate reviewer's valuable suggestions and recognization of this important problem. Following reviewer's concern, we believe we have solved all the concerns.
Q1: The validity of ADPR and checked the results manually.
This concern is related to the effectiveness of the proposed metric ADPR. As reviewer mentioned, more information is under this metric. ADPR evaluates collaboration effectiveness through executable cross-component dependency checks, which assess both upstream prerequisite functionality (U) and downstream dependent behavior (D). Since these U/D checks underpin ADPR, analyzing their outcomes allows us to further investigate whether differences in ADPR reflect coordination-related failures. We therefore examined 104 matched dependencies from two strong models, DeepSeek and Qwen3.8, comparing Single and Async-Manager.
Table 4.1. The U/D statistical result of strong models
The results show that upstream correctness remains nearly unchanged, while downstream performance declines under Async-Manager. In particular, U-pass/D-fail cases increase from 2 to 12, suggesting that collaboration introduces additional challenges in correctly utilizing upstream functionality, beyond upstream implementation failures.
To further validate this observation, we randomly selected five U-pass/D-fail task cases involving DeepSeek and Qwen3.8 and manually inspected the agents' execution trajectories and generated code diffs. All five cases exhibited coordination-related failures. A recurring pattern was that downstream agents incorrectly invoked or incorporated upstream functionality, produced incorrect implementations, and failed to correct them in subsequent iterations.
These findings provide both quantitative and execution-level evidence supporting ADPR as a meaningful measure of cross-component collaboration effectiveness, with U/D analysis revealing how coordination failures contribute to unresolved dependencies.
Q2: DRS Definition and Computational Budget
We thank the reviewer for this question. The motivation for DRS is to capture dependency-resolution dynamics that final ADPR cannot reveal. Rather than measuring execution speed, DRS tracks when cross-agent dependencies first become satisfied during integration. Since different protocols generate different numbers of integration checkpoints, we normalize the first-resolution step by each run's checkpoint count to characterize relative resolution progress. This distinction is meaningful: as shown in Section 5.3, Qwen3.6-27B and Qwen3.8-27B achieve similar ADPR under Async-Manager (63.16% vs. 64.91%) but different DRS values (0.641 vs. 0.719), revealing differences in their resolution trajectories that final coverage alone would miss.
Regarding computational budget, we apply the same limit of 100 model calls per agent, rather than a fixed token cap, to accommodate differences in token consumption across models. We further report actual token usage for all 50 configurations in Tables 10 and 11 and explicitly analyze quality–cost trade-offs in Section 5.5. Our Pareto analysis shows that greater token consumption does not necessarily improve dependency resolution, with 5 of 10 models exhibiting different frontiers under ADPR and Test Pass Rate. Thus, DRS characterizes relative dependency-resolution progress, while our token-based analysis separately accounts for computational efficiency across protocols.