- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces REVIBE, a benchmark for evaluating multi-agent teams building complete software applications from natural-language requirements. The benchmark includes ten full-stack projects and ten collaboration modes. LEGOGENT executes development and deployment, while TAGENT provides weighted evaluation through UI, API, and source-code checks. The experiments record completion scores, latency, token costs, and multi-round refinement trajectories, showing that collaboration modes affect outcomes and that no mode consistently dominates across projects.
Critical strengths
The research question has clear practical relevance. Rather than only comparing models or individual agent systems, the paper investigates how to organize agent teams. The ten collaboration modes provide a concrete and insightful design space. If their advantages and applicable conditions can be reliably validated, they could inform practical software-development workflows.
Comparing complete organizational policies is a reasonable research choice. Practitioners typically adopt complete workflows encompassing ownership, concurrency, decision-making, and handoff rules. I appreciate the authors’ decision to compare these modes as packages and do not consider identifying the causal contribution of every component a prerequisite for obtaining valuable findings.
The evaluation covers the software-delivery process and considers quality, time, and cost together. Checking deployed applications through UI, API, and source-code evidence is more convincing than relying on agents’ completion claims. Recording multi-round refinement, regressions, and cache-aware token costs also improves the benchmark’s practical relevance.
Critical weaknesses
- TAGENT’s scoring reliability on agent-generated software has not been sufficiently validated.
Section 5 reports a Spearman correlation of 0.87 between TAGENT and TA grades on 97 student projects, while explicitly leaving validation on agent-generated artifacts to future work. This is useful preliminary evidence, but student projects and agent-generated projects may have different error distributions. Overall rank correlation also does not establish item-level correctness, absolute scoring error, or the reliability of comparisons between collaboration modes.
This directly affects the paper’s core results because the mode differences and their interpretations depend on TAGENT. I suggest selecting representative samples of existing agent-generated artifacts across projects, modes, and score ranges for human review blinded to the collaboration mode, alongside repeated grading of some fixed artifacts. The priority should be to test whether mode differences reflected in automated scores persist under independent assessment, rather than requiring human reevaluation of every run.
- Evidence for the stability of cross-project findings is limited, and some mechanistic interpretations should be more cautious.
According to Section 4, each project–mode configuration in the full grid is run once per model, while three repeated trials are concentrated on instant messaging. Appendix B further uses the average standard deviation on this project to support the claim that the mode differences in Table 4 reflect real effects. However, run-to-run variance on one project does not sufficiently establish the stability of differences on other projects.
I appreciate that Sections 4.2 and 4.5 treat small average gaps as ties rather than emphasizing fine-grained rankings. Nevertheless, explanations such as requirement chains favoring QA-first and clean stage boundaries favoring pipeline remain primarily post hoc interpretations of observed outcomes. Section 5 also explicitly acknowledges that the paper describes rather than predicts mode–project fit.
I suggest repeating a small set of representative modes on several projects with different requirement structures and reporting paired differences and their uncertainty. Relationships between task structure and mode advantages should meanwhile be presented as interpretations requiring validation. The paper need not introduce a mode selector, but it should distinguish descriptive findings from selection rules applicable to new projects.
- The claim that multi-agent teams generally outperform single agents is stronger than the available controls support.
Table 3 provides same-model single-agent comparisons on instant messaging, which is a strength. However, the cross-project comparison in Table 7 contrasts a Claude single agent with the best team mode averaged over three other models, so the difference cannot be independently attributed to team organization. Selecting the best team mode for each project after observing results also differs from choosing a workflow in advance in practice.
Comparing single agents and teams under the same wall-clock limit is reasonable. However, teams can consume more aggregate inference resources, so these results do not also establish an advantage in budget efficiency. I suggest narrowing the general statement in Section 4.1 and Appendix B that the question is which mode to use rather than whether to form a team. If the stronger claim is retained, same-model comparisons on a small set of representative projects would help, with a clear distinction between quality gains under equal time limits and efficiency gains under equal monetary budgets.
Questions for authors
The data and aggregation procedures need clarification. Table 3 reports a trajectory of 73/88/94 for Claude feature squad on instant messaging, whereas Table 13 reports 79.22/93.07/96.95; Appendix H nevertheless states that the worked example matches the corresponding row in Table 3. Do these come from different repeated trials? Please explain the correspondence and aggregation across the main tables, worked examples, and repeated experiments.
Please clarify the interpretation of the Functional Completion Score. I accept the paper’s focus on functional completion and do not require it to comprehensively evaluate every dimension of software quality. However, UI functionality checks do not directly establish visual quality or usability, and averaging partial scores does not mean that complete user workflows are operational. Please clarify what the aggregate score represents and separately report failures of critical workflows or security requirements.
Refinement gains should retain their stated qualification. The authors acknowledge that TAGENT provides both feedback and scoring, so refinement gains are gains on its rubric. This protocol is reasonable, but it does not establish corresponding improvements in software quality on unseen scenarios. If resources permit, independent acceptance testing of some final artifacts, or additional hidden scenarios consistent with the original requirements, would strengthen the findings.
Dynamic delegation would be a valuable practical baseline. As an extension, a coordinator could be allowed to decide task partitioning, subagent invocation, and integration timing within the same framework, and then be compared with the fixed modes. This would help establish the benefits of prescribed organizational policies relative to dynamic orchestration. However, I do not consider this baseline necessary to complete the paper’s comparison of fixed modes.
Representative audit materials would be helpful. Appendix A states that generated repositories and raw handoff logs are not released. Providing a small set of sanitized artifacts, scoring evidence, and collaboration traces would facilitate verification of scores, mode adherence, and failure mechanisms without requiring the release of all raw records.
4 comments
Response to Reviewer 5m7f [2/2]
[Question 1] Correspondence between the P00 table and appendix examples.
We clarify the different reporting units used in the main table and the worked examples. Table 3 reports per-round averages over three repeated P00 team trials, giving 73/88/94 for Claude feature squad. Appendices H and I present the detailed trajectory of one of these three trials, with scores of 79.22/93.07/96.95, illustrating token accounting and requirement-level repair. The PM example likewise describes a score change within an individual execution. The main table reports averages across trials, and the appendices show the development process and requirement-level changes within a trial included in those averages.
[Question 2] Meaning of Functional Completion Score and critical failures.
We address score interpretation by presenting weighted requirement completion together with critical failure evidence. FCS sums pass/partial/fail credit by requirement weight and averages scores from multiple evaluator classes for the same requirement. It therefore measures weighted completion, with partial credit possible when only part of a user workflow succeeds. Live UI checks contribute evidence of observed functionality, while visual quality and usability remain separate evaluation dimensions. However, FCS does require an operational workflow. For example, if TAgent fails to register an account, the user login FCS will also drop to 0.
Section 4.4 and Appendix G separately report authentication/session/permission failures in 46% of analyzed runs, prerequisite cascades in 39%, and security/transport failures in 78%. These categories can overlap within a run. We also mark the supplementary P08 exception in which TAgent awarded 100 despite a million-document claim backed by only 13 physical documents. Reporting these failures alongside FCS exposes where a high aggregate score can conceal an unmet critical requirement. It also identifies concrete improvement targets: stateful integration tests, prerequisite-aware repair, and security checks at deployment.
[Question 3] Refinement gains and unseen scenarios.
We make the evaluation target explicit throughout the refinement analysis. TAgent supplies both the repair feedback and the rubric used to measure subsequent progress, so the reported gains describe improvement on that visible rubric. Prerequisite-aware UI tests exercise stateful workflows within it, and the requirement traces show both repaired checks and regressions. Independent hidden-scenario testing would extend this validation to transfer beyond the feedback rubric. We also present the fourth-round continuations as exploratory because their graders and scoring denominators change. The supported result is evidence about how refinement changes measured requirement completion over the evaluated rounds.
[Question 4] Dynamic delegation as a baseline.
In our observations, current models using dynamic delegation commonly assign separate features to parallel agents, yielding a feature-squad pattern. Choosing when to delegate and choosing how agents collaborate are distinct decisions. REVIBE addresses the latter by comparing ten prescribed workflows under a shared protocol. These comparisons reveal substantial performance differences within the same task and model, with the leading observed policy varying across projects. They motivate adaptive selection of collaboration policies beyond a default feature-squad pattern. Evaluating a native dynamic coordinator against these reference outcomes would test whether online delegation achieves such adaptation.
[Question 5] Representative audit materials.
We clarify the available audit material and its purpose in Appendix A. The current release contains requirements, rubrics, benchmark code, aggregated metrics, pricing data, and analysis scripts. The manuscript adds a collaboration snapshot and worked examples containing UI states, API responses, and source locations. These materials let readers inspect the illustrated handoffs, scoring evidence, and repair decisions. Our current release scope covers these benchmark assets and representative excerpts.
We commit to releasing a more comprehensive set of representative audit materials upon acceptance. Releasing them now is difficult because sanitizing the repositories for double-anonymous review, on top of the current revision workload, would require substantial additional effort.
The resulting findings connect aggregate performance to evidence that readers can inspect. The blinded audit establishes score agreement within the agent-artifact sample, the P00 comparisons reveal substantial organizational quality-cost trade-offs, and the failure analysis identifies requirements that aggregate scores can conceal. Together, these results make REVIBE useful for both comparing workflows and identifying concrete targets for improving software-development agents.
Thank you for the authors' detailed response and the revisions made to the paper. The newly added experiments and the clarification of the scope of the conclusions have addressed some of my concerns, but I still have two main concerns.
-
Limited scientific insight. Although the paper conducts experiments on a relatively large scale, the overall analysis remains largely descriptive and lacks systematic, generalizable scientific findings. Some conclusions are relatively intuitive, while findings of greater research value, such as the relationship between task structure and collaboration patterns, lack sufficient validation. Although narrowing the claims alleviates the problem of overclaiming, it does not resolve the issue of limited research contribution.
-
Insufficient reliability and robustness of the benchmark. I acknowledge the newly added human blind evaluation experiment, but consistency at the overall score level does not sufficiently guarantee the correctness of the evaluation of key requirements, such as the case in Appendix K that receives a full score despite containing only 13 physical documents. In addition, the newly added cross-project repetition experiment suffers from inconsistencies in evaluators and execution conditions, and is still insufficient to verify the stability of performance differences across collaboration patterns.
Overall, I acknowledge the research motivation, engineering effort, and the authors' efforts during the rebuttal stage. However, as an empirical analysis study, its scientific insight remains limited; as a benchmark, its reliability and robustness have not yet been sufficiently validated. Therefore, I decide to maintain my original score.
Thank you for reconsidering the paper and for acknowledging the motivation, engineering effort, and revision. We respectfully argue that the revised paper has resolved both remaining concerns.
1. Scientific insight
The central finding is not intuitive: average rankings hide large, task-dependent policy effects. With project and model held fixed, DeepSeek v4 Pro scores 88 under pipeline and 48 under open-source review on P00, a 40-point gap from the collaboration policy alone (Section 4.5, Table 3). Across the full grid, the ten policies differ by more than 10 points in 25 of 30 project-model cells, with a median spread of 29.9 points. Yet the six leading policies are within two points on cross-project means, and the project-level leader changes from task to task (Section 4.5, Appendix C). A leaderboard of averages would conclude that organization barely matters, while the per-task data show it is often the largest controllable factor.
This makes collaboration a training objective for agentic models, not only an orchestration setting. The same model weights gain or lose up to 40 points depending on how the work is organized, and no fixed policy is best everywhere. Selecting the collaboration structure for a task is therefore a capability in its own right, and REVIBE supplies what is needed to train and evaluate it: a fixed-policy reference grid and requirement-level feedback. And we use Figure 5 and a large portion of Section 4.2 to describe what we find, which is not, as you claimed, "intuitively" or "lack of validation".
On task structure and collaboration pattern, the existence and size of the task dependence are validated by the controlled grid. The mechanism is stated as testable hypotheses, anchored by the failure taxonomy: 56% of failed checks trace to missing features or incomplete logic, authentication and prerequisite failures to stateful integration, and security failures to deployment checks (Section 4.4, Appendix G).
2. Benchmark reliability and robustness
Our validation protocol follows established practice. Checking an automated judge against human grading on a sampled subset is the standard validation for such evaluators. MT-Bench reports over 80% agreement between LLM judges and humans [1], Agent-as-a-Judge validates requirement-level verdicts against a human evaluation baseline [2], and RealDevWorld, the closest design to TAgent, reports a correlation of 0.85 with expert assessment [3].
Requirement-level correctness. Our blinded audit covers 20 previously unaudited repositories, two drawn at random from each project. Reviewers see neither the policy label nor the TAgent score, and two teaching assistants cross-check the grades. The maximum total-score difference is 5 points, Spearman ρ = 0.88, and subjectively judged UI/code items differ by at most 2 points (Section 3.3, Appendix F). This is at the level reported in [1–3].
The Appendix K case. We agree the P08 run should not have received full credit on that requirement. Our own manual audit found it, and we report it as an explicit exception (Appendix K). It concerns one requirement in one project and does not affect cross-policy comparisons, which rest on spreads of tens of points across 30 cells.
Stability of policy differences. This claim rests on the controlled grid, not on the supplementary repetitions. The 30 P00 team cells each have three trials under identical evaluator and execution conditions, with a mean best-score standard deviation of 2.8 points (Section 4.2, Appendix B). The 40-point P00 gap is more than ten times this variation. We agree the cross-project repetitions differ in evaluator and recovery conditions, which is why they are reported separately and are not used to support the stability claim (Table 4, Appendix K).
In summary, automated scores agree with a blinded human audit under a standard validation protocol, the one known rubric failure is disclosed and bounded, and policy effects are far larger than measured trial variation. We hope you will reconsider your assessment in this light.
[1] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks.
[2] Zhuge et al. Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934.
[3] Bian et al. You Don't Know Until You Click: Automated GUI Testing for Production-Ready Software Evaluation. arXiv:2508.14104.
Response to Reviewer 5m7f [1/2]
Thank you for identifying the value of comparing complete organizational policies and how to strengthen the evidence. We hope the following responses address your concern. Substantive manuscript additions are marked in red.
[Weakness 1] Reliability of TAgent on agent-generated software.
We address the difference between student projects and agent-generated software through a blinded audit of existing agent artifacts. We randomly select two previously unaudited repositories from each project, giving 20 repositories. Human reviewers receive neither the collaboration-mode label nor the TAgent score, and two teaching assistants cross-check the grades. Against the existing automated scores, the maximum total-score difference is 5 points, Spearman ρ = 0.88, and the mean automated-minus-human difference is −0.5 points. Subjectively judged UI/code items differ by at most 2 points. These results provide direct agreement evidence on the artifacts evaluated by REVIBE, complementing the earlier validation on 97 student projects.
We report the blinded audit separately from the grading procedure used for exceptional disagreements. Two initial TAgent passes with a difference above 10 points trigger a third pass and human review. This occurred for three artifacts. We select the existing automated score closest to the human grade in those cases. Among retained agreeing pairs, the mean absolute difference is 3.7 points and the maximum is 5. Section 3.3 and Appendix F distinguish these post-adjudication statistics from the blinded audit and initial-pass variability. The resulting validation addresses both absolute score differences and rank agreement within the audited sample. We use this evidence at the sample level and interpret close mode rankings through the variability analysis below.
[Weakness 2] Cross-project stability and mechanistic interpretations.
We address stability by matching each conclusion to the relevant comparison and repetition evidence. The main grid contains 300 project-mode-model cells, with P00 team cells averaging three trials. Across the 30 P00 team cells from the three full-grid models, the mean best-score standard deviation is 2.8 points. We use this statistic to describe variation on P00.
We also examine the supplementary records beyond P00. They contain two team observations each for P01/pipeline, P02/PM oversight, and P04/QA-first, plus additional feature-squad observations on P03, P05, and P09. We report both results for each repeated cell and document their coding routes, graders, and recovery settings. Because these conditions vary, the supplementary records provide descriptive coverage of the selected configurations.
Our analysis shows substantial performance differences between collaboration policies within evaluated projects, while the leading observed policy varies across projects. For example, DSv4 Pro scores 88 under pipeline and 48 under open-source review on P00 with the project and model held fixed. Across projects, the six leading modes have mean scores within two points of one another. On live teaching, feature squad and pipeline score 75.1 and 74.2, respectively. We highlight these performance patterns and interpret small score gaps as near ties. Requirement structure helps frame hypotheses about acceptance planning and ownership, providing concrete directions for evaluating task-dependent mode selection.
[Weakness 3] General team-benefit and budget-efficiency claims.
We address the comparison-design concern by separating the repeated same-model control from the broader cross-project reference. On P00, Claude's single-agent mean best score is 64.2 over three runs, and every Claude team policy has a higher mean. Feature squad reaches approximately 94 under the same model, tools, time cap, deployment path, and feedback. The single-agent best scores are 52.750, 91.000, and 48.875 (sample SD 23.283), making its observed variability explicit. Mean coding-agent costs are about USD 92 for the single agent and USD 617 for feature squad. The supported finding is a measured quality gain with additional inference expenditure under shared time and round limits.
We label the cross-project table according to its different comparison design. It compares Claude single agents with the best team-mode mean over DeepSeek Flash, DeepSeek Pro, and GLM-5.2, selected after observing the grid. The caption identifies both the model differences and the selection rule, so readers can interpret it as a descriptive reference.
We also report raw scores, denominators, and the associated grading and coding conditions on single-agent runs for P01-P09. We retain both observations for repeated team cells. The P00 Claude comparison therefore provides direct evidence of higher team mean quality at higher inference cost, illustrating the organizational trade-offs that REVIBE measures. Comparing equal-dollar efficiency is hard since we cannot find the correct pausing point, which may incur compilation error.