acceptodds
Under review as a conference paper at ICLR 2027

REVIBE: Quantitatively Evaluating Multi-Agent Vibe Coding on Real-World Software Tasks

Abstract

Vibe coding is reshaping software development, and multi-agent teams promise to compress delivery time by dividing labor across parallel workers. Yet existing multi-agent benchmarks rely on simulated settings that entangle reasoning with communication, only reward coarse task completion, and ignore the time and monetary costs that matter most in practice. We introduce REVIBE, a benchmark that quantitatively evaluates multi-agent vibe coding on real-world software tasks. REVIBE crosses 10 full-stack projects, each given to agents only as a natural-language requirement document and graded by a weighted 100-point rubric, with 10 explicitly defined collaboration modes: complete organizational policies that fix how a four-agent team allocates ownership, concurrency, decision authority, and handoff artifacts, while the model, team size, tools, round budget, and evaluator stay fixed. Two components make the grid runnable: LEGOGENT executes each mode through periodic sync communication and a native CI/CD pipeline, and TAGENT discovers and probes each unseen deployment to report a functional score together with latency and prefix-cached token cost. Across 100 project–mode configurations and more than 300 runs, the collaboration mode moves the average best score about as much as the choice among three models does, and far more on a given task: with the project and model fixed, switching the mode shifts the score by up to 40 points and the time to the best round by up to 4.1. No mode dominates: six modes lie within two points of each other across projects, and the winning mode depends on the project's requirement structure. REVIBE makes these speed, cost, and quality trade-offs measurable and reproducible. Code and benchmark are available at https://anonymous.4open.science/r/revibe_repo-8828/.

Primary area:
datasets and benchmarks
Keywords:
multi-agent framework, vibe coding, benchmark, software engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5m7fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REVIBE, a benchmark for evaluating multi-agent teams building complete software applications from natural-language requirements. The benchmark includes ten full-stack projects and ten collaboration modes. LEGOGENT executes development and deployment, while TAGENT provides weighted evaluation through UI, API, and source-code checks. The experiments record completion scores, latency, token costs, and multi-round refinement trajectories, showing that collaboration modes affect outcomes and that no mode consistently dominates across projects.

Critical strengths

The research question has clear practical relevance. Rather than only comparing models or individual agent systems, the paper investigates how to organize agent teams. The ten collaboration modes provide a concrete and insightful design space. If their advantages and applicable conditions can be reliably validated, they could inform practical software-development workflows.

Comparing complete organizational policies is a reasonable research choice. Practitioners typically adopt complete workflows encompassing ownership, concurrency, decision-making, and handoff rules. I appreciate the authors’ decision to compare these modes as packages and do not consider identifying the causal contribution of every component a prerequisite for obtaining valuable findings.

The evaluation covers the software-delivery process and considers quality, time, and cost together. Checking deployed applications through UI, API, and source-code evidence is more convincing than relying on agents’ completion claims. Recording multi-round refinement, regressions, and cache-aware token costs also improves the benchmark’s practical relevance.

Critical weaknesses
  1. TAGENT’s scoring reliability on agent-generated software has not been sufficiently validated.

Section 5 reports a Spearman correlation of 0.87 between TAGENT and TA grades on 97 student projects, while explicitly leaving validation on agent-generated artifacts to future work. This is useful preliminary evidence, but student projects and agent-generated projects may have different error distributions. Overall rank correlation also does not establish item-level correctness, absolute scoring error, or the reliability of comparisons between collaboration modes.

This directly affects the paper’s core results because the mode differences and their interpretations depend on TAGENT. I suggest selecting representative samples of existing agent-generated artifacts across projects, modes, and score ranges for human review blinded to the collaboration mode, alongside repeated grading of some fixed artifacts. The priority should be to test whether mode differences reflected in automated scores persist under independent assessment, rather than requiring human reevaluation of every run.

  1. Evidence for the stability of cross-project findings is limited, and some mechanistic interpretations should be more cautious.

According to Section 4, each project–mode configuration in the full grid is run once per model, while three repeated trials are concentrated on instant messaging. Appendix B further uses the average standard deviation on this project to support the claim that the mode differences in Table 4 reflect real effects. However, run-to-run variance on one project does not sufficiently establish the stability of differences on other projects.

I appreciate that Sections 4.2 and 4.5 treat small average gaps as ties rather than emphasizing fine-grained rankings. Nevertheless, explanations such as requirement chains favoring QA-first and clean stage boundaries favoring pipeline remain primarily post hoc interpretations of observed outcomes. Section 5 also explicitly acknowledges that the paper describes rather than predicts mode–project fit.

I suggest repeating a small set of representative modes on several projects with different requirement structures and reporting paired differences and their uncertainty. Relationships between task structure and mode advantages should meanwhile be presented as interpretations requiring validation. The paper need not introduce a mode selector, but it should distinguish descriptive findings from selection rules applicable to new projects.

  1. The claim that multi-agent teams generally outperform single agents is stronger than the available controls support.

Table 3 provides same-model single-agent comparisons on instant messaging, which is a strength. However, the cross-project comparison in Table 7 contrasts a Claude single agent with the best team mode averaged over three other models, so the difference cannot be independently attributed to team organization. Selecting the best team mode for each project after observing results also differs from choosing a workflow in advance in practice.

Comparing single agents and teams under the same wall-clock limit is reasonable. However, teams can consume more aggregate inference resources, so these results do not also establish an advantage in budget efficiency. I suggest narrowing the general statement in Section 4.1 and Appendix B that the question is which mode to use rather than whether to form a team. If the stronger claim is retained, same-model comparisons on a small set of representative projects would help, with a clear distinction between quality gains under equal time limits and efficiency gains under equal monetary budgets.

Questions for authors

The data and aggregation procedures need clarification. Table 3 reports a trajectory of 73/88/94 for Claude feature squad on instant messaging, whereas Table 13 reports 79.22/93.07/96.95; Appendix H nevertheless states that the worked example matches the corresponding row in Table 3. Do these come from different repeated trials? Please explain the correspondence and aggregation across the main tables, worked examples, and repeated experiments.

Please clarify the interpretation of the Functional Completion Score. I accept the paper’s focus on functional completion and do not require it to comprehensively evaluate every dimension of software quality. However, UI functionality checks do not directly establish visual quality or usability, and averaging partial scores does not mean that complete user workflows are operational. Please clarify what the aggregate score represents and separately report failures of critical workflows or security requirements.

Refinement gains should retain their stated qualification. The authors acknowledge that TAGENT provides both feedback and scoring, so refinement gains are gains on its rubric. This protocol is reasonable, but it does not establish corresponding improvements in software quality on unseen scenarios. If resources permit, independent acceptance testing of some final artifacts, or additional hidden scenarios consistent with the original requirements, would strengthen the findings.

Dynamic delegation would be a valuable practical baseline. As an extension, a coordinator could be allowed to decide task partitioning, subagent invocation, and integration timing within the same framework, and then be compared with the fixed modes. This would help establish the benefits of prescribed organizational policies relative to dynamic orchestration. However, I do not consider this baseline necessary to complete the paper’s comparison of fixed modes.

Representative audit materials would be helpful. Appendix A states that generated repositories and raw handoff logs are not released. Providing a small set of sanitized artifacts, scoring evidence, and collaboration traces would facilitate verification of scores, mode adherence, and failure mechanisms without requiring the release of all raw records.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5m7f [1/2]

Thank you for identifying the value of comparing complete organizational policies and how to strengthen the evidence. We hope the following responses address your concern. Substantive manuscript additions are marked in red.

[Weakness 1] Reliability of TAgent on agent-generated software.

We address the difference between student projects and agent-generated software through a blinded audit of existing agent artifacts. We randomly select two previously unaudited repositories from each project, giving 20 repositories. Human reviewers receive neither the collaboration-mode label nor the TAgent score, and two teaching assistants cross-check the grades. Against the existing automated scores, the maximum total-score difference is 5 points, Spearman ρ = 0.88, and the mean automated-minus-human difference is −0.5 points. Subjectively judged UI/code items differ by at most 2 points. These results provide direct agreement evidence on the artifacts evaluated by REVIBE, complementing the earlier validation on 97 student projects.

We report the blinded audit separately from the grading procedure used for exceptional disagreements. Two initial TAgent passes with a difference above 10 points trigger a third pass and human review. This occurred for three artifacts. We select the existing automated score closest to the human grade in those cases. Among retained agreeing pairs, the mean absolute difference is 3.7 points and the maximum is 5. Section 3.3 and Appendix F distinguish these post-adjudication statistics from the blinded audit and initial-pass variability. The resulting validation addresses both absolute score differences and rank agreement within the audited sample. We use this evidence at the sample level and interpret close mode rankings through the variability analysis below.

[Weakness 2] Cross-project stability and mechanistic interpretations.

We address stability by matching each conclusion to the relevant comparison and repetition evidence. The main grid contains 300 project-mode-model cells, with P00 team cells averaging three trials. Across the 30 P00 team cells from the three full-grid models, the mean best-score standard deviation is 2.8 points. We use this statistic to describe variation on P00.

We also examine the supplementary records beyond P00. They contain two team observations each for P01/pipeline, P02/PM oversight, and P04/QA-first, plus additional feature-squad observations on P03, P05, and P09. We report both results for each repeated cell and document their coding routes, graders, and recovery settings. Because these conditions vary, the supplementary records provide descriptive coverage of the selected configurations.

Our analysis shows substantial performance differences between collaboration policies within evaluated projects, while the leading observed policy varies across projects. For example, DSv4 Pro scores 88 under pipeline and 48 under open-source review on P00 with the project and model held fixed. Across projects, the six leading modes have mean scores within two points of one another. On live teaching, feature squad and pipeline score 75.1 and 74.2, respectively. We highlight these performance patterns and interpret small score gaps as near ties. Requirement structure helps frame hypotheses about acceptance planning and ownership, providing concrete directions for evaluating task-dependent mode selection.

[Weakness 3] General team-benefit and budget-efficiency claims.

We address the comparison-design concern by separating the repeated same-model control from the broader cross-project reference. On P00, Claude's single-agent mean best score is 64.2 over three runs, and every Claude team policy has a higher mean. Feature squad reaches approximately 94 under the same model, tools, time cap, deployment path, and feedback. The single-agent best scores are 52.750, 91.000, and 48.875 (sample SD 23.283), making its observed variability explicit. Mean coding-agent costs are about USD 92 for the single agent and USD 617 for feature squad. The supported finding is a measured quality gain with additional inference expenditure under shared time and round limits.

We label the cross-project table according to its different comparison design. It compares Claude single agents with the best team-mode mean over DeepSeek Flash, DeepSeek Pro, and GLM-5.2, selected after observing the grid. The caption identifies both the model differences and the selection rule, so readers can interpret it as a descriptive reference.

We also report raw scores, denominators, and the associated grading and coding conditions on single-agent runs for P01-P09. We retain both observations for repeated team cells. The P00 Claude comparison therefore provides direct evidence of higher team mean quality at higher inference cost, illustrating the organizational trade-offs that REVIBE measures. Comparing equal-dollar efficiency is hard since we cannot find the correct pausing point, which may incur compilation error.

AuthorsOpenReview ↗

Response to Reviewer 5m7f [2/2]

[Question 1] Correspondence between the P00 table and appendix examples.

We clarify the different reporting units used in the main table and the worked examples. Table 3 reports per-round averages over three repeated P00 team trials, giving 73/88/94 for Claude feature squad. Appendices H and I present the detailed trajectory of one of these three trials, with scores of 79.22/93.07/96.95, illustrating token accounting and requirement-level repair. The PM example likewise describes a score change within an individual execution. The main table reports averages across trials, and the appendices show the development process and requirement-level changes within a trial included in those averages.

[Question 2] Meaning of Functional Completion Score and critical failures.

We address score interpretation by presenting weighted requirement completion together with critical failure evidence. FCS sums pass/partial/fail credit by requirement weight and averages scores from multiple evaluator classes for the same requirement. It therefore measures weighted completion, with partial credit possible when only part of a user workflow succeeds. Live UI checks contribute evidence of observed functionality, while visual quality and usability remain separate evaluation dimensions. However, FCS does require an operational workflow. For example, if TAgent fails to register an account, the user login FCS will also drop to 0.

Section 4.4 and Appendix G separately report authentication/session/permission failures in 46% of analyzed runs, prerequisite cascades in 39%, and security/transport failures in 78%. These categories can overlap within a run. We also mark the supplementary P08 exception in which TAgent awarded 100 despite a million-document claim backed by only 13 physical documents. Reporting these failures alongside FCS exposes where a high aggregate score can conceal an unmet critical requirement. It also identifies concrete improvement targets: stateful integration tests, prerequisite-aware repair, and security checks at deployment.

[Question 3] Refinement gains and unseen scenarios.

We make the evaluation target explicit throughout the refinement analysis. TAgent supplies both the repair feedback and the rubric used to measure subsequent progress, so the reported gains describe improvement on that visible rubric. Prerequisite-aware UI tests exercise stateful workflows within it, and the requirement traces show both repaired checks and regressions. Independent hidden-scenario testing would extend this validation to transfer beyond the feedback rubric. We also present the fourth-round continuations as exploratory because their graders and scoring denominators change. The supported result is evidence about how refinement changes measured requirement completion over the evaluated rounds.

[Question 4] Dynamic delegation as a baseline.

In our observations, current models using dynamic delegation commonly assign separate features to parallel agents, yielding a feature-squad pattern. Choosing when to delegate and choosing how agents collaborate are distinct decisions. REVIBE addresses the latter by comparing ten prescribed workflows under a shared protocol. These comparisons reveal substantial performance differences within the same task and model, with the leading observed policy varying across projects. They motivate adaptive selection of collaboration policies beyond a default feature-squad pattern. Evaluating a native dynamic coordinator against these reference outcomes would test whether online delegation achieves such adaptation.

[Question 5] Representative audit materials.

We clarify the available audit material and its purpose in Appendix A. The current release contains requirements, rubrics, benchmark code, aggregated metrics, pricing data, and analysis scripts. The manuscript adds a collaboration snapshot and worked examples containing UI states, API responses, and source locations. These materials let readers inspect the illustrated handoffs, scoring evidence, and repair decisions. Our current release scope covers these benchmark assets and representative excerpts.

We commit to releasing a more comprehensive set of representative audit materials upon acceptance. Releasing them now is difficult because sanitizing the repositories for double-anonymous review, on top of the current revision workload, would require substantial additional effort.

The resulting findings connect aggregate performance to evidence that readers can inspect. The blinded audit establishes score agreement within the agent-artifact sample, the P00 comparisons reveal substantial organizational quality-cost trade-offs, and the failure analysis identifies requirements that aggregate scores can conceal. Together, these results make REVIBE useful for both comparing workflows and identifying concrete targets for improving software-development agents.

Thank you for the authors' detailed response and the revisions made to the paper. The newly added experiments and the clarification of the scope of the conclusions have addressed some of my concerns, but I still have two main concerns.

  1. Limited scientific insight. Although the paper conducts experiments on a relatively large scale, the overall analysis remains largely descriptive and lacks systematic, generalizable scientific findings. Some conclusions are relatively intuitive, while findings of greater research value, such as the relationship between task structure and collaboration patterns, lack sufficient validation. Although narrowing the claims alleviates the problem of overclaiming, it does not resolve the issue of limited research contribution.

  2. Insufficient reliability and robustness of the benchmark. I acknowledge the newly added human blind evaluation experiment, but consistency at the overall score level does not sufficiently guarantee the correctness of the evaluation of key requirements, such as the case in Appendix K that receives a full score despite containing only 13 physical documents. In addition, the newly added cross-project repetition experiment suffers from inconsistencies in evaluators and execution conditions, and is still insufficient to verify the stability of performance differences across collaboration patterns.

Overall, I acknowledge the research motivation, engineering effort, and the authors' efforts during the rebuttal stage. However, as an empirical analysis study, its scientific insight remains limited; as a benchmark, its reliability and robustness have not yet been sufficiently validated. Therefore, I decide to maintain my original score.

Thank you for reconsidering the paper and for acknowledging the motivation, engineering effort, and revision. We respectfully argue that the revised paper has resolved both remaining concerns.

1. Scientific insight

The central finding is not intuitive: average rankings hide large, task-dependent policy effects. With project and model held fixed, DeepSeek v4 Pro scores 88 under pipeline and 48 under open-source review on P00, a 40-point gap from the collaboration policy alone (Section 4.5, Table 3). Across the full grid, the ten policies differ by more than 10 points in 25 of 30 project-model cells, with a median spread of 29.9 points. Yet the six leading policies are within two points on cross-project means, and the project-level leader changes from task to task (Section 4.5, Appendix C). A leaderboard of averages would conclude that organization barely matters, while the per-task data show it is often the largest controllable factor.

This makes collaboration a training objective for agentic models, not only an orchestration setting. The same model weights gain or lose up to 40 points depending on how the work is organized, and no fixed policy is best everywhere. Selecting the collaboration structure for a task is therefore a capability in its own right, and REVIBE supplies what is needed to train and evaluate it: a fixed-policy reference grid and requirement-level feedback. And we use Figure 5 and a large portion of Section 4.2 to describe what we find, which is not, as you claimed, "intuitively" or "lack of validation".

On task structure and collaboration pattern, the existence and size of the task dependence are validated by the controlled grid. The mechanism is stated as testable hypotheses, anchored by the failure taxonomy: 56% of failed checks trace to missing features or incomplete logic, authentication and prerequisite failures to stateful integration, and security failures to deployment checks (Section 4.4, Appendix G).

2. Benchmark reliability and robustness

Our validation protocol follows established practice. Checking an automated judge against human grading on a sampled subset is the standard validation for such evaluators. MT-Bench reports over 80% agreement between LLM judges and humans [1], Agent-as-a-Judge validates requirement-level verdicts against a human evaluation baseline [2], and RealDevWorld, the closest design to TAgent, reports a correlation of 0.85 with expert assessment [3].

Requirement-level correctness. Our blinded audit covers 20 previously unaudited repositories, two drawn at random from each project. Reviewers see neither the policy label nor the TAgent score, and two teaching assistants cross-check the grades. The maximum total-score difference is 5 points, Spearman ρ = 0.88, and subjectively judged UI/code items differ by at most 2 points (Section 3.3, Appendix F). This is at the level reported in [1–3].

The Appendix K case. We agree the P08 run should not have received full credit on that requirement. Our own manual audit found it, and we report it as an explicit exception (Appendix K). It concerns one requirement in one project and does not affect cross-policy comparisons, which rest on spreads of tens of points across 30 cells.

Stability of policy differences. This claim rests on the controlled grid, not on the supplementary repetitions. The 30 P00 team cells each have three trials under identical evaluator and execution conditions, with a mean best-score standard deviation of 2.8 points (Section 4.2, Appendix B). The 40-point P00 gap is more than ten times this variation. We agree the cross-project repetitions differ in evaluator and recovery conditions, which is why they are reported separately and are not used to support the stability claim (Table 4, Appendix K).

In summary, automated scores agree with a blinded human audit under a standard validation protocol, the one known rubric failure is disclosed and bounded, and policy effects are far larger than measured trial variation. We hope you will reconsider your assessment in this light.

[1] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks.

[2] Zhuge et al. Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934.

[3] Bian et al. You Don't Know Until You Click: Automated GUI Testing for Production-Ready Software Evaluation. arXiv:2508.14104.

Reviewer_oA57OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a benchmark to evaluate multi-agent mechanisims. The authors design different modes of multi-agent collaborations and evaluate them with a setting involving four agents. The benchmark tasks are from university software engineer projects, covering a wide range of domains. Multiple models, though not frontier at current stage, are evaluated in terms of task score and cost.

Critical strengths
  1. The multi-agent is a future trend and it indeed needs corresponding evaluation protocols and benchmarks.
  2. This work provides detailed taxonomy of collaboration of multi-agent under a specific setting involving four agents.
Critical weaknesses

The multi-agent setting seem to be too specific and limits the impact and generalization of the relevant observations and conclusions. It involves four predefined agents. Moreover, such a setting is quite different from the general multi-agent architectures where agents are spawned online as needed.

The communication between different agents are not well studied. The context exchange is essential for multi-agent collaboration, which influences the costs and final results. The evaluation protocol and experimental analysis do not seem to give sufficient attention to this aspect.

The evaluation does not cover general multi-agent executions that are implemented in harnesses like Codex, Claude Code or other tailored multi-agent harnesses.

Analysis, e.g. section 4.4, are coarse and do not provide much information for enhancement of model/agent capabilities.

Why not do the evaluation on existing long-horizon benchmarks? What are the superiority of the tasks over existing ones? Are there anything that the proposed tasks can measure while existing ones cannot?

Questions for authors

Since it is found that no mode dominates, why not let the agent decide itself in a task-wise manner?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oA57 [1/2]

Thank you for the suggestions on scope, communication, and analysis. We address your concerns with the following responses. Substantive manuscript additions are marked in red.

[Weakness 1] Fixed four-agent policies and generalization to online spawning.

We clarify the design choice by identifying what each policy changes and what the experiment holds fixed. Every mode specifies ownership, concurrency, decision authority, and required handoffs. Team size, tools, time limits, deployment, and evaluation remain shared. This lets us compare concrete workflows such as feature squads, pipeline handoff, and pull-request review under a common setup. The resulting observations concern adoption of complete four-agent policies, including their associated parallelism and coordination work.

We also make model coverage explicit. DeepSeek v4 Flash, DeepSeek v4 Pro, and GLM-5.2 cover the full project grid. Claude Opus 4.8, GPT-5.5, and Qwen3.6-Flash additionally cover the P00 team configurations. This reference grid, covering mainstream models during the development period of this work, supplies measured outcomes against which variable team size and online spawning could be evaluated under the same task requirements and scoring protocol.

[Weakness 2] Communication and context exchange.

We address communication by describing the shared channels, their contents, and a concrete execution trace. LegoGent provides a four-minute synchronization snapshot and an active mailbox for targeted peer questions. Each snapshot contains completed work, test evidence, blockers, and next steps, while mode-specific artifacts provide checkable handoffs. We also give instructions to each agent on what should or should not be passed. Appendix E illustrates an approximately 244-second synchronization interval and 52 peer-mailbox threads.

We account for context exchange within the reported token and time totals, so the measured resource use includes coordination work. Section 3.2 specifies complete policies as the unit of analysis. Attributing tokens to individual channels and varying communication frequency or context size would extend the analysis to individual communication mechanisms. The shared communication interface and concrete trace make the execution process inspectable, while the aggregate measurements expose the total resource consequences of each workflow.

Our messaging mechanism is inherited from Claude Code. With the communication executing in the background, we never witnessed mailbox overflow or communication overhead in message transportation in our evaluation.

[Weakness 3] Coverage of Codex, Claude Code, and other harnesses.

We clarify that every LegoGent agent already runs as an isolated, resumable Claude Code process. Section 3.2 now explains the roles of the two components: Claude Code supplies the common coding interface, and LegoGent applies organizational policies, synchronization channels, deployment checks, and round gates. This identifies the practical execution environment behind the results.

Using one shared harness keeps the coding interface consistent across the ten policies. Our conclusions therefore concern organizational choices within a specific harness, following the example of other benchmarks (e.g., AsyncCodeBench). Comparing native orchestration in Codex or another harness would extend the study through an adapter with specified tools, budgets, and evaluator access. REVIBE supplies the common task and measurement protocol needed to make that comparison interpretable.

AuthorsOpenReview ↗

Response to Reviewer oA57 [2/2]

[Weakness 4] Failure analysis and capability improvements.

We strengthen the failure analysis by linking each major category to a development capability and its observable test target. Section 4.4 and Appendix G classify 1,440 agent-attributable failed checks after excluding 80 grader-side errors. Missing features and incomplete logic account for 56% of these failures, making requirement coverage tracking and executable acceptance checks concrete improvement targets. Authentication and prerequisite failures account for approximately 23%, pointing to stateful integration tests and dependency-aware repair. Security/transport failures account for 8.5%, identifying TLS, redirects, and secure WebSockets as deployment checks that deserve explicit attention.

The worked examples connect these categories to API responses, UI states, and source locations. Readers can inspect both the observed defect and the check needed to detect its repair. These diagnostic targets define concrete interventions for subsequent evaluation. They connect observed performance patterns to specific behaviors that model and agent developers can improve.

[Weakness 5] Relationship to existing long-horizon benchmarks.

We clarify the additional measurement enabled by REVIBE: how alternative organizational policies deliver the same application from requirements under a shared deployment and evaluation process. Agents start from natural-language requirements and an empty repository, define their interfaces, deploy the resulting application, and repair it after live UI/API/code inspection. Repeating this process across ten policies records their functional outcomes, failure patterns, and resource use on a common basis.

We make the relationship to existing benchmarks concrete. MultiAgentBench includes software-level coding and evaluates a serialized solution, while REVIBE probes a deployed application's live UI and APIs. CloudDevBench evaluates a fixed assistant with more detailed, predefined API specifications. Our capstone tasks require agents to coordinate both interface design and integration as part of software delivery. Existing long-horizon tasks could also be adapted to this policy-comparison protocol. The contribution lies in making organizational outcomes measurable across that full process.

[Question 1] Why not let an agent choose the mode for each task?

Our results motivate training models to choose collaboration policies for each task. With DeepSeek v4 Pro on P00, pipeline scores 88 and open-source review 48, showing substantial performance differences under the same model and task. The leading observed mode also varies across projects. In our observations, current models tend to default to a relatively fixed feature-squad pattern. We therefore view adaptive collaboration as a capability that deserves explicit training. REVIBE supplies the evaluation framework and reference outcomes for this direction. Training large models is a next step requiring additional hardware capacity and suitable training data beyond our current resources.

The observed variation across projects motivates adaptive orchestration, and REVIBE supplies a reference for evaluating it. Our shared execution interface makes policy comparisons interpretable, communication traces expose coordination behavior, and requirement-level failures identify specific improvement targets. The contribution is a benchmark that connects organizational decisions to deployed functionality and resource use, with fixed-policy results that future adaptive methods can be compared against.

Reviewer_4fYLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REVIBE, a benchmark designed to evaluate how different multi-agent collaboration patterns affect the quality, time, and cost of full-stack software development. It systematically combines ten software projects with ten four-agent collaboration patterns, using LEGOGENT to execute development and deployment, and TAGENT to conduct requirement-driven inspections of the running applications across UI, API, and code levels. The framework provides weighted scoring and supports up to three rounds of iterative feedback.

The experimental results demonstrate that both collaboration patterns and model choices influence the quality–cost trade-off, and that the optimal collaboration pattern varies across projects. Iterative refinement generally improves outcomes, although it can also introduce functional regressions.

The main limitations concern the experimental design and the strength of the resulting claims. First, repeated experiments are concentrated on a single project, which limits the robustness and generalizability of the findings. Second, the cross-project comparison between multi-agent and single-agent settings conflates the effects of model differences with post hoc selection of the best-performing collaboration pattern, making it difficult to isolate the actual contribution of the collaboration strategy. Third, the conclusions regarding stopping strategies are not directly validated beyond the three-round feedback limit, leaving their broader applicability insufficiently established. These limitations weaken the generalizability of some of the practical recommendations.

Nevertheless, integrating collaboration-strategy comparisons, actual deployment-based validation, iterative feedback, and resource consumption tracking into a unified benchmark represents a meaningful research contribution. Overall, I find the benchmark design and experimental coverage valuable and potentially impactful. However, the experimental reporting and supporting evidence are not yet sufficiently rigorous, and several conclusions require further clarification regarding statistical methodology, the correspondence between reported results and their underlying experimental settings, and the appropriate scope of the claims.

Critical strengths
  • Strength1: A practically meaningful benchmark for comparing collaboration policies throughout the software delivery process. REVIBE integrates ten software projects and ten explicitly defined four-agent collaboration modes with deployment, requirement-based evaluation, iterative feedback, and resource accounting. The shared execution and evaluation framework enables comparisons of complete organizational policies under consistent conditions (Section 3 and Table 2). This combination provides a useful foundation for studying how agent organization affects functional software delivery and constitutes the strongest contribution supporting acceptance.

  • Strength2: Substantial experimental coverage with informative quality–cost and refinement analyses. The full experimental grid covers three models, supplemented by additional models and repeated trials on instant messaging. Reporting functional scores alongside latency, monetary cost, and per-round trajectories exposes trade-offs and regressions that a final success metric would miss (Tables 3–4 and Section 4.3). The requirement-level evaluation evidence also supports more detailed failure analysis, increasing the benchmark’s practical research value.

Critical weaknesses
  • Weakness1: Human–evaluator agreement is insufficiently validated on agent-generated applications. Section 5 reports a Spearman correlation of 0.87 between TAGENT scores and teaching-assistant grades on 97 student projects, providing useful initial validation. However, these human-written projects differ from the agent-generated applications evaluated in REVIBE, and the paper explicitly leaves validation on agent artifacts to future work. Moreover, rank correlation alone does not establish agreement on absolute scores or individual requirement verdicts, which matters when interpreting small differences between collaboration modes. Since TAGENT supplies both refinement feedback and the final evaluation, systematic evaluator errors could affect both development trajectories and reported comparisons. A blinded human audit of representative agent-generated applications, reporting score discrepancies and requirement-level agreement, would strengthen confidence in the benchmark’s conclusions.

  • Weakness2 : Some conclusions about mode suitability and stability exceed the statistical evidence. Repeated trials are concentrated on instant messaging, yet Appendix B uses its average standard deviation of 2.6 to support claims about cross-project differences in Table 4. Although the paper appropriately treats small average gaps as ties and describes mode suitability as observational, some explanations remain overly categorical. For example, Appendix B states that only vertical ownership preserves coherent real-time functionality on live teaching, whereas feature squad exceeds pipeline by only 0.9 points (Table 8). These results suggest potentially useful associations but do not establish stable project-specific rankings or the proposed mechanisms. Repeated comparisons on additional representative projects, or more restrained interpretations, would strengthen these claims.

  • Weakness3 : The evidence for general team benefits is limited by comparison design and unclear result aggregation. Table 3 provides useful same-model comparisons on instant messaging, but Table 7 compares a Claude single agent against the best team mode selected after averaging over three different models. This combines model differences with post hoc mode selection. Furthermore, equal time limits do not imply comparable inference expenditure: the Claude feature-squad configuration scores 94 at approximately $617, versus 64 at $92 for the single agent. Several reported trajectories also require clarification: Table 3 lists 73/88/94 for Claude feature squad, while Appendix I reports 79.22/93.07/96.95, and Appendix H describes its worked example as matching Table 3. These differences may reflect individual trials versus aggregated results, but that relationship is not clearly specified. They do not show the experiments are incorrect, but they reduce confidence in the quantitative comparisons and the broad recommendations to form teams.

  • Weakness4 : Reporting inconsistencies and inaccurate references require correction. Several numerical examples are not clearly reconciled with the main results. For Claude feature squad on instant messaging, Table 3 reports scores of 73/88/94, whereas Appendix I reports 79.22/93.07/96.95; Appendix H nevertheless states that its worked example matches the corresponding Table 3 row. Similarly, Section 4.3 describes a DeepSeek Pro PM-oversight regression from 80.9 to 66.0, while Table 3 reports 73/76/71. These differences may reflect individual trials versus aggregated results, but the aggregation and correspondence should be explicitly documented. The treatment of prior work also requires correction: Table 1 marks MultiAgentBench as not supporting coding, despite its coding scenario in Appendix A.7, and its bibliography entry incorrectly lists “Daisy Zhe Wang” and “Robert Tang” instead of “Zhe Wang” and “Xiangru Tang.” These issues do not invalidate the benchmark contribution, but they reduce confidence in reporting.

Questions for authors

Questions for Authors Q1:Repeated trials and result aggregation. Are repeated-run results available for projects beyond P00? What supports extending the average standard deviation of 2.6 in Figure 7 to the cross-project comparisons in Table 4? Please also clarify whether Tables 3 and 7 report individual trials or averages across repetitions, and how the worked example in Appendix H relates to Table 3. Q2:Team benefits and stability. Are same-model single-agent versus team comparisons available beyond P00, including repeated-run variability for Claude in both settings? If not, would you qualify the claim that the question is which collaboration mode to use, rather than whether to form a team, as an observation limited to the evaluated configurations? Q3:Cost accounting and stopping claims. Do the reported costs include TAGENT calls, deployment, and evaluation of both candidates in competitive mode? Would the quality–cost frontier materially change if historical best scores were paired with the actual cost of completing all rounds? Is the claim that three rounds lie near the knee of the improvement curve supported by any evaluations beyond three rounds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4fYL [1/2]

Thank you for highlighting the value of the benchmark. We hope the following responses address your concern. Substantive manuscript additions are marked in red.

[Weakness 1] Human-evaluator agreement on agent-generated applications.

We address the problem through a blinded audit of existing agent artifacts. We randomly select two previously unaudited repositories from each project, giving 20 repositories. Human reviewers receive neither the collaboration-mode label nor the TAgent score, and two teaching assistants cross-check the grades. Against the existing automated scores, the maximum total-score difference is 5 points, Spearman ρ = 0.88, and the mean automated-minus-human difference is −0.5 points. Subjectively judged UI/code items differ by at most 2 points. Human and TAgent give the exact same score on over 91.2% of requirements. These results provide direct agreement evidence on the artifacts evaluated by REVIBE, complementing the earlier validation on 97 student projects.

We report the blinded audit separately from the grading procedure used for exceptional disagreements. Two initial TAgent passes with a difference above 10 points trigger a third pass and human review. This occurred for three artifacts. We select the existing automated score closest to the human grade in those cases. Among retained agreeing pairs, the mean absolute difference is 3.7 points and the maximum is 5. Section 3.3 and Appendix F distinguish these post-adjudication statistics from the blinded audit and initial-pass variability. The resulting validation addresses both absolute score differences and rank agreement within the audited sample. We use this evidence at the sample level and interpret close mode rankings through the variability analysis below.

[Weakness 2] Stability and interpretations of mode suitability.

We address stability by matching each conclusion to the relevant comparison and repetition evidence. The main grid contains 300 project-mode-model cells, with P00 team cells averaging three trials. Across the 30 P00 team cells from the three full-grid models, the mean best-score standard deviation is 2.8 points. We use this statistic to describe variation on P00.

We also examine the supplementary records beyond P00. They contain two team observations each for P01/pipeline, P02/PM oversight, and P04/QA-first, plus additional feature-squad observations on P03, P05, and P09. We report both results for each repeated cell and document their coding routes, graders, and recovery settings. Because these conditions vary, the supplementary records provide descriptive coverage of the selected configurations.

Our analysis shows substantial performance differences between collaboration policies within evaluated projects, while the leading observed policy varies across projects. For example, DSv4 Pro scores 88 under pipeline and 48 under open-source review on P00. Across projects, the six leading modes have mean scores within two points of one another. On live teaching, feature squad and pipeline score 75.1 and 74.2, respectively. We interpret small score gaps as near ties. Requirement structure helps frame hypotheses about acceptance planning and ownership, providing concrete directions for evaluating task-dependent mode selection.

[Weakness 3] Team benefits, comparison design, and inference expenditure.

We address the comparison-design concern by separating the repeated same-model control from the broader cross-project reference. On P00, Claude's single-agent mean best score is 64.2 over three runs, and every Claude team policy has a higher mean. Feature squad reaches approximately 94 under the same model, tools, time cap, deployment path, and feedback. The single-agent best scores are 52.750, 91.000, and 48.875 (sample SD 23.283), making its observed variability explicit. Mean coding-agent costs are about USD 92 for the single agent and USD 617 for feature squad. The supported finding is a measured quality gain with additional inference expenditure under shared time and round limits.

We label the cross-project table according to its different comparison design. It compares Claude single agents with the best team-mode mean over DeepSeek Flash, DeepSeek Pro, and GLM-5.2, selected after observing the grid. The caption identifies both the model differences and the selection rule, so readers can interpret it as a descriptive reference.

For the single-agent observations on P01–P09, we report raw scores, denominators, and the associated grading and coding conditions. We retain both observations for repeated team cells. This reporting makes the empirical contribution concrete: REVIBE measures the quality and resource trade-offs of organizational choices, with the P00 Claude comparison providing direct evidence of higher team mean quality at higher inference cost. Equal-dollar efficiency remains a separate comparison requiring a common monetary budget, which is hard to implement to ensure it stops at an appropriate place.

AuthorsOpenReview ↗

Response to Reviewer 4fYL [2/2]

[Weakness 4] Numerical provenance and prior-work comparison.

We clarify the different reporting units used in the main table and the worked examples. Table 3 reports per-round averages over three repeated P00 team trials, giving 73/88/94 for Claude feature squad. Appendices H and I present the detailed trajectory of one of these three trials, with scores of 79.22/93.07/96.95, illustrating token accounting and requirement-level repair. For DeepSeek v4 Pro with PM oversight, the example follows a score change from 80.9 to 66.0 within an individual execution, while Table 3 reports a mean trajectory of 73/76/71. The main table reports averages across trials, and the appendices show the detailed progression of a trial included in those averages. The single-agent row is identified separately as the submitted control cohort, including the three-trial Claude aggregate.

We also clarify the prior-work comparison. MultiAgentBench includes software-level coding, and Table 1 now marks its Software entry as partial support. Its coding scenario scores a serialized solution, while REVIBE additionally deploys a full-stack application and probes live UI and API behavior. The bibliography uses the official ACL author names, Zhe Wang and Xiangru Tang, instead of the Google Scholar trace.

[Question 1] Repetitions beyond P00 and aggregation across tables.

We report the available supplementary observations beyond P00 in Table 4 and Appendix K, as detailed under Weakness 2. For the main tables, P00 team cells average three trials, and project means weight cells equally. The single-agent references use the submitted control cohort, including the three-trial Claude P00 aggregate and two-trial Claude aggregates on P05/P06/P07. We identify the appendix trace as a constituent trial and label later supplementary cohorts by their execution conditions. This gives each result an explicit aggregation unit and confines the P00 variability estimate to the project it measures.

[Question 2] Same-model comparisons and qualification of the team-benefit claim.

We base the team-benefit conclusion on the repeated Claude comparison on P00 and report its observed variability and inference cost together. The broader project results serve as descriptive references, with supplementary DeepSeek observations reported under their own execution conditions. The resulting practical finding is specific and measurable: for the evaluated P00 Claude configurations, teams deliver higher mean completion scores under the shared time limit while spending more on inference. This gives practitioners evidence about the quality-cost trade-off of those workflows.

[Question 3] Cost coverage, complete-run accounting, and stopping evidence.

The reported costs cover coding-agent inference for all agents, including both teams in competitive mode. TAgent evaluation and deployment are additional cost categories. We pair each best-round score with the coding-agent expenditure through that round.

  1. For runs that already score highly at an intermediate round, subsequent rounds mainly address the few remaining issues in TAgent's feedback. These targeted edits add relatively little time and token usage, which is negligible.
  2. For lower-scoring runs, the highest-scoring round is determined retrospectively from the completed trajectory. An early stopping decision must therefore use the partial trajectory to predict whether further repairs will improve the score, which is impossible.

We use three rounds as a common evaluation horizon. Appendix K reports five exploratory fourth-round continuations, with their grading conditions and score denominators documented, none improved by more than 5 points

The revised findings show why organization deserves explicit evaluation alongside model choice. For DeepSeek v4 Pro on P00, pipeline and open-source review differ by 40 score points. The Claude comparison shows higher team mean quality with higher inference expenditure, and the broader grid shows changing leading policies across projects. We support these findings with direct score-agreement evidence on audited agent artifacts and explicit comparison rules. REVIBE provides a common protocol for measuring these organizational trade-offs and evaluating future policies against concrete reference outcomes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Changes

We thank the reviewers for their careful reading and constructive suggestions. We appreciate their recognition of the problem's practical importance, the value of comparing complete collaboration policies, and the evaluation of deployed software through quality, time, cost, and refinement.

REVIBE makes collaboration policy an explicit variable in software-agent evaluation. We compare ten policies on ten full-stack projects under a shared execution and evaluation protocol. Our central finding is that policy choice can produce substantial performance differences with the task and model held fixed, and this guides us that collaboration policy is an important training goal other than coding capability itself, which is not addressed by prior works. On P00, DeepSeek v4 Pro scores 88 under pipeline and 48 under open-source review. The leading observed policy varies across projects, while the six leading cross-project means are within two points. We emphasize these substantial differences and interpret small gaps as near ties. This provides concrete reference outcomes for evaluating task-dependent policy selection (Sections 4.2 and 4.5 and Appendix C).

We address the reviewers' concerns through the following revisions. Substantive manuscript additions are marked in red.

  • Direct evaluator validation. We add a blinded audit of 20 previously unaudited agent-generated repositories, randomly selecting two per project. Human reviewers see neither the policy label nor TAgent score, and two teaching assistants cross-check the grades. The maximum total-score difference is 5 points, Spearman ρ = 0.88, and the mean automated-minus-human difference is −0.5 points. We distinguish this audit from the separate procedure for large grading disagreements, which selects the existing automated score closest to the human grade for three artifacts (Section 3.3 and Appendix F).

  • Clearer comparisons and resource accounting. We clarify the repeated same-model comparison on P00. Claude feature squad reaches approximately 94 versus 64.2 for the single agent, with mean coding-agent costs of about $617 versus $92 under shared time and round limits. This demonstrates higher mean quality with additional inference expenditure. We distinguish this comparison from the broader cross-project reference and report supplementary repetitions with their execution conditions (Section 4.1, Tables 3 and 4, and Appendices B and K). We also specify that costs cover all coding agents, including both competitive teams, through the selected checkpoint. TAgent evaluation and deployment are separate cost categories (opening of Section 4 and Appendix H).

  • Actionable interpretation of scores and repairs. We clarify that FCS measures weighted requirement completion and connect critical failures to concrete improvement targets, including requirement coverage, stateful integration tests, and deployment security checks (Section 3.3, Section 4.4, and Appendices F and G). The supplementary P08 audit illustrates why a high score should be read alongside requirement evidence (Appendix K). We describe refinement gains as progress on the feedback rubric and report the fourth-round continuations with their grading conditions, making the scope of the stopping evidence explicit (Section 4.3 and Appendix K).

  • Explicit policies, execution, and reporting. We clarify ownership, concurrency, decision authority, and handoffs, identify Claude Code as the shared harness, and explain synchronization and peer communication (Sections 3.1 and 3.2 and Appendices C and E). We clarify that Table 3's P00 team entries average three trials and that the Claude example in the appendices follows one of those trials (Table 3 caption and Appendices H and I). We also refine the comparison with prior benchmarks and specify the available benchmark assets and representative evidence (Section 2, Table 1, and Appendix A).

These revisions connect our main contribution to direct evaluator validation, explicit comparison rules, and inspectable failure evidence. REVIBE provides both empirical findings about the consequences of collaboration policy and a common protocol for evaluating future adaptive orchestration. We thank the reviewers again for helping us strengthen this contribution, and we hope that the revision can address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.