acceptodds
Under review as a conference paper at ICLR 2027

DeliverBench: Benchmarking AI Agents on Delivering Realistic Multi-Stage Professional Projects

Abstract

As LLM agents grow more capable, delegation is expanding from individual assistance to end-to-end professional project delivery, with potential to support enterprise automation and even one-person companies. Some existing benchmarks cover professional tasks and extended workflows, but inadequately assess how projects evolve and whether their final multi-file deliverables meet acceptance criteria. Thus, we introduce DeliverBench, which takes a multi-stage professional project as its evaluation unit. Reconstructed from real-world projects with domain experts, it includes 11 projects across seven domains and 150 interdependent subtasks. Agents work in a persistent workspace, incorporating new materials and client feedback released along a project timeline while reusing, revising, and integrating their own earlier artifacts. Expert-written rubrics assess checkpoint submissions and evaluate final packages for deliverable quality, cross-file consistency, and package acceptance. Across seven model–harness systems, we find that mean final-delivery scores range from 48.1 to 70.0 out of 100, and subtask pass rates at a 90-point threshold range from 13.3% to 50.0%. Every system's mean final-delivery score falls below its mean process score, revealing a gap between local task performance and project delivery. Correctness and usability are the weakest dimensions, while case studies highlight sustained quality and artifact repair. Runtime and token use vary widely, with greater expenditure not consistently associated with better delivery. These findings indicate that task-level scores alone cannot establish project readiness, motivating evaluation of agents’ ability to sustain professional work and deliver coherent, usable packages.

Primary area:
datasets and benchmarks
Keywords:
llm agent, benchmark, professional project

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JtMPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DeliverBench, a benchmark harness for evaluating agents that are tasked with completing large projects involving many tasks and perturbing inputs. These projects are typically managed at the executive level of organizations, which distinguishes the benchmark from other benchmarks based on individuals and small teams. DeliverBench includes 11 projects constructed from real anonymized cases by domain experts in domains like law, finance, medicine. DeliverBench organizes tasks within each use to make dependencies explicit, and manages the introduction of new data using a predetermined schedule, simulating what happens in real-world engagements. The paper evaluates 7 model harnesses and shows that end-to-end project delivery is challenging for current agents, with local task scores not being indicative of project completeness.

Critical strengths
  • Paper looks at timely topic of running agents at scale on complex non-coding tasks that normally staffed by large teams of humans
  • Construction of domain specific tasks appears sound, though laborious
  • Shows data that professional projects with many tasks is challenging for state-of-the-art agent harnesses and that the “sum-of-parts does not equal the whole”
  • Paper is well-written
Critical weaknesses
  • Limited number of cases in benchmark makes it less convincing that results generalize (though this is acknowledged in the paper)
  • Newer benchmark research demonstrates that outcome based is limited. The trajectory of the agents at the task level should be monitored to mitigate hallucinations, misbehavior, and leakage that may bias judging.
Questions for authors
  • Who assigns roles of agents when performing a task? Is this fixed or self-organizing
  • What are the prompts used for agents and judges?
3 comments

W1 — Benchmark Scale and Generalization

Thank you for your concern about project scale and the generalizability of the results. We understand that project count affects conclusions about cross-project generalization, and the paper discusses this limitation. During construction, we sought to balance domain coverage and project depth, preserving real-world task dependencies and repeated handoffs within the available expert time and budget. Twelve domain experts contributed real projects, and eight construction experts organized and reviewed the materials. Through expert interviews and cross-checks, we reconstructed professional work spanning weeks to months into executable, evaluable projects. Each project required checking earlier deliverables, subsequent tasks, and acceptance requirements individually, involving considerable professional input and review.

The current 11 projects span seven domains and contain 150 subtasks with concrete assignments, delivery requirements, and evaluation criteria. For comparison of task counts, TheAgentCompany contains 175 workplace tasks, and GDPval's public gold set contains 220 professional tasks. DeliverBench further preserves the business dependencies among these subtasks, allowing us to examine whether models can carry work across different stages through to complete delivery in multiple professional settings.

Project-level evaluations also use relatively small numbers of complete instances. For example, PaperBench, discussed in the paper, contains 20 paper-replication projects, each requiring the agent to understand a paper, implement code, and run experiments. DeliverBench's 11 projects similarly contain multiple interdependent work components. We hope that considering project count, internal task coverage, and complete-delivery requirements together gives a fuller account of the breadth and depth of work covered by the current evaluation.

W2 — Trajectory Monitoring and Evaluation Integrity

Thank you for this suggestion. We agree that incorporating execution trajectories enables a fuller examination of agent behavior, particularly when verifying whether material access, actual operations, and submitted content correspond to one another. DeliverBench currently leverages fine-grained, expert-authored rubrics to assess each submission and the final deliverable package. This is a well-established and widely adopted strategy for evaluating complex professional work. For instance, APEX-Agents adopts expert-authored rubrics paired with an LLM judge to evaluate completed work criterion by criterion, and GDPval integrates detailed task rubrics into its evaluation pipeline. Following this paradigm, DeliverBench assesses all subtask submissions and the final deliverable against clear professional requirements.

To cover professional delivery requirements as comprehensively as possible, we constructed 5,322 unique criteria spanning five quality dimensions, correctness, completeness, consistency, professionalism, and usability. They separately assess subtask submissions, final artifacts, cross-file relationships, and package acceptance requirements. This supports concrete checks of factual and computational correctness, whether corrections are applied to the relevant files, and whether final delivery meets project requirements. The authenticity of an artifact's own claims about sources or actions still requires verification against source materials and execution records.

Following your suggestion, we inspected six runs from two systems across three projects involving audit, forensic investigation, and data delivery, examining material access, file writes, and the correspondence between submitted content and actual operations. In the inspected trajectories, we found no evidence of access to this evaluation's private grading materials or tampering with the grading environment. Trajectories also helped verify concrete repairs. For example, in P05, the agent omitted a transaction-reconciliation workbook in an earlier task, then reconstructed it from retained parsing results and used it for subsequent entity checks. The tool outputs, actual workbook, and downstream references corresponded, confirming that the repair occurred.

We believe your suggestion can further improve this type of evaluation. Building on the existing trajectory records and inspections, we plan to explore more systematic trajectory evaluation of material provenance, tool operations, and their correspondence with submitted content, complementing the current artifact-based evaluation.

Q1 — Role Assignment and Agent Autonomy

Thank you for this question. Domain experts specify the agent's professional role in advance based on the real-world work setting and state it explicitly in the project instructions. The role usually corresponds to the expert's professional responsibilities in that project. For example, P02 is constructed from actual project materials supplied by a financial expert. Its instructions assign the agent the role of the Project Mars financial due-diligence team, providing analysis for the transaction team, lenders, and subsequent valuation work.

All evaluated systems receive the same role and business requirements for a given project. Within this setting, the agent autonomously decides how to inspect materials, organize its analysis, and prepare deliverables.

Q2 — Agent and Judge Prompts

1. Agent prompts. Each system uses its harness's native system prompt and tool interface, together with the benchmark-supplied project brief and subtask instructions. These specify the professional role, objective, currently available materials, information cut-off, and required submissions. Later subtasks continue in the persistent workspace, using the agent's earlier artifacts and newly released materials.

2. Judge prompts. Each request combines the review instructions, project and subtask context, and the criteria to assess. The Judge receives a read-only evidence view containing the assessed submission, preserved earlier submissions, released source materials, and project instructions. It returns criterion-level judgments, reasons, and supporting evidence, with evidence located to specific files and relevant pages, sections, paragraphs, tables, or cells.

The instructions also specify how to inspect files, report insufficient evidence, and handle embedded attempts to alter grading rules. Such embedded instructions are treated as data to verify. The evaluation program validates the returned judgments and applies the predefined scoring rules.

Appendix D.2 now describes this prompt construction and evaluation interface. Figure 10 provides concrete P02 examples, including the project brief, the T09 instruction that uses the agent's T08 workbook, selected Judge instructions, a complete rubric criterion, and the evidence scope. Scoring and evaluator settings are detailed in Appendix C.3-C.4.

AuthorsOpenReview ↗

Follow-up on Our Responses and Revised Manuscript

Dear Reviewer JtMP,

As the discussion period draws to a close, we would like to ensure that we have addressed your concerns satisfactorily. We have carefully responded to your comments and updated the manuscript with additional experiments and clarifications. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

If our responses and the additional evidence have satisfactorily addressed your main concerns, we would be grateful if you could reconsider your assessment and rating in light of these updates.

Thank you again for your time and thoughtful review.

Reviewer_CD56OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a novel bencmark DeliverBench, a benchmark for multi-stage professional project delivery, including 11 expert-reconstructed projects across seven domains with 150 interdependent subtasks and 111 required final deliverables. The authors evaluate seven model–harness systems, showing that mean final-delivery scores range from 48.1 to 70.0, while the fraction of subtasks scoring at least 90 ranges from 13.3% to 50.0%. Evaluator validation compares 478 binary rubric judgments against expert judgments on one system’s final outputs from three projects, obtaining 93.31% agreement.

Critical strengths
  1. The scenario of the benchmark is novel and meaningful, derived from daily work in professional fields.
  2. The experiments are sufficient and comprehensive. In addition to the final score, the paper provides a phased execution trajectory, specific failure and remediation cases, as well as a joint analysis of delivery quality, runtime, and token usage.
Critical weaknesses

None

Questions for authors

I am curious about how different combination of models and harness would affect the final score.

2 comments

Q1 — Model-Harness Combinations

Thank you for your positive assessment of DeliverBench and for raising the question of model-harness combinations. We added crossed evaluations on three projects and observed that changing the harness affects final scores, with effects that vary across models and projects.

The original experiments prioritized each model's associated native harness. In the supplementary experiments, we selected P02 refinancing, P08 pharmacovigilance, and P10 legal work from finance, medicine, and law. All seven models were evaluated with Codex and Claude Code, alongside results from Qwen Code, Kimi CLI, and Gemini CLI. The table reports equally weighted mean final-delivery scores across the three projects.

ModelCodexClaude CodeOther native harness
Claude Opus 547.0153.50—
GPT-5.6 Sol59.7255.23—
Qwen3.8-Max39.9944.8344.39 (Qwen Code)
DeepSeek V4.1 Flash45.2650.03—
Kimi K341.8242.1541.97 (Kimi CLI)
GLM-5.235.0337.43—
Gemini 3.8 Flash23.7115.3322.02 (Gemini CLI)
  1. Different models benefit from different harnesses. Claude and GPT achieve higher three-project means with their respective Claude Code and Codex harnesses. Other models can also benefit from changing harnesses. For example, Qwen performs better with Claude Code than with Qwen Code on the pharmacovigilance project. Exploring alternative pairings is also worthwhile for models with an associated harness.
  2. The effect of a harness on the same model varies across projects. DeepSeek's scores under the two harnesses differ by approximately 15 points on the legal project and are very close on the other two projects. Kimi's best-performing harness also differs across all three projects. This shows that cross-project means alone can obscure the strengths or weaknesses of a pairing on a particular project.
  3. Substantial performance differences between models remain under a shared harness. On these three projects, GPT, Claude, and DeepSeek maintain relatively strong overall performance with either Codex or Claude Code. These results allow us to compare models in a common execution environment and separately examine the effect of changing the harness for a given model.

Appendix D.6 of the revised manuscript includes the model-harness comparison. Table 6 reports the three-project means, and Tables 7-9 provide the complete results for P02, P08, and P10, respectively.

Thank you for your additional experiments, my major concerns have been addressed and I will raise the score.

Reviewer_jTfrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeliverBench to evaluate AI agents on realistic, multi-stage professional projects spanning 11 real-world tasks across seven domains. The authors find that while agents perform passably on individual intermediate steps, every tested system scores significantly worse on the final delivered package (48.1 to 70.0%), demonstrating that individual task success does not guarantee overall project readiness.

Critical strengths

S1. The tasks come from real-world professional projects, with interdependent subtasks, staged information releases, and a multi-file final package. This is richer than single-task benchmarks.

S2. Scoring checkpoint submissions separately from the final package, with quality-dimension labels, allows useful diagnosis, such as the P05 artifact-repair case study.

S3. The resource analysis (time and tokens versus score) is informative, and current agents are far from saturating the benchmark.

Critical weaknesses

W1. The benchmark hands agents a fixed, expert-authored decomposition. The subtasks, their order, the information releases, and the feedback are all prescribed. Studies of programming by chat suggest that real users work differently. Tang et al. (2026, arXiv:2604.00436) analyze 11,579 real IDE chat sessions and report that developers externalize plans into persistent artifacts and refine requirements progressively instead of specifying complete tasks upfront. In this style of work, the agent is asked to break the project down, and the plan is kept and reused while the agent executes it step by step. Planning is therefore part of the job in realistic use, and DeliverBench removes it. The benchmark measures execution of a pre-segmented workflow, not project delivery. The "realistic" claim in the title and abstract, and the "CEO sets goals and accepts delivery / one-person company" framing in Figure 1, therefore overstate what is measured.

W2. The authors should add a condition where the agent builds and maintains its own plan from the project brief, or narrow the claims. Scale compounds the problem. There are only 11 projects, and four of the seven domains have a single project, so domain-level observations are anecdotal. The paper also does not quantify how much of each project is original real-world material and how much is author reconstruction and scenario assumption.

W3. The paper calls the rubrics "expert-written", but the AI Use Statement discloses that generative AI drafted and refined a subset of the criteria, and the size of that subset is not given. The paper does not report any verification of the AI-drafted criteria. It gives no inter-annotator agreement among experts, no human–AI agreement on criteria, and no rate of criteria that experts edited or rejected. It also reports no check for ambiguous or non-verifiable criteria. "Experts review and refine" (Step 5 of the construction pipeline) is a statement of intent, not a verification protocol. Because the rubrics define every score, their acceptance as ground truth cannot be taken for granted. W3. The model family that drafted the criteria may also be the family that judges them (gpt), which risks circularity. The main text also does not show example criteria, weights, or acceptance gates, so this crucial part is left to the appendix.

W4. The paper says experts have "several years of professional experience" but does not say how many took part. It is unclear whether each project was built and reviewed by multiple experts or by a single expert, whether experts worked across projects or one per domain, and whether authors and reviewers were different people. No annotation guideline is reported. The paper does not say what instructions experts received for decomposing projects into subtasks, deciding what information is released at each stage, writing instructions, or writing and scoring rubric criteria. Without shared guidelines, nothing ensures that granularity, strictness, weighting, and the meaning of the quality dimensions are consistent across the 11 projects, seven domains, and 4,313 criteria. Differences in scores between projects or domains could then reflect differences between experts rather than differences in difficulty. If there was a single expert per project, there is no cross-check at all. "Expert-guided" and "expert-reviewed" therefore cannot be assessed.

W5. Human agreement is checked on only 3 of 11 projects (P01, P05, P10), on one agent system (GPT-5.6 Sol with Codex), and on final-delivery judgments only. The paper admits that checkpoint submissions and other systems need separate assessment. Yet 3,828 of the 4,313 criteria are process criteria, so the process scores, the process–final gap, and the 90-point pass rates rest on judgments that were never validated. Only raw agreement is reported (446/478 = 93.3%), with no chance-corrected measure such as Cohen's kappa and no direction of errors (too lenient or too harsh). The judge (GPT-5.6 Luna) and the validated system (GPT-5.6 Sol) are from the same family, so self-preference is possible, and the GPT/Claude tie at about 70 may be affected. It is unclear who the human judges were and whether they saw the judge's verdicts.

W6. The central finding is that final-delivery scores are lower than process scores for every system. But the two scores use different artifact versions, criteria, and acceptance gates, as the authors acknowledge. A lower final score may partly reflect stricter or differently scaled criteria rather than a weaker ability to integrate work. No control applies identical criteria to the same artifacts at both stages. The conclusion that task-level scores cannot establish project readiness is plausible but not demonstrated. W7. Each system–project pair has a single run, with no confidence intervals, repeated runs, or significance tests, over only 11 projects. Several systems are within a few points of each other. Models and harnesses are confounded, since seven different model–harness pairs are compared, so claims about "stronger agents" cannot be separated from harness effects. The case studies use three hand-picked projects (P01, P05, P11) with no selection criterion, and the cost–quality analysis rests on seven points.

Questions for authors

Please have a look at W1-W6, specifically,

  • Have you tested a condition where the agent produces and externalizes its own plan, as in real programming-by-chat use, instead of receiving the expert decomposition? How does the choice of fixed task division affect results?
  • How many experts took part, and was each project built and reviewed by multiple experts or by one? What written guidelines did they follow to keep decomposition, information release, and rubric writing consistent across projects and domains? What fraction of each project is original versus reconstructed?
  • What fraction of the criteria was AI-drafted, and what were the expert edit and rejection rates? Is there any inter-expert or human–AI agreement on rubrics, and were the drafting and judging models from the same family?
  • Can you validate the judge on checkpoint submissions, on more projects and systems, with kappa and a judge from a different family?
  • Does the delivery gap persist when identical criteria are applied at both stages? Can you add repeated runs, confidence intervals, and significance tests?
8 comments

W1, W2 & Q1 — Workflow Structure, Planning, and Claim Scope

Thank you for emphasizing the importance of autonomous planning and for pointing out wording that could cause misunderstanding. We would like to clarify that DeliverBench reconstructs the intermediate deliveries required by a professional engagement; the agent must independently plan how to complete them. Specifying these business obligations does not provide an execution plan for fulfilling them.

1. Business-Driven Subtasks and Agent Autonomy (W1)

In real-world projects, clients and other participants request particular documents, analyses, and reviews at different points. Their requirements often specify the content and format of the deliverables. Our experts reconstruct these requirements along the source project's timeline. The subtask sequence reflects business dependencies and handoffs, while information releases and feedback reproduce the arrival of new evidence and requests for revision.

For example, in P02, due-diligence findings support a lender presentation and an investment-committee review; revised financing terms must then be incorporated into the financial model before subsequent coordination and signing preparation. These are materials that the engagement requires during the project. Delivering only a final package would leave those intermediate obligations unmet. Appendix B.1 describes this workflow. In the revised benchmark, P02 also generates lender clarification questions from the agent's submitted documents, so the feedback content can differ between submissions.

The agent still decides how to inspect the evidence, resolve inconsistencies, organize calculations, use tools, and prepare the requested files. The delivery sequence provides business constraints; it does not prescribe the steps needed to produce a satisfactory result.

2. Planning and Progressive Requirements in Tang et al. (W1)

We agree with Tang et al.'s observations about persistent planning artifacts and progressively refined requirements. These observations are compatible with the setting we model. A client can request an intermediate report and later supply additional evidence or revisions, while the agent plans how to meet each request. The existence of those requests does not eliminate the need for planning.

DeliverBench likewise releases new materials and requirements as work progresses, and its persistent workspace allows the agent to retain and revise its own plans and intermediate files. For example, receiving updated financing terms requires the agent to identify affected calculations and documents and organize the necessary revisions. Thus, the agent must plan both the work within a subtask and the updates needed across dependent deliverables. The business deliveries and the agent's working plan serve different purposes: one defines what the engagement requires, while the other organizes how to accomplish it.

3. Planning and the Choice of Evaluation Setting (W1, W2, Q1)

We agree with your emphasis on creating, retaining, and continually revising plans. DeliverBench's persistent workspace allows the agent to retain its own plans and intermediate files, while subsequent tasks require it to incorporate new information and continue the work. Planning in this setting includes organizing the execution of the current task, identifying how earlier results affect later work, and deciding which analyses and files need revision.

Regarding Q1, the current experiments consistently use business milestones and delivery sequences reconstructed by experts from real projects. These milestones reflect the actual requirements of client reviews, material handoffs, and subsequent work, providing project-level structure. The agent still needs to organize its analyses and operations autonomously within these constraints and adjust its work plan as new materials and feedback arrive.

The condition you propose, in which the agent divides the project into tasks, has research value. In DeliverBench, intermediate deliveries also provide materials to other participants, elicit feedback, and initiate subsequent work. Letting the agent determine these milestones and their order would require corresponding changes to information releases, handoffs, and intermediate acceptance. In this evaluation, we aim to address a specific question of whether an agent can carry interdependent professional work through to complete delivery under established business collaboration requirements. We therefore retain this experimental setting and remove wording such as CEO and one-person company to align the claims with the capabilities evaluated.

W2 — Project Scale, Domain Coverage, and Representativeness

Thank you for your concern about project scale and domain representativeness. We understand that the small number of projects in some domains limits conclusions about performance across those domains. The depth of project construction, the work covered within each project, and observations across projects also help explain the representativeness of the current evaluation.

1. Project Scale and Construction Depth (W2)

Thank you for your concern about project scale and the generalizability of the results. We understand that project count affects conclusions about cross-project generalization, and the paper discusses this limitation. During construction, we sought to balance domain coverage and project depth, preserving real-world task dependencies and repeated handoffs within the available expert time and budget. Twelve domain experts contributed real projects, and eight construction experts organized and reviewed the materials. Through expert interviews and cross-checks, we reconstructed professional work spanning weeks to months into executable, evaluable projects. Each project required checking earlier deliverables, subsequent tasks, and acceptance requirements individually, involving considerable professional input.

Full-project evaluations also use relatively small numbers of complete instances. For example, PaperBench, discussed in the paper, contains 20 paper-replication projects, each requiring the agent to understand a paper, implement code, and run experiments. DeliverBench's 11 projects similarly contain multiple interdependent work components.

At the subtask level, the current 11 projects span seven domains and contain 150 subtasks with concrete assignments, delivery requirements, and evaluation criteria. For comparison of task counts, TheAgentCompany contains 175 workplace tasks, and GDPval's public gold set contains 220 professional tasks. DeliverBench organizes these subtasks into complete projects with business dependencies, covering continuous work from material analysis and intermediate delivery through feedback-driven revision and final handoff. We hope that considering project count, internal task coverage, and complete-delivery requirements together gives a fuller account of the scope and depth of the current evaluation.

We will continue to expand DeliverBench with additional projects across professional domains to support a more comprehensive evaluation of cross-project generalization.

2. Professional Work and Domain Coverage (W2)

The 11 projects cover strategy consulting, financing and due diligence, audit and investigation, design coordination, quantitative research, drug safety, compliance and legal services, and data acceptance and delivery. Table 2 in Appendix B.2 of the revised manuscript lists the names of P01-P11, followed by descriptions of their work, task dependencies, and final-delivery requirements.

Within each project, the work further includes evidence verification, modeling, interviews, risk assessment, document drafting, feedback-driven revision, and final handoff. Each subtask has a concrete assignment, delivery requirements, and evaluation criteria, and its outputs are used in subsequent work. Project count describes coverage of independent business scenarios, while the 150 subtasks capture the breadth and depth of work within those scenarios.

3. Cross-Project Evidence and Domain-Level Interpretation (W2)

In the current results, all seven evaluated systems have lower mean final-delivery scores than mean process scores. This gap is also observed in three different project types involving strategy consulting, forensic investigation, and data delivery. Artifact and trajectory analyses provide concrete explanations. For example, in P05, the agent needed to reconstruct an omitted transaction-reconciliation workbook to restore the dependency required for subsequent entity analysis.

These results show that, across the professional scenarios covered, maintaining earlier work, incorporating later changes, and producing a usable final delivery after completing individual tasks remain shared challenges for multiple systems. For domains represented by a single project, we interpret observations in terms of that project's tasks and delivery requirements. Our overall discussion of project-delivery capability draws on performance across all projects and the concrete cases.

W2 & Q2 — Original Materials and Project Reconstruction

Thank you for your interest in the provenance of the project materials. We have added counts of the business input files by source for each project and explain the purposes of material adaptation and project reconstruction.

1. Input-File Provenance (W2, Q2)

Counting business input files, the 11 projects contain 669 files, comprising 535 real files (79.97%) and 134 synthetic or adapted files (20.03%). The project-level breakdown is provided below.

ProjectBusiness input filesReal filesSynthetic/adapted filesReal-file share
P016056493.33%
P0259491083.05%
P0326161061.54%
P047162987.32%
P055752591.23%
P06159660.00%
P077366790.41%
P082924582.76%
P0927270100.00%
P103529682.86%
P112171457266.82%
Total66953513479.97%

2. Anonymization and Workflow Reconstruction (W2, Q2)

Some files were reconstructed or adapted for anonymization, preserving task-relevant business information and professional constraints. Under the guidance of domain experts, the construction team also organized task instructions, intermediate delivery requirements, and material-release schedules according to actual project workflows, converting ongoing professional work into executable, evaluable multi-stage projects. The table reports material provenance by file count. Task dependencies and acceptance requirements were confirmed through the business context supplied by experts and cross-review.

W3 & Q3 — Rubric Construction and Cross-Family Judge Validation

Thank you for your concern about rubric provenance, review procedures, and scoring reliability. We clarify the respective roles of experts and AI in constructing criteria and provide additional cross-family Judge comparisons.

1. Expert Criteria, AI Assistance, and Review (W3, Q3)

We thank the reviewer for raising this important concern. We agree that the term "expert-written" did not sufficiently clarify the role of generative AI in rubric construction or the extent of expert verification.

Specifically, domain experts provide the original criteria based on the business requirements and acceptance standards of real projects. Generative AI assists in organizing these criteria into a common rubric format and making them atomic and self-contained. This includes splitting compound requirements, adding context needed for assessment, and specifying applicability.

The resulting rubrics are reviewed and revised by experts before adoption. This review examines their business meaning, evidential basis, and verifiability, rather than formatting alone. Ambiguous or difficult-to-verify requirements are removed during this process. The adopted rubrics therefore combine expert-provided criteria, AI-assisted structuring, and expert verification.

2. Criterion Agreement Across Judge Families (W3, Q3)

Our choice of Judge balances evaluation quality, execution speed, and cost. To examine the potential effect of model family on scoring, we used Claude Sonnet 5.5 to reassess existing artifacts from seven systems on P02, P08, and P10 and compared its results with the original GPT-5.6 Luna Judge. Tasks, rubrics, evidence files, Judge instructions, and scoring rules were fixed, covering 15,363 paired decisions.

Evaluation scopePaired decisionsCriterion-activation agreementCohen’s κ
Process evaluation12,16492.64%0.8488
Final-delivery evaluation3,19992.78%0.8549
Overall15,36392.67%0.8514

Agreement exceeds 92% in both evaluation stages, with chance-corrected kappa values around 0.85, supporting consistent application of these criteria across different model families.

3. Sensitivity of Final Scores to Judge Choice (W3, Q3)

On the 0-100 scale, the mean change across 21 final-delivery scores is +0.90 points, and the mean absolute change is 2.85 points, with 19 changes within five points. These results quantify the effect of Judge choice on scores and provide empirical support for cross-family scoring stability across the tested projects and systems. Criterion-level agreement and all 21 paired final-delivery scores are provided in Appendix F.2, Tables 12-13 of the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewers,

We would like to follow up on our rebuttal and revised manuscript and ensure that we have addressed your concerns satisfactorily. We have carefully worked through the comments from all reviewers, provided detailed responses, and updated the manuscript with additional experiments and clarifications. We believe these responses and revisions provide a thorough and substantive response to the concerns raised.

We would greatly appreciate your taking the time to review these updates. If our responses and the additional evidence have satisfactorily addressed your main concerns, we would be grateful if you could reconsider your assessments and ratings in light of these updates. If any points remain unclear or would benefit from further discussion, please let us know.

Thank you all for your time, thoughtful feedback, and consideration of our work.

Best regards,

The Authors

AuthorsOpenReview ↗

Rebuttal Summary and Request for Consideration

Dear Program Chairs, Senior Area Chair, Area Chair, and Reviewers,

Thank you for your time and consideration. We would like to summarize our contribution, the work completed during rebuttal, and the current discussion status.

1. Contribution of DeliverBench

DeliverBench takes multi-stage professional projects as its evaluation unit. It comprises 11 projects with 150 interdependent subtasks. As instructions, new materials, and client feedback arrive along the project timeline, agents complete successive subtasks and produce new deliverables while reusing, revising, and integrating their own earlier artifacts in a persistent workspace. Expert-reviewed rubrics assess process submissions and evaluate final packages for artifact quality, cross-file consistency, and package acceptance.

Our evaluation of seven model–harness systems reveals a gap between single task performance and project delivery: every system's mean final-delivery score falls below its mean process score. These results suggest that task-level capability does not simply compose into delivery; delivery also depends on keeping the work coherent as it evolves.

2. Rebuttal Responses and Additional Evidence

We have responded point by point to all three reviews and supplemented the manuscript with:

  • Model–harness comparisons using Codex, Claude Code, and relevant native harnesses.
  • Judge validation through human-expert comparisons and cross-family rejudging of existing outputs.
  • Repeatability analyses through independent complete runs, confidence intervals, and full-benchmark statistical tests.
  • Methodological clarifications covering material provenance, rubric construction and expert verification, agent autonomy, scoring procedures, and prompts, alongside sampled trajectory checks.

These additions strengthen the empirical support and transparency of the evaluation. Detailed harness comparisons, Judge validation, and repeatability analyses are provided in Appendices D.6, F, and G.

3. Discussion Status and Request for Consideration

During the discussion, one reviewer increased their rating from 3 to 4. The other two ratings remain at 2. At the time of writing, we have not yet received follow-up comments from those two reviewers on our rebuttal or supplementary experiments, including after our reminder comments. We would appreciate their feedback on whether any concerns remain unresolved.

We respectfully ask the chairs to consider the complete response record, revised manuscript, and additional evidence when assessing the paper. We would also be grateful if the reviewers could examine these updates at their earliest convenience and reconsider their assessments and ratings where the new evidence addresses their concerns.

Thank you again for your time and thoughtful consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.