- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents DeliverBench, a benchmark harness for evaluating agents that are tasked with completing large projects involving many tasks and perturbing inputs. These projects are typically managed at the executive level of organizations, which distinguishes the benchmark from other benchmarks based on individuals and small teams. DeliverBench includes 11 projects constructed from real anonymized cases by domain experts in domains like law, finance, medicine. DeliverBench organizes tasks within each use to make dependencies explicit, and manages the introduction of new data using a predetermined schedule, simulating what happens in real-world engagements. The paper evaluates 7 model harnesses and shows that end-to-end project delivery is challenging for current agents, with local task scores not being indicative of project completeness.
Critical strengths
- Paper looks at timely topic of running agents at scale on complex non-coding tasks that normally staffed by large teams of humans
- Construction of domain specific tasks appears sound, though laborious
- Shows data that professional projects with many tasks is challenging for state-of-the-art agent harnesses and that the “sum-of-parts does not equal the whole”
- Paper is well-written
Critical weaknesses
- Limited number of cases in benchmark makes it less convincing that results generalize (though this is acknowledged in the paper)
- Newer benchmark research demonstrates that outcome based is limited. The trajectory of the agents at the task level should be monitored to mitigate hallucinations, misbehavior, and leakage that may bias judging.
Questions for authors
- Who assigns roles of agents when performing a task? Is this fixed or self-organizing
- What are the prompts used for agents and judges?
3 comments
Q1 — Role Assignment and Agent Autonomy
Thank you for this question. Domain experts specify the agent's professional role in advance based on the real-world work setting and state it explicitly in the project instructions. The role usually corresponds to the expert's professional responsibilities in that project. For example, P02 is constructed from actual project materials supplied by a financial expert. Its instructions assign the agent the role of the Project Mars financial due-diligence team, providing analysis for the transaction team, lenders, and subsequent valuation work.
All evaluated systems receive the same role and business requirements for a given project. Within this setting, the agent autonomously decides how to inspect materials, organize its analysis, and prepare deliverables.
Q2 — Agent and Judge Prompts
1. Agent prompts. Each system uses its harness's native system prompt and tool interface, together with the benchmark-supplied project brief and subtask instructions. These specify the professional role, objective, currently available materials, information cut-off, and required submissions. Later subtasks continue in the persistent workspace, using the agent's earlier artifacts and newly released materials.
2. Judge prompts. Each request combines the review instructions, project and subtask context, and the criteria to assess. The Judge receives a read-only evidence view containing the assessed submission, preserved earlier submissions, released source materials, and project instructions. It returns criterion-level judgments, reasons, and supporting evidence, with evidence located to specific files and relevant pages, sections, paragraphs, tables, or cells.
The instructions also specify how to inspect files, report insufficient evidence, and handle embedded attempts to alter grading rules. Such embedded instructions are treated as data to verify. The evaluation program validates the returned judgments and applies the predefined scoring rules.
Appendix D.2 now describes this prompt construction and evaluation interface. Figure 10 provides concrete P02 examples, including the project brief, the T09 instruction that uses the agent's T08 workbook, selected Judge instructions, a complete rubric criterion, and the evidence scope. Scoring and evaluator settings are detailed in Appendix C.3-C.4.
Follow-up on Our Responses and Revised Manuscript
Dear Reviewer JtMP,
As the discussion period draws to a close, we would like to ensure that we have addressed your concerns satisfactorily. We have carefully responded to your comments and updated the manuscript with additional experiments and clarifications. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
If our responses and the additional evidence have satisfactorily addressed your main concerns, we would be grateful if you could reconsider your assessment and rating in light of these updates.
Thank you again for your time and thoughtful review.
Thank you for your concern about project scale and the generalizability of the results. We understand that project count affects conclusions about cross-project generalization, and the paper discusses this limitation. During construction, we sought to balance domain coverage and project depth, preserving real-world task dependencies and repeated handoffs within the available expert time and budget. Twelve domain experts contributed real projects, and eight construction experts organized and reviewed the materials. Through expert interviews and cross-checks, we reconstructed professional work spanning weeks to months into executable, evaluable projects. Each project required checking earlier deliverables, subsequent tasks, and acceptance requirements individually, involving considerable professional input and review.
The current 11 projects span seven domains and contain 150 subtasks with concrete assignments, delivery requirements, and evaluation criteria. For comparison of task counts, TheAgentCompany contains 175 workplace tasks, and GDPval's public gold set contains 220 professional tasks. DeliverBench further preserves the business dependencies among these subtasks, allowing us to examine whether models can carry work across different stages through to complete delivery in multiple professional settings.
Project-level evaluations also use relatively small numbers of complete instances. For example, PaperBench, discussed in the paper, contains 20 paper-replication projects, each requiring the agent to understand a paper, implement code, and run experiments. DeliverBench's 11 projects similarly contain multiple interdependent work components. We hope that considering project count, internal task coverage, and complete-delivery requirements together gives a fuller account of the breadth and depth of work covered by the current evaluation.
Thank you for this suggestion. We agree that incorporating execution trajectories enables a fuller examination of agent behavior, particularly when verifying whether material access, actual operations, and submitted content correspond to one another. DeliverBench currently leverages fine-grained, expert-authored rubrics to assess each submission and the final deliverable package. This is a well-established and widely adopted strategy for evaluating complex professional work. For instance, APEX-Agents adopts expert-authored rubrics paired with an LLM judge to evaluate completed work criterion by criterion, and GDPval integrates detailed task rubrics into its evaluation pipeline. Following this paradigm, DeliverBench assesses all subtask submissions and the final deliverable against clear professional requirements.
To cover professional delivery requirements as comprehensively as possible, we constructed 5,322 unique criteria spanning five quality dimensions, correctness, completeness, consistency, professionalism, and usability. They separately assess subtask submissions, final artifacts, cross-file relationships, and package acceptance requirements. This supports concrete checks of factual and computational correctness, whether corrections are applied to the relevant files, and whether final delivery meets project requirements. The authenticity of an artifact's own claims about sources or actions still requires verification against source materials and execution records.
Following your suggestion, we inspected six runs from two systems across three projects involving audit, forensic investigation, and data delivery, examining material access, file writes, and the correspondence between submitted content and actual operations. In the inspected trajectories, we found no evidence of access to this evaluation's private grading materials or tampering with the grading environment. Trajectories also helped verify concrete repairs. For example, in P05, the agent omitted a transaction-reconciliation workbook in an earlier task, then reconstructed it from retained parsing results and used it for subsequent entity checks. The tool outputs, actual workbook, and downstream references corresponded, confirming that the repair occurred.
We believe your suggestion can further improve this type of evaluation. Building on the existing trajectory records and inspections, we plan to explore more systematic trajectory evaluation of material provenance, tool operations, and their correspondence with submitted content, complementing the current artifact-based evaluation.