acceptodds
Under review as a conference paper at ICLR 2027

DeliverBench: Evaluating Multimodal Agents on Program-Verifiable Deliverables

Abstract

Much of the work delegated to multimodal agents turns a batch of heterogeneous material into a structured artifact that downstream software consumes, where a single mishandled item can make the artifact unusable. We introduce DeliverBench, 129 tasks that each supply a batch of authentic material, including phone screenshots, photographed invoices and nameplates, spreadsheets, audio, and video, and require such a deliverable. Deliverables are verified item by item against 411 rubric criteria under a fixed set of 37 general tools, with every call logged. Across 27 frontier models, the strongest agent completes only 41.1% of tasks, while human annotators scored by the same judge reach 87.0%; 53 tasks are solved by no model. The main empirical reversal is a batch–horizon inversion. Batch-processing tasks succeed at 16.0%, while long-horizon interactive tasks, commonly treated as the frontier difficulty, succeed at 46.0%. We hypothesize that interaction helps because it exposes verifiable intermediate states and clear stopping conditions; batch work offers neither, so one inconsistent item can silently invalidate the artifact. Two further findings concern how agents are measured. Countable efficiency summaries, such as tool calls, tokens, steps, and error rate, show no significant model-level association with success and should not be read as proxies for capability; a rubric-based rating of how each task was attempted does track success (r = +0.677). Tool-use strategy varies 19.6-fold and clusters strongly by vendor, yet is not significantly associated with success. Six shortcut screens and a paired cross-harness control support the validity of these measurements. We release the task pack, the rubric criteria, and the evaluation logs needed to reproduce every reported number.

Primary area:
datasets and benchmarks
Keywords:
multimodal agents, agent benchmark, tool use, vision-language models, executable verification, long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A3fUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeliverBench, a benchmark comprising 129 tasks that require multimodal agents to turns a batch of heterogeneous material into a structured artifact that downstream software consumes. The authors claim that batch-processing tasks are more challenging than long-horizon interactive tasks, a phenomenon they term “batch-horizon inversion.” However, this conclusion may be an artifact of the benchmark's limited task coverage rather than a generalizable phenomenon.

Critical strengths
  1. The authors conduct extensive experiments across 26 models and provide human performance results as a reference.

  2. The proposed tasks align well with real-world needs for batch processing heterogeneous data.

Critical weaknesses
  1. The claimed batch-horizon inversion may result from limited and imbalanced task coverage, with 62 batch-processing tasks but only 16 long-horizon interactive tasks.

  2. Why is a threshold of 0.999 used for Pass Rate? Would relaxing it to 0.95 or 0.90 change the claimed batch-horizon inversion?

  3. Could using only one judge model introduce evaluation bias, such as preferences toward models from the same family?

  4. Relying solely on an LLM judge to evaluate deliverables requires human validation to demonstrate the reliability of LLM-as-a-judge in this setting.

Questions for authors

See Weaknesses.

Reviewer_EJHrOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeliverBench evaluates multimodal agents on 129 tasks requiring heterogeneous source materials to be transformed into structured deliverables. It uses a shared set of 37 tools, 411 outcome criteria, and a separate trajectory rubric. The paper evaluates 27 models and a human baseline, reporting lower perfect-score rates on batch-processing tasks than on interactive tasks. Validation includes shortcut screening, ranking comparisons on shared task subsets, and paired cross-harness experiments.

Critical strengths
  1. Practically valuable tasks. The benchmark includes useful workflows that turn heterogeneous materials, such as receipts and screenshots, into structured deliverables. These tasks provide concrete settings for studying whether multimodal agents can produce complete, accurate, and internally consistent outputs (§3.1).

  2. Relevant evaluation target. The focus on consistently processing multiple source items into structured deliverables captures practical failures that aggregate quality scores and trajectory inspection can obscure.

Critical weaknesses
  1. The validity of the primary evaluation is insufficiently established. Most criteria rely on a single model judge, but no independent expert–judge agreement or adjudication error analysis is reported; the human baseline uses the same evaluator. Concrete grading inconsistencies remain in 41 cells across two spreadsheet tasks. For example, GPT-6-Astra-Max on customer-chat-screenshot-quality-scorecard-hard-01 receives C-002 = 0 for allegedly missing formulas while C-003 = 1 confirms the formula requirements. These scores remain in the released grid despite the revised rubric excluding that penalty. Under the ≥0.999 passing threshold, such errors can affect task outcomes (§3.5; Appendix F–H; supplementary criterion cache).

  2. The analysis provides limited explanatory insight beyond descriptive performance differences. The central batch–horizon comparison contrasts 62 batch tasks with 16 interactive tasks without controlling for item count, perceptual difficulty, or grading requirements. The interactive tasks primarily involve visual puzzles and target-based artifact editing, which limits the interpretation of “long-horizon” difficulty. Although the feedback explanation is explicitly presented as a hypothesis, no failure attribution or controlled intervention tests it, and human performance by task mode remains unreported. Thus, the evidence does not establish cross-item consistency or missing intermediate feedback as the dominant bottleneck (§6.3; Appendix E).

  3. The benchmark’s representativeness and additional capability coverage are unclear. The paper provides limited justification for how the selected tasks and category balance represent the intended class of practical work. Its distinction from existing artifact-oriented benchmarks also needs stronger support: GDPval already evaluates heterogeneous reference files transformed into deliverables, while TOBench includes multimodal artifact creation and verification. GDPval is omitted from Table 1 despite this relevance. The paper does not sufficiently establish what additional capability coverage its batch-focused collection provides, limiting the broader significance of its findings (§2–§3).

  4. The experimental controls and released evidence do not support a consistently reproducible evaluation. The supplementary harness report states that only 24–29 baseline cells per model were regraded, while alternative harnesses used corrected evaluators, confounding harness effects with grading changes. Appendix B also labels 37 wins / 60 losses / 221 ties as perfect-score outcomes, whereas the released pairs yield 13 / 25 / 280 at the stated threshold; the former counts correspond to continuous-score differences exceeding ±0.05. Missing source assets, per-task human results, and report files required by the statistics script further prevent independent verification.

Questions for authors

See before.

Reviewer_hmo3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeliverBench evaluates whether multimodal agents can turn batches of raw materials into structured, verifiable deliverables. It includes 129 tasks, a fixed set of 37 general-purpose tools, and 411 weighted evaluation criteria. Final outputs and execution processes are scored separately. Across 27 evaluated models, the best model achieves a 41.1% pass rate, compared with 87.0% for humans under the same scoring pipeline. Within this benchmark, batch-processing tasks have both lower pass rates and lower mean scores than long-horizon interactive tasks. Simple measures such as tool-call counts show no significant association with success at the model level. The results highlight how difficult it remains for agents to process many inputs accurately and produce a fully correct deliverable.

Critical strengths
  1. The benchmark addresses a practical problem. Turning multiple source materials into a complete, usable file is a common real-world task. Checking the deliverable itself can catch errors that are easy to miss when looking only at an agent’s response or execution trace.
  2. The evaluation distinguishes partial progress, complete delivery, and process quality. Reporting both mean scores and full pass rates makes it clear when agents get most of the work done but still fail to meet all requirements. The separate process scores also help show that an orderly execution does not necessarily lead to a correct result.
  3. The paper includes useful validation and controls. The human baseline shows that most tasks can be completed under the same scoring pipeline. The six shortcut checks, paired cross-harness comparisons, and robustness checks on the common task subset also give me more confidence in the reported results.
Critical weaknesses
  1. It is still unclear what “consistency” diagnoses beyond item-level errors. The paper emphasizes cross-item consistency, but the examples of incorrect field extraction and receipt classification could also come from individual perception or reasoning errors. They do not necessarily show that an agent applied different rules across items. The current metrics and examples do not clearly separate these possibilities. They support the finding that agents struggle to complete batches reliably, but provide less support for identifying cross-item inconsistency as the main bottleneck.
  2. The performance gap between task modes is not yet well explained. Batch processing and interaction can coexist, and differences in batch size or task composition could affect the comparison. Full pass rates are particularly sensitive to the number of opportunities for error. Lower mean scores on batch tasks show that the gap is not just a consequence of requiring a perfect score, but its cause remains unclear. The authors appropriately label missing feedback as a hypothesis; with only 16 interactive tasks, stronger claims about the underlying capability gap would need controlled comparisons.
Questions for authors
  1. What observable errors count as “cross-item inconsistency”? How does the evaluation distinguish inconsistent rule application from individual perception or reasoning errors, omissions, and duplicate processing?
  2. How are tasks classified when they involve both batch processing and interaction? Does the gap persist after controlling for item count and basic perceptual difficulty? Could you report human performance by task mode, or compare the same tasks with and without feedback?
  3. Section 3.5 motivates the perfect-score threshold using ledgers, and Appendix H adjusts the visual scoring procedure. For reports and visual layouts, is there evidence from users or downstream use that a non-perfect output should be considered unusable? How stable are the main findings under different thresholds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.