- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
StartupBench contains professional workflow tasks reconstructed from interviews and the use cases of commercially adopted AI products. Agents must produce complete artifacts, which are assessed through separate sessions for individual rubric items. The paper evaluates several models and harnesses, compares general-purpose and specialized systems, and validates automatic judgments against expert annotations.
Critical strengths
-
Starting from workflows people already use gives task construction a sensible practical anchor. Requiring an actual spreadsheet, report, or other deliverable also exposes failures that a plausible written answer can hide. The expert construction and cross-review process is a meaningful investment in making these tasks useful.
-
The evaluator receives more validation than is common in agent benchmarks. Experts check outputs across the full task set for two models, and the paper directly compares separate rubric judgments with judging everything in one session. The additional harness comparison also helps establish that the observed difficulties are not peculiar to a single execution framework.
Critical weaknesses
-
[Major] The success criterion does not establish that a deliverable is professionally usable. A task passes when its weighted rubric score reaches 90%, even though the paper defines Core requirements as conditions whose violation can compromise correctness, safety, or usability. A missed Core requirement can still fit within the permitted score loss. The expert agreement study checks decisions under this same threshold, so it validates consistency of scoring without independently validating acceptance for real use. This matters because the paper's central conclusion is about completing work, not just accumulating rubric credit. Have users or experts accept or reject the artifacts independently of the numerical rule, and check whether critical failures should prevent a task from passing.
-
[Major] The benchmark itself is not available through the supplied release link. At the time of this review, the anonymous repository contains only a README with the project title. I could not inspect the tasks, workspaces, reference deliverables, rubric sets, or the promised task-to-product mapping. For a benchmark paper, these are the main contribution: without them, readers cannot assess how faithfully the interview scenarios became tasks or whether the rubrics admit reasonable alternative solutions. The construction narrative and selected examples are helpful, but cannot substitute for a reviewable release. Please provide the actual benchmark and evaluation configuration.
Questions for authors
See weaknesses
We thank reviewer Ez3a for the insightful feedback and insightful comments. Below we provide point-by-point responses to the concerns raised:
Weakness 1 snd Question 1
We sincerely thank reviewer for recognizing the real-world relevance of StartupBench and for raising this important concern regarding its novelty and scientific contributions. We would like to clarify our distinctive evaluation objectives and empirical findings in two aspects.
1.Complementary evaluation objectives beyond existing E2E benchmarks.
While StartupBench shares the focus on real-world professional workflows with GDPval[1] and Workspace-Bench[2], the three benchmarks are motivated by fundamentally different research questions.
GDPval is economic value-oriented: it identifies economically significant industries and occupations and recruits domain experts to construct representative professional tasks, aiming to assess AI's potential contribution to real-world economic productivity.
Workspace-Bench is workspace learning-oriented: it constructs realistic, large-scale digital workspaces and curates tasks emphasizing file dependencies, contextual reasoning, heterogeneous file understanding, and lineage tracing, aiming to evaluate agents' ability to operate within complex professional workspaces.
StartupBench is AI adoption-oriented: rather than starting from occupational importance or predefined target capabilities, we investigate commercially adopted AI products and interview their actual users to identify professional workflows that users already seek to delegate to AI.
As summarized in the table below, the three benchmarks differ in their research perspectives, task discovery methodologies, and primary evaluation objectives, offering complementary rather than competing insights into E2E agent capabilities. Importantly, economically valuable work, work involving complex workspace dependencies, and work that users actually seek to delegate to AI are related but not necessarily equivalent. StartupBench therefore adopts a bottom-up, demand-driven methodology, using real-world AI adoption as the starting point for task discovery and capability evaluation. This distinctive perspective enables us to systematically investigate the gap between demonstrated real-world demand for AI services and the actual capabilities of current general-purpose agents.
Our contribution is not to claim that individual task categories or capabilities are absent from existing benchmarks, but to provide a complementary evaluation perspective: measuring the gap between demonstrated real-world demand for AI services and the actual capabilities of current general-purpose agents. We believe this perspective is particularly valuable for understanding the practical readiness of general-purpose agents and identifying capability gaps that matter to real-world AI users. We will clarify the differences between StartupBench and existing Benchmarks more claerly in the revised version.
References
[1] Patwardhan T, Dias R, Proehl E, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks[C]//International Conference on Learning Representations. 2026, 2026: 24005-24040.
[2]Tang Z, Zhou X, Liu Y, et al. Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies[J]. arXiv preprint arXiv:2605.03596, 2026.