acceptodds
Under review as a conference paper at ICLR 2027

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

Abstract

Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce StartupBench, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks. Our data is available at https://anonymous.4open.science/r/Startup_bench-1CB6.

Primary area:
datasets and benchmarks
Keywords:
benchmark, real-world end-to-end task

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_17LfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StartupBench contains professional workflow tasks reconstructed from interviews and the use cases of commercially adopted AI products. Agents must produce complete artifacts, which are assessed through separate sessions for individual rubric items. The paper evaluates several models and harnesses, compares general-purpose and specialized systems, and validates automatic judgments against expert annotations.

Critical strengths
  1. Starting from workflows people already use gives task construction a sensible practical anchor. Requiring an actual spreadsheet, report, or other deliverable also exposes failures that a plausible written answer can hide. The expert construction and cross-review process is a meaningful investment in making these tasks useful.

  2. The evaluator receives more validation than is common in agent benchmarks. Experts check outputs across the full task set for two models, and the paper directly compares separate rubric judgments with judging everything in one session. The additional harness comparison also helps establish that the observed difficulties are not peculiar to a single execution framework.

Critical weaknesses
  1. [Major] The success criterion does not establish that a deliverable is professionally usable. A task passes when its weighted rubric score reaches 90%, even though the paper defines Core requirements as conditions whose violation can compromise correctness, safety, or usability. A missed Core requirement can still fit within the permitted score loss. The expert agreement study checks decisions under this same threshold, so it validates consistency of scoring without independently validating acceptance for real use. This matters because the paper's central conclusion is about completing work, not just accumulating rubric credit. Have users or experts accept or reject the artifacts independently of the numerical rule, and check whether critical failures should prevent a task from passing.

  2. [Major] The benchmark itself is not available through the supplied release link. At the time of this review, the anonymous repository contains only a README with the project title. I could not inspect the tasks, workspaces, reference deliverables, rubric sets, or the promised task-to-product mapping. For a benchmark paper, these are the main contribution: without them, readers cannot assess how faithfully the interview scenarios became tasks or whether the rubrics admit reasonable alternative solutions. The construction narrative and selected examples are helpful, but cannot substitute for a reviewable release. Please provide the actual benchmark and evaluation configuration.

Questions for authors

See weaknesses

Reviewer_uFbDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors want to create a dataset that better represents real professional workflows. To this end, they sourced tasks from commercial AI adoption users and surveyed them. These were turned into a dataset of reproducible professional workflows.

The dataset has n=97 workflows and ~2.5k associated rubric items in total. A tool-enabled llm as a judge judges each item and assigns pass/fail to each. Each item has a weight which leads to a total percentage score. A run is seen as successful if the score is greater than 90.

The authors find that agents can often satisfy many individual requirements, but less often reliably reach the completion threshold.

Critical strengths

The dataset methodology (based on user interviews of commercial adoption of AI to mine tasks) is novel and quite possibly valuable. This data also is currently available anonymously. Rubric-level and task-success-level agreements are at 90+% against expert annotations, which supports evaluator reliability.

Critical weaknesses

Authors show limited evidence of this dataset specifically filling a gap in comparison to other existing datasets, such as GDPval. Cursory investigation of the dataset available shows substantial overlap in topics, such as multi-file financial anslysi, payroll models, and constructing legal memos. GDPval also has detailed, weighted rubrics. GDPval is also a dataset constructed from experts, though building a dataset from the learnings from a survey is different.

Questions for authors

n/a

Reviewer_Ez3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StartupBench, a benchmark evaluating general-purpose agents on end-to-end professional workflows derived from commercially adopted AI products. It contains 97 tasks across six domains, constructed through startup surveys, user interviews, and expert annotations. The authors evaluate nine frontier models using fine-grained rubrics and find that even the strongest agents achieve only around 30% task success.

Critical strengths
  1. Sourcing tasks from commercially adopted AI products is a meaningful attempt to improve the real-world relevance of agent benchmarks.

  2. The benchmark covers diverse professional workflows and evaluates complete deliverables using detailed, expert-designed rubrics.

  3. The paper includes human–judge agreement analysis, comparisons across agent harnesses, and informative failure case studies.

Critical weaknesses
  1. Limited novelty and scientific insights. Beyond providing another model leaderboard, the paper does not clearly demonstrate what new capabilities StartupBench measures compared with existing E2E benchmarks such as GDPval and Workspace-Bench. Its main findings, including difficulties with complex instruction following, domain expertise, and artifact verification, are largely consistent with existing observations.

  2. Although the benchmark provides fine-grained evaluation, the failure analysis remains mostly descriptive. It does not sufficiently identify underlying capability bottlenecks or demonstrate how the benchmark can guide improvements in agent performance.

Questions for authors
  1. What unique insights does StartupBench provide beyond existing E2E agent benchmarks? Can the authors demonstrate capabilities or failure modes that existing benchmarks fail to capture?

  2. Can the fine-grained evaluation be used to identify actionable bottlenecks or guide concrete improvements in agent performance?

2 comments

We thank reviewer Ez3a for the insightful feedback and insightful comments. Below we provide point-by-point responses to the concerns raised:

Weakness 1 snd Question 1

We sincerely thank reviewer for recognizing the real-world relevance of StartupBench and for raising this important concern regarding its novelty and scientific contributions. We would like to clarify our distinctive evaluation objectives and empirical findings in two aspects.

1.Complementary evaluation objectives beyond existing E2E benchmarks.

While StartupBench shares the focus on real-world professional workflows with GDPval[1] and Workspace-Bench[2], the three benchmarks are motivated by fundamentally different research questions.

GDPval is economic value-oriented: it identifies economically significant industries and occupations and recruits domain experts to construct representative professional tasks, aiming to assess AI's potential contribution to real-world economic productivity.

Workspace-Bench is workspace learning-oriented: it constructs realistic, large-scale digital workspaces and curates tasks emphasizing file dependencies, contextual reasoning, heterogeneous file understanding, and lineage tracing, aiming to evaluate agents' ability to operate within complex professional workspaces.

StartupBench is AI adoption-oriented: rather than starting from occupational importance or predefined target capabilities, we investigate commercially adopted AI products and interview their actual users to identify professional workflows that users already seek to delegate to AI.

As summarized in the table below, the three benchmarks differ in their research perspectives, task discovery methodologies, and primary evaluation objectives, offering complementary rather than competing insights into E2E agent capabilities. Importantly, economically valuable work, work involving complex workspace dependencies, and work that users actually seek to delegate to AI are related but not necessarily equivalent. StartupBench therefore adopts a bottom-up, demand-driven methodology, using real-world AI adoption as the starting point for task discovery and capability evaluation. This distinctive perspective enables us to systematically investigate the gap between demonstrated real-world demand for AI services and the actual capabilities of current general-purpose agents.

DimensionGDPvalWorkspace-BenchStartupBench (Ours)
Research perspectiveEconomic value-orientedWorkspace learning-orientedAI adoption-oriented
Starting pointEconomically significant industries and occupationsReal-world office scenarios and complex digital workspacesCommercially adopted AI products
Task discoveryOccupational analysis and expert experienceUser surveys and workspace dependency analysisMarket adoption evidence and in-depth user interviews
Task constructionExpert-authored representative occupational tasksReconstructed workspace tasks with explicit file dependenciesExpert-reconstructed workflows grounded in actual AI usage
Evaluation focusEconomically valuable professional workWorkspace understanding and cross-file reasoningE2E completion of user-delegated professional workflows
Primary research valueAssessing AI's potential economic impactIdentifying bottlenecks in workspace learningIdentifying gaps between demonstrated AI demand and general-purpose agent capabilities

Our contribution is not to claim that individual task categories or capabilities are absent from existing benchmarks, but to provide a complementary evaluation perspective: measuring the gap between demonstrated real-world demand for AI services and the actual capabilities of current general-purpose agents. We believe this perspective is particularly valuable for understanding the practical readiness of general-purpose agents and identifying capability gaps that matter to real-world AI users. We will clarify the differences between StartupBench and existing Benchmarks more claerly in the revised version.

References

[1] Patwardhan T, Dias R, Proehl E, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks[C]//International Conference on Learning Representations. 2026, 2026: 24005-24040.

[2]Tang Z, Zhou X, Liu Y, et al. Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies[J]. arXiv preprint arXiv:2605.03596, 2026.

2. Distinctive empirical insights beyond existing observations.

While we acknowledge that some of our observations are consistent with prior studies, StartupBench also provides several distinctive empirical findings that have received relatively limited attention in existing E2E agent benchmarks:

  • The gap between partial progress and successful E2E delivery (Section 4.2). Although leading models achieve average scores above 73%, none achieves a task success rate (score ≥ 90%) exceeding one-third. This reveals a substantial gap between satisfying individual requirements and reliably delivering work products that meet practical acceptance criteria, highlighting an important remaining challenge for general-purpose agents.

  • Disproportionate failures on core user requirements (Section 4.4.1). Our importance-aware rubric analysis reveals that models exhibit higher satisfaction rates for Auxiliary requirements (68.67%) than Important (65.89%) and Core requirements (63.45%), with 8 out of 9 models exhibiting this trend. This identifies a more specific and underexplored bottleneck: agents often satisfy peripheral requirements while failing to fulfill the core requirements that matter most to users, helping explain why substantial partial progress does not necessarily translate into successful E2E delivery.

  • The persistent gap between general-purpose and commercial specialized agents (Section 4.5.2). Unlike most existing E2E benchmarks, StartupBench directly compares general-purpose agents with the commercially deployed specialized agents from which the benchmark tasks are derived, using identical tasks and evaluation criteria. Specialized agents substantially outperform general-purpose agents (83.50 vs. 64.31 in average score; 39.18% vs. 19.74% in success rate). This advantage persists even under an oracle best-of-three setting for general-purpose agents, highlighting the remaining gap between general-purpose capabilities and specialized production systems.

Finally, our objective is to faithfully characterize the capabilities and limitations of current agents on market-validated workflows. While some findings corroborate prior observations, independently validating their generalizability in a distinct, adoption-grounded evaluation setting also provides scientific value. Taken together, these findings demonstrate that StartupBench offers empirical insights beyond simply establishing another model leaderboard.

Comments on the paper

1 comment

Summary of our Work and Responses to Reviewers' Main Concerns

We sincerely thank all reviewers for their constructive feedback and for recognizing the practical relevance, expert-driven construction, and evaluation rigor of StartupBench. To facilitate further discussion, we summarize our main contributions, the reviewers' primary concerns, and our corresponding clarifications below.

1. Our Contributions

StartupBench introduces an AI-adoption-driven benchmark that evaluates whether general-purpose agents can complete professional workflows that real-world users already seek to delegate to AI. We construct 97 E2E tasks across 6 professional domains, grounded in commercially adopted AI products and user interviews, with expert-designed fine-grained rubrics. Our experiments reveal substantial gaps between partial requirement satisfaction and successful end-to-end delivery, as well as between general-purpose agents and commercially deployed specialized systems.

2. Reviewers' Primary Concerns

  1. Novelty and differentiation: The distinctive contributions of StartupBench relative to existing professional-work benchmarks, particularly GDPval and Workspace-Bench, needs to be sufficiently established.

  2. Scientific insights and actionable findings: Some findings appear consistent with existing observations, and the failure analysis does not sufficiently demonstrate how the benchmark can guide concrete improvements in agent capabilities.

  3. Evaluation validity and reproducibility: The weighted 90% success threshold may not guarantee satisfaction of all Core requirements or professional usability. Additionally, concerns were raised regarding the accessibility of the released benchmark data.

3. Our Responses to the Concerns

  1. Distinctive research motivation and evaluation perspective.

Unlike GDPval, which starts from economically significant occupations, or Workspace-Bench, which emphasizes complex workspace dependencies, StartupBench starts from demonstrated real-world AI adoption to identify tasks that users actually seek to delegate to AI. Our task discovery process involves investigating over 20 commercially validated AI products, interviewing over 30 actual users, and reconstructing workflows with over 50 domain experts. We do not claim exclusive coverage of particular task categories; instead, StartupBench provides a complementary, demand-driven perspective that measures the gap between real-world AI demand and general-purpose agent capabilities.

  1. Distinctive empirical findings and actionable implications.

Beyond aggregate model comparisons, StartupBench provides several important findings:

  • Partial completion versus E2E success: Leading models achieve average scores above 73%, yet none exceeds a one-third task success rate, exposing the gap between satisfying individual requirements and producing acceptable deliverables.
  • Requirement prioritization: Agents satisfy Auxiliary requirements more frequently than Core requirements (68.67% vs. 63.45%), revealing a bottleneck in fulfilling users' most critical needs.
  • General-purpose versus specialized agents: Commercial specialized agents substantially outperform general-purpose agents on identical tasks (83.50 vs. 64.31 in average score), suggesting opportunities to internalize domain-specific workflows, reusable skills, and tool orchestration strategies into foundation models.

These findings provide actionable directions for targeted post-training, reward design, and agent development, although the effectiveness of the proposed training interventions remains to be experimentally validated.

  1. Evaluation criteria and benchmark accessibility.

Our success criterion is based on expert-designed, importance-weighted rubrics, where a score of at least 90% is intended to represent a professionally usable deliverable while tolerating minor imperfections. We acknowledge that weighted aggregation may still permit individual Core requirements to remain unsatisfied. To address this concern, we will additionally report the proportion of tasks satisfying all Core requirements and compare it with the original success rates in the revised version.

Regarding reproducibility, we have rechecked and confirmed that the benchmark data are accessible through our anonymous repository provided in the Abstract. We will also provide a non-anonymous public repository containing the complete benchmark data and evaluation code in the revised version.

We appreciate the reviewers' constructive suggestions again and will incorporate these clarifications and additional analyses into the revised version. We hope this consolidated response helps clarify the distinctive contributions, empirical value, and evaluation reliability of StartupBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.