- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Argo-Bench, a benchmark for evaluating data agents on enterprise-scale analytical workflows.
Unlike existing text-to-SQL benchmarks, Argo-Bench constructs a simulated New York City food-delivery platform and projects it into an Oracle E-Business Suite-style warehouse containing 235 tables and 7.49 billion rows. The benchmark includes 210 tasks spanning trust and safety, finance planning, marketplace operations, accounting, and growth analytics.
The key design choice is that agents do not simply answer SQL questions. They must perform end-to-end workflows using SQL, Python, statistical analysis, machine learning, and finally submit decisions such as fraud bans, forecasts, budget allocations, or dashboard sources.
The benchmark evaluates agents against hidden simulator ground truth rather than manually constructed answers. The strongest evaluated model solves 32.9% of tasks completely and achieves an average score of 59.3.
Critical strengths
- Strong motivation for moving beyond text-to-SQL evaluation A major strength is identifying that enterprise data work is not merely query generation. Real workflows often require:
- discovering relevant tables,
- reconstructing business logic,
- fitting models,
- forecasting future outcomes,
- making decisions.
Argo-Bench captures this by requiring agents to execute complete workflows rather than returning SQL outputs.
This is a meaningful improvement over many existing benchmarks.
- Objective evaluation through hidden simulator state The benchmark avoids a common weakness in data benchmarks: unreliable human-written answer keys. Because the simulator maintains latent ground truth, tasks such as fraud detection and forecasting can be evaluated based on actual consequences rather than approximate answers. This is a strong methodological choice.
- Enterprise-scale environment The scale of the benchmark is impressive. The warehouse contains:
- 235 tables
- 7.49 billion rows
- ERP-style organization
and includes custom tables representing business-specific workflows.
This makes it significantly closer to enterprise settings than many academic datasets.
Critical weaknesses
- Limited evidence that benchmark difficulty reflects real enterprise complexity My main concern is whether the simulated environment sufficiently represents actual enterprise data systems. Although the benchmark is large, it is still a clean synthetic warehouse. The authors acknowledge that it differs from complex ERP deployments because it lacks:
- foreign currency complexity,
- multiple business lines,
- long historical evolution,
- multiple ERP formats.
Real enterprise warehouses often contain: - duplicated legacy tables,
- inconsistent definitions,
- undocumented transformations,
- organizational knowledge dependencies. Therefore, success on Argo-Bench may measure reasoning ability in a well-defined synthetic environment rather than real enterprise analytics capability.
- Domain limitation may restrict generalization The benchmark focuses on a single food-delivery marketplace. Although this domain contains interesting problems such as fraud, forecasting, and incentives, enterprise workflows differ substantially across industries. For example:
- healthcare requires privacy constraints and clinical reasoning,
- banking requires regulatory interpretation,
- manufacturing requires supply-chain dependencies. The paper should provide stronger arguments that the proposed task abstraction transfers beyond food delivery.
- Agent performance may depend heavily on hidden simulator assumptions
Because the benchmark uses a simulator, correctness depends on whether the simulator accurately models real-world causal relationships.
The authors acknowledge that simulation introduces assumptions and may make tasks easier than real environments.
A model could potentially optimize for benchmark-specific patterns rather than learning generally useful enterprise reasoning.
Questions for authors
Q1. How well does Argo-Bench correlate with real enterprise performance? The benchmark is carefully designed, but it remains unclear whether performance on Argo-Bench predicts success in actual enterprise environments. Have the authors conducted any validation with practitioners or real enterprise workflows?
Q2. How sensitive are results to simulator assumptions? Since task correctness depends on the simulator's hidden state:
- How much would model rankings change under a different simulator seed?
- Are the learned strategies robust to alternative business assumptions?
Q3. Can the benchmark separate different agent capabilities? Many tasks require multiple skills:
- SQL reasoning
- schema exploration
- statistical modeling
- optimization
- business understanding Can the authors provide a more fine-grained breakdown of which capabilities dominate performance?
6 comments
W3: Agent performance may depend heavily on hidden simulator assumptions. Because the benchmark uses a simulator, correctness depends on whether the simulator accurately models real-world causal relationships. The authors acknowledge that simulation introduces assumptions and may make tasks easier than real environments. A model could potentially optimize for benchmark-specific patterns rather than learning generally useful enterprise reasoning.
A3: We realize that a simulated world cannot perfectly match the real world. However, we note the extensive calibration that we have done to verify the simulator’s assumptions against real datasets (see our reply to reviewer 5ydS’s W1/Q3). We also, as previously noted, carefully document all of the assumptions made in the creation of the simulator. Here, we would like to specifically address concerns about optimizing for benchmark-specific patterns rather than general enterprise reasoning. There are two variants of this risk: memorizing instances and learning the simulator’s mechanisms.
To guard against the risk of memorization, we generate variations of the world based on different seeds. A model trained on the released public warehouse could learn and memorize its specific answers. Thus, the leaderboard results that we provide are based on data, tasks, and scores of a separate private world, generated with a different seed. Scores carry over from one world to the other, with a Spearman score of 0.972 as per the General Comment.
The second risk, learning the simulator’s mechanisms, is the more important one. Regardless of the seed used, each world uses the same rules, which have been grounded to approximate real-life relationships. This risk cannot be fully mitigated, but the design of the world and its grading limits it. First, recognizing a pattern the simulator generates does not pay on its own: the world plants benign look-alikes that share the surface pattern of the fraud they resemble, and the grader penalizes acting on innocent accounts. For example, on one refund-collusion task, GPT-6 Astra (high) held all 10 colluding storefronts and none of the innocent ones, scoring 90.6, while Claude Sonnet 5 (high) acted on the same pattern without separating the two groups, holding 9 of the 10 targets but also 77 innocent storefronts, and scored 5.2. Second, we default to industry standards whenever possible, such as by ensuring that 159 of the 235 warehouse tables are standard Oracle EBS tables. For example, quarter-to-date balances are stored as of the start of the period, in line with Oracle convention. Third, we consulted with ERP practitioners in the design of the warehouse schema. The practitioners rated the realism of the tasks (see A4), scoring the realism of task quirks (what makes a task difficult) 4 or 5 out of 5 in 14 of 16 finance tasks. Thus, the tasks test challenges seen in real systems, and solving them tests real, transferable capabilities. Finance is only one task area (others include forecasting, fraud detection, etc.), but model rankings largely agree across kinds of task (pairwise Spearman 0.76–0.80 between the bookkeeping, account-and-budget action, and forecasting task groups, over the 12 Table 2 models).
Every simulated benchmark carries the risk of models learning benchmark-specific patterns, but we audit run logs for shortcuts and design tasks to test realistic challenges to mitigate this effect.
Q1: How well does Argo-Bench correlate with real enterprise performance? The benchmark is carefully designed, but it remains unclear whether performance on Argo-Bench predicts success in actual enterprise environments. Have the authors conducted any validation with practitioners or real enterprise workflows?
A4: It is true that Argo-Bench’s correlation with real enterprise performance has not been tested. The paper claims only to compare models’ performance in a realistic simulated environment and not that this can predict success in a real enterprise environment. However, as stated in section 2.2, “we designed the schema with three ERP consultants with 14 to 31 years of experience in the domain.” A key part of our methodology involved partnering with real practitioners deeply familiar with enterprise ERPs to design a warehouse that closely resembles real systems, with the caveat that we are forced to make some assumptions on more granular behaviors (to our knowledge, there are no publicly available large-scale food delivery datasets with per-order granularity). We include real observed events, such as an April 2024 minimum pay increase, and real entities, via actual NYC restaurants and addresses. We have also calibrated the simulator to the city’s quarterly delivery-app reports.
We have also conducted validation with the practitioners. We conducted a more comprehensive survey of these consultants, asking them to audit each of our 16 finance tasks included in the benchmark, with model results omitted to avoid bias. They rated each of the finance tasks 1-5 on the following qualities (with provided descriptions):
- Realism: Does a finance or ERP team actually do this, and how often?
- Fidelity: Are the Oracle EBS tables, terms, and accounting treatments correct?
- Clarity: Would a competent analyst know what’s being asked (and, would they stop to ask a question, and if so, where)?
- Difficulty: Is this easier than, about the same as, or harder than doing it on a real deployment?
- Trap realism: Is the quirk that this task tests something they have seen in real systems?
- Time: How long would an experienced analyst with SQL access take?
For each quality, a score of 1 is worst (unrealistic, incorrect, unclear, significantly less difficult/time consuming than a real task) while a score of 5 is best (very realistic, completely correct, very clear, significantly more difficult/time-consuming than a real task).
Across our 16 questions, we had an average score of 4.31 for realism, 4.00 for fidelity, 3.68 for clarity, 2.88 for difficulty, 4.38 for trap realism, and 2.63 for time. Note that the scores for difficulty and time indicate that tasks are, on average, slightly less difficult and less time-consuming than real-world deployments. This is consistent with your W1 notes.
Notably, to address your concern for realism, all but three tasks were rated 4 or 5 for realism and all but two tasks were rated 4 or 5 for trap realism. Ten of the tasks were rated 4 or 5 for fidelity of EBS accounting terms. We are working with our consultants to conduct similar audits across more task domains and improve the clarity and correctness of the remaining questions.
Thank you for recognizing the strengths of our work and for your detailed comments and questions. We address the weaknesses and your questions below.
A1: We do acknowledge that the simulated environment differs from a real enterprise data system. You mention the following complexities of a real ERP system:
You also note that real warehouses often contain duplicated legacy tables, inconsistent definitions, undocumented transformations, and organizational knowledge dependencies. We address some of these areas; the rest are acknowledged gaps between the environment of Argo-Bench and a real system:
Notably, we leave out duplicate legacy tables on purpose. Our benchmark aims to test data science tasks that are hard because evidence is hard to find, or where putting together context clues and reasoning is tricky. We do not want difficulty to come from the answer being unclear, which would make a deterministic grading impossible or unfair. We ensure that all tasks have a single best answer and that answers do not depend on unwritten rules (the few deliberately underspecified tasks are paired with a hinted twin that states the rule. See our response to reviewer 5ydS’s W2).
A2: Argo-Bench is scoped to a single food delivery marketplace in New York. However, far from restricting generalization, there is a surprising amount of complexity and variability that remains true across industries (we do, however, acknowledge that differences remain, such as your point to healthcare’s privacy constraints and clinical reasoning, or banking’s requirements for regulatory interpretation). For generalization across domains, we note that 159 of our 235 tables are standard Oracle EBS tables. Additionally, the five action groups (report figures, publish a data source, act on accounts, allocate a budget, and forecast) are useful to any business. Finally, we note that even within our environment, we do test the complexity of reading and interpreting regulation. For instance, some tasks involve applying minimum-pay and back-pay regulations.