acceptodds
Under review as a conference paper at ICLR 2027

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Abstract

Real-world enterprise data science and analytics workflows typically require reasoning across dozens of tables, performing statistical analysis and intermediate computations, and taking actions based on findings. Established benchmarks do not represent realistic, full-scale enterprise data warehouses due to the highly sensitive and confidential nature of such data. We introduce Argo-Bench, an evaluation framework comprising 210 complex data science and analytics tasks in a simulated environment. We simulate a terabyte-scale consumer food delivery platform in New York City with grounded economics and well-known patterns ranging from fraud scenarios to marketplace financial incentives based on public data sources, peer-reviewed industry literature, and regulatory data disclosures and filings. We then export this world to an ERP warehouse modeled on the Oracle E-Business Suite schema with 235 tables and 7.49 billion rows. By withholding the simulator's ground-truth state from the warehouse projection exposed to the evaluated agent, we can design realistic tasks that require reconstructing facts through the warehouse's idiosyncrasies before acting on them. Every task has an executable reference solution that demonstrates solvability using only the warehouse projection. Argo-Bench goes beyond text-to-SQL, requiring the agent to file actions such as banning users or discontinuing promotions. The strongest of 12 frontier and open-weight models reaches a score of at least 95 out of 100 on only 32.9% of tasks and averages 59.3 points. We propose that progress on Argo-Bench demonstrates meaningful progress towards understanding, navigating, and taking action within real-world data environments.

Primary area:
datasets and benchmarks
Keywords:
LLM Benchmark, Data Science, Data Analytics, Code Generation, Text-to-SQL, ERP, Simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YCA3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Argo-Bench, a benchmark for evaluating data agents on enterprise-scale analytical workflows. Unlike existing text-to-SQL benchmarks, Argo-Bench constructs a simulated New York City food-delivery platform and projects it into an Oracle E-Business Suite-style warehouse containing 235 tables and 7.49 billion rows. The benchmark includes 210 tasks spanning trust and safety, finance planning, marketplace operations, accounting, and growth analytics.
The key design choice is that agents do not simply answer SQL questions. They must perform end-to-end workflows using SQL, Python, statistical analysis, machine learning, and finally submit decisions such as fraud bans, forecasts, budget allocations, or dashboard sources.
The benchmark evaluates agents against hidden simulator ground truth rather than manually constructed answers. The strongest evaluated model solves 32.9% of tasks completely and achieves an average score of 59.3.

Critical strengths
  1. Strong motivation for moving beyond text-to-SQL evaluation A major strength is identifying that enterprise data work is not merely query generation. Real workflows often require:
  • discovering relevant tables,
  • reconstructing business logic,
  • fitting models,
  • forecasting future outcomes,
  • making decisions. Argo-Bench captures this by requiring agents to execute complete workflows rather than returning SQL outputs.
    This is a meaningful improvement over many existing benchmarks.
  1. Objective evaluation through hidden simulator state The benchmark avoids a common weakness in data benchmarks: unreliable human-written answer keys. Because the simulator maintains latent ground truth, tasks such as fraud detection and forecasting can be evaluated based on actual consequences rather than approximate answers. This is a strong methodological choice.
  2. Enterprise-scale environment The scale of the benchmark is impressive. The warehouse contains:
  • 235 tables
  • 7.49 billion rows
  • ERP-style organization and includes custom tables representing business-specific workflows.
    This makes it significantly closer to enterprise settings than many academic datasets.
Critical weaknesses
  1. Limited evidence that benchmark difficulty reflects real enterprise complexity My main concern is whether the simulated environment sufficiently represents actual enterprise data systems. Although the benchmark is large, it is still a clean synthetic warehouse. The authors acknowledge that it differs from complex ERP deployments because it lacks:
  • foreign currency complexity,
  • multiple business lines,
  • long historical evolution,
  • multiple ERP formats.
    Real enterprise warehouses often contain:
  • duplicated legacy tables,
  • inconsistent definitions,
  • undocumented transformations,
  • organizational knowledge dependencies. Therefore, success on Argo-Bench may measure reasoning ability in a well-defined synthetic environment rather than real enterprise analytics capability.
  1. Domain limitation may restrict generalization The benchmark focuses on a single food-delivery marketplace. Although this domain contains interesting problems such as fraud, forecasting, and incentives, enterprise workflows differ substantially across industries. For example:
  • healthcare requires privacy constraints and clinical reasoning,
  • banking requires regulatory interpretation,
  • manufacturing requires supply-chain dependencies. The paper should provide stronger arguments that the proposed task abstraction transfers beyond food delivery.
  1. Agent performance may depend heavily on hidden simulator assumptions Because the benchmark uses a simulator, correctness depends on whether the simulator accurately models real-world causal relationships. The authors acknowledge that simulation introduces assumptions and may make tasks easier than real environments.
    A model could potentially optimize for benchmark-specific patterns rather than learning generally useful enterprise reasoning.
Questions for authors

Q1. How well does Argo-Bench correlate with real enterprise performance? The benchmark is carefully designed, but it remains unclear whether performance on Argo-Bench predicts success in actual enterprise environments. Have the authors conducted any validation with practitioners or real enterprise workflows?

Q2. How sensitive are results to simulator assumptions? Since task correctness depends on the simulator's hidden state:

  • How much would model rankings change under a different simulator seed?
  • Are the learned strategies robust to alternative business assumptions?

Q3. Can the benchmark separate different agent capabilities? Many tasks require multiple skills:

  • SQL reasoning
  • schema exploration
  • statistical modeling
  • optimization
  • business understanding Can the authors provide a more fine-grained breakdown of which capabilities dominate performance?
6 comments

Thank you for recognizing the strengths of our work and for your detailed comments and questions. We address the weaknesses and your questions below.

W1: Limited evidence that benchmark difficulty reflects real enterprise complexity. My main concern is whether the simulated environment sufficiently represents actual enterprise data systems. Although the benchmark is large, it is still a clean synthetic warehouse. The authors acknowledge that it differs from complex ERP deployments because it lacks:

  • foreign currency complexity,
  • multiple business lines,
  • long historical evolution,
  • multiple ERP formats.

Real enterprise warehouses often contain:

  • duplicated legacy tables,
  • inconsistent definitions,
  • undocumented transformations,
  • organizational knowledge dependencies.

Therefore, success on Argo-Bench may measure reasoning ability in a well-defined synthetic environment rather than real enterprise analytics capability.

A1: We do acknowledge that the simulated environment differs from a real enterprise data system. You mention the following complexities of a real ERP system:

  • Foreign currency complexity
  • Multiple business lines
  • Long historical evolution
  • Multiple ERP formats

You also note that real warehouses often contain duplicated legacy tables, inconsistent definitions, undocumented transformations, and organizational knowledge dependencies. We address some of these areas; the rest are acknowledged gaps between the environment of Argo-Bench and a real system:

  • Our environment and tasks do reflect conflicting definitions: for instance, tasks that concern ‘membership’ can use a definition backed by contract dates or by billing history. The model must choose the appropriate definition based on the task context and data.
  • We also address hidden transformations: the data warehouse stores hourly check-ins, while the app sends updates every half hour, so some tasks require that the model notices the discrepancy between the two rates of update.
  • Company knowledge is also tested in some tasks: explaining what quests are for raises the mean from 16.7 to 59.9.

Notably, we leave out duplicate legacy tables on purpose. Our benchmark aims to test data science tasks that are hard because evidence is hard to find, or where putting together context clues and reasoning is tricky. We do not want difficulty to come from the answer being unclear, which would make a deterministic grading impossible or unfair. We ensure that all tasks have a single best answer and that answers do not depend on unwritten rules (the few deliberately underspecified tasks are paired with a hinted twin that states the rule. See our response to reviewer 5ydS’s W2).

W2: Domain limitation may restrict generalization. The benchmark focuses on a single food-delivery marketplace. Although this domain contains interesting problems such as fraud, forecasting, and incentives, enterprise workflows differ substantially across industries. For example:

  • healthcare requires privacy constraints and clinical reasoning,
  • banking requires regulatory interpretation,
  • manufacturing requires supply-chain dependencies.

The paper should provide stronger arguments that the proposed task abstraction transfers beyond food delivery.

A2: Argo-Bench is scoped to a single food delivery marketplace in New York. However, far from restricting generalization, there is a surprising amount of complexity and variability that remains true across industries (we do, however, acknowledge that differences remain, such as your point to healthcare’s privacy constraints and clinical reasoning, or banking’s requirements for regulatory interpretation). For generalization across domains, we note that 159 of our 235 tables are standard Oracle EBS tables. Additionally, the five action groups (report figures, publish a data source, act on accounts, allocate a budget, and forecast) are useful to any business. Finally, we note that even within our environment, we do test the complexity of reading and interpreting regulation. For instance, some tasks involve applying minimum-pay and back-pay regulations.

W3: Agent performance may depend heavily on hidden simulator assumptions. Because the benchmark uses a simulator, correctness depends on whether the simulator accurately models real-world causal relationships. The authors acknowledge that simulation introduces assumptions and may make tasks easier than real environments. A model could potentially optimize for benchmark-specific patterns rather than learning generally useful enterprise reasoning.

A3: We realize that a simulated world cannot perfectly match the real world. However, we note the extensive calibration that we have done to verify the simulator’s assumptions against real datasets (see our reply to reviewer 5ydS’s W1/Q3). We also, as previously noted, carefully document all of the assumptions made in the creation of the simulator. Here, we would like to specifically address concerns about optimizing for benchmark-specific patterns rather than general enterprise reasoning. There are two variants of this risk: memorizing instances and learning the simulator’s mechanisms.

To guard against the risk of memorization, we generate variations of the world based on different seeds. A model trained on the released public warehouse could learn and memorize its specific answers. Thus, the leaderboard results that we provide are based on data, tasks, and scores of a separate private world, generated with a different seed. Scores carry over from one world to the other, with a Spearman score of 0.972 as per the General Comment.

The second risk, learning the simulator’s mechanisms, is the more important one. Regardless of the seed used, each world uses the same rules, which have been grounded to approximate real-life relationships. This risk cannot be fully mitigated, but the design of the world and its grading limits it. First, recognizing a pattern the simulator generates does not pay on its own: the world plants benign look-alikes that share the surface pattern of the fraud they resemble, and the grader penalizes acting on innocent accounts. For example, on one refund-collusion task, GPT-6 Astra (high) held all 10 colluding storefronts and none of the innocent ones, scoring 90.6, while Claude Sonnet 5 (high) acted on the same pattern without separating the two groups, holding 9 of the 10 targets but also 77 innocent storefronts, and scored 5.2. Second, we default to industry standards whenever possible, such as by ensuring that 159 of the 235 warehouse tables are standard Oracle EBS tables. For example, quarter-to-date balances are stored as of the start of the period, in line with Oracle convention. Third, we consulted with ERP practitioners in the design of the warehouse schema. The practitioners rated the realism of the tasks (see A4), scoring the realism of task quirks (what makes a task difficult) 4 or 5 out of 5 in 14 of 16 finance tasks. Thus, the tasks test challenges seen in real systems, and solving them tests real, transferable capabilities. Finance is only one task area (others include forecasting, fraud detection, etc.), but model rankings largely agree across kinds of task (pairwise Spearman 0.76–0.80 between the bookkeeping, account-and-budget action, and forecasting task groups, over the 12 Table 2 models).

Every simulated benchmark carries the risk of models learning benchmark-specific patterns, but we audit run logs for shortcuts and design tasks to test realistic challenges to mitigate this effect.

Q1: How well does Argo-Bench correlate with real enterprise performance? The benchmark is carefully designed, but it remains unclear whether performance on Argo-Bench predicts success in actual enterprise environments. Have the authors conducted any validation with practitioners or real enterprise workflows?

A4: It is true that Argo-Bench’s correlation with real enterprise performance has not been tested. The paper claims only to compare models’ performance in a realistic simulated environment and not that this can predict success in a real enterprise environment. However, as stated in section 2.2, “we designed the schema with three ERP consultants with 14 to 31 years of experience in the domain.” A key part of our methodology involved partnering with real practitioners deeply familiar with enterprise ERPs to design a warehouse that closely resembles real systems, with the caveat that we are forced to make some assumptions on more granular behaviors (to our knowledge, there are no publicly available large-scale food delivery datasets with per-order granularity). We include real observed events, such as an April 2024 minimum pay increase, and real entities, via actual NYC restaurants and addresses. We have also calibrated the simulator to the city’s quarterly delivery-app reports.

We have also conducted validation with the practitioners. We conducted a more comprehensive survey of these consultants, asking them to audit each of our 16 finance tasks included in the benchmark, with model results omitted to avoid bias. They rated each of the finance tasks 1-5 on the following qualities (with provided descriptions):

  • Realism: Does a finance or ERP team actually do this, and how often?
  • Fidelity: Are the Oracle EBS tables, terms, and accounting treatments correct?
  • Clarity: Would a competent analyst know what’s being asked (and, would they stop to ask a question, and if so, where)?
  • Difficulty: Is this easier than, about the same as, or harder than doing it on a real deployment?
  • Trap realism: Is the quirk that this task tests something they have seen in real systems?
  • Time: How long would an experienced analyst with SQL access take?

For each quality, a score of 1 is worst (unrealistic, incorrect, unclear, significantly less difficult/time consuming than a real task) while a score of 5 is best (very realistic, completely correct, very clear, significantly more difficult/time-consuming than a real task).

Across our 16 questions, we had an average score of 4.31 for realism, 4.00 for fidelity, 3.68 for clarity, 2.88 for difficulty, 4.38 for trap realism, and 2.63 for time. Note that the scores for difficulty and time indicate that tasks are, on average, slightly less difficult and less time-consuming than real-world deployments. This is consistent with your W1 notes.

Notably, to address your concern for realism, all but three tasks were rated 4 or 5 for realism and all but two tasks were rated 4 or 5 for trap realism. Ten of the tasks were rated 4 or 5 for fidelity of EBS accounting terms. We are working with our consultants to conduct similar audits across more task domains and improve the clarity and correctness of the remaining questions.

Reviewer_5ydSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Argo-Bench, a benchmark for evaluating data agents on enterprise-scale data analysis and business decision-making tasks. It simulates a New York City food delivery platform with 235 tables, 7.49 billion rows, and 210 tasks covering fraud detection, forecasting, financial analysis, and dashboard generation. Unlike traditional benchmarks that primarily evaluate answer correctness, Argo-Bench evaluates agents' ability to reason over complex enterprise data and make business decisions based on their consequences. Experiments across 12 models show that the best-performing model, Claude Opus 5.5, achieves an average score of 59.3/100 and fully solves only 32.9% of tasks, highlighting substantial limitations in current data agents.

Critical strengths

1.The paper targets a real-world problem and constructs a large-scale simulated food delivery platform for evaluation.

2.The benchmark evaluates the economic impact of agents' decisions instead of relying solely on answer correctness.

3.The paper evaluates a diverse set of models and provides informative failure analyses.

Critical weaknesses

1.The simulator is calibrated to aggregate real-world statistics, yet the validity of its simulated behaviors and resulting business outcomes is not well justified. This is important because the benchmark scores depend on these assumptions.

2.Some tasks have ambiguous descriptions or inconsistencies between prompts and graders. For example, Appendix I.4 acknowledges ambiguous task descriptions in the Kitchen Capacity task, while Appendix J reports a mismatch between the prompt and grading criteria in the refund-collusion tasks. While I appreciate the transparency, I hope these problems can be addressed in the official benchmark.

3.It is unclear which capabilities primarily drive the benchmark's difficulty. Although the paper provides several insightful failure case studies, it lacks a systematic analysis of whether agent failures stem from schema discovery, data analysis, business reasoning, or long-horizon planning. A more fine-grained breakdown would help clarify what capabilities Argo-Bench actually measures.

Questions for authors

1.How long does it take to run the full benchmark, and how many tasks are executed in parallel?

2.Why are recent, closely related benchmarks such as AvalancheBench, The Era by Eon, and ERPBench discussed in the Related Work section but omitted from Table 1? Including them would help clarify Argo-Bench's contributions relative to recent work.

3.How exactly was the simulator calibrated using real-world data? The paper mentions iterative calibration but provides no details regarding the calibration process. Can you provide further details?

4.How were the reference solutions constructed? Were they developed independently of the simulator's hidden ground truth, or were the ground-truth answers used to guide their development?

5 comments

We appreciate your thoughtful comments on the strengths and weaknesses of our work! We address the stated weaknesses and questions:

W1: The simulator is calibrated to aggregate real-world statistics, yet the validity of its simulated behaviors and resulting business outcomes is not well justified. This is important because the benchmark scores depend on these assumptions.

A1: We agree that matching aggregate statistics does not by itself validate the simulated behaviors. We are careful not to conflate outcome with mechanism. We check that the values produced by our simulated world match our datasets, but that does not imply that the mechanism that drives those values is correct. Thus, we record all assumptions made alongside our anchors. We also check behavior at various levels: at the level of individual mechanisms, comparing each component with its data, as well as at the level of emergent phenomena. For example, when calculating restaurant wait time, we take as input the individual component of time per delivery. We then verify the derived wait time to ensure that it fits plausibly with restaurant preparation times and deliveries per hour from independent data sources. While there are assumptions in this system, they are closely calibrated to reflect observed data, and we compare models within our approximation of the world rather than claiming to reproduce every mechanism of a real platform.

W2: Some tasks have ambiguous descriptions or inconsistencies between prompts and graders. For example, Appendix I.4 acknowledges ambiguous task descriptions in the Kitchen Capacity task, while Appendix J reports a mismatch between the prompt and grading criteria in the refund-collusion tasks. While I appreciate the transparency, I hope these problems can be addressed in the official benchmark.

A2: In the revised version of the benchmark, the refund-collusion tasks are re-run with corrected prompts, with a maximum overall score change of 0.67 and no changes to the model ranking (see the General Comment and response to reviewer 7CP5’s W2). We’d like to address ambiguous task descriptions separately. A very small set of tasks are intentionally underspecified, so as to test LLM abilities to infer the correct definition even when the prompt does not specify the grading criteria, as typical enterprise users do not always use the most precise language. These tasks have a “hinted twin”, which does not underspecify the grading criteria and allows us to judge model performance side-by-side. Besides the Kitchen Capacity task (sig-01-kitchen-full-turnaways-hard), this applies to four Operations Research tasks (or-01b-coverage-cut-january-bare, or-02-quest-cut, or-03-quest-cut-weeks, or-04b-coverage-add-january-bare). Over all 30 settings, the mean score on these tasks rises from 6.1 to 29.2 between the unhinted and hinted versions. In our error analysis (see General Comment), 1.3% of errors are attributable to disputable answer keys.

W3: It is unclear which capabilities primarily drive the benchmark’s difficulty. Although the paper provides several insightful failure case studies, it lacks a systematic analysis of whether agent failures stem from schema discovery, data analysis, business reasoning, or long-horizon planning. A more fine-grained breakdown would help clarify what capabilities Argo-Bench actually measures.

A3: We thank the reviewer for this suggestion. Based on it, we present a new quantitative error analysis in the General Comment, which classifies the first decisive error of every failed run by capability (schema exploration, business understanding, statistical modeling, SQL logic, optimization) and will be added to the revised manuscript. Here we offer a complementary breakdown of the benchmark’s difficulty. We can group tasks by what we ask the agent to file:

FilingNumber of tasksTable 2 mean scoreTable 2 solvedOpus 5.5 and Astra, mean score
Report figures3251.038.0%81.0
Publish a data source3625.124.3%68.1
Act on accounts6128.54.8%52.0
Allocate a budget917.74.6%34.0
Forecast726.41.7%31.5

We can also characterize the behavior per filing type (listed in Appendix G, table 6):

  • For data sources (480 expectations - note that many tasks involve more than one expectation), 86% pass the structural contract, 25.0% are right, 53.5% have fewer than 90% of rows right, 7.7% have 90% or more but not all, and 11.2% publish nothing.
  • For enforcement (540 expectations), 12.2% act on nobody, 34.1% find under 20% of the targets, and 40.2% find 20 to 80%. 6.7% find at least 80% but lose the savings to wrong actions, and 5.2% are solved. Pooled recall and precision are 70% and 82% for Opus 5.5 and 55% and 78% for Astra.
  • For forecasts (1,090 series), in 62.6% of series, the point is closer to the observed outcome than the reference point. However, the 80% interval covers the outcome in only 41.7%. Where the point beats the reference, the interval still covers only 53.5%.
  • For reported figures (576 expectations), 46.4% are right, 25.7% have every figure outside tolerance, and 11.5% file nothing.

Concerning tool usage across agent runs, a model is fitted in 55% of forecast runs, 19% of allocation runs and 17% of enforcement runs. An optimizer is used in 18% of allocation runs.

We also provide a more detailed breakdown of various failure modes: running into time/turn/context limits, filing nothing, filing a partially correct result, and filing a completely wrong result.

Stage (% of failed runs)All 30 settingsTable 2Opus 5.5 and Astra
Stopped by a limit (60 minutes, 500 turns, context window)6.9%8.5%0.0%
Ended on its own and filed nothing1.3%1.2%0.3%
Filed, but left out a required filing3.7%2.9%3.0%
Complete filing, but scored below 9588.0%87.3%96.7%
Failed runs5,5872,214301

These are the submitted runs. In the revised manuscript, runs stopped by the 60-minute limit are continued to completion (see the General Comment). Of runs in the last row, about 44% score zero or less (all 30 settings) and the rest earn partial credit. We can also characterize failures by whether or not the model queries the tables required to solve the task, which provides a sense of whether the model simply could not find the relevant data, or interpreted it incorrectly.

For the 71 tasks that are fully solved by at least 3 settings, we also analyze the tables that successful runs queried. We can then analyze, for failed runs with complete filings, the share of runs that never queried at least one table that every solving run used:

SettingShare (n)
Opus 5.5 low / medium / high65.7% (35), 37.5% (16), 18.2% (11)
Astra low / medium / high58.8% (34), 40.0% (25), 35.0% (20)
Sol low / medium / high61.0% (41), 45.0% (40), 48.6% (37)
Sonnet 5 low / medium / high68.3% (63), 65.1% (63), 52.5% (61)
Haiku 4.571.6% (67)
All 30 settings50.3% (1,255)

In general, we find a Spearman score of -0.59 between a model’s score and its miss share, over 12 models pooled over efforts.

We hope that this analysis provides a more insightful breakdown of model capability and characterizes model behavior across tasks.

Q1: How long does it take to run the full benchmark, and how many tasks are executed in parallel?

A4: 6299 runs (a total of 30 model/effort settings, on 210 tasks, less one unscored run) took a total of 1683 run-hours. We did execute runs in parallel. Runs were in flight for 20.3 hours for all 30 settings. On average, we ran 83 runs in parallel, with a peak of 529 runs in parallel. There are no intrinsic features of the benchmark that prevent parallel runs, as tasks and settings are independent of one another. Concurrency was capped per provider and per model, with 27 to 86 runs at once per model at its peak. The median run took 9.6 minutes, the average run took 16.0 minutes, and the 90th percentile run took 39.5 minutes. Our architecture enforced a 60-minute limit and 317 of the 6299 runs reached this limit. All 317 were re-run without the limit for the revised manuscript (see the General Comment). To give a better sense of the per-model runtime, to run the full 210 task suite on a single model, for one effort level, we provide the following timings for a subset of models:

  • 16 run-hours for Claude Haiku 4.5
  • 27 run-hours for GPT-6 Astra
  • 42 run-hours for Claude Opus 5.5
  • 181 run-hours for Gemini 3.8 Flash

Q2: Why are recent, closely related benchmarks such as AvalancheBench, The Era by Eon, and ERPBench discussed in the Related Work section but omitted from Table 1? Including them would help clarify Argo-Bench’s contributions relative to recent work.

A5: We provide the following comparisons to AvalancheBench, The Era by Eon, and the two ERPBench benchmarks:

BenchmarkTables per DBRows per DBDataCoherent systemPython and MLActions gradedGround truth
AvalancheBench (Kłeczek et al., 2026)3.7n/rGenerated; LLM-written text✗n/r✗Latent world, 3 LLM judges
The Era by Eon (Gruenbaum et al., 2026)n/r¹n/rGenerated, calibrated²✗³✗✗Computed from records
ERPBench (Zhang et al., 2026)n/r⁴n/rSimulated game✗n/r✓Company valuation
ERPBench (Bhagtani et al., 2026)n/rn/rSeeded records✓✗✓Expected DB fields
Argo-Bench (ours)2357.49BSimulated✓✓✓Latent state

n/r = not reported

¹No SQL database, 66 simulated SaaS products and about 290 read-only MCP tools.

²Realism scored against Eon’s operational data and published statistics.

³Separate SaaS simulators projected from one shared entity graph.

⁴The agent queries tools and returns JSON actions; there are no tables.

Argo-Bench is the only benchmark with all four properties. In the revised manuscript, we will update Table 1 accordingly.

Reviewer_7CP5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Argobench evaluates data agents in a simulated food delivery business with about 7.5 b rows and 210 tasks. Financial/operational tables belong to the same business. Agents use SQL and Python to analyze the warehouse. The required outputs include forecasts/fraud actions/reports/resource decisions. The grader checks these submissions using simulator information hidden from the agent.

Critical strengths

(1) It is a plus that historical enforcement is not treated as the true fraud label. The simulator includes innocent accounts with similar early behavior. A good score against past enforcement may still lead to wrong actions.

(2) The cash example separates payment dates from bank clearance. A correct year end balance does not ensure a correct cash timeline even when the explanation sounds reasonable. Filing/cell checks show this error.

(3) The tasks in Table 5 use the same pay rule change with data available up to different dates; some tasks ask for a forecast before the change. Later ones ask the agent to recover the rule or check payments.

Critical weaknesses

(1) It seems that App.G properness claim does not hold for the final forecast grade. WIS itself is proper. Clipping the score can favor an overconfident forecast. It would be good to distinguish WIS from the final grade.

(2) In App.J, the prompt gave equal costs. The grader instead weights storefront savings more than customer closure f1. The reported runs used the old wording before the correction. It would be good to check main comparison without these tasks.

(3) The public runner uses a different world from the same simulator and changes the entities in 15 prompts. A rerun checks the procedure rather than the exact task instances in the main table. Keeping test keys private is reasonable. It is not clear how public scores can be checked with a fixed grader version outside the best effort service.

Questions for authors

please see weakness part.

6 comments

We thank you for your constructive comments and critiques and address each weakness and question in turn.

W1: It seems that App.G properness claim does not hold for the final forecast grade. WIS itself is proper. Clipping the score can favor an overconfident forecast. It would be good to distinguish WIS from the final grade.

A1: Thank you for raising this point. It is true that in our grading of forecasting tasks, clipping the score at 0 distorts the final grade. This grade was calculated as 100⋅max⁡(0,1−WIS/WISref)100 · \max(0, 1 − WIS/WIS_{ref}), further divided by a ceiling between 0.72 and 0.94 and capped at 100 for 25 of the 98 forecast expectations, so each forecast series received a score between 0 and 100. We have since revised our scoring methodology for forecasting tasks to use the following grading formula instead: 100⋅(1−WISg/0.623)100 · (1 − WIS_g/0.623), where WISgWIS_g is the WIS scaled on asinh, g(v)=asinh⁡((v−r)/s)g(v) = \operatorname{asinh}((v-r)/s). 0.623 is a constant denoting the median, over all series, of the reference forecast’s own WIS on that scale. (We then floor the score at -200, to bound each forecast series to receive a score between -200 and 100. We score the case where a model does not file an answer at -200. In a model’s overall score, its forecasting tasks taken together contribute no less than 0.) On this new scale, an exact forecast scores 100, a forecast that is as far off as the median reference gets 0, a forecast that is twice as far off as the reference gets -100, and three times as far off gets -200. Across the 2,789 series across all models on the 72 forecasting tasks, the distribution of scores is as follows:

Clipping at 0Revised formula with clipping at -200
Series at the lower clip (of 2,789 filed)1,141 (40.9%) at 0286 (10.3%) at -200
Filing the true quantiles maximizes expected grade18.4% of series96.5% of series
Filing a narrower interval gives a higher score72.0% of series3.5% of series
Moving the point toward the reference raises the score29.3% of series0.0% of series

The submitted scoring methodology does incentivize filing an overconfident forecast. It also makes each series’ score dependent on the performance of the reference forecast for that series. Our revised formula largely addresses these issues as shown above. Additionally, we provide the following scores to compare the submitted formulation, revised formulation (which also includes the answer-key fixes described in the General Comment), and unclipped formulations:

Model (high effort where offered)As submittedRevised forecast gradeRevised, no floor
Claude Opus 5.559.354.553.1
GPT-6 Astra52.848.245.4
GPT-6 Sol38.931.428.6
Qwen 3.8 Max34.324.112.1
Kimi K330.721.513.1
Muse Spark 1.330.620.57.5
DeepSeek V4.1 Flash30.020.011.6
GLM 5.3 Flash25.816.3-0.9
Claude Sonnet 522.413.42.8
GPT-6 Luna21.212.7-1.5
Gemini 3.8 Flash16.312.0-29.7
Claude Haiku 4.510.05.2-47.3

Under the revised grading, the 12 models maintain the same order as the submitted version (Spearman 1.000, or 0.988 over all 30 settings). If we remove the floor entirely, as shown in the third column, the top three models keep their place while the middle models reorder. (See also the General Comment.)

W2: In App.J, the prompt gave equal costs. The grader instead weights storefront savings more than customer closure f1. The reported runs used the old wording before the correction. It would be good to check main comparison without these tasks.

A2: When we drop the affected tasks (four col-22 tasks out of a total of 210 tasks), we see the following changes on the revised scores from A1:

  • The largest overall Table 2 score change is 0.87 points.
  • The ordering of models stays as is, with the exception of DeepSeek V4.1 Flash moving 0.02 points above Muse Spark 1.3.
  • The fraud column’s biggest score change is 3.4 points (Opus 5.5’s score increases from 57.7 to 61.1).

In the revised version of the benchmark, we reran the col-22 tasks on every model setting with wording updated to reflect the grader’s true behavior. We incorporate these changes into the revised manuscript, with a maximum overall score change of 0.67 and no changes to the model ranking. The misleading wording did change model behavior. With the corrected wording, models close more accounts:

Across 120 paired runsOld wordingCorrected wording
Customer accounts closed (median)3458 (more closed in 78 of 120 pairs, fewer in 31)
Customer-closure F1 component19.324.4
Storefront component14.015.6
Composite task score15.818.6

The composite task score increases by 2.8 points (95% CI -3.5 to +8.7) and more accounts are closed in 78 of 120 pairs. This indicates that the score effect is small. Additionally, we conducted an audit of all remaining tasks’ answer keys and graders, leading to 21 answer-key changes (see the General Comment.)

W3: The public runner uses a different world from the same simulator and changes the entities in 15 prompts. A rerun checks the procedure rather than the exact task instances in the main table. Keeping test keys private is reasonable. It is not clear how public scores can be checked with a fixed grader version outside the best effort service.

A3: It is true that the world described in the paper is not the same as the one released publicly, and thus different entities are named in some of the prompts. Reruns on the public world also check the procedure rather than exact instances. In the 15 tasks that the reviewer refers to, couriers, storefronts, or promotion codes are selected per world using a fixed property rule. Of these, five prompts show different names, and ten show different counts based on the selected entities.

We address concerns about reproducibility by comparing runs between the paper’s world and the released public world (see the cross-seed table in the General Comment). For every setting, the interval for the difference contains zero.

We agree that relying on the authors to check submissions on a best-effort basis is a flawed approach. We instead provide an API-based, automated, version-controlled grading service open to researchers on request. Grading only requires the filings and takes a few seconds per task. Each task in the public repository carries a version hash, and every grade is computed with the answer keys and grader code frozen for that release.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment [1/2]

We are grateful to all reviewers for their constructive feedback and valuable suggestions. In the revised manuscript, we make the following changes to the benchmark:

  • Addressing reviewer 7CP5's W1, we floor each forecast series at -200 instead of 0, so as not to favor overconfident forecasts. The 0 floor now applies only to the average over the forecasting tasks as a whole, so forecasts aren't chosen between model and reference series by series with hindsight. We report the unfloored forecast averages separately.
  • We apply the answer-key fixes made since submission (21 of 210 tasks; no Table 2 score moves by more than 0.74 from these alone).
  • Addressing reviewer 7CP5's W2 and 5ydS's W2, we re-run the refund-collusion task group with prompt wording updated to reflect the grader's true behavior.
  • Addressing reviewer 5ydS's Q1, we remove effects of parallel runs by lifting the time constraints and allowing runs that were cut off by the 1-hour limit to complete.

Table 2 will be updated as follows:

Model (high)As submitted+ forecast grade / key fixes+ col-22 reruns+ resumed runsRankSolved %
Claude Opus 5.559.354.554.554.51 → 132.9 → 31.0
GPT-6 Astra52.848.248.248.62 → 226.2 → 25.7
GPT-6 Sol38.931.430.730.73 → 316.2 → 15.7
Qwen 3.8 Max34.324.124.124.44 → 517.6 → 17.1
Kimi K330.721.521.821.95 → 611.9 → 10.5
Muse Spark 1.330.620.519.919.96 → 711.9 → 10.5
DeepSeek V4.1 Flash30.020.019.819.87 → 89.0 → 8.1
GLM 5.3 Flash25.816.316.717.08 → 98.1 → 7.6
Claude Sonnet 522.413.413.413.49 → 105.2 → 4.3
GPT-6 Luna21.212.712.612.810 → 115.2 → 4.8
Gemini 3.8 Flash16.312.012.226.011 → 410.0 → 16.2
Claude Haiku 4.510.05.25.15.112 → 121.4 → 1.0

We also acknowledge reviewer 7CP5's W3 and YCA3's Q2, and the revised version will introduce an additional Appendix section and table covering cross-seed reproducibility, with cross-seed results for all models. For now, we've conducted re-runs on the publicly released world for the following model settings:

ModelPaper's seedReleased seedDifference [95% interval over task families]
Claude Opus 5.5, high54.554.2−0.4 [−5.2, +4.1]
Claude Opus 5.5, medium48.951.3+2.4 [−3.7, +9.3]
Claude Opus 5.5, low31.732.5+0.8 [−3.7, +4.9]
GPT-6 Sol, high30.631.3+0.7 [−5.0, +6.1]
GPT-6 Sol, medium27.826.8−1.0 [−7.3, +5.9]
DeepSeek V4.1 Flash, high19.515.2−4.3 [−9.0, +2.5]
GPT-6 Sol, low19.315.9−3.3 [−7.7, +1.0]
Claude Sonnet 5, high13.417.4+4.0 [−0.3, +7.7]
GPT-6 Luna, high12.913.6+0.8 [−1.6, +4.6]
GPT-6 Luna, medium12.211.5−0.7 [−3.3, +1.7]
GPT-6 Luna, low5.85.0−0.8 [−3.0, +1.4]
Claude Haiku 4.55.14.9−0.2 [−1.7, +1.3]

Results on the two seeds are consistent: the 12 settings rank almost identically across the two seeds (Spearman 0.972, or 0.991 without DeepSeek V4.1 Flash). The only change in order is that Claude Sonnet 5 (high) moves from eighth to sixth place and DeepSeek V4.1 Flash (high) from sixth to eighth. DeepSeek's drop traces to a change in how its provider now serves the model. 13% of its runs on the released seed ended without an answer after a reply reached the 65,536-token output limit, which happened in none of its 13,993 model calls in the paper's runs. Leaving out the 29 tasks whose re-run ended on such a reply, its difference is +0.3 [−4.3, +7.5].

AuthorsOpenReview ↗

General Comment [2/2]

To address reviewer 5ydS's W3 and YCA3's W3/Q3, we classify the primary error cause for all 1,933 failed runs that submitted filings. Given a run's task definition, answer-key code, per-metric scores, and agent trace, a frontier model selects one of eight categories:

  • Schema exploration. The data required for the task was not found (e.g., a similarly named table was used instead) or the semantics of a column were misunderstood.
  • Business understanding. The agent misinterpreted the task or a business quantity (e.g., misread a process described in the prompt or selected the wrong population).
  • SQL logic. The SQL or pandas code did not compute what the agent intended (e.g., join fan-out, an incorrect filter or date boundary, de-duplication, aggregation grain, null handling).
  • Statistical modeling. The statistical method was inadequate (e.g., a detection rule with low recall/precision or a forecast ignoring seasonality, trend or a known change).
  • Optimization. The relevant quantities were estimated correctly but budgets were allocated poorly (e.g., in proportion to volume rather than marginal return).
  • Filing or execution. The analysis was adequate but the result was filed incorrectly (e.g., wrong format or key column).
  • Answer key disputable. Given only the prompt and the warehouse, the agent's interpretation is at least as defensible as the answer key's.
  • Undetermined. The trace does not show which of the above applies.

In the revised manuscript, we will introduce an additional section with the aggregate results of this analysis as well as breakdowns by filing type and model. Additionally, two authors, blind to the LLM-assigned causes, are hand-classifying a representative subset of 200 traces from a range of models and task categories. We will report human-LLM agreement.

Aggregate results:

First decisive errorShareRuns
Statistical modeling48.6%939
Business understanding27.2%525
Schema exploration20.5%396
SQL logic1.7%32
Answer key disputable1.3%25
Optimization0.4%7
Filing or execution0.4%7
Undetermined0.1%2

By filing type, as a share of that type’s failed runs:

FilingSchema explorationBusiness understandingStatistical modelingRuns
Report figures53.3%33.0%9.0%212
Publish a data source55.6%35.7%1.1%277
Act on accounts7.0%30.4%61.8%570
Allocate a budget8.0%45.5%38.6%88
Forecast10.4%18.2%67.6%786

And by model:

ModelSchema explorationBusiness understandingStatistical modelingRuns
Claude Opus 5.56.5%20.9%69.1%139
GPT-6 Astra8.6%22.4%64.5%152
GPT-6 Sol14.7%30.0%51.2%170
Qwen 3.8 Max11.6%35.7%49.1%112
Kimi K319.7%25.3%51.7%178
Muse Spark 1.319.5%29.3%48.9%174
DeepSeek V4.1 Flash20.3%29.9%44.4%187
GLM 5.3 Flash26.9%24.2%43.5%186
Claude Sonnet 525.8%25.8%45.9%194
GPT-6 Luna26.3%24.7%46.2%186
Gemini 3.8 Flash12.1%41.4%44.8%58
Claude Haiku 4.537.1%27.4%31.0%197
Top two7.6%21.6%66.7%291
Middle five17.7%29.6%49.0%821
Bottom five27.9%26.7%41.8%821

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.