acceptodds
Desk rejected at ICLR 2027

CommerceAgentBench: Benchmarking Long-Horizon Agents in High-Fidelity, Stateful, and Reproducible Replicas of Real Online Services

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Whether large language model agents can carry out sustained operational work remains a central evaluation challenge. Such work spans tools and systems, changes shared state, and depends on repeated judgment. CommerceAgentBench (CAB) evaluates this ability in high-fidelity, stateful, reproducible replicas of real services across browser UI, API/MCP, isolated command-line, replay, and open-source appliance forms. Task-surface fidelity aligns the declared routes, controls, validations, visible outputs, and state transitions with upstream evidence and regression tests. We evaluate a fixed set of 107 cases spanning commerce, sourcing, office suites, marketplace operations, logistics, and ERP-like reconciliation. Human authors reconstruct workflows from request examples or curated procedures, ground workspaces in public or officially sourced synthetic records, and define outcome contracts. Agent actions mutate durable state, and verifiers read final state or artifacts. Scoring combines strict workflow completion with macro-averaged capacity over 834 checks: 799 non-judge checks and 35 grounded boolean model-judge checks. Thirteen models run on three agent frameworks with three independent runs per model–framework pair. The strongest model achieves a 59.7% mean pass rate and 0.892 capacity across frameworks. On a selected 15-cell diagnostic panel, supplying object-selection decisions or target-system state enables previously failed workflows to complete. A 14-verifier state audit and one-change replays on 12 of 32 HARD cases test score responses to controlled errors. These results characterize complete workflows, partial progress, and the effect of targeted interventions. Our data and code are available at https://anonymous.4open.science/r/CommerceAgentBench-5E42.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model Agents, Agent Benchmarks, Operational Tasks, Stateful Evaluation, Reproducible Environments

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6sVTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CommerceAgentBench (CAB), a fixed set of 107 operational tasks. The tasks run in replicas of real services in five forms: captured browser pages with a stateful backend, HTTP/MCP mocks, daemon-isolated CLI clones, replay snapshots, and OSS appliances. Each task is scored by final-state verifiers, using a binary pass and a macro-averaged partial "capacity" over 834 checks. The authors evaluate 13 models on three agent harnesses (Accio, Pi, OpenClaw) with 3 runs each; the best model reaches 59.7% pass and 0.892 capacity.

Critical strengths
  • Resettable copies of commercial services. Most prior benchmarks use open-source substitutes. The authors instead copy the commercial products. This is different from the existing approaches; for example, WebArena uses Magento as the open-source alternative, and Amazon-Bench uses the live Amazon website, and MCP-Universe uses live MCP servers.
Critical weaknesses
  • The paper does not show novelty beyond prior benchmarks. Its three defining properties, its partial-credit metric and its findings are already established, and new domain coverage alone is not an ICLR-level contribution. The introduction presents three properties as CAB's motivation:

    • The agent operates through the surfaces used in the real workflow.
    • Writes change what later reads and final-state verifiers observe.
    • The world is resettable and replayable. Each property, and the partial-completion metric, is already a design principle of peer-reviewed benchmarks, several of which CAB itself cites, for example, WebArena and TheAgentCompany. So I would argue that no novel metric is introduced to bring new findings in understanding the model x harness performances.
  • The per-run results are very unlikely under independent runs. Until it is explained, I cannot trust the main results (Tables 1 and 16).

    • Tables 16–17 give three runs (R1–R3) for each of the 39 model–harness pairs (13 models × 3 harnesses). For each pair, I computed the gap between its best and worst run, i.e., max − min of the three pass counts. A majority of these gaps are exactly 3 tasks. While this explains why "±1.5" is dominant in Table 1, the probability it could happen under independent runs is small.

    • In Table 16, Relative to the R1, the other two runs repeat the same offsets. R2 = R1 + 2 and R3 = R1 − 1, e.g., Opus 5 on Accio is 66 → 68, 65 and DeepSeek V4 Pro on Accio is 54 → 56, 53.

  • "Long-horizon" is not justified in the paper. The paper reports no steps, tokens, wall-clock time, or cost per task. In contrast, OSWorld 2.0 on average requires 318 tool calls and 1.6 human-hours for task completion.

  • Terms and acronyms are used without being properly defined, for example, "ILP trimming" (Line 205), "granularity migration audit" (Line 234). , and "HARD" cases (Line 377).

Questions for authors
  • Which capability, failure mode, or measurement can CAB capture that existing benchmarks (WebArena, AppWorld, tau-bench etc) failed to capture? A side-by-side comparison table, rather than a citation list, would answer this directly.

  • For the models that overlap with existing leaderboards, how strongly does the CAB ranking correlate with theirs? If it correlates highly, what does a practitioner learn from CAB that they could not learn from those benchmarks? If it does not, what explains the difference?

  • How does "capacity" differ, conceptually or empirically, from TheAgentCompany's and ToolSandbox's reported metrics?

  • Please provide the per-task steps, time, and token statistics to justify the long-horizon claim.

  • Please explain the statistical pattern reported in Tables 1 and 16.

2 comments

Thank you for the concrete concerns. We address run provenance first.

1. Repeated-run provenance. Rechecking Tables 16-17 confirms that 32 of 39 model-harness pairs have a pass-count range of exactly three, and 16 pairs have offsets (+2, -1) relative to R1. The original run configurations and execution logs are needed to explain this pattern and verify independent sampling.

The 21 available R1 reports reproduce seven-model, three-harness aggregates over the same 107 tasks. This audit covers the supplied reports; the full 117-run provenance remains to be verified. Archive labels such as "bestof" and "folded" also require their actual inclusion rules to be documented.

We will regenerate tables from provenance-verified task records and correct or remove unsupported repeated-run statistics.

2. What CAB adds. We agree that real interfaces, persistent state, reset, and partial completion are established benchmark principles. We will sharpen the contribution around evidence-aligned replicas of commercial-service task surfaces and the operational contracts implemented on them.

BenchmarkMain evaluation setting
WebArenaSelf-hosted websites and browser-based functional tasks
TheAgentCompanySimulated workplace with web, terminal, files, and coworker interaction
AppWorldStateful everyday apps accessed programmatically through APIs
tau-benchPolicy-constrained customer-service dialogue and domain APIs
ToolSandboxStateful tools and dependency-aware conversational milestones
CABDeclared commercial-service surfaces across browser/API/MCP/CLI/replay/OSS, with operational state and artifact contracts

CAB tests whether selecting the right supplier or billing object leads to the required updates and cross-system associations while preserving unrelated records. These capabilities have precedents; CAB adds service-specific workflows and their implementation evidence. The 107 tasks were curated from approximately 2,000 deduplicated workflow candidates grounded in production traces and user interviews, using demand, reproducibility, and outcome verifiability. Fidelity covers declared task surfaces supported by upstream evidence and regression tests.

3. Capacity and ranking correlation. Capacity is an interpretable macro-average of Boolean requirement completion. It is conceptually close to TheAgentCompany's partial task completion. ToolSandbox instead aligns intermediate/final milestones to a trajectory under a dependency graph, with milestone similarities; CAB primarily aggregates declared outcome checks, including event requirements where specified.

Cross-benchmark ranking correlation remains to be measured using matched model versions and documented harnesses, budgets, benchmark versions, and overlap size. CAB could help locate service-specific failures even when rankings are highly correlated; a matched-model comparison would assess its effect on model-selection decisions.

4. Observed horizon. We extracted the "Tool calls" fields from the 21 available R1 reports, covering 2,247 task records. Mean/median/P90 are 58.1/40/131 calls, with maximum 344. For 2,099 records with reported trajectory duration, median/P90 are 7.63/30.95 minutes; 148 durations are missing. The reports do not expose a consistent token-usage field.

These measurements include failed and budget-limited runs. A tool invocation can contain multiple operations, and harness accounting differs; comparison with OSWorld's action counts requires a shared counting unit. We will provide per-task measurements, label missing values, and describe the scope as multi-step workflows with dependent decisions and writes.

5. Presentation and anonymity. We will consolidate the repeated passages in A.4/A.5 and describe the concrete task-construction process. HARD is the historical seven-model OpenClaw failure screen, with maximum capacity >0.3 and at least five checks. The "granularity migration audit" concerns regrouping low-level checks and checking binary-verdict preservation.

We will remove the earlier-name linkage and nonessential model-release discussion in B.2 from the anonymized manuscript, audit identifying links, and ensure any required AI-assistance disclosure accurately reflects actual use. The authors take responsibility for the final text and its evidence.

Reviewer_6sVTOpenReview ↗

Official Comment by Reviewer 6sVT

Thank you for the response. Overall, my assessment of the paper has not substantially changed, and I will maintain my original rating.

Reviewer_W9bgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CommerceAgentBench introduces 107 operational tasks across commerce, sourcing, office tools, and logistics. Tasks run in browser, API/MCP, CLI, and open-source service enviroments with shared durable state and outcome verifiers (834 checks total, 799 non-judge). The paper evaluates 13 models across 3 harnesses (3 runs each), reporting both strict workflow completion and macro-averaged check completion ("capacity"), where the best config reaches 59.7% pass rate and 0.892 capacity. The authors also run targeted interventions on object selection and target state, alongside verifier audits on incorrect outcomes.

Critical strengths
  1. The environment infrastructure is a substantial and useful contribution. The replicas connect task-relevant interfaces to shared durable state, so a write can affect later reads and the final outcome. Fidelity is supported by interface-specific evidence rather than appearance alone: captured pages and interaction tests for browser environments, and upstream fact sheets or golden comparisons for six CLI families, with 540 CLI regression checks across seven families (Section 3.2 and Appendix A.3). The scope is appropriately defined as the declared task surface. This provides a practical substrate for studying operational workflows.

  2. The evaluation combines meaningful scale with inspectable outcomes. The same 107 cases are evaluated in 117 complete benchmark runs, exposing both model and harness effects rather than reporting a single agent loop. Strict pass and capacity distinguish completed workflows from partial progress. The selected verifier audit also checks semantic properties: all 14 audited verifiers accept correct states and reject incomplete writes, surface-only completion, and wrong-object bindings; the one-change replays isolate the affected check in 12 HARD cases. These are useful checks on what the reported scores measure.

Critical weaknesses
  1. Ambiguities in the diagnostic interventions (A1/A3). Because A3 pre-fills target-system state, some checks are likely satisfied before the agent even runs, yet this pre-agent score is not reported. As a result, the best-of-3 flips (22/33 for A3, 8/12 for A1) only show that simplified tasks are easier rather than isolating a specific failure mode. Also, the reported capacities don't quite match up: Table 2 gives 0.92/0.93 for A1/A3 (~0.927 across the 15 tasks), whereas the three models in Table 3 average ~0.947.

  2. Missing details on how the 107 tasks were filtered. The paper mentions two authoring routes and ILP trimming, but omits the initial pool size, route split, and ILP rules. Adding custom requirements like traceability manifests also makes it hard to seperate core workflow difficulty from instruction-following overhead. I currently lean weak rejection.

Questions for authors
  1. For A1/A3, what is the check score immediately after the intervention vs. after the agent finishes, and what are the single-run success rates (beyond best-of-3)? Please also clarify the capacity mismatch between Table 2 (~0.927) and Table 3 (~0.947) and state the sample size for A2.

  2. Could you provide basic stats on task selection (initial pool size, source breakdown, ILP setup, and whether model results were used in filtering), along with typical action lengths per task?

2 comments

Thank you. The archives clarify the interventions and reveal two reporting corrections.

1. A3 supplies a structured target-state plan. The supplied artifact is execution_plan.json: it describes target objects, fields, and relations. Services start from the task seeds, and the agent must inspect runtime IDs and execute the writes. We will replace the ambiguous phrase "supplies target-system state" with "supplies a structured target-state plan."

Some requirements already hold in the seeds. The available audit gives:

TaskBefore agent writesArchived final outcome
Todoist19/7575/75 in all nine initial runs across the three models
Jira2/1515/15 in eight of nine runs; 11/15 in the other
Stripe1/1313/13 in the inspected GPT-5.5 T1 run

Todoist initially passes 17 label checks, card-set preservation, and readability; execution adds 36 routing checks, 19 label checks, and Backlog clearance. Jira initially passes setup and distractor preservation; successful runs add 13 checks. The 11/15 run adds 10 and loses one initial pass. Stripe adds 12 under the archived verifier; Q98T's coverage concerns require correction.

Todoist/Jira use offline SQL-seed reconstruction, with readability conditional on state loading. Stripe uses archived CLI snapshots before the first write. These examples document the before-and-after changes in the inspected tasks.

2. Aggregation and trial accounting. The archived capacities take each task-model pair's maximum checks_passed/checks_total over trials, then average pairs equally. Each selected score comes from one complete trial.

Table 2's capacities inadvertently include a fourth model, Qwen 3.7 Plus, while its flip counts and Table 3 use three models. The four-model values are 0.918347 (A1) and 0.933183 (A3). With the same three models as Table 3, they are 0.926646 and 0.948737. Their weighted mean, (4 x 0.926646 + 11 x 0.948737)/15 = 0.942846, matches the mean of Table 3's unrounded model values. We will correct the inconsistent model scope.

The trial count also needs correction: 18 task-model pairs include three extra rerunA records. The published 8/12 and 22/33 counts summarize success across three or six archived attempts per pair.

Run inclusion, three modelsA1 successful runsA1 pairs with successA3 successful runsA3 pairs with success
Initial three runs only16/36 (44.4%)6/1247/99 (47.5%)21/33
All archived attempts18/54 (33.3%)8/1248/135 (35.6%)22/33

For the initial three runs, best-run macro-capacity is 0.907796/0.939646 for A1/A3; trial-mean capacity is 0.798047/0.871804. We will separate this uniform-attempt analysis from the archival reconstruction. The second row's pooled rates are descriptive because extra attempts are unevenly allocated.

The handoffs facilitate completion on selected simplified tasks.

3. A2 denominator. A2 is an exploratory cargo-insurance pilot outside the 15-task panel: one task, four completed model runs, zero binary flips. An additional invocation failed before task execution. The pilot also exposed reason-keyword scoring effects, so its interpretation is specific to this task and scoring setup.

4. Source and selection. Our construction records summarize this funnel:

StageScale and procedure
Production samplingAbout 1.6 million de-identified user-task conversation records over two months; each aggregates exchanges and retries
Trajectory filteringAbout 200,000 multi-step trajectories retained using user activity and task length
Workflow abstractionAbout 2,000 candidates after clustering/deduplication by operation sequences and target systems
Executable suite107 tasks selected using workflow frequency, interview-informed demand, reproducibility, and outcome verifiability

Nearly 100 core-user interview records clarified business intent, implicit constraints, and acceptance criteria, including labor-intensive or error-prone workflows underrepresented by frequency alone. The approximately 2,000 candidates formed the workflow abstraction pool for executable-task curation. This demand-informed curation reflects the source user population and implementation feasibility.

The diagnostic panel is separately outcome-selected: HARD denotes failure in all seven historical OpenClaw lanes, maximum check capacity >0.3, and at least five checks. Interviews ground business acceptance criteria; we will identify added evaluation deliverables separately. For observed execution effort, the 2,247 task records in 21 available R1 reports have median/P90 tool-call counts of 40/131.

Thank you for the detailed clarifications and for acknowledging the evaluation issues. However, the verifier gaps, unresolved run provenance, and inconsistent trial accounting still limit my confidence in the reported results. More significant emprical results are needed to assess their impact. Therefore, my final rating is still, 2.

Reviewer_Q98TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CommerceAgentBench (CAB), a benchmark for evaluating long-horizon agents in online services in high-fidelity, stateful, reproducible replicas of real services across browser UI, API/MCP, isolated command-line, replay, and opensource appliance forms. They evaluate a fixed set of 107 cases spanning commerce, sourcing, office suites, marketplace operations, logistics, and ERP-like reconciliation. Human authors reconstruct workflows from request examples or curated procedures, ground workspaces in public or officially sourced synthetic records, and define outcome contracts. across thirteen models and three frameworks on a fixed 107-case evaluation set, the strongest model reaches a 59.7% mean workflow pass rate and 0.892 macro-averaged capacity.

Critical strengths
  • The tasks are practical and valuable. They involve cross tool decision making and modifications to the system state.
  • The released code includes a comprehensive set of tasks, mock services, and verifiers, reflecting a significant engineering effort.
  • The evaluation is comprehensive.
Critical weaknesses
  • The evaluation container image is not provided. per-task experimental results are also missing in the released repo.
  • Lacks a systematic comparison with existing evaluation frameworks or benchmarks.
  • Some tasks have discrepancies between the verifier and the task requirements. (see Question for authors)
Questions for authors
  • What key capabilities do CAB introduce compared to the similar existing benchmarks? can you add a table to compare with existing benchmarks?

  • Some tasks have discrepancies between the verifier and the task requirements. For example, in the "Stripe billing-audit-repair" task, the requirements explicitly specify coupon validation periods, the relation between subscriptions and coupons, and the exact quantities of objects. However, the current verifier does not fully validate these constraints. Thus, the final states violating these requirements might still pass all checks. Could you double check this (and other tasks if possible) and whether it will affect the reported results?

1 comment

We thank the reviewer for recognizing the practical tasks, state-changing workflows, and engineering contribution. Your Stripe example identifies a real coverage gap. We address it directly and clarify CAB's contribution and reproducibility.

1. Contribution and comparison. We agree that stateful execution, reset, and partial credit have precedents. CAB's contribution is an evaluation resource that connects evidence-aligned commercial-service task surfaces to durable business state and inspectable outcome contracts. Fidelity is scoped to declared routes, commands, fields, and transitions, with documented deviations. The 107 tasks were curated from about 2,000 deduplicated workflow candidates grounded in production traces and user interviews, using demand, reproducibility, and outcome verifiability.

BenchmarkPrincipal environment and evaluation emphasis
WebArenaSelf-hosted web applications; realistic browser tasks and functional outcome evaluation.
TheAgentCompanyA simulated workplace combining web, terminal, files, and coworker interaction; task and subtask completion.
AppWorldStateful implementations of everyday apps; programmatic API interaction and state-based task evaluation.
tau-benchSimulated customer-service conversations, domain APIs, policies, and repeated-trial reliability.
ToolSandboxStateful conversational tools; intermediate/final milestones with dependency-aware trajectory evaluation.
CABCommercial-service task-surface replicas across browser, API/MCP, isolated CLI, replay, and OSS forms; operational outcomes involving object bindings, required writes, and exclusions.

The additional evaluation target is whether an agent can complete an operational contract through these evidence-aligned service surfaces: for example, reconcile a supplier record across systems while preserving unrelated records and producing the required evidence. The distinction lies in the implemented service/workflow coverage and its alignment evidence. Section 3.2 reports surface-specific regression evidence, including 540 CLI checks across seven families.

2. Stripe verifier coverage. We inspected the bundled billing-audit-repair verifier and reproduced the problem with controlled offline state replays. Starting from its accepted reference fixture, changing EARLYBIRD to an incorrect duration, assigning an incorrect subscription coupon, or adding an extra price/coupon/subscription still gives 13/13. For the extra-object tests, the audit metadata was left unchanged; the verifier compares it with expected constants and needs a check against independently recomputed live-state values.

The necessary corrections are to check coupon duration and duration_in_months, the required coupon-to-subscription bindings, exact active-object cardinalities, and audit metadata recomputed from final state. Customer counting must distinguish the five business customers from the additional audit customer. The task's preservation requirements also need explicit coverage. We will add one-violation controls for these requirements and use the same versioned verifier to rescore archived states across models and harnesses.

The existing 14-verifier audit validates the specific perturbations tested. We will make that scope explicit and extend the requirement-to-check audit to analogous constraints in other tasks.

If Stripe alone changes and all other task scores and the 107-task denominator remain fixed, the maximum absolute change per complete run is 1/107 in capacity, or 0.935 percentage points in pass rate. The original-versus-corrected per-task results should accompany the revision to quantify the actual impact.

3. Reproducibility. Reproducibility requires a runnable environment and traceable evaluation records. The available 21 R1 reports contain the same 107 task IDs and reproduce the corresponding seven-model, three-harness pass counts and stored capacity summaries. This audit covers the aggregates in the supplied reports.

The release needs an anonymously accessible evaluation image (or fully pinned build), image digest, task/verifier revision, run configurations, and machine-readable per-task/check results linked to trajectories. We will supply these together with the corrected scoring results. Image publication and the full repeated-run provenance remain to be confirmed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.