- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces CommerceAgentBench (CAB), a fixed set of 107 operational tasks. The tasks run in replicas of real services in five forms: captured browser pages with a stateful backend, HTTP/MCP mocks, daemon-isolated CLI clones, replay snapshots, and OSS appliances. Each task is scored by final-state verifiers, using a binary pass and a macro-averaged partial "capacity" over 834 checks. The authors evaluate 13 models on three agent harnesses (Accio, Pi, OpenClaw) with 3 runs each; the best model reaches 59.7% pass and 0.892 capacity.
Critical strengths
- Resettable copies of commercial services. Most prior benchmarks use open-source substitutes. The authors instead copy the commercial products. This is different from the existing approaches; for example, WebArena uses Magento as the open-source alternative, and Amazon-Bench uses the live Amazon website, and MCP-Universe uses live MCP servers.
Critical weaknesses
-
The paper does not show novelty beyond prior benchmarks. Its three defining properties, its partial-credit metric and its findings are already established, and new domain coverage alone is not an ICLR-level contribution. The introduction presents three properties as CAB's motivation:
- The agent operates through the surfaces used in the real workflow.
- Writes change what later reads and final-state verifiers observe.
- The world is resettable and replayable. Each property, and the partial-completion metric, is already a design principle of peer-reviewed benchmarks, several of which CAB itself cites, for example, WebArena and TheAgentCompany. So I would argue that no novel metric is introduced to bring new findings in understanding the model x harness performances.
-
The per-run results are very unlikely under independent runs. Until it is explained, I cannot trust the main results (Tables 1 and 16).
-
Tables 16–17 give three runs (R1–R3) for each of the 39 model–harness pairs (13 models × 3 harnesses). For each pair, I computed the gap between its best and worst run, i.e., max − min of the three pass counts. A majority of these gaps are exactly 3 tasks. While this explains why "±1.5" is dominant in Table 1, the probability it could happen under independent runs is small.
-
In Table 16, Relative to the R1, the other two runs repeat the same offsets. R2 = R1 + 2 and R3 = R1 − 1, e.g., Opus 5 on Accio is 66 → 68, 65 and DeepSeek V4 Pro on Accio is 54 → 56, 53.
-
-
"Long-horizon" is not justified in the paper. The paper reports no steps, tokens, wall-clock time, or cost per task. In contrast, OSWorld 2.0 on average requires 318 tool calls and 1.6 human-hours for task completion.
-
Terms and acronyms are used without being properly defined, for example, "ILP trimming" (Line 205), "granularity migration audit" (Line 234). , and "HARD" cases (Line 377).
Questions for authors
-
Which capability, failure mode, or measurement can CAB capture that existing benchmarks (WebArena, AppWorld, tau-bench etc) failed to capture? A side-by-side comparison table, rather than a citation list, would answer this directly.
-
For the models that overlap with existing leaderboards, how strongly does the CAB ranking correlate with theirs? If it correlates highly, what does a practitioner learn from CAB that they could not learn from those benchmarks? If it does not, what explains the difference?
-
How does "capacity" differ, conceptually or empirically, from TheAgentCompany's and ToolSandbox's reported metrics?
-
Please provide the per-task steps, time, and token statistics to justify the long-horizon claim.
-
Please explain the statistical pattern reported in Tables 1 and 16.
2 comments
Official Comment by Reviewer 6sVT
Thank you for the response. Overall, my assessment of the paper has not substantially changed, and I will maintain my original rating.
Thank you for the concrete concerns. We address run provenance first.
1. Repeated-run provenance. Rechecking Tables 16-17 confirms that 32 of 39 model-harness pairs have a pass-count range of exactly three, and 16 pairs have offsets (+2, -1) relative to R1. The original run configurations and execution logs are needed to explain this pattern and verify independent sampling.
The 21 available R1 reports reproduce seven-model, three-harness aggregates over the same 107 tasks. This audit covers the supplied reports; the full 117-run provenance remains to be verified. Archive labels such as "bestof" and "folded" also require their actual inclusion rules to be documented.
We will regenerate tables from provenance-verified task records and correct or remove unsupported repeated-run statistics.
2. What CAB adds. We agree that real interfaces, persistent state, reset, and partial completion are established benchmark principles. We will sharpen the contribution around evidence-aligned replicas of commercial-service task surfaces and the operational contracts implemented on them.
CAB tests whether selecting the right supplier or billing object leads to the required updates and cross-system associations while preserving unrelated records. These capabilities have precedents; CAB adds service-specific workflows and their implementation evidence. The 107 tasks were curated from approximately 2,000 deduplicated workflow candidates grounded in production traces and user interviews, using demand, reproducibility, and outcome verifiability. Fidelity covers declared task surfaces supported by upstream evidence and regression tests.
3. Capacity and ranking correlation. Capacity is an interpretable macro-average of Boolean requirement completion. It is conceptually close to TheAgentCompany's partial task completion. ToolSandbox instead aligns intermediate/final milestones to a trajectory under a dependency graph, with milestone similarities; CAB primarily aggregates declared outcome checks, including event requirements where specified.
Cross-benchmark ranking correlation remains to be measured using matched model versions and documented harnesses, budgets, benchmark versions, and overlap size. CAB could help locate service-specific failures even when rankings are highly correlated; a matched-model comparison would assess its effect on model-selection decisions.
4. Observed horizon. We extracted the "Tool calls" fields from the 21 available R1 reports, covering 2,247 task records. Mean/median/P90 are 58.1/40/131 calls, with maximum 344. For 2,099 records with reported trajectory duration, median/P90 are 7.63/30.95 minutes; 148 durations are missing. The reports do not expose a consistent token-usage field.
These measurements include failed and budget-limited runs. A tool invocation can contain multiple operations, and harness accounting differs; comparison with OSWorld's action counts requires a shared counting unit. We will provide per-task measurements, label missing values, and describe the scope as multi-step workflows with dependent decisions and writes.
5. Presentation and anonymity. We will consolidate the repeated passages in A.4/A.5 and describe the concrete task-construction process. HARD is the historical seven-model OpenClaw failure screen, with maximum capacity >0.3 and at least five checks. The "granularity migration audit" concerns regrouping low-level checks and checking binary-verdict preservation.
We will remove the earlier-name linkage and nonessential model-release discussion in B.2 from the anonymized manuscript, audit identifying links, and ensure any required AI-assistance disclosure accurately reflects actual use. The authors take responsibility for the final text and its evidence.