- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces E-Commerce Bench, a long-horizon benchmark in which LLM agents operate e-commerce businesses over a simulated year. The environment combines multi-store operation, market dynamics, inventory and cash-flow management, repeated supplier negotiation, fraud risks, and persistent memory. A deterministic economic model and seeded negotiation kernel are used to improve reproducibility, while natural-language supplier interactions are retained. The paper evaluates 18 models using end-of-year assets together with six diagnostic dimensions, showing substantial differences across models and strong trade-offs between profitability, negotiation, fraud avoidance, efficiency, solvency, execution, and learning.
Critical strengths
A substantially richer and more consequential long-horizon evaluation setting. The benchmark couples negotiation, inventory, pricing, fulfillment, liquidity, dynamic events, and memory within one persistent environment. Decisions therefore create delayed consequences and dependencies rather than behaving as largely independent subtasks. This makes the benchmark a meaningful contribution to agent evaluation.
Strong emphasis on reproducibility and diagnostic evaluation. Separating the supplier's economic policy from language realization is a sensible design choice, and reporting multiple capability dimensions reveals failures that would be hidden by the financial objective alone. The evaluation across 18 models also provides a useful empirical snapshot of current long-horizon agent capabilities.
Critical weaknesses
The benchmark currently contains information leakage and environment inconsistencies that weaken construct validity. Most importantly, an observable supplier identifier can systematically reveal whether a supplier is fraudulent, making a core hidden-state reasoning problem mechanically exploitable. There are also mismatches between agent-facing descriptions and underlying numerical parameters, as well as released fields that do not affect the simulator as described.For a benchmark paper, these are consequential because future agents can optimize against artifacts rather than the intended capabilities, particularly fraud detection.
The evidence for “learning over the horizon” and stable model rankings is not yet sufficiently controlled. The proposed learning metric measures sequential price discipline, but the paper explicitly acknowledges that the observed improvements cannot isolate memory use, context effects, or genuine adaptation. Controlled comparisons such as removing historical information, disabling persistent memory, or providing perfect historical retrieval would be important for validating what this metric actually measures. Moreover, only five episodes are run per model despite very large within-model variance, which limits confidence in several ranking claims.
Questions for authors
After removing the supplier-identifier leakage and correcting the identified environment inconsistencies, do the fraud-avoidance and overall model rankings remain qualitatively unchanged? A corrected rerun would materially affect my assessment of benchmark validity.
Can the authors provide controlled ablations or baselines that isolate long-horizon learning—for example, agents without persistent memory/history, agents given explicit historical prices, or scripted policies—and show that the proposed learning metric reliably distinguishes these conditions? Additional runs or uncertainty analysis for the main rankings would also substantially strengthen the conclusions.