acceptodds
Under review as a conference paper at ICLR 2027

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

Abstract

Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps. We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. GPT-5.6 Sol earns the most, growing the ¥100,000 opening stake into ¥1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with ¥416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders.

Primary area:
datasets and benchmarks
Keywords:
Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_AssHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces E-Commerce Bench, a long-horizon benchmark in which LLM agents operate e-commerce businesses over a simulated year. The environment combines multi-store operation, market dynamics, inventory and cash-flow management, repeated supplier negotiation, fraud risks, and persistent memory. A deterministic economic model and seeded negotiation kernel are used to improve reproducibility, while natural-language supplier interactions are retained. The paper evaluates 18 models using end-of-year assets together with six diagnostic dimensions, showing substantial differences across models and strong trade-offs between profitability, negotiation, fraud avoidance, efficiency, solvency, execution, and learning.

Critical strengths

A substantially richer and more consequential long-horizon evaluation setting. The benchmark couples negotiation, inventory, pricing, fulfillment, liquidity, dynamic events, and memory within one persistent environment. Decisions therefore create delayed consequences and dependencies rather than behaving as largely independent subtasks. This makes the benchmark a meaningful contribution to agent evaluation.

Strong emphasis on reproducibility and diagnostic evaluation. Separating the supplier's economic policy from language realization is a sensible design choice, and reporting multiple capability dimensions reveals failures that would be hidden by the financial objective alone. The evaluation across 18 models also provides a useful empirical snapshot of current long-horizon agent capabilities.

Critical weaknesses

The benchmark currently contains information leakage and environment inconsistencies that weaken construct validity. Most importantly, an observable supplier identifier can systematically reveal whether a supplier is fraudulent, making a core hidden-state reasoning problem mechanically exploitable. There are also mismatches between agent-facing descriptions and underlying numerical parameters, as well as released fields that do not affect the simulator as described.For a benchmark paper, these are consequential because future agents can optimize against artifacts rather than the intended capabilities, particularly fraud detection.

The evidence for “learning over the horizon” and stable model rankings is not yet sufficiently controlled. The proposed learning metric measures sequential price discipline, but the paper explicitly acknowledges that the observed improvements cannot isolate memory use, context effects, or genuine adaptation. Controlled comparisons such as removing historical information, disabling persistent memory, or providing perfect historical retrieval would be important for validating what this metric actually measures. Moreover, only five episodes are run per model despite very large within-model variance, which limits confidence in several ranking claims.

Questions for authors

After removing the supplier-identifier leakage and correcting the identified environment inconsistencies, do the fraud-avoidance and overall model rankings remain qualitatively unchanged? A corrected rerun would materially affect my assessment of benchmark validity.

Can the authors provide controlled ablations or baselines that isolate long-horizon learning—for example, agents without persistent memory/history, agents given explicit historical prices, or scripted policies—and show that the proposed learning metric reliably distinguishes these conditions? Additional runs or uncertainty analysis for the main rankings would also substantially strengthen the conclusions.

Reviewer_gT28OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

E-Commerce Bench asks an LLM agent to operate up to four stores over a simulated year, starting from ¥100,000. It combines procurement negotiation, inventory, fulfillment, returns, delayed settlement, promotions, and supplier fraud. A seeded kernel determines bargaining outcomes and an LLM supplies the dialogue. Eighteen models run five episodes each. The paper reports final account assets and six diagnostics, finding that asset growth does not track negotiation quality, fraudulent spending, efficiency, or repeat-purchase price discipline consistently.

Critical strengths
  1. Delayed settlement, explicit withdrawals, inventory costs, and shared cash across stores connect decisions across time. An agent can remain profitable on paper while running out of spendable funds, which gives the benchmark a concrete source of long-horizon dependence.
  2. The 90-episode analysis includes bankrupt runs and specifies the scoring and final-settlement rules. The case studies and metric caveats help explain what the reported rankings do and do not measure.
Critical weaknesses
  1. Supplier metadata undermine the intended fraud test. Appendix B.5 (pp.20–21, lines 1077–1095) states that emails contain supplier identifiers and that honest suppliers have lower identifiers than fraudulent suppliers within every category. This permits screening by metadata instead of dialogue or delivery evidence. The paper does not show results after removing the shortcut. It does not follow that the evaluated models exploited it, or that every financial ranking is wrong; those effects need to be measured.
  2. The capability interpretations would benefit from operating-policy controls and robustness tests. The five runs use one fixed world, and AnchorRatio measures the ordering of prices for selected honest repeat purchases. The paper explicitly acknowledges that this does not identify the causal effects of memory or planning (§4.3; Conclusion). Heuristic and explicit-memory baselines would help calibrate that interpretation. Appendix I.4.2 discloses provider-default decoding for supplier dialogue and advertised delivery times generated outside the episode seed. Quantifying their effect would help assess ranking robustness; this variation does not by itself invalidate the observed outcomes.
Questions for authors
  1. After removing the identifier leakage, how do fraudulent spending and final assets change for representative LLMs? A simple identifier-based screening baseline on the original version would help quantify the available shortcut.
  2. Is there a full-year fixed-policy reference, with and without explicit purchase-price memory? Such a limited control would help determine whether AnchorRatio captures the intended adaptation.
  3. Can the supplier-renderer outputs, advertised delivery times, and membership-classifier decisions be fixed and replayed? How sensitive are the reported rankings to the remaining sources of interaction variation?
Reviewer_Y4fiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces E-Commerce Bench, a 365-day simulation where LLM agents run up to four online stores with ¥100,000 in starting capital. Agents source from 576 suppliers offering 6,886 products, including 152 fraudulent suppliers. Negotiations follow a seeded kernel based on TERMS-Bench, while customer demand, returns, and cash flows are simulated under fixed rules.

Performance is measured by end-of-year assets, along with six metrics covering negotiation, fraud avoidance, solvency, efficiency, operations, and learning. The authors test 18 models across five runs each. No model dominates across all metrics: GPT-5.6 Sol achieves the highest assets (14.3× initial capital) but ranks 16th in fraud avoidance. Notably, 15 of 18 models show worse price discipline than a shuffled baseline on repeat purchases, suggesting that experience alone does not improve their decisions.

Critical strengths
  1. Clear separation between economic decisions and language. A deterministic kernel controls prices, concessions, and deal outcomes, with agent acceptances checked against quotes (±0.005\pm 0.005). The LLM only generates dialogue. This makes results auditable and allows surplus and repeat-order regret to be measured against known references, unlike Vending-Bench 2. I see this as the paper's main contribution, as it separates agent performance from counterparty randomness.

  2. Transparency and detailed evaluation. The appendix documents the simulation rules and kernel equations, acknowledges data inconsistencies (§B.5), and discusses metric limitations. The results reveal useful trade-offs, such as high profits despite fraud exposure and strong bargaining without improvement on repeat orders. I also checked several figures, including 15 models with zanchor<−1.96z_{\text{anchor}} < -1.96, 8,647 re-orders in Table 16, and the fraud and efficiency ratios in §4.3. All were internally consistent.

Critical weaknesses
  1. Limited statistical evidence and untested generalization. All runs use the same seed (20260122), suppliers, products, and event calendar. With only n=5n=5 episodes per model, results vary substantially: Qwen3.7-Max ranges from roughly ¥6k to ¥359k, while GPT-5.5 averages ¥702k with an SD of ¥689k. Yet the rankings and 311 pairwise rank reversals lack confidence intervals or significance tests. Even on the learning axis, only 53 of 153 model pairs are statistically distinguishable.

    The paper also tests no alternative worlds, unlike YC-Bench, which includes multiple seeds and difficulty settings. This limits generalization and raises overfitting concerns. More importantly, the supplier email suffix leak (§B.5, item 2) could make fraud detection trivial once the benchmark is public. All models also share one agent harness, despite MerchantBench showing that framework choice can substantially affect performance.

  2. Missing baselines and questionable evaluation metrics. The paper provides neither a simple rule-based baseline nor an oracle policy, making results like 14.3×14.3\times initial capital hard to interpret. MerchantBench, RetailBench, and Vending-Bench 2 all offer such reference points.

    Several metrics are also problematic:

    • Drawdown: Excludes inventory from assets, treating stock purchases as losses. This may explain why 49 of 90 episodes hit their minimum assets in January.
    • Operations execution: Effectively measures pricing-induced returns, since the shipping component is always zero (§E.5). Higher prices can increase both returns and profits, so the metric may penalize good decisions. GPT-5.6 Sol ranks 17th here but 1st in assets.
    • BadSpend%: Counts all spending with post-deal scammers as harmful, even when purchases at the honest cost floor remain profitable.
    • Profit per call: Closely follows asset rankings, differing by just 0.7 places on average, and adds little new information.
    • CSE$^+$ and half-lift: Depend on which suppliers each agent chooses. Figure 16 shows supplier-template effects can be comparable to differences between models.

    These concerns, along with inconsistent reasoning-effort settings across providers (max / xhigh / unset / not sent), make it difficult to conclude that the six metrics capture distinct agent capabilities.

Questions for authors
  1. Can you report bootstrap CIs on mean assets and state which pairwise differences in the asset ranking are statistically distinguishable?

  2. How stable are the rankings across different environment seeds, supplier rosters, or calendars? Will the released benchmark include held-out worlds?

Reviewer_6xbjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Research contribution: Integrates year-long, multi-store retail operations with shared cash and time constraints, repeated negotiation, adversarial suppliers, and delayed consequences. A seeded kernel separates supplier economic decisions from dialogue generation (§3; Appendix D).

Evaluation: Tests existing frontier models across assets and six capability dimensions, showing that financial success can coexist with weak fraud avoidance and operational inefficiency (§§4–5).

Novelty relative to prior work: Sustained retail operations, negotiation, and adversarial suppliers exist in earlier benchmarks. The contribution is their integration with controlled supplier economics and diagnostic measurement (§2.2; Table 1).

Acknowledged label leakage: The authors recognize that supplier email identifiers reveal fraud labels, permitting screening without dialogue (Appendix B, p. 20, lines 1077–1079). This leakage should be actively addressed and removed, as it potentially confounds fraud-avoidance and financial results.

Critical strengths

Paper's strength is the integration of economically consequential, long-horizon retail decisions with controlled supplier negotiation. The variety/breadth of the environment and multidimensional evaluation make it a useful testbed for identifying operational weaknesses beyond aggregate financial performance.

The environment contains 6,886 products, 60 categories, 576 suppliers, and 12 store types. Catalog attributes and sales bands come from a real platform; other parameters are synthesized from platform-log analyses.

Long-horizon dependencies: Purchases commit cash, inventory must generate later revenue, and deliveries and returns affect subsequent decisions. This tests maintaining a functioning business across interconnected actions.

Controlled supplier negotiation: A seeded kernel governs economic decisions while an LLM renders dialogue. This provides a more controlled basis for studying repeated bargaining, although dialogue itself can vary.

Critical weaknesses

Acknowledged label leakage: The authors recognize that supplier email identifiers reveal fraud labels, permitting screening without dialogue (Appendix B, p. 20, lines 1077–1079). This leakage should be actively addressed and removed, as it potentially confounds fraud-avoidance and financial results.

Applicability and practical alignment with large retailers:

To support category merchants and online sales teams using agents to automate planning and deliberation, the benchmark could better reflect the following demand and procurement decisions.

Demand dynamics demand shaping and assortment strategy:

Current formulation. The authors combine category-level base demand with price elasticity, weekends, promotions, fixed monthly seasonality by store type, events, and reputation, followed by category and store capacity constraints and inventory limits (Appendix C.2, pp. 21–24; Appendix I.3.3, p. 60, Eq. 47). Appendix B.2 specifies one seasonal multiplier per store type and calendar month (p. 18, lines 967–971; Figure 7, p. 19).

Within-season trends and clearance. Consider evolving product popularity, declining seasonal demand, and markdown decisions for residual inventory after a season.

Customer perception and destination building. Consider assortment investments intended to change how customers perceive the store and which shopping needs they associate with it.

Cross-category shopping: Consider introducing categories that encourage customers to purchase across the assortment within one store or destination, with effects on demand over time.

Extensibility to more complex demand shaping environments: Enable modular demand and customer-behavior mechanisms that the authors or other researchers can contribute later, supporting cumulative research on richer merchant objectives.

Procurement lead times and advance commitments

Current formulation. Lead times are fixed when orders are placed; goods already in transit retain their arrival dates. Supply disruptions apply lead-time multipliers, including 2.5× for winter storms, 4× for the factory fire, and 5× for logistics shutdowns (Appendix B.3, p. 19, lines 1004–1006; Table 6, p. 20). Promotions are announced up to seven days ahead, while enrollment accepts a target 30 days ahead; this is an enrollment window, not a procurement lead-time limit (p. 19, lines 999–1000).

Retailer alignment: Establish whether procurement lead times and purchasing horizons represent large retailers’ advance inventory commitments, and enable longer commitments where appropriate.

Questions for authors

expressed as comments/weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.