acceptodds
Under review as a conference paper at ICLR 2027

StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents

Abstract

Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward is a terminal verdict, and the pass bar is set arbitrarily. We introduce StoreBench, a live-commerce environment in which an agent runs a mid-size online apparel store on a production-grade commerce backend, testing long-horizon planning and economic judgment under uncertainty. Customers order around the clock, suppliers reprice and fail, and market shocks arrive with partial or no warning. The agent acts through the same 29 merchant tools a human operator would use, under a windowed operation budget that makes simulated time a function of actions taken, so model latency cannot influence simulated time. Pass thresholds are calibrated against scripted anchor policies, the reward is hardened against a catalogue of reward hacks, and every episode replays identically given a sequence of actions. We evaluate seven frontier LLMs on 11 scenarios of 30 to 45 days and a full simulated year, over three world seeds at matched reasoning effort. No model matches the scripted smart-triage policy on average: the best, DeepSeek-V4-Pro, passes 49% of task-seed cells against the heuristic's 97%. Human experts working through the same tools and budgets outscore every model (mean composite 0.708 vs. 0.700). Over a full simulated year under the Claude Code harness, most models show dramatic performance improvement. In a GRPO post-training run, Qwen3.5-27B trained on only five disjoint tasks raises its mean composite on the held-out evaluation tasks from 0.136 to 0.373. We release five example training-split tasks, ten sample trajectories, and the scoring and verification tooling; the full environment and evaluation suite are withheld to keep the benchmark uncontaminated.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agentic benchmarks, RL environments, interactive environments, benchmark calibration, long-horizon decision making, reinforcement learning, reward hacking, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kCKAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Authors introduce a novel benchmark that evaluates how well an agent can agent operate a midsized apparel store for 30 to 265 simulated days. The benchmark has exploit resistant evaluation. They show that even strong models reach <49% pass rate.

Critical strengths
  • A new long-horizon benchmark with reward-hacking calibrations.
  • Calibration for reward-hacking guard.
  • Post-trained Qwen3-27B with GRPO to show that the benchmark tasks can be used as RL environments and the post-trained model generalizes on the held-out ID set.
Critical weaknesses
  • Not enough info on benchmark sourcing, construction, verification.
  • The scripted policy (smart-triage) already beats human baseline.
Questions for authors
  • Where the tasks/scenarios/hidden market model are coming from?
  • If a scripted policy such as smart-triage already beats human baseline maybe the best approach is just to let the agent to improve the scripted policy?
  • Why confidence intervals are not reported if there were multiple attempts/seeds per task?
  • Why the learning rate was increased 2x during the GRPO training in the middle of the training run as well as the reward?
  • Why GRPO run was using only 5 tasks? Was it intended to overfit?
  • Will the benchmark code, data and other artifacts will be released?
4 comments
AuthorsOpenReview ↗

reply to the review by kCKA 1/3

Thank you for the direct questions, and for recognizing the reward-hacking calibration and the post-training experiment. Your soundness concern rests on two points: construction detail, and a scripted policy that outscores the human reference.

W1 / Q1. Where do the tasks, scenarios, and hidden market model come from, and how were they verified? The information was in the submission but scattered across two sections and four appendices. The revision consolidates it into one subsection, "Construction and verification" (Section 3.5). In summary: Sourcing and construction.

  • Backend. The store runs on Medusa v2, an open-source production commerce platform, and the agent acts through the same administrative API surface a human operator would use.
  • History. Eighteen months of operating history (3,320 orders, about 2,500 customers, a 14.7% return rate) are produced by seeded procedural generators and written into the backend's own database.
  • Supplier market. Forty vendors offer about 5,200 SKUs in tiers that differ in price, lead time, minimum quantity, defect rate, and reliability. Prices follow a bounded random walk.
  • Hidden market model. Demand is a parametric model that we wrote: per-line popularity, price elasticity, shelf attractiveness, seasonality, copy quality, and stockout censoring. No language model runs inside the simulation. We set the parameters by hand to reproduce known regularities of online apparel retail rather than fitting them to a proprietary dataset: concentrated product popularity with a tail that never sells (about 15% of lines), evening and Sunday-to-Monday order peaks, a Black Friday spike, heavy-tailed repeat purchase, and a 14.7% return rate led by size and fit. The seed-quality suite asserts each regularity as a range, so a generator change cannot move one silently. Price elasticity (2.2 per product, about 2 at store level) was set so that raising every price at once loses money. At an earlier value, a blanket 50% markup raised profit, and we closed that as an exploit.
  • Tasks. Each of the 12 tasks is a small configuration over the same world, authored by us: a horizon, an operation budget, and an event script delivered on announced, rumored, or silent channels. Verification.
  • World quality. A suite of 31 distributional and coherence audits gates every published image.
  • Task validity. Each of the 36 reported task-seed cells, and 24 held-out cells, must pass four calibration gates: anchor ordering, a minimum gap between judgment and volume policies, a noise margin around the threshold, and exploit suppression. The gates are enforced: one candidate task failed the judgment-gap gate during development and was rebalanced.
  • Reward integrity. An audit of 756 pre-hardening trials and a code-level exploit hunt produced a nine-entry catalogue. Each entry is closed, pinned by a regression test, and replayed by the exploiter policy at every calibration. The exploiter now averages 0.04.
  • Determinism. Replaying an action sequence reproduces the ledger byte for byte, checked in CI on every image.
  • Human playability. Human participants completed 36 shifts through the same tool surface.
  • Resistance to cheap search. On one task, 300 random-search and 300 hill-climbing episodes reached 0.21 and 0.40 against a 0.72 threshold. We hope this shows that the construction is documented and checked, and we agree it should have been presented in one place.

(continued in the next comment)

AuthorsOpenReview ↗

reply to the review by kCKA 2/3

W2 / Q2. Smart-triage already beats the human baseline; why not let the agent improve the scripted policy? We understand why this reads as a weakness, and would respectfully offer a different interpretation, in four parts. It is a calibration instrument, and it is meant to be strong. Smart-triage was written by the environment's authors with full knowledge of its rules and settlement mechanics. Its role is to fix the scale: the pass threshold on every cell is set from the two best honest scripted policies. A weak reference would make the threshold trivial. It is also deterministic and narrow, scoring between 0.70 and 0.78 on every cell. It is not a ceiling. The scale tops out at 1.0, and the evidence that there is room above the anchor has grown since submission:

  • In the extended evaluation, the top frontier model scores 0.835, significantly above smart-triage (gap 0.071, 95% CI 0.045 to 0.099).
  • Human participants reach up to 1.00 on individual cells and outscore smart-triage on cold-start (0.794 against 0.741) and slow-drift (0.799 against 0.768).
  • On the full-year task, a separate setup, four models score above 0.96. The Limitations section already states that scripted anchors "bound checklist and heuristic play, not optimal play". The human number is a reference, and a conservative one. Participants used an unfamiliar operation-metered console, not the storefront tools they normally use, with a median of 5.1 hours per shift. We disclose the selection procedure in full. The revision renames it "human expert reference" throughout and states that it is not an upper bound on human performance. On improving the scripted policy. This is an interesting direction, and agents can already attempt part of it: the harness gives every agent a shell, and a scripted call costs the same operation as a native one. Where models had both interfaces, those that lean on scripting do worse, because batched scripts act on stale state: one model averages 0.34 when shell-driven against 0.43 natively, another 0.55 against 0.89. (For three models the tools failed to load in our goose build, leaving the shell as their only route to the store; the revision discloses this.) More fundamentally, the agent is not told the scoring rule, the thresholds, or the event scripts. Writing smart-triage required exactly the knowledge that the benchmark asks the agent to infer while operating. A separate policy-synthesis track, in which the agent may iterate on a policy across episodes, would be a valuable extension, and we add it to the Discussion as future work.

Q3. Why are confidence intervals not reported? This was an omission in the leaderboard table; only the post-training result reported one (+0.237, 95% CI +0.19 to +0.29). Every leaderboard row now has a 95% bootstrap CI over the 33 task-seed cells with 10,000 resamples. For example: smart-triage 0.764 [0.758, 0.768], DeepSeek-V4-Pro 0.700 [0.670, 0.732], Claude Opus 4.8 0.668 [0.642, 0.691], and the human reference 0.708 [0.654, 0.760]. Pass rates carry intervals as well (DeepSeek-V4-Pro 49% [39, 60]). Adjacent models often overlap, so the revision no longer orders models whose intervals overlap and calls a difference significant only when its interval excludes zero, as for every submitted model sitting below smart-triage.

AuthorsOpenReview ↗

reply to the review by kCKA 3/3

Q4. Why were the learning rate and the reward changed mid-run? Both changes are disclosed in the post-training appendix, which the revision expands with the explanation below (Appendix L). Reward. Steps 1 to 4 used the official composite and produced no learning signal. The base model almost never trades at a profit, so its business term was clipped to 0 in essentially every rollout. Rollouts within a group then scored alike and their advantages vanished. From step 5 the training reward clips normalized business to [-2, 1] in place of [0, 1], which ranks losing stores against each other. It is the same quantity with a wider clip. Every reported number uses the official composite, on held-out tasks. Learning rate. From the step-15 checkpoint we ran two branches as a learning-rate comparison, at 5e-5 and 1e-4. At step 21 we also cut the schedule from ten optimizer updates per step to one. The ten-update schedule let the trainer's weights drift from the sampling policy and collapsed the 5e-5 branch. With compute for one branch, we continued the 1e-4 branch on the one-update schedule. We returned it to 5e-5 at step 27, when policy entropy kept rising while training reward stayed flat. We agree that a hand-adjusted schedule is not a training recipe, and the Limitations section says the study shows "that the environment supports hill-climbing with transfer to unseen scenarios, not a tuned training recipe". Two facts bound the concern. Every adjustment used training-batch statistics only: reward, entropy, and the importance ratio. And we report the final checkpoint, step 36. We evaluated checkpoints 32, 34, and 36 on the held-out tasks, and all three show the gain: checkpoint 34 scores 0.341 with 31 of 33 cells improved, and a partial evaluation of checkpoint 32 agrees.

Q5. Why only five training tasks? Was it intended to overfit? No; the design tests the opposite. The five training tasks are disjoint from the 11 evaluation tasks, have different event scripts, and run 7 to 15 days against 30 to 45 in evaluation. The small set reflects rollout cost: each episode is a long multi-turn interaction, and the run totals 1,080 episodes. If the policy had overfit to the training tasks, held-out scores would not move. They rise on 31 of 33 cells. One caveat, which we state in Limitations: the evaluation uses held-out tasks on the same three world seeds, not held-out seeds.

Q6. Will the code, data, and artifacts be released? We will release the code and public sets in the final camera ready version. The anonymized supplementary material already contains five training-split tasks with prompts, configurations, and calibration, ten trajectories with graded ledgers, the scorer, and a re-grader that reproduces each score, under an MIT license. The evaluation suite and its event scripts stay private so the benchmark remains a held-back test set. Thank you again for the close reading. We hope the consolidated construction section and the clarified role of the scripted anchors resolve your soundness concerns, and we would be grateful if you would reconsider your scores in that light.

Thank you for addressing my concerns and for the open source commitment.

Reviewer_J7BHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces StoreBench, a benchmark for testing LLM capabilities at running an online apparel store through a commerce backend software, and compare several frontier models on it. They then train an open source model with GRPO to increase its score.

Critical strengths

The use of a production-grade commerce software makes the benchmark interesting and not a toyish setting anymore.

Critical weaknesses

Comparison and positioning with previous similar benchmarks and papers is not very thorough and it's missing several key citations (e.g., Vending-bench 2, Vending-bench arena, RetailBench, and more).

Heuristics and tools are under-explained in the paper.

Harness comparisons are not entirely fair, models should be tested with their harnesses where possible (GPT or Gemini should be tested through Codex and Antigravity rather than Claude Code)

No statistical significance computed anywhere.

Not open-sourcing the benchmark means that its usefulness for the broader community risks being very limited. I'd advise the authors to release at least part of it, and potentially keep some held-out tracks internally to avoid saturation.

Questions for authors
  1. Can the authors better place their benchmark amongst the broader landscape of similar benchmarks that have come out in the past year or so? Particularly what are the critical difference that StoreBench brings?
  2. Can the authors compute standard error or 95% confidence intervals for the results? I'd expect several results to be within error bars of each other.
  3. Can the authors produce tables/plots for performance pareto frontier with tokens usage and estimated cost for each model?
  4. Can the authors better describe the heuristic policies and tools?
  5. Can the authors test models with their specific harness instead of Claude Code or goose where available? The difference in performance between shell and native tool usage for some models seems to point out that harness and tooling can make a big difference.
  6. How much of the improvements from the GRPO training are from the model actually getting better at the task, versus the model just getting more familiar with the harness?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer J7BH, comment 1 of 3

Thank you for the concrete and actionable questions. Most of them led to direct additions in the revision. We answer each question in order; W1 to W5 map onto Q1, Q4, Q5, Q2, and the release point at the end.

Q1 (W1). Positioning against recent benchmarks, and the critical differences. We agree the related work was too thin and was bounded by the page limit. The revision cites and discusses Vending-Bench 2, Vending-Bench Arena, and RetailBench, and adds a comparison table to Related Work (Table 1). RetailBench (Zhang et al., 2026) is closest in scope: agents run one simulated supermarket for 180 days and are ranked by days survived and final net worth against a privileged oracle. StoreBench differs in three ways. Agents act through a production commerce backend, not a purpose-built simulator. Scores are calibrated per task and seed, with a pass threshold. Events arrive on announced, rumored, or silent channels. The differences we consider critical are methodological:

Vending-Bench familyRetailBenchStoreBench
ScoreTerminal dollars, no ceilingDays survived, then final net worth; no ceiling0 to 1, calibrated per task and seed
Pass thresholdNoneNoneSet from six scripted policies under four gates
CounterpartiesLLM-simulated suppliersSimulation grounded in real grocery recordsSeeded simulation with no LLM; replays byte for byte
Time and tokensOutput tokens are billed (v2)Tokens reported, not scoredTime metered in operations; latency and tokens do not affect score
ScenariosOne business settingOne store configuration12 tasks across steady state, supply, demand, cash, and competition shocks
Runs per model5 (Vending-Bench); 4 per Arena roundBest run reported from up to three agent frameworks99 (11 tasks, 3 seeds, 3 attempts)
WorldOne machine with 12 slots; agent finds products by web search; starts with $50096 products in 20 categories, 5 suppliers each3,448 variants, 40 suppliers, 18 months of history on a production backend
Exploit hardeningExploits reported after the factNot reportedEach exploit closed, pinned by a regression test, and replayed by an exploiter policy at every calibration

In short, the Vending-Bench line asks how much money an agent ends with in an open-ended world with LLM counterparties. StoreBench asks how well an agent operates relative to measured reference policies, in a world that is deterministic enough to audit and to train against. We view the two as complementary. Vending-Bench Arena studies agent-versus-agent competition, which StoreBench does not; our rivals are scripted, and we say so in Limitations.

Q2 (W4). Standard errors or 95% confidence intervals. You are right that the leaderboard table lacked them; only the post-training result reported an interval. Every row of the leaderboard (Table 2) now has a 95% bootstrap CI over the 33 task-seed cells with 10,000 resamples. For the seven submitted models and the references:

Policy / modelComposite [95% CI]Pass rate [95% CI]
smart-triage (scripted)0.764 [0.758, 0.768]97% [91, 100]
Human expert reference0.708 [0.654, 0.760]39% [24, 58]
DeepSeek-V4-Pro0.700 [0.670, 0.732]49% [39, 60]
Claude Opus 4.80.668 [0.642, 0.691]36% [25, 47]
rule-based (scripted)0.638 [0.577, 0.688]12% [3, 24]
Claude Fable 50.629 [0.586, 0.673]29% [16, 42]
Qwen3.8-Max0.587 [0.546, 0.630]20% [9, 32]
Gemini 3.8 Flash0.552 [0.517, 0.584]15% [8, 23]
GPT-5.6 Sol0.538 [0.502, 0.575]19% [11, 28]
blast-list (scripted)0.463 [0.418, 0.500]0%
Muse Spark 1.20.387 [0.338, 0.448]8% [1, 17]

Your expectation is correct. The intervals of every adjacent pair from DeepSeek-V4-Pro down to GPT-5.6 Sol overlap, and so do those of the human reference and DeepSeek-V4-Pro. The revision no longer orders adjacent models in the text and no longer says that humans outscore every model. What the intervals do support: all seven submitted models sit below smart-triage (the highest upper bound is 0.732 against a lower bound of 0.758), among them, models three or more ranks apart are separated, and Muse Spark 1.2 is separated from the rest. The comparisons the paper relies on are the ones against the anchors. In the extended 13-model evaluation, the top model exceeds smart-triage by 0.071 (95% CI 0.045 to 0.099), and the next two models (0.772 and 0.762 against 0.764) are within the margin of error. The revision calls a difference significant only in cases like these.

(continued in the next comment)

AuthorsOpenReview ↗

Reply to Reviewer J7BH, comment 2 of 3

(continued)

Q3. Pareto frontier of performance against tokens and cost. Per-trial token usage and metered cost are logged for every run, and the configuration appendix summarizes them (Appendix I). The revision adds a cost-performance table for the seven submitted models (Table 8). A short-horizon rollout costs about $1 for GPT-5.6 Sol and Gemini 3.8 Flash, $4.5 for DeepSeek-V4-Pro, $7.5 for Claude Opus 4.8, $15 for Muse Spark 1.2, $28 for Qwen3.8-Max, and $31 for Claude Fable 5. A full-year rollout averages $146, from about $10 (Muse Spark 1.2, DeepSeek-V4-Pro) to $670 (Claude Fable 5). All costs use each provider's published list price. For the five models whose harness did not price their tokens, we apply that list price to the logged token counts. One design note: cost is reported beside the score and is not part of it. Billing tokens inside the score would tie the ranking to provider pricing, which changes over time.

Q4 (W2). Better description of the heuristic policies and tools. Agreed. The appendix lists the 29 tools by family with their operation cost (Appendix B), and the revision adds an appendix that describes all six scripted policies and the pass-threshold rule (Appendix E). In brief:

  • do-nothing never attends; absentee attends every window and never acts.
  • blast-list ships every order it can and makes no other decision.
  • rule-based works through a fixed checklist each window.
  • smart-triage applies hand-written decision rules that prioritize the most urgent obligations and restock needs within the operation budget.
  • exploiter plays every closed reward exploit in the catalogue.

Q5 (W3). Testing each model in its vendor's harness. We agree that the harness matters; the submission's own Discussion calls it "a potential confound, one often left unexplored by public benchmarks". We would respectfully explain why a single pinned harness is still the right primary protocol:

  • Vendor harnesses differ in system prompt, built-in tools, and context management. Running each model in its own would compare model plus scaffold, and a difference could not be assigned to either.
  • First-party agent harnesses are not available for every model in the field, and those that exist differ in maturity, so a per-vendor protocol cannot cover the field uniformly.
  • The short suite uses goose, which is open-source and vendor-neutral. Claude Code was used only for the separately reported full-year task, where all seven models ran under it.
  • Because time is metered in operations, a slower or more verbose harness gains or loses no simulated time. The shell-versus-native observation needs a correction, which the revision makes. In the goose 1.51.0 build we used, the MCP extension failed to load for Claude Opus 4.8, Claude Fable 5, and GPT-5.6 Sol, so those models reached the store only by scripting calls through the shell. Their shell use is a harness artifact, not a model preference, and the revised failure-mode analysis and Discussion say so. The within-model comparison holds only where a model had both interfaces in the same harness: Muse Spark 1.2 averages 0.34 when shell-driven against 0.43 natively, and Gemini 3.8 Flash 0.55 against 0.89.

Q6. How much of the GRPO gain is the task, and how much is familiarity with the harness? Both contribute, and the submission says the gain comes "partly from turn efficiency under the 150-turn cap". Base and tuned policies run under the same harness, but only the tuned one was trained in it, so your question is fair. The composite decomposes exactly, which gives a first answer:

ComponentBaseTunedContribution to +0.237
Business (weight 0.7)0.0050.236+0.162
Reliability (weight 0.2)0.4010.669+0.054
Continuity (weight 0.1)0.5210.741+0.022

Continuity, the term that directly rewards getting further through the episode, accounts for under a tenth of the gain. The tuned policy also does better per obligation: on-time shipping rises from 0.65 to 0.89, returns handled from 0.16 to 0.45, and profitable episodes from 3% to 72%. These rates still benefit from reaching more windows, so the split is not clean. Scoring both policies on their first 33 windows only, the window the base model reaches on average, the gain persists: the business term, computed from net assets at window 33 before clipping, rises by +0.36 (95% CI +0.27 to +0.44), and 30 of 33 cells improve (Appendix L). The evaluation tasks are also disjoint from the training tasks and two to six times longer, so the policy did not learn these scenarios.

(continued in the next comment)

AuthorsOpenReview ↗

Reply to Reviewer J7BH, comment 3 of 3

(continued)

W5. Release. Your suggestion, release part and hold the rest back, matches the structure we chose: a public training split and a private evaluation suite. The supplementary material in Zip (uploaded to Openreview) contains five training-split tasks, ten trajectories with ledgers, the scorer, and a re-grader. The gap you identify is real: the public split cannot be run without the engine. We do not release the engine, because the held-back tasks run on the same demand model, and its source would expose the quantities those tasks test. The evaluation suite and engine stay private to prevent contamination, as in other held-back benchmarks; details are in our reply to mfmg.

Thank you again. We believe the revision addresses each question, and we would be grateful if you would consider updating your assessment.

Reviewer_mfmgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StoreBench evaluates agents that operate an online apparel store. It uses a Medusa commerce backend with simulated demand, suppliers, orders, returns, and market events. Agents have 29 tools and a limited number of operations in each time window. The score combines business results, service reliability, and continuity, with thresholds set using scripted policies. The paper evaluates seven frontier models, provides a human reference, and reports a GRPO training experiment on Qwen3.5-27B.

Critical strengths
  1. The environment includes several practical difficulties: cash constraints, delayed returns, supplier problems, and changing demand. The commerce backend supports detailed operations and obligations that agents need to handle over time.
  2. The reward auditing is useful. The paper gives concrete examples of exploits and explains the fixes. Scripted reference policies and ledger-based grading also help readers understand what the scores measure.
Critical weaknesses
  1. The available trajectories and scorer support some checks on reported scores, but readers cannot run another agent or reproduce the training experiment.
  2. The full-year experiment changes the harness, reasoning settings, context management, operation schedule, and market events together. Scripted shocks are also removed. The higher scores therefore do not tell us how much each change contributes.
  3. The post-training experiment uses one base model and one training run. This is not enough to show that the improvement carries over to a new store or market.
  4. The tuned model reaches more time windows and handles routine obligations better, but rarely reprices or restocks. It passes only 3 of 99 evaluation episodes. The results show better task execution, but provide limited evidence of better business decisions.
  5. The analysis gives few detailed operating trajectories. Cash, inventory, purchasing decisions, and responses to shocks need to be examined together to explain why an agent succeeds or fails.

After rebuttal: The profit results make the GRPO proof of concept more convincing, and the revised framing of the full-year experiment is clearer. I still keep my rating at 2. The training evidence comes from one run in the same simulated store, and the released traces let readers check scores but still do not provide a way to evaluate a new agent.

Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Reply to reviewer mfmg, comment 1 of 2

Thank you for the careful review, and for noting the practical difficulties the environment captures and the value of the reward auditing. We respond to each weakness in order.

W1. Readers cannot run another agent or reproduce the training experiment. This is a deliberate trade-off, and we would love to explain it more clearly than the submission did. StoreBench measures how well an agent operates under hidden dynamics: demand, elasticity, and event scripts are observable only through the store’s own sales. Releasing the engine would make those quantities readable from source, and the benchmark would then measure whether an agent had seen the code, not whether it can operate a store. Keeping the engine and test split private also protects the benchmark from contamination and saturation, in line with the private evaluation sets of ARC-AGI-2 (Chollet et al., 2025) and the unreleased tasks of FrontierCode (Cognition, 2026). Private does not mean unverifiable, as the released material below shows. What we release, under an MIT license, lets readers inspect real runs and verify our scoring:

  • Five training-split tasks with prompts, configurations, and calibration.
  • Ten trajectories, each with its graded terminal ledger and run summary.
  • The composite scorer and a standalone re-grader that reproduces each score from the exported ledger. References
  • Chollet, F., Knoop, M., Kamradt, G., Landers, B., and Pinkard, H. ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems. arXiv:2505.11831, 2025.
  • Cognition. Introducing FrontierCode. Blog post, June 2026. cognition.com/blog/frontier-code

W2. The full-year experiment changes harness, reasoning settings, context management, schedule, and events together. We agree, and the submission takes the same position. The full-year task is reported in its own table and never averaged with the short suite. The table caption states that the delta "contrasts them rather than isolating horizon", and the Discussion states that we "cannot isolate the harness's effect". We make no claim about how much each change contributes. One clarification: the absence of scripted shocks is the definition of the full-year task (it has zero scripted events in the task table), not a change applied on top of another task. Seasonal demand, returns, fines, and carrier availability still run all year. We accept that the framing invited a stronger reading. In the revision:

  • The abstract no longer says models show "dramatic performance improvement"; it says the full-year setting differs in harness and effort and is not comparable with the short suite.
  • The section now leads with what the experiment does support without any cross-setup comparison: two long-horizon failure modes that never appear in short tasks. In 4 of 9 attempts one model waits on a background process that does not exist (about 0.1, against 0.799 when it operates). In 1 of 9 attempts another model declares completion at window 185 of 365.

W3. One base model and one training run do not show carry-over to a new store or market. We agree with this scope, and our Limitations section states it: one base model, one hand-adjusted run, held-out tasks and not held-out seeds. We do not claim transfer to a new store or market. The claim is narrower: the environment's reward supports policy improvement that transfers to scenarios the policy never trained on. Within that scope the evidence is reasonably strong for a proof of concept:

  • The 11 evaluation tasks are disjoint from the 5 training tasks, use different event scripts, and run 30 to 45 days against 7 to 15 in training.
  • The paired gain is +0.237 per cell (bootstrap 95% CI +0.19 to +0.29), and 31 of 33 cells improve.
  • Checkpoint 34, evaluated independently, scores 0.341 and also improves 31 of 33 cells, so the result does not hinge on one checkpoint. The revision labels the section a proof of concept in its opening paragraph (Section 7) and in the abstract.

(continued in the next comment)

AuthorsOpenReview ↗

Reply to reviewer mfmg, comment 2 of 2

(continued)

W4. The tuned model handles obligations better but rarely reprices or restocks, and passes 3 of 99 episodes. We agree, and we report this ourselves: the Discussion says the gains "come primarily from operating discipline rather than commercial strategy". We would only add that the improvement is not confined to task execution. The business component rises from 0.005 to 0.236, and the share of episodes ending at a profit rises from 3% to 72%. The policy reaches better economic outcomes, through execution and not yet through pricing or sourcing judgment. Two points of context. The run is 1,080 episodes with a rank-32 adapter on a 27B model. The pass bar is one that frontier models clear on only 8% to 49% of cells in the submission, so 3 of 99 from a base of 0 of 99 is in line with a small run. The revision states plainly that the experiment shows the reward is learnable and that learning transfers to held-out tasks, and that it does not show a trained policy with good commercial judgment.

W5. Few detailed operating trajectories. We agree that this analysis is valuable, and the data for it already exist: every trial exports per-window balances, inventory, purchase orders, and the event log. The ten released trajectories in the supplementary material carry the same exports, so readers can already plot cash, inventory, and purchasing against events for any of them. Section 6.4 traces one such run, in which the agent places no restock, inventory drains from 1,603 units to zero, and 61 orders are lost. Within this revision's page limit we prioritized the confidence intervals, positioning, and construction detail requested across the reviews. We agree that examining cash, inventory, purchasing, and responses to shocks together is the right lens for explaining success and failure, and the released exports are intended to support exactly that analysis.

We hope these clarifications and revisions address your concerns, and we would be grateful if you would consider them in your final assessment.

AuthorsOpenReview ↗

Review Rebuttal Follow-Up

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.

We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.

We are happy to provide further clarification and look forward to your feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

We thank the three reviewers for their careful and constructive reviews. We are glad the reviewers found the production-grade backend (mfmg, J7BH), the reward-exploit auditing and calibration (mfmg, kCKA), and the post-training experiment (kCKA) to be useful contributions. The concerns converge on four themes, and we have revised the paper on each.

(1) Statistical uncertainty (J7BH, kCKA). Every leaderboard entry (Table 2) now carries a 95% bootstrap confidence interval, resampling the 33 task-seed cells 10,000 times. Scripted policies are deterministic, so their intervals reflect task and seed variation only. The text no longer orders models whose intervals overlap, and calls a difference significant only when its interval excludes zero.

(2) Positioning and construction detail (J7BH, kCKA). Related work now covers Vending-Bench 2, Vending-Bench Arena, and RetailBench, with a side-by-side comparison table in Related Work (Table 1). A new subsection consolidates how the world, the hidden market model, and the tasks were built and verified (Section 3.5), and a new appendix describes all six scripted policies and the pass-threshold rule (Appendix E).

(3) Scope of the full-year and post-training claims (mfmg, J7BH, kCKA). We agree that the full-year result cannot attribute its gains to any one factor and that a single GRPO run is a proof of concept. The submission says both, but in the discussion and appendix. The revision states them in the abstract and the section openings, and removes wording that could be read as a stronger claim.

(4) Release and reproducibility (all reviewers). Every released score is independently verifiable. The anonymized supplementary material, under an MIT license, contains five training-split tasks, ten trajectories with graded ledgers, the composite scorer, and a standalone re-grader that reproduces each score from its ledger. We will also release the code/public set on GitHub in the camera ready version of the paper. The evaluation suite and engine stay private to prevent contamination, as in other held-back benchmarks; details are in our reply to mfmg.

Extended evaluation. Since submission we evaluated six newer frontier models under the unchanged protocol, for 13 in total. The top model now reaches a composite of 0.835 with an 89% pass rate, and is the only model whose lead over smart-triage (0.764) is statistically significant (gap 0.071, 95% CI 0.045 to 0.099). The next two models (0.772 and 0.762) are indistinguishable from smart-triage. We have updated the abstract accordingly: the benchmark still separates models across its full range, and the scripted anchor is a calibrated bar, not a ceiling.

We also add a cost-performance table (J7BH; Table 8). Point-by-point replies follow under each review.

AuthorsOpenReview ↗

Summary of Rebuttal and Additional Evidence for the Area Chair

Dear Area Chair,

Thank you for overseeing our submission. We summarize below how the rebuttal and revision address each reviewer's concerns, to facilitate your assessment.

Two requests were shared across reviews and are resolved in the revision for all three reviewers. Every leaderboard row now carries a 95% bootstrap confidence interval, and the text no longer orders models whose intervals overlap. On release, the supplementary material lets readers verify every released score from its ledger, and we commit to releasing the code and public task set with the camera-ready version, while the evaluation suite stays private to prevent contamination.

Reviewer kCKA. The reviewer's soundness concern rested on undocumented benchmark construction and on a scripted policy that outscores the human reference. We consolidated construction and verification into one subsection and clarified that smart-triage is a calibration anchor and not a ceiling: in the extended 13-model evaluation, the top model significantly exceeds it (0.835 against 0.764, gap 95% CI 0.045 to 0.099). We also explained each mid-run change to the training schedule and showed that an earlier checkpoint, evaluated independently, shows the same held-out gain.

Reviewer mfmg. The reviewer's main concern was that the full-year and post-training experiments claim more than they support. We agree with the scope and had stated it in the discussion and limitations; the revision moves it into the abstract and section openings, describing the full-year setting as not comparable with the short suite and the post-training run as a proof of concept. Within that scope, the held-out gain is +0.237 (95% CI +0.19 to +0.29) with 31 of 33 cells improving. The reviewer also noted that the tuned model improves on obligations more than on commercial judgment, which we report ourselves, and asked for more operating trajectories, which the released per-window exports support.

Reviewer J7BH. The reviewer asked six concrete questions, and each led to an addition in the revision: a comparison table against the Vending-Bench family and RetailBench, a cost-performance table, and an appendix describing all six scripted policies. On testing each model in its vendor's harness, we explained why a single pinned, open-source harness is the right primary protocol, and we disclosed a tool-loading artifact that affected three models. On whether the post-training gain reflects harness familiarity, a new matched-window analysis shows it persists when both policies are scored on the same first 33 windows (+0.36, 95% CI +0.27 to +0.44).

We hope this summary is useful in weighing the original reviews against the revision.

We sincerely appreciate your consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.