- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Authors introduce a novel benchmark that evaluates how well an agent can agent operate a midsized apparel store for 30 to 265 simulated days. The benchmark has exploit resistant evaluation. They show that even strong models reach <49% pass rate.
Critical strengths
- A new long-horizon benchmark with reward-hacking calibrations.
- Calibration for reward-hacking guard.
- Post-trained Qwen3-27B with GRPO to show that the benchmark tasks can be used as RL environments and the post-trained model generalizes on the held-out ID set.
Critical weaknesses
- Not enough info on benchmark sourcing, construction, verification.
- The scripted policy (smart-triage) already beats human baseline.
Questions for authors
- Where the tasks/scenarios/hidden market model are coming from?
- If a scripted policy such as smart-triage already beats human baseline maybe the best approach is just to let the agent to improve the scripted policy?
- Why confidence intervals are not reported if there were multiple attempts/seeds per task?
- Why the learning rate was increased 2x during the GRPO training in the middle of the training run as well as the reward?
- Why GRPO run was using only 5 tasks? Was it intended to overfit?
- Will the benchmark code, data and other artifacts will be released?
4 comments
reply to the review by kCKA 2/3
W2 / Q2. Smart-triage already beats the human baseline; why not let the agent improve the scripted policy? We understand why this reads as a weakness, and would respectfully offer a different interpretation, in four parts. It is a calibration instrument, and it is meant to be strong. Smart-triage was written by the environment's authors with full knowledge of its rules and settlement mechanics. Its role is to fix the scale: the pass threshold on every cell is set from the two best honest scripted policies. A weak reference would make the threshold trivial. It is also deterministic and narrow, scoring between 0.70 and 0.78 on every cell. It is not a ceiling. The scale tops out at 1.0, and the evidence that there is room above the anchor has grown since submission:
- In the extended evaluation, the top frontier model scores 0.835, significantly above smart-triage (gap 0.071, 95% CI 0.045 to 0.099).
- Human participants reach up to 1.00 on individual cells and outscore smart-triage on cold-start (0.794 against 0.741) and slow-drift (0.799 against 0.768).
- On the full-year task, a separate setup, four models score above 0.96. The Limitations section already states that scripted anchors "bound checklist and heuristic play, not optimal play". The human number is a reference, and a conservative one. Participants used an unfamiliar operation-metered console, not the storefront tools they normally use, with a median of 5.1 hours per shift. We disclose the selection procedure in full. The revision renames it "human expert reference" throughout and states that it is not an upper bound on human performance. On improving the scripted policy. This is an interesting direction, and agents can already attempt part of it: the harness gives every agent a shell, and a scripted call costs the same operation as a native one. Where models had both interfaces, those that lean on scripting do worse, because batched scripts act on stale state: one model averages 0.34 when shell-driven against 0.43 natively, another 0.55 against 0.89. (For three models the tools failed to load in our goose build, leaving the shell as their only route to the store; the revision discloses this.) More fundamentally, the agent is not told the scoring rule, the thresholds, or the event scripts. Writing smart-triage required exactly the knowledge that the benchmark asks the agent to infer while operating. A separate policy-synthesis track, in which the agent may iterate on a policy across episodes, would be a valuable extension, and we add it to the Discussion as future work.
Q3. Why are confidence intervals not reported? This was an omission in the leaderboard table; only the post-training result reported one (+0.237, 95% CI +0.19 to +0.29). Every leaderboard row now has a 95% bootstrap CI over the 33 task-seed cells with 10,000 resamples. For example: smart-triage 0.764 [0.758, 0.768], DeepSeek-V4-Pro 0.700 [0.670, 0.732], Claude Opus 4.8 0.668 [0.642, 0.691], and the human reference 0.708 [0.654, 0.760]. Pass rates carry intervals as well (DeepSeek-V4-Pro 49% [39, 60]). Adjacent models often overlap, so the revision no longer orders models whose intervals overlap and calls a difference significant only when its interval excludes zero, as for every submitted model sitting below smart-triage.
reply to the review by kCKA 3/3
Q4. Why were the learning rate and the reward changed mid-run? Both changes are disclosed in the post-training appendix, which the revision expands with the explanation below (Appendix L). Reward. Steps 1 to 4 used the official composite and produced no learning signal. The base model almost never trades at a profit, so its business term was clipped to 0 in essentially every rollout. Rollouts within a group then scored alike and their advantages vanished. From step 5 the training reward clips normalized business to [-2, 1] in place of [0, 1], which ranks losing stores against each other. It is the same quantity with a wider clip. Every reported number uses the official composite, on held-out tasks. Learning rate. From the step-15 checkpoint we ran two branches as a learning-rate comparison, at 5e-5 and 1e-4. At step 21 we also cut the schedule from ten optimizer updates per step to one. The ten-update schedule let the trainer's weights drift from the sampling policy and collapsed the 5e-5 branch. With compute for one branch, we continued the 1e-4 branch on the one-update schedule. We returned it to 5e-5 at step 27, when policy entropy kept rising while training reward stayed flat. We agree that a hand-adjusted schedule is not a training recipe, and the Limitations section says the study shows "that the environment supports hill-climbing with transfer to unseen scenarios, not a tuned training recipe". Two facts bound the concern. Every adjustment used training-batch statistics only: reward, entropy, and the importance ratio. And we report the final checkpoint, step 36. We evaluated checkpoints 32, 34, and 36 on the held-out tasks, and all three show the gain: checkpoint 34 scores 0.341 with 31 of 33 cells improved, and a partial evaluation of checkpoint 32 agrees.
Q5. Why only five training tasks? Was it intended to overfit? No; the design tests the opposite. The five training tasks are disjoint from the 11 evaluation tasks, have different event scripts, and run 7 to 15 days against 30 to 45 in evaluation. The small set reflects rollout cost: each episode is a long multi-turn interaction, and the run totals 1,080 episodes. If the policy had overfit to the training tasks, held-out scores would not move. They rise on 31 of 33 cells. One caveat, which we state in Limitations: the evaluation uses held-out tasks on the same three world seeds, not held-out seeds.
Q6. Will the code, data, and artifacts be released? We will release the code and public sets in the final camera ready version. The anonymized supplementary material already contains five training-split tasks with prompts, configurations, and calibration, ten trajectories with graded ledgers, the scorer, and a re-grader that reproduces each score, under an MIT license. The evaluation suite and its event scripts stay private so the benchmark remains a held-back test set. Thank you again for the close reading. We hope the consolidated construction section and the clarified role of the scripted anchors resolve your soundness concerns, and we would be grateful if you would reconsider your scores in that light.
Thank you for addressing my concerns and for the open source commitment.
reply to the review by kCKA 1/3
Thank you for the direct questions, and for recognizing the reward-hacking calibration and the post-training experiment. Your soundness concern rests on two points: construction detail, and a scripted policy that outscores the human reference.
W1 / Q1. Where do the tasks, scenarios, and hidden market model come from, and how were they verified? The information was in the submission but scattered across two sections and four appendices. The revision consolidates it into one subsection, "Construction and verification" (Section 3.5). In summary: Sourcing and construction.
(continued in the next comment)