- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a framework for building decision-making environments that combine LLM-generated transitions with rules implemented in code. It introduces three environments, evaluates eleven models, and assesses simulator validity through human judgments, historical data, and comparison with an existing simulator.
Critical strengths
The controlled replacement of coded dynamics in YC Bench is a useful attempt to test whether LLM generated transitions preserve conclusions about agent performance.
Critical weaknesses
The paper does not establish a substantial advance over existing work. Using an LLM to interpret actions and determine their consequences is already central to Concordia, while LLM-assisted simulator construction has been explored in WorldCoder and FactorSim. The contribution here is therefore the particular combination of these ideas, its construction guidelines, and the new environments. However, the paper does not show that its guidelines or architecture produce more reliable or useful environments than existing approaches.
The experiments do not establish generalization of the proposed framework. Evaluating eleven models in three environments provides results within those environments; it does not show that the construction method reliably transfers to new decision problems. The paper does not identify the conditions under which LLM-based dynamics can be trusted, or demonstrate that its validation procedure predicts reliability in environments not used to develop the framework. The concern is not simply the number of environments, but the absence of a tested principle that extends beyond them. The startup scoring exploit further illustrates why plausible simulations are not necessarily valid measures of decision quality. Ranking agreement in YC-Bench is useful evidence, but does not resolve this broader concern.
The paper demonstrates that these environments can be built and used. It does not sufficiently establish what generalizable knowledge or reliable evaluation capability they add. I therefore do not find the contribution sufficient for acceptance.
Questions for authors
What demonstrated capability does the proposed framework add beyond existing LLM-based simulation and environment-construction approaches? Which reported experiment isolates that advantage?
What evidence shows that the benchmark rewards better decisions rather than strategies that exploit its particular dynamics and scoring rules?
3 comments
Response to Reviewer YVpS (2/2)
(W2) […] it does not show that the construction method reliably transfers to new decision problems.
(W3) The paper does not identify the conditions under which LLM-based dynamics can be trusted, or demonstrate that its validation procedure predicts reliability […]
(W4, part 1) The concern is not simply the number of environments, but the absence of a tested principle that extends beyond them.
Our framework rests on 2 principles, and we test both.
(1) Environment design: We use (1) typed state and action representations, and (2) hybrid dynamics in which an LLM judges fuzzy changes and code handles rule-based ones. Ours outperforms the baselines in W1, and reproduces YC-Bench's coded simulator: agent rankings correlate at , vs. between 2 runs of the original.
(2) Calibration and validation. We calibrate each environment on development data (human judgments and real-world outcomes), then validate it on held-out data. We designed this procedure before building the environments and applied it to each. In our new Startup validation experiment, we replay the decisions of 12 held-out test startups: simulated final valuations rank the companies at Spearman against reality. On held-out test NFL coaching tenures, the environment's per-tenure title error () is close to that of preseason betting lines (, an oracle baseline that knows each season's real roster, while our replay knows only the first season's. We note that it’s impossible to measure true fidelity over all possible decisions because we cannot observe counterfactuals in the real world.
Transfer to new problems. We showed that both principles of our framework generalize by using it to build 3 environments for different fuzzy decision problems, Career, Startup, and NFL (Section 4, Table 1). A third-party tested our builder and framework (new) to create an AI safety policymaking environment that simulates a staffer working a fictional state bill through a 2-year session (Appendix F). In about 60 minutes, with expert input on 15 decisions, it produced 32 state variables, 17 action parameters, 23 initial states, and dynamics calibration cases.
(Q2) What evidence shows that the benchmark rewards better decisions rather than strategies that exploit its […] scoring rules?
(W4, part 2) The startup scoring exploit […]
We show 3 pieces of evidence.
- Humans agree with the environment's ranking. 5 human raters compared pairs of Career action sequences, with utilities and outcomes hidden, and picked the one that should score higher utility. They agreed with the environment in 82.7% of comparisons.
- The environments rank real-world outcomes correctly on held-out startups and NFL tenures (W3).
- Privileged search stress-tests each environment. We ran several rounds of privileged search to find exploits and fix them (Section 4.1). In Startup, it found that in hot financing climates, AI startups with low revenue can reach very high valuations. Re-running with founders' stake or revenue based utilities keeps the model ranking stable with
Thank you again for your review. As the discussion period ends soon, we wanted to ask whether our response addressed your concerns, or if there's anything else we can clarify.
Response to Reviewer YVpS (1/2)
Thank you for your review. We address your concerns below.
Our framework differs from prior work in 3 ways.
(1) Goal. We evaluate open-ended, long-term decisions, such as which career to pursue or which markets a business should target. Prior work like Concordia (Vezhnevets et al., 2023) simulates social interactions, and WorldCoder (Tang et al., 2024) and FactorSim (Sun et al., 2024) generate code simulators for games and gridworlds, all unrealistic settings with known rules. We simulate parts of the real world, where no ground-truth simulator exists and how the world works is complex and unknown. This makes our environments much harder to design and validate.
(2) Design (Sections 3.1, 3.2, 4.1) and (3) validation against held-out data (Section 6), detailed under W3.
We add 2 new experiments that show the advantage of LLM-as-Environment:
4.4× lower error than Concordia. On YC-Bench, where the coded simulator provides ground truth, we replay 24 fixed action sequences with LLM-as-Environment vs Concordia. We implement Concordia by doing two game-master calls to resolve the structured YC-Bench command and describe the resulting event, followed by a third call that uses the event, original state, and command to predict structured state updates. We hold the dynamics LLM, state and action space, and agent interface fixed. Mean relative final-state error is 0.00319 for ours vs 0.0141 for Concordia.
LLM-only simulator. We had 5 human raters compare trajectories from our Career Planning environment and from an LLM simulator that simulates outcome of four years in one call. They preferred ours in 90 of comparisons. To check that raters could tell simulators apart, we planted errors in one trajectory of a test pair, and every rater marked it as less realistic.