acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Open-Ended Decision-Making with LLM-as-Environment

Abstract

We study how to evaluate open-ended decision-making. Because outcomes may take years to become clear and alternative choices cannot be observed, evaluating these decisions requires simulating how different choices shape future outcomes. We introduce , a framework for constructing environments for this purpose. Our key idea is to use an LLM as part of the environment to simulate changes that are difficult to specify with fixed rules, while code handles calculations, constraints, and state updates. We also develop design principles and skill files that guide coding agents in constructing, calibrating, and stress-testing these environments. Using this framework, we build , with environments for undergraduate career planning, startup strategy, and NFL head coaching. We validate the environments using human judgments, historical data, and a controlled comparison with an existing rule-based simulator. Replacing rule-based transitions of a programmatic startup simulator, YC-Bench, with LLM-predicted transitions preserves model rankings across eight models (Spearman ), compared with between independent runs. We then evaluate 11 frontier language models across all three environments. Even the strongest models close only – of the utility gap between a scripted baseline and a privileged search reference, leaving substantial room for improvement. Advances in LLM capabilities enable a new approach to building decision-making environments: LLMs can help construct them and also serve as part of the environment itself. We release the environments, evaluation harness, and builder so that researchers can refine these environments and construct new ones for other decision problems.

Primary area:
datasets and benchmarks
Keywords:
language models, strategic decision-making, agent evaluation, simulation, POMDPs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YVpSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a framework for building decision-making environments that combine LLM-generated transitions with rules implemented in code. It introduces three environments, evaluates eleven models, and assesses simulator validity through human judgments, historical data, and comparison with an existing simulator.

Critical strengths

The controlled replacement of coded dynamics in YC Bench is a useful attempt to test whether LLM generated transitions preserve conclusions about agent performance.

Critical weaknesses

The paper does not establish a substantial advance over existing work. Using an LLM to interpret actions and determine their consequences is already central to Concordia, while LLM-assisted simulator construction has been explored in WorldCoder and FactorSim. The contribution here is therefore the particular combination of these ideas, its construction guidelines, and the new environments. However, the paper does not show that its guidelines or architecture produce more reliable or useful environments than existing approaches.

The experiments do not establish generalization of the proposed framework. Evaluating eleven models in three environments provides results within those environments; it does not show that the construction method reliably transfers to new decision problems. The paper does not identify the conditions under which LLM-based dynamics can be trusted, or demonstrate that its validation procedure predicts reliability in environments not used to develop the framework. The concern is not simply the number of environments, but the absence of a tested principle that extends beyond them. The startup scoring exploit further illustrates why plausible simulations are not necessarily valid measures of decision quality. Ranking agreement in YC-Bench is useful evidence, but does not resolve this broader concern.

The paper demonstrates that these environments can be built and used. It does not sufficiently establish what generalizable knowledge or reliable evaluation capability they add. I therefore do not find the contribution sufficient for acceptance.

Questions for authors

What demonstrated capability does the proposed framework add beyond existing LLM-based simulation and environment-construction approaches? Which reported experiment isolates that advantage?

What evidence shows that the benchmark rewards better decisions rather than strategies that exploit its particular dynamics and scoring rules?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YVpS (1/2)

Thank you for your review. We address your concerns below.

(W1) The paper does not establish a substantial advance over existing work. […] (Q1) What demonstrated capability does the proposed framework add […]? Which reported experiment isolates that advantage?

Our framework differs from prior work in 3 ways.

(1) Goal. We evaluate open-ended, long-term decisions, such as which career to pursue or which markets a business should target. Prior work like Concordia (Vezhnevets et al., 2023) simulates social interactions, and WorldCoder (Tang et al., 2024) and FactorSim (Sun et al., 2024) generate code simulators for games and gridworlds, all unrealistic settings with known rules. We simulate parts of the real world, where no ground-truth simulator exists and how the world works is complex and unknown. This makes our environments much harder to design and validate.

(2) Design (Sections 3.1, 3.2, 4.1) and (3) validation against held-out data (Section 6), detailed under W3.

We add 2 new experiments that show the advantage of LLM-as-Environment:

  1. 4.4× lower error than Concordia. On YC-Bench, where the coded simulator provides ground truth, we replay 24 fixed action sequences with LLM-as-Environment vs Concordia. We implement Concordia by doing two game-master calls to resolve the structured YC-Bench command and describe the resulting event, followed by a third call that uses the event, original state, and command to predict structured state updates. We hold the dynamics LLM, state and action space, and agent interface fixed. Mean relative final-state error is 0.003190.00319 for ours vs 0.01410.0141 for Concordia.

  2. LLM-only simulator. We had 5 human raters compare trajectories from our Career Planning environment and from an LLM simulator that simulates outcome of four years in one call. They preferred ours in 9090% of comparisons. To check that raters could tell simulators apart, we planted errors in one trajectory of a test pair, and every rater marked it as less realistic.

AuthorsOpenReview ↗

Response to Reviewer YVpS (2/2)

(W2) […] it does not show that the construction method reliably transfers to new decision problems.

(W3) The paper does not identify the conditions under which LLM-based dynamics can be trusted, or demonstrate that its validation procedure predicts reliability […]

(W4, part 1) The concern is not simply the number of environments, but the absence of a tested principle that extends beyond them.

Our framework rests on 2 principles, and we test both.

(1) Environment design: We use (1) typed state and action representations, and (2) hybrid dynamics in which an LLM judges fuzzy changes and code handles rule-based ones. Ours outperforms the baselines in W1, and reproduces YC-Bench's coded simulator: agent rankings correlate at ρ=0.857\rho=0.857, vs. 0.7380.738 between 2 runs of the original.

(2) Calibration and validation. We calibrate each environment on development data (human judgments and real-world outcomes), then validate it on held-out data. We designed this procedure before building the environments and applied it to each. In our new Startup validation experiment, we replay the decisions of 12 held-out test startups: simulated final valuations rank the companies at Spearman ρ=0.93\rho=0.93 against reality. On held-out test NFL coaching tenures, the environment's per-tenure title error (2.882.88) is close to that of preseason betting lines (2.14)2.14), an oracle baseline that knows each season's real roster, while our replay knows only the first season's. We note that it’s impossible to measure true fidelity over all possible decisions because we cannot observe counterfactuals in the real world.

Transfer to new problems. We showed that both principles of our framework generalize by using it to build 3 environments for different fuzzy decision problems, Career, Startup, and NFL (Section 4, Table 1). A third-party tested our builder and framework (new) to create an AI safety policymaking environment that simulates a staffer working a fictional state bill through a 2-year session (Appendix F). In about 60 minutes, with expert input on 15 decisions, it produced 32 state variables, 17 action parameters, 23 initial states, and dynamics calibration cases.

(Q2) What evidence shows that the benchmark rewards better decisions rather than strategies that exploit its […] scoring rules?

(W4, part 2) The startup scoring exploit […]

We show 3 pieces of evidence.

  1. Humans agree with the environment's ranking. 5 human raters compared pairs of Career action sequences, with utilities and outcomes hidden, and picked the one that should score higher utility. They agreed with the environment in 82.7% of comparisons.
  2. The environments rank real-world outcomes correctly on held-out startups and NFL tenures (W3).
  3. Privileged search stress-tests each environment. We ran several rounds of privileged search to find exploits and fix them (Section 4.1). In Startup, it found that in hot financing climates, AI startups with low revenue can reach very high valuations. Re-running with founders' stake or revenue based utilities keeps the model ranking stable with ρ=0.87–0.90.\rho=0.87–0.90.

Thank you again for your review. As the discussion period ends soon, we wanted to ask whether our response addressed your concerns, or if there's anything else we can clarify.

Reviewer_YmXGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes LLM-as-Environment, using an LLM as part of the environment to simulate changes that are hard to specify with fixed rules. It provides a framework that combines LLM transition predictions with a programmatic engine, along with design principles and skill files for constructing such environments. It also uses the framework to construct FuzzyDecisionBench, comprising three decision-problem environments. The environments are validated through comparisons with hardcoded simulators, human evaluations, and historical data.

Critical strengths
  1. The paper addresses a practically relevant evaluation challenge involving long-horizon choices whose consequences are hard to observe and effects are difficult to formalize. Using an LLM for the qualitative parts of the environment while keeping exact mechanics in code is a useful methodological contribution that provides a practical construction path for evaluation environments in settings where fixed rules are insufficient.
  2. The experimental design is multi-faceted, which strengthens the credibility of the method. The benchmark spans three environments and combines three complementary validation strategies: a transition-replacement study on YC-Bench, human plausibility judgments, and historical replay. Together, these provide multiple lines of evidence for the credibility of the LLM-based simulation approach and its environments, making the benchmark's conclusions more credible than a single validation check would.
Critical weaknesses
  1. The paper does not provide a convincing argument for the advantage of its approach over existing LLM-based methods for simulating open-ended environments, such as Concordia [1], where an LLM game master interprets actions and generates consequences. Although the paper provides multi-faceted internal validation of fidelity and stability, none of this evidence establishes whether the proposed method is more effective than these alternatives in terms of simulation fidelity, stability, or evaluation validity. As a result, the framework's effectiveness relative to prior work remains undemonstrated, which directly affects the assessment of the paper's contribution.
  2. The presentation would benefit from greater clarity and conciseness. The paper also contains numerous formatting issues.

[1] Vezhnevets A S, Agapiou J P, Aharon A, et al. Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia[J]. arXiv preprint arXiv:2312.03664, 2023.

Questions for authors
  1. In the College Career Planning environment, one action is taken per semester as a single time step. Is this time span too large to realistically simulate college students' decision-making?
  2. Since you have already used LLMs to implement part of the environmental change increments, why do you still use a programmatic engine for part of the state transitions? Have you tried an experiment in which state changes are implemented entirely with LLMs?
  3. Considering that this is a simulation of open-ended events, have you considered incorporating low-probability events, such as a sudden shift in industry trends in the College Career Planning environment, to make the simulated environment more realistic?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YmXG

Thank you for your review! We address your concerns below, and have revised our paper in response to your comments.

(W1) Prior works such as Concordia also use LLMs to simulate environment.

Concordia simulates social interactions among many generative agents, such as its phone simulator. We build environments that simulate how the world responds to real-life decision-making scenarios that are hard to evaluate. Each environment has a typed state and a fixed utility, so different agents are scored and ranked on the same terms.

(W1) None of this evidence establishes whether the proposed method is more effective than these alternatives in terms of simulation fidelity, stability, or evaluation validity.

We compare LLM-as-Environment with Concordia (Vezhnevets et al., 2023) on YC-Bench, where the coded simulator provides ground truth. In Concordia, an adapter turns each YC-Bench action into natural language, a game master LLM narrates the outcome, and a second call extracts the resulting state. LLM-as-Environment passes the typed state and action directly, and the LLM returns typed changes. Both use the same LLM and description of YC-Bench's rules. On the 24 fixed action sequences from Section 6.1 (8 agents, 3 seeds each), mean relative final-state error against the coded simulator is 0.00319 for LLM-as-Environment vs. 0.0141 for Concordia, 4.4× lower.

(W2) Would benefit from better presentation.

Thank you for raising this. We've cleaned up the presentation and formatting in Section 2, Tables 1–3, Figures 1 and 3, and the appendix.

(Q1) In the College Career Planning environment, one action is taken per semester as a single time step. Is this time span too large […]?

One step is one academic quarter, not a semester. All of our environments support 2 types of actions: (1) discrete decisions made once per step (a college quarter, NFL season, or startup month), such as applying to an internship or raising funding, and (2) allocations that describe what happens continuously during the step, such as dividing weekly hours among coursework, research, projects, career preparation, networking, sleep, and socializing. Most career-relevant decisions happen about once per quarter, such as choosing courses, joining a lab, or applying for internships. We chose the longest time step that preserves these tradeoffs, while the weekly time allocation captures how the student spends each week.

(Q2) […] why do you still use a programmatic engine for part of the state transitions? Have you tried an experiment in which state changes are implemented entirely with LLMs?

We use code to update state variables that follow fixed rules or equations, such as cash or founder dilution in Startup, for 2 reasons:

(i) there is no need for an LLM to do this arithmetic, and small LLM calculation errors compound over time, which makes utility and decision scoring inaccurate.

(ii) code shortens the dynamics LLM's context and saves tokens and API cost.

Yes. In College Career Planning (updated environment), we tested an LLM-only simulator in which each quarter is one LLM call that updates the state, with no code engine. Its small calculation errors build up over the trajectory. For example, degree progress, which the engine computes from credited units, was off by up to 21 % (about 37 of 180 units) in a single quarter. In some trajectories the student never graduated despite completing enough units, which likely lowered the utility: the same trajectory scored 0.310.31 under an LLM-only simulator and 0.620.62 under the other. Human raters also preferred our environment's trajectories over an LLM-only simulator in 90 % of comparisons (App. B.7).

(Q3) have you considered incorporating low-probability events, such as a sudden shift in industry trends […]?

Yes. Our environments simulate exogenous state variables, such as the financing climate or job market. In College Career Planning, variables such as tech hiring demand and research funding are sampled at the start of each episode and held fixed for the 4 years. Startup Strategy changes them during the episode: the engine moves the financing climate and sector heat with a seeded random drift, and each month the dynamics LLM judges whether an outside event begins, such as a competitor's launch or fundraise, a supplier change, or a technology shift.

Thank you again for your review. As the discussion period ends soon, we wanted to ask whether our response addressed your concerns, or if there's anything else we can clarify.

Thank you for the clarification. Some key concerns remain unresolved.

W1. The authors’ clarification of the key differences from Concordia does not address my concerns. I remain inclined to view this work as an extension of Concordia. Moreover, the additional experiments do not provide convincing evidence of the proposed method’s advantages in fidelity and stability.

W2. The presentation remains unclear in places. For example, the arrows in Fig. 1(a) are confusing, and the text annotation in Fig. 1(b) overlap with the plotted lines.

Given these unresolved issues, I will maintain my original score.

Reviewer_dcMcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LLM-as-Environment, a framework for constructing and evaluating long-horizon decision environments in which an LLM predicts the "fuzzy" consequences of actions—effects that resist fixed rules—while a programmatic engine enforces exact constraints, state updates, and seeded draws. The transition is a three-stage pipeline: code pre-processes the action, the LLM predicts typed changes to the variables it owns, and code validates, clamps, and applies rule-based consequences. Using this framework, the authors build FuzzyDecisionBench, a suite of three environments: College Career Planning (15 quarters, advisor role), Startup Strategy (36 months, founder/CEO), and NFL Team Management (up to 10 seasons, head coach). They evaluate 11 frontier models, normalizing utility between a hand-coded scripted policy and a privileged simulator-aware search. The strongest models close 55.6%, 25.1%, and 26.5% of the reference gap in the three environments, respectively.

Critical strengths
  1. The core observation—that many real decision problems have important effects that resist fixed rules, and that LLMs can fill this gap while code enforces exact mechanics—is both practically compelling and conceptually clean.
  2. College Career Planning, Startup Strategy, and NFL Team Management differ substantially in horizon, state complexity, action types, and utility. Table 1 documents the differences clearly (78/36/25 state variables; 15 quarters / 10 seasons / 36 months). The environments are not toy problems: the startup environment alone has 25 structured variables with about 190 fields, and the NFL environment has 36 variables with seeded roster draws.
  3. The paper does not rely on a single validation strategy. The YC-Bench replacement study is a particularly good idea: by replacing a known coded simulator with LLM dynamics and checking rank preservation, the authors provide a stress test that is more informative than plausibility judgments alone.
Critical weaknesses
  1. The Startup Strategy benchmark has an identified exploit that the paper does not fix. The paper explicitly states: "utility uses the latest priced-round valuation without charging for founder dilution, so repeated fundraising can dominate realized revenue." The privileged search exploits this (nine financings, $1.6B valuation, $3.54M revenue), and the model rankings are visibly driven by valuation rather than operating progress (Claude Fable 5.1 at $190.2M with 25.1%, GPT-6 Astra at $111.6M with 21.0%, Claude Opus 5.5 at $529.2M with 19.3%). A benchmark with a known exploit that the reference policy exploits is not measuring what it claims to measure. The authors should either fix the utility (e.g., charge for dilution, weight revenue) or explicitly scope the startup results as "financing ability" rather than "company-building ability."
  2. The "fuzzy" motivation is plausible but not rigorously established. The core claim is that these environments require LLM dynamics because their effects "resist fixed rules." The YC-Bench replacement study is suggestive, but it shows that LLM dynamics can substitute for some coded dynamics, not that the FuzzyDecisionBench environments could not be modeled by a carefully designed stochastic simulator. A stronger argument would compare against a hand-coded stochastic baseline for at least one environment and show that it either fails to capture the relevant effects or requires disproportionate engineering effort.
  3. Compute cost is not reported. Evaluating 11 models × three environments × multiple states × multiple seeds, with LLM dynamics calls at every transition, is expensive. The paper does not report token or dollar cost, which is a practical barrier to adoption and a missing piece of the benchmark's usability story.
  4. Small human sample. Two raters for plausibility, four raters for the generated-game preference study in the related work (not this paper, but relevant context). The human validation is thin relative to the scale of the benchmark.
Questions for authors

Leakage. Can you provide a fully held-out validation split—e.g., fresh companies or coaching tenures never used in development—and report the validation results on that split? This would substantially strengthen the claim that the validation generalizes.

Startup utility. Given that the utility has an identified exploit that the privileged search exploits, can you report results with a dilution-adjusted or revenue-weighted utility, and show whether the model rankings change? If the rankings are stable, the exploit is less damaging; if not, the startup results should be scoped accordingly.

Human reliability. With κ = 0.185/0.089/0.000, how should readers interpret the LLM judges' 89–94% agreement with humans? Would a larger rater pool, or a different agreement metric (e.g., Krippendorff's α with ordinal labels), change the picture?

Fuzzy motivation. Could a carefully designed stochastic simulator with hand-coded rules capture the effects in at least one of these environments? If so, what does the LLM dynamics add beyond convenience?

Breadth. Do you plan to add environments in other domains (clinical, policy, scientific project selection)? What would be required to extend the framework?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dcMc (1/2)

Thank you for your review. We address your concerns below, and have revised our paper in response to your comments.

(W1) Startup benchmark has an exploit.

(Q2) Startup utility. Can you report results with a dilution-adjusted or revenue-weighted utility, and show whether the model rankings change?

We first note that the startup environment privileged upper bound you're referring to is not that far off from reality. The evaluation case is a hot financing climate for AI research startups, in which the privileged search raises $ 808M over nine financings and reaches a $ 1.6B valuation with $ 3.54M in revenue. It's not unusual nowadays to raise at high valuations with low revenue. The exploit mainly affects the approximate oracle upper bound (privileged search) used to normalize all models' utilities, not the model rankings.

We re-scored the same trajectories with two different utilities, without running new simulations: (1) Founder Stake, which accounts for dilution (terminal valuation × founders' ownership), and (2) Revenue (10 × terminal sustained ARR). Model rankings remain similar, with high rank correlations: ρ=0.90\rho=0.90 for Founder Stake and ρ=0.87\rho=0.87 for Revenue. We added these results to Section 5.3 and Appendix C.4 in our revision, and report both final valuation and founders’ stake utilities.

AgentValuation (old)Founder StakeRevenue
Claude Fable 5.125.1%15.9%26.5%
GPT-6 Astra21.0%15.7%22.3%
Claude Opus 5.519.3%13.4%44.5%
GPT-5.6 Sol15.7%17.7%27.4%
Kimi K38.5%6.4%25.8%
Gemini 3.8 Flash8.5%8.9%3.0%
DeepSeek V4 Pro7.9%3.8%12.9%
GPT-5.6 Luna7.3%5.2%3.6%
Claude Sonnet 53.2%4.5%2.7%
GLM-5.33.0%1.0%1.3%
Qwen3.8-Max0.3%1.3%-1.0%
Spearman with original ranking--0.900.87

(W2) YC-Bench shows that LLM can simulate hard-coded dynamics but not fuzzy decisions. Build a hard-coded stochastic simulator for at least 1 environment and compare it with the current LLM-as-environment one. Show that it either fails to capture the relevant effects or requires disproportionate engineering effort.

(Q4) Fuzzy motivation. Could a carefully designed stochastic simulator with hand-coded rules capture the effects in at least one of these environments? If so, what does the LLM dynamics add beyond convenience?

A hand-coded stochastic simulator can capture the effects that its designer writes rules for. Using LLM dynamics allows the agent to submit “open-ended” decisions via free-form text because the LLM dynamics can interpret it.

For example, in the Startup environment the agent can write, in natural language, its startup idea, business plan, and pitch to investors. This is not supported in existing long-horizon benchmarks that test startup or business strategy, such as Vending-Bench (Backlund & Petersson, 2025), EcoGym (Hu et al., 2026), YC-Bench (He et al., 2026), and CEO-Bench (Chen et al., 2026).

(W3) Compute cost.

We now report cost in Table 1 and Appendix A.4. Evaluating one model on all scored states of all 3 environments with 3 seeds each including the dynamics LLM, costs $ 258 -- $ 741 at list API prices.

Compared with other long-horizon agent benchmarks, this is equivalent to a few Vending-Bench 2 runs, and far less than a full PaperBench evaluation. The dynamics LLM accounts for $ 160 -- $ 236 of our cost; the rest depends on the price of the model being evaluated.

BenchmarkAPI costScope
Vending-Bench 2~$ 100 –- $ 1,500+ per run, depending on model
PaperBench$ 400 per task + $ 66 grading
PaperBench, full evaluation$ 9,320
Ours: Claude Sonnet 5$ 258
Ours: Claude Opus 5.5$ 408
Ours: Claude Fable 5.1$ 741
AuthorsOpenReview ↗

Response to Reviewer dcMc (2/2)

(Q1) Can you provide a fully held-out validation split—e.g., fresh companies or coaching tenures never used in development—and report the validation results on that split? This would substantially strengthen the claim that the validation generalizes.

Thank you for raising this. Our wording in the submission was imprecise. To clarify, we kept the dev and test sets separate and did not use the test set to calibrate the environment. However, some figures show dev and test results together in the same table or plot.

We've revised the paper to report the test set in separate plots and also validate on more test cases for Startup (Appendix C.6 and D.5).

For Startup, we test on 12 held-out startups, using an updated version of the environment. Our environment's simulated final valuations correspond with reality with Spearman coefficient ρ=0.93\rho = 0.93 ρ=0.82\rho=0.82 for the 7 startups founded before the dynamics model's knowledge cutoff (GPT-5.1, September 2024), and ρ=1.00\rho=1.00 for the 5 startups founded afterwards. For final ARR, ρ=0.81\rho=0.81. The environment underpredicts final valuation and ARR by 0.410.41 and 0.170.17, respectively.

For NFL, we replay 10 real head-coaching tenures (73 seasons), 5 for development and 5 held out for testing, and report the test tenures separately (Appendix D.5). We primarily care that our environments distinguishes between better and worse decisions. Therefore we measure ranking correlation with reality. Our environment predicts tenure rankings with ρ=0.825\rho=0.825. Among the test tenures, our environment correctly gives 0 titles to both coaches who were fired. The environment predicts 3.763.76 Super Bowl titles in total (ground-truth is 4). Total error on test is 2.882.88. For reference, an approximate oracle upper bound, which are preseason betting lines in which expert NFL forecasters see additional information that our environment doesn’t (i.e. the real roster at every season/time step) gets 2.142.14 error.

(W4/Q3) Human reliability. Use more human raters. For context, a related work on LLM-generated games uses 4 raters. With κ = 0.185/0.089/0.000, how should readers interpret the LLM judges' 89–94% agreement with humans? Would a larger rater pool, or a different agreement metric (e.g., Krippendorff's α with ordinal labels), change the picture?

The low values come from label imbalance because raters mostly select “correct” for ratings. When one label dominates, Cohen's is low even when raw agreement is high (the "kappa paradox" per Feinstein & Cicchetti, 1990). Krippendorff's faces the same problem, because it corrects for chance using the label marginals. We thus report Gwet’s AC1, which adjusts for label imbalance and is suited for settings like ours.

We also expanded the human evals to 5 expert raters (i.e. are current or former undergraduates at the university modeled in Career Planning).

Fidelity. Humans rate 94.7% of direction changes correct, 84.9% of magnitudes about right, and 91.7% of trajectories coherent. Agreement on direction between human raters is AC1 = 0.905.

Policy ranking. Raters see pairs of action sequences, with utilities and simulated states hidden, and pick the one that better serves the student's objective. They agree with the environment's ranking in 82.7% of comparisons and with each other in 77.1% (AC1 = 0.745).

(Q5) Breadth. Do you plan to add environments in other domains (clinical, policy, scientific project selection)? What would be required to extend the framework?

Yes. We will open-source the environment builder so experts can build new decision environments for their domains. The builder drafts the state, actions, utility, starting states, dynamics and eval cases, and asks the expert for input on objective, environment calibration, and environment scope. As a test, a third party (a researcher outside of the project) used it to build an AI-safety policymaking environment (Appendix F): a staffer working a fictional state bill through a two-year session. In about 60 minutes with expert input on 15 decisions, the builder produced a validated environment: 32 state factors (11 LLM-judged, 21 engine-owned or derived), 17 action parameters, 23 starting states and a 20-case dynamics eval pack. It flagged 15 design decisions that need an expert, such as how to weigh a passed but weakened bill against a vetoed strong one.

Reviewer_dcMcOpenReview ↗

Response to Authors

Thank you for the thorough and constructive rebuttal. The robustness check demonstrating that model rankings remain highly stable (ρ = 0.87–0.90) under dilution-adjusted and revenue-weighted utilities effectively resolves my concern regarding the startup exploit. I also find the clarification that LLM dynamics are fundamentally necessary to process free-form, open-ended text inputs (which hard-coded simulators cannot parse) to be a highly compelling justification for the framework. Finally, the expansion of the human evaluation using domain experts and Gwet's AC1 to address the kappa paradox, the reporting of compute costs, and the addition of a fully held-out validation set substantially strengthen the paper's empirical foundation. I am satisfied with the revisions and am upgrading my score to a 3 (Weak Acceptance).

Thank you for engaging with our rebuttal! We appreciate your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their feedback. We're glad that reviewers found our approach practical and clean (dcMC, YmXG) and our environments diverse, interesting, and not toy (dcMC). They also noted that we validate in several complementary ways (dcMC, YmXG), and that the LLM-dynamics experiment on YC-Bench is a useful stress test (dcMC, YVpS, YmXG).

We respond to the main concerns below, and have revised our paper accordingly (changes in blue).

1. LLM-as-Environment contributions and comparison with prior work (YVpS, YmXG). Our goal is to evaluate long-term, open-ended decision-making. Because we cannot observe counterfactuals in the real world, we evaluate with environments that simulate how the world responds to real-life decisions. Our framework rests on 2 principles:

(i) Careful environment construction that uses an LLM as part of the dynamics to predict fuzzy state changes, which lets us use detailed, flexible state and action spaces, including free-text actions that the LLM dynamics can read.

(ii) Calibration and validation against real-world data, including real decisions that people made and their outcomes (historical replay in Startup and NFL), and judgments by human and LLM raters (Career and Startup).

This is different from prior work on LLM-based simulators, which target synthetic settings and do not validate against real-world data. Concordia uses LLMs to narrate everyday social scenarios among agents, and WorldCoder and FactorSim use LLMs to write code for games and gridworld environments with programmatic rules.

New: we directly compare LLM-as-Environment with Concordia on simulating a programmatic simulator (YC-Bench). LLM-as-Environment has 4.4× lower final-state error (0.003190.00319 vs. 0.01410.0141).

2. Hybrid vs. code-only and LLM-only simulators (dcMC, YmXG). A hand-coded simulator cannot interpret open-ended decisions expressed in free text, such as a startup idea or pitch to investors. An LLM-only simulator can, but it makes small calculation and bookkeeping errors that compound over time.

New: in an LLM-only version of College Career Planning, degree progress was off by up to 0.21 in a single step, and in some trajectories the student never graduated despite completing enough units. Human raters found our environment more realistic compared with an LLM-only simulator in 9090 % comparisons (App. B.7).

3. Do the environments reward good decisions? (YVpS). We show 3 pieces of evidence.

(1) New: 5 human raters compared pairs of Career action sequences, with utilities and outcomes hidden, and agreed with the environment's ranking in 82.782.7 % of comparisons (App. B.7).

(2) The environments rank held-out real-world outcomes correctly (point 4).

(3) New: the Startup model ranking is robust to the scoring exploit (point 5).

4. Held-out validation (dcMC, YVpS). We kept the dev and test sets separate and did not use the test set to calibrate the environment, but some figures show dev and test results together. We've revised the paper to clarify this.

New: for Startup, we test on 12 test startups using an updated version of the environment, and simulated final valuations mathc reality with ρ=0.93\rho=0.93 (App. C.6). For NFL, our environment ranks test tenures with ρ=0.825\rho=0.825, and its title error is 2.882.88, close to the 2.142.14 of preseason betting lines, which is an approximate upper bound that sees the real roster every season, which our environment doesn't (App. D.5).

5. The Startup utility exploit (dcMC, YVpS). New: we re-ran model evals with 2 different utilities: (1) Founder Stake, which accounts for dilution, and (2) Revenue (10 x final ARR). Model rankings remain similar, with ρ=0.90\rho=0.90 for Founder Stake and ρ=0.87\rho=0.87 for Revenue (App. C.4). We now report both final valuation and founders' stake utilities.

The exploit mainly affects the privileged-search upper bound used to normalize all models' utilities, not the model rankings. It is also not that unrealistic: an AI startup in a hot financing climate reaches a $ 1.6B valuation with only $ 3.5M in ARR, which is not unusual today.

6. Generalization (YVpS). We used our framework and builder to build environments for 3 diverse fuzzy decision problems (Career, Startup, and NFL), and an LLM-dynamics version of YC-Bench. New: a third party used our builder to create an AI safety policymaking environment in about 60 minutes, with expert input on 15 decisions (App. F).

7. Human evals metric (dcMC). The low κ\kappa values come from label imbalance, because raters mostly select "correct." We thus report Gwet's AC1, which adjusts for label imbalance. We thus report Gwet's AC1, which adjusts for label imbalance. New: we also expanded the human evals to 5 raters (App. B.7). Humans rate 94.794.7% of simulated direction changes correct, 84.984.9% of magnitudes about right, and 91.791.7% of trajectories coherent, with inter-rater AC1 = 0.9050.905.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.