acceptodds
Withdrawn from ICLR 2027

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

LLM-based environment simulators offer a flexible substrate for agent training, but their usefulness depends on whether generated feedback and state updates faithfully reflect agent actions. In practice, LLM-simulated environments suffer from hallucinations, logical inconsistencies, and silent state drift—failures that corrupt agent reward signals and compound the construction costs that the paradigm was designed to eliminate. To address this gap, we propose EnvSimBench with four contributions: (1) We provide the first formal definition and operationalization of Environment Simulation Ability (EnvSim Ability) as a quantifiable research objective. (2) We construct EnvSimBench, a rigorous benchmark covering 400 samples across 167 diverse environments, equipped with verifiable labels and fine-grained difficulty stratification along three axes. (3) Evaluations of seven frontier language models reveal a pronounced decline in Config Match on state-changing operations relative to state-preserving operations, which we term the state-change cliff. This finding identifies a limitation in predicting the state updates required by the evaluated environments. (4) We design a constraint-driven simulation pipeline that substantially reduces hallucination, boosts environment synthesis yield by 6.8%, and cuts costs by over 90%. Overall, EnvSimBench serves as both a diagnostic framework and a practical optimization path for reliable LLM-based environment simulation, establishing a solid foundation for scalable agent training. Code and data are available at \url{https://anonymous.4open.science/r/EnvSimBench-

Primary area:
datasets and benchmarks
Keywords:
environment simulation, benchmarks, state-transition prediction, agent training, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dJdCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EnvSimBench asks an LLM to predict a tool call’s response and state changes from the current configuration and implementation code. Executed transitions provide references for separate feedback and configuration match scores. The main set has 400 samples from 167 environments. Seven-model evaluation finds that correct responses can conceal wrong state updates, particularly on multi-field transitions; reasoning-mode and 1,300-sample tests show a similar pattern. A fine-tuned Qwen3-4B reaches 45.3% configuration match versus 42.3% for the best main frontier baseline, and raises admitted EnvScaler environments from 191 to 204. The benchmark measures a useful failure mode, though some inputs omit runtime values needed for exact prediction.

Critical strengths
  1. The separation between feedback and state accuracy is useful. A simulator can return a plausible response while silently corrupting its state; evaluating only what the agent sees would miss this failure. Executable references make the mismatch easy to inspect.

  2. The supplementary tests strengthen the empirical finding: the authors evaluate reasoning-enabled models, exclude benchmark environments from SFT training, and expand the test set to 100 examples per change count.

Critical weaknesses
  1. Some examples require information absent from the model’s inputs. The true transition includes runtime values, but the predictor receives only state, action, and code. Appendix D.3 finds runtime-dependent fields in 23–28% of configuration failures with at least three changes. Exact matching therefore penalizes some predictions that cannot be recovered from the input. Whole-state matching also compounds errors as more fields change, so the proposed cliff needs to be separated from this effect and from differences in update structure.

  2. The downstream evidence is less convincing than the transition evaluation. The paper acknowledges overlap in distribution between the synthesis filter and SFT data, so higher admission yield does not establish independent quality. There is no paired uncertainty analysis for the 45.3% versus 42.3% gain. Single-step evaluation also leaves accumulated drift and the effect on agent training untested.

Questions for authors
  1. How do rankings and change-count curves change if runtime values are supplied or canonicalized? Please report results on examples whose outputs are fully determined by the inputs.
  2. Can per-field scores and examples matched for update structure and code complexity distinguish a cliff from compounding errors?
  3. Is the 45.3% versus 42.3% gain statistically reliable? Has an independent audit checked the 204 admitted environments?
Reviewer_zdGsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EnvSimBench to evaluate whether LLMs can correctly simulate tool feedback and state transitions given the current state and implementation code. The benchmark contains 400 transitions across 167 environments, with a 1,300-sample extension. Results show that plausible feedback often hides incorrect state updates. A fine-tuned 4B simulator improves configuration accuracy and increases the number of environments the synthesis pipeline retains.

Critical strengths
  1. Execution-grounded labels and held-out environments provide a controlled evaluation. This justifies the benchmark curation process.
  2. The error analysis and training ablations reveal concrete simulator failure modes. This strengthens the soundness of the work.
Critical weaknesses
  1. Lack of model training in the simulated environment. From the motivation the key to the simulated environment is to better build trianing dataset/gyms. However, this is not measured end-to-end and cannot well justify the motivation of the work.
  2. Some targets depend on runtime timestamps/identifiers unavailable to the model. Exact matching therefore weakens the interpretation of benchmark scores.
  3. The claimed state-change threshold is not sufficiently isolated. Different components like state matching and runtime dependencies, all affect difficulty; the authors should further justify the difficulty design.
Questions for authors

See weakness.

Reviewer_AsTkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether LLMs can predict state changes produced by an agent's tool call, the ability of environment simulator. EnvSimBench built on EnvScaler, supplies the recorded before-state, action, and implementation code, and evaluates feedback equality and reconstructed-state equality against executor-produced references, though EnvScaler is also a synthetic environment suite which has no guarantee of correct state transitions. The authors also fine-tune a Qwen 4B model using the data from EnvScaler and report improved performance on the benchmark.

Critical strengths
  1. This paper studies an interesting problem: measuring the capability of LLMs to act as a environment simulator. The evaluation checks both feedback and the reconstructed after-state against executable references, instead of relying on an LLM judge for these labels
  2. The proposed method relies on existing synthetic environments, which may save costs to build this evaluation benchmark, though the fidelity may be limited by the quality of EnvScaler.
Critical weaknesses
  1. This paper heavily relies on EnvScaler, which is a synthetic environment suite that has no guarantee of correct state transitions. The benchmark should provide more evidence that the reference states are correct, or at least provide a discussion of the potential impact of incorrect references on the evaluation results. And I also suggest the authors to use human crafted environments such as WebArena or tau-bench or AFWorld to collect reference data.
  2. The used models whatever for collecting trajs (GPT-4o-mini) or for evaluation are relatively outdated, and the authors should provide more recent models for evaluation such as GPT-5.6, Opus-5 and etc. which are released several months ago before ICLR submission.
  3. The proposed evaluation metric heavily relies on the reference of the correctness of executable environments. Feedback Match requires exact string equality with the executor's reference output, and the paper shows this mostly detects serialization conventions rather than environment feedback correctness. Another limitation is that the main motivation is that unfaithful simulators corrupt agent reward signals, yet the evaluation never connects FM or CM to agent learning. I suggest the authors add training run in a high-CM and a low-CM simulator.
  4. The downstream experiment does not independently establish better environment quality, because the authors acknowledge overlap between the quality-filter distribution and the SFT data.
  5. The related work section can be enhanced, since there are many recent works on environment synthesis and LLM simulators, and should discuss the differences and unique contributions of this work. For example, Simia and EnvScaler.
Questions for authors

Please see the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.