acceptodds
Under review as a conference paper at ICLR 2027

WorldSimBench: Benchmarking Large Language Models as Environment Simulators

Abstract

Training agents at scale requires large volumes of interaction trajectories, which are costly to collect from real environments. LLM-based generative simulators offer a scalable alternative, but existing public evaluations are often limited to narrow interaction settings and short trajectories, or rely on open-ended LLM-based scoring. We introduce WorldSimBench, a benchmark for evaluating LLMs as environment simulators across diverse tasks and long-horizon interactions. It comprises 3,580 tasks from 322 environments across ToolUse, Terminal, Game, Web, and Search. To construct long-horizon trajectories with fixed reference outcomes, we use story-driven plans to organize interactions and executable environments to produce the resulting states and observations. ToolUse, Terminal, Game, and Search use deterministic code-based or rule-based verifiers, whereas Web uses a fixed LLM verifier to check predefined requirements for components and values. Human assessment of representative tasks across all five environments yields an average rating of 9.15 out of 10. Across 32 model, the best achieves an unweighted macro-average success rate of 81.39%. Reasoning configurations achieve higher macro-average success rates in all 13 paired comparisons under the reported mode-specific inference settings. We find that world model performance is positively correlated with agent performance. Context truncation experiments on two models further show that early history is important for ToolUse, Terminal, and Search, whereas retaining only recent interactions improves accuracy over full histories in Game. These findings suggest that reliable simulation depends on preserving task-relevant state and limiting interference from irrelevant history. Code and sample data are available at https://anonymous.4open.science/r/worldsimbench/.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, environment interaction, simulated environments, tool use, benchmark, action prediction, state tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YuW3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldSimBench, a benchmark evaluating LLMs as environment simulators across five domains (ToolUse, Terminal, Game, Web, Search). It comprises 3,580 tasks with trajectories up to 199 steps, constructed via code-grounded story simulation to ensure deterministic reference outcomes. The authors evaluate 32 model configurations, finding that provider-recommended reasoning settings outperform non-reasoning settings in all 13 paired comparisons. Additional analyses examine context retention, scaling with model size, and failure patterns.

Critical strengths
  1. The benchmark comprises 3,580 tasks distributed across 322 environments and 5 interaction types. The trajectory lengths significantly exceed those in prior work (up to 199 steps, compared to typically fewer than 25 steps in the cited baselines).
  2. The environment and data construction, based on code-driven story simulations, effectively decouple planning from execution. Furthermore, systematic quality control filters out non-deterministic or under-specified instances.
  3. Regarding evaluation, four out of the five domains employ precise code/rule-based verifiers, avoiding the reliance on LLM-as-a-judge common in most assessments.
  4. Experimentally, the authors evaluated 32 model configurations and conducted stability checks (5 runs), context truncation experiments, complexity quadrant analysis, and a detailed review of failure modes.
Critical weaknesses
  1. Unlike other domains, the Web domain uses GPT-5.6-sol for verification. Despite the presence of scoring rubrics, this introduces potential evaluator bias.
  2. The error category distribution (Figure 7) is derived solely from DeepSeek-V4-Flash-Think-High, which limits the generalizability of the diagnostic conclusions.
  3. The paired thinking vs. non-thinking comparison simultaneously alters multiple variables, including temperature, top-p, repetition penalty, and reasoning budget.
Questions for authors
  1. Could you provide an ablation study that isolates the reasoning effect by toggling only the thinking switch while keeping all other sampling parameters (temperature, top-p, repetition penalty) constant?
  2. Were the 10 human-evaluated samples per environment selected randomly, or were they chosen to represent the difficulty distribution?
  3. What is the estimated accuracy of the GPT-5.6-sol Web judge relative to human raters?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer YuW3 (Part 1)

Thank you for your thoughtful comments on evaluation reliability and experimental controls. We address your concerns point by point below.

Weakness 1 and Question 3: Reliability of the Web judge

GPT-5.6-sol achieved 97.78% agreement with human judgments on 180 sampled predictions.

For each of three models evaluated in thinking mode—DeepSeek-V4-Pro-High, GLM-5.1, and Gemini-3.1-Pro-Preview—we randomly sampled 30 predictions judged correct and 30 judged incorrect by GPT-5.6-sol. Human annotators checked each required component and its value against the reference answer, and we then compared the resulting sample-level judgments with those of the judge.

Among the four disagreements, two were clear judging errors, while the other two were labeled uncertain by human annotators because of conflicting evidence or rubric ambiguity. We counted these uncertain cases as disagreements.

The annotation tool, annotation results, and case analyses are available in the web_judge_human_audit/ directory of our anonymous repository.

We have added the audit result to Section 4.1 and documented the sampling protocol, disagreements, and interpretation of the agreement rate in Appendix A.1.

Weakness 3 and Question 1: Controlling sampling parameters in the thinking comparison

The performance advantage of the thinking configurations persists after aligning the sampling settings. The macro-average scores of the six non-thinking configurations decreased by only 0.08–1.29 percentage points. Compared with the corresponding thinking results in Table 4, the thinking configurations still score 13.22–50.13 percentage points higher, consistently outperforming their non-thinking counterparts across all six comparisons.

We re-evaluated five Qwen models and Kimi-K2.6 with thinking disabled, matching temperature and top_p to their respective thinking configurations and leaving repetition_penalty at its default, as in the original thinking runs. The table below reports results from a single run.

ModelTemperatureTop-PRepetition penaltyToolAllTerminalGameWebSearchAvg
Qwen3.8-Max10.95—14.19 (+0.81)46.97 (−1.51)6.72 (−0.37)51.04 (+1.04)24.09 (−0.36)28.60 (−0.08)
Qwen3.7-Max0.70.9—10.90 (+0.47)40.91 (+1.01)2.99 (−1.49)43.23 (−1.56)8.27 (−0.25)21.26 (−0.36)
Qwen3.7-Plus0.70.9—8.76 (+0.09)43.94 (−3.03)1.49 (0.00)32.29 (+1.04)8.15 (−0.97)18.93 (−0.57)
Qwen3.6-Plus0.70.9—8.43 (+0.76)42.42 (−2.53)1.87 (−1.12)32.29 (−2.61)9.25 (−0.97)18.85 (−1.29)
Qwen3.5-397B-A17B0.70.9—6.48 (−0.66)36.36 (−2.02)1.49 (+0.37)33.85 (−0.53)9.49 (+1.10)17.54 (−0.34)
Kimi-K2.610.95—8.10 (+0.91)27.27 (−3.03)1.49 (−0.38)34.90 (+2.61)9.25 (−1.82)16.20 (−0.34)

Scores are percentages. Parentheses report the difference from the corresponding “Without thinking” result in Table 4, in percentage points. Avg is the unweighted mean of success rates across the five domains.

We have summarized the results in Section 4.2 and provided the settings, full results, and limitations in Appendix A.2.1.

AuthorsOpenReview ↗

Author Response to Reviewer YuW3 (Part 2)

Weakness 2: Generalizability of the error analysis

The additional analysis shows that some error patterns are shared across models, while their distributions differ. We extended the error-attribution analysis to GLM-5.3, GLM-5.3-Flash, Qwen3.8-Max, and Kimi-K3 in thinking mode. Instruction-following errors are the largest category in ToolUse across all five configurations, whereas error distributions differ substantially on Web. We therefore clarify that Figure 7 describes the error distribution of the analyzed DeepSeek configuration and provide cross-model results rather than generalizing its proportions to all models.

Each cell lists the percentages in the following order: Instruction Following / Similarity Interference / Numeric Error / Missing Evidence / Format Error / Misinterpreted Evidence. Values are separated by slashes, with percent signs omitted.

ModelToolUseGameTerminalWebSearch
DeepSeek-V4-Flash-High (Paper)47.9/6.3/8.5/3.0/26.0/8.337.2/38.2/0.0/16.7/1.0/6.920.6/0.0/26.5/20.6/14.7/17.635.4/1.5/1.5/46.2/10.8/4.64.1/8.1/5.1/43.9/19.4/19.4
GLM-5.350.9/5.1/10.8/7.8/17.9/7.533.3/12.5/0.0/2.1/27.1/25.038.1/14.3/11.9/2.4/19.0/14.341.67/55.56/0/0/2.78/024.6/24.1/1.6/7.8/33.0/8.9
GLM-5.3-Flash46.5/4.9/13.0/5.2/20.0/10.476.4/5.5/5.4/0.9/0.9/10.934.7/12.2/18.4/14.3/12.2/8.256.86/29.41/0/3.92/9.80/019.2/27.5/0.8/12.1/31.7/8.7
Qwen3.8-Max55.1/4.7/10.2/2.8/20.4/6.877.1/11.4/0.0/0.0/2.9/8.630.0/6.7/16.7/3.3/26.7/16.655.17/44.83/0/0/0/023.8/24.2/4.4/0.9/30.0/16.7
Kimi-K354.0/6.3/11.5/5.4/16.1/6.780.0/10.0/0.0/0.0/2.5/7.536.4/15.1/9.1/6.1/18.2/15.175.00/20.00/0/0/5.00/030.5/26.1/3.5/4.9/25.1/9.9

Section 5.6 now explains which error patterns are shared across models and which differ. Appendix D.2 provides the full comparison. Figure 7 still shows the results for the original DeepSeek configuration.

Question 2: Selection of samples for human evaluation

The samples were selected through difficulty-stratified random sampling, accounting for trajectory length and tool complexity.

Our original procedure selected 10 samples from each of ToolUse, Terminal, Game, Web, and Search, for a total of 50 samples. Within each domain, we divided the data into four difficulty strata and randomly selected samples within these strata. We clarify the existing procedure below:

  • Terminal, Game, Web, and Search: We divided the range of step counts within each domain into four equal-width intervals.
  • ToolUse: We formed four strata by combining short versus long trajectories (S ≤ 45 versus S ≥ 49) with fewer versus more available tools (T ≤ 10 versus T ≥ 13).

The sampling code is available in the human_evaluation/ directory of our anonymous repository.

We have clarified how the samples were selected in Section 3.5. Appendix B.9 provides the grouping rules, how samples were allocated across groups, and the random seed.

Thanks to the authors for their detailed rebuttal and the additional experiments. The Web judge audit (Q3) and the stratified sampling clarification (Q2) are convincing and directly address my concerns. The controlled inference ablation (Q1) is a valuable addition; however, because it relies on a single run rather than the repeated evaluations used elsewhere, I recommend reporting variance or explicitly noting this limitation in the final version. Similarly, while the expanded failure analysis across five models (Weakness 2) usefully demonstrates that error profiles are model-dependent, it reinforces my original point that the diagnostic conclusions drawn from a single configuration have limited generalizability. I appreciate the authors’ transparency in presenting these differences.

While the responses to Q2 and Q3 are satisfactory, the core methodological concern regarding the thinking vs. non-thinking comparison (Q1) is only partially addressed due to the lack of multi-run stability data for the ablation. Furthermore, the expanded failure analysis confirms rather than resolves the limited generalizability of the diagnostic claims. Therefore, I maintain my original score.

AuthorsOpenReview ↗

Author Response to Reviewer YuW3’s Follow-up Comments

Thank you for your follow-up. Below, we clarify what the error analysis shows and report repeated runs of the additional ablation.

1. Purpose and scope of the error analysis

Our error analysis aims to identify each model’s weaknesses. It does not assume that all models have the same error distribution. Using the same error categories makes comparisons possible, while differences in their proportions help identify where each model needs improvement.

We have added four thinking configurations—GLM-5.3, GLM-5.3-Flash, Qwen-3.8-Max, and Kimi-K3—to the analysis in Appendix D.2. The results show both shared weaknesses and differences: instruction-following errors are the largest category in ToolUse for all five analyzed configurations, whereas the Web error distributions vary across models. Figure 7 presents the distribution for the original model; Appendix D.2 shows the cross-model results.

Such differences have also been reported in prior work: on the same HumanEval task set, wrong logical direction accounts for 10.9% of CodeGen-16B’s errors and 35.0% of GPT-4’s errors (Wang et al., 2025, Figure 1).

Our main contribution is a benchmark for evaluating how accurately LLMs predict environment outcomes, using long interaction trajectories constructed from story outlines and actual environment execution. The error analysis provides additional diagnostic information; the benchmark does not depend on error proportions being identical across models.

2. Stability of the additional ablation

The repeated non-thinking evaluations show small run-to-run variation, and all six configurations remain below their corresponding thinking configurations in Table 4.

Table 5 in the original paper reports means and standard deviations over five runs for eight configurations. For the additional ablation, we now report five runs for each of the six non-thinking configurations below.

With thinking disabled, we set temperature and top_p to the values used for the corresponding thinking configuration. As in the original thinking runs, we left repetition_penalty unspecified, using the service default. We kept the task set and inference settings fixed across the five runs.

Across the six configurations and five domains, the standard deviations range from 0.26 to 1.76 percentage points. The five-run macro-average scores are 13.55–50.33 percentage points below the corresponding thinking scores originally reported in Table 4. These results support the performance advantage of the thinking configurations under the tested settings.

Each domain column reports mean ± standard deviation across five runs, in percent. Avg is the equal-weight mean across the five domains; its standard deviation is not reported. Parentheses show the change from the original non-thinking score in Table 4, in percentage points.

ModelToolAll (ToolUse)TerminalGameWebSearchAvg
Qwen3.8-Max13.55 ± 1.10 (+0.17)46.66 ± 1.22 (−1.82)6.42 ± 0.87 (−0.67)51.46 ± 1.21 (+1.46)23.89 ± 1.28 (−0.56)28.40 (−0.28)
Qwen3.7-Max11.33 ± 0.94 (+0.90)42.12 ± 0.94 (+2.22)2.98 ± 0.91 (−1.50)42.92 ± 1.12 (−1.87)8.00 ± 0.85 (−0.52)21.47 (−0.15)
Qwen3.7-Plus8.78 ± 0.26 (+0.11)44.14 ± 1.04 (−2.83)1.42 ± 0.64 (−0.07)31.35 ± 1.52 (+0.10)7.64 ± 0.53 (−1.48)18.67 (−0.83)
Qwen3.6-Plus8.23 ± 0.39 (+0.56)43.13 ± 1.62 (−1.82)1.42 ± 0.44 (−1.57)31.04 ± 0.96 (−3.86)8.76 ± 0.46 (−1.46)18.52 (−1.63)
Qwen3.5-397B-A17B6.87 ± 0.51 (−0.27)35.96 ± 1.61 (−2.42)1.49 ± 0.62 (+0.37)33.54 ± 1.46 (−0.84)9.27 ± 0.91 (+0.88)17.43 (−0.46)
Kimi-K2.67.78 ± 0.65 (+0.59)26.87 ± 1.17 (−3.43)1.57 ± 0.76 (−0.30)34.48 ± 1.76 (+2.19)9.32 ± 0.47 (−1.75)16.00 (−0.54)
Reviewer_mGF1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldSimBench, which consists of 3580 prediction tasks from 322 environments across diverse domains. The pipeline is LLMs will generate environments and interactions plans and environment-specific backends for execution. The experiment evaluate 32 model configurations and show reasoning models generally outperform their non-thinking match. Additional analyses about examining context retention, model size, external benchmark correlations, and failure categories are conducted.

Critical strengths
  1. The executable grounding will make the benchmark more reproducible and auditable.
  2. The benchmark covers several distinct interaction settings and bring performance comparison across different categories, which aggregated leaderboard cannot be done. 32 models comparison make the experiments solid.
  3. The extra context-retention analysis produces informative, qualified findings, which bring insights for future developments.
Critical weaknesses
  1. For reliable long-horizon simulation, the paper does not measure accumulated state drift, repeated simulation consistency, or robustness.
  2. It seems the AgentWorldBench, introduced in the related work, already evaluates history-conditioned observation prediction across seven domains, the unique contribution of this paper should be specified. 3.Broadening your experiment model family.
Questions for authors
  1. May I know what's the simulation LLM you are using? If use the same model in the experiment, it may brings advantages for the performance.
  2. Missing related work, such as Chuang, Yun-Shiuan, et al. "Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026). 2026.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer mGF1 (Part 1)

Thank you for your thoughtful comments. We address your concerns about long-horizon evaluation, the contribution relative to AgentWorldBench, model coverage, and data construction below.

Weakness 1: Accumulated state drift, repeated simulation consistency, and robustness

Endpoint correctness and state drift. Our accuracy metric can reflect the effects of accumulated state drift: drift is an underlying cause, while final-answer accuracy measures the outcome. Errors in intermediate state tracking or updates that lead to an incorrect final answer are counted as prediction failures.

Table 5 (Evaluation Stability) already reports stability across repeated evaluations. We evaluated eight representative model configurations five times each, using fixed task sets and inference settings, and reported the mean and sample standard deviation of success rates across the five environments. All standard deviations are at most 3.91 percentage points, supporting the stability of evaluation scores across repeated runs for these configurations.

The macro-average ranking of all six configurations remains unchanged across the original prompt and both prompt variants. We added a robustness experiment covering all 3,580 tasks. V1 paraphrases the instructions while preserving their meaning, whereas V2 reorders the instruction blocks. All other evaluation conditions remain unchanged. For five configurations, the absolute change in the macro-average score is no more than 2.64 percentage points; GLM-5.3-Flash improves by 5.45 and 5.98 percentage points, respectively. These results support the stability of the overall ranking under the tested prompt variants, while showing that absolute scores for some configurations remain sensitive to prompt changes.

Each cell below reports Original / V1 / V2 success rates (%). Original results are from Table 4; Avg is the unweighted mean across the five domains.

ModelThinkingToolAllTerminalGameWebSearchAvg
GLM-5.3Enabled70.33 / 70.95 / 69.9077.78 / 69.70 / 71.7281.72 / 80.60 / 75.7568.75 / 68.23 / 68.7576.76 / 77.13 / 76.0375.07 / 73.32 / 72.43
GLM-5.3-FlashEnabled61.48 / 65.52 / 62.6275.25 / 80.30 / 81.8258.96 / 65.30 / 69.7855.21 / 64.58 / 61.4670.80 / 73.24 / 75.9164.34 / 69.79 / 70.32
Qwen-3.8-MaxEnabled75.52 / 75.57 / 76.6284.85 / 82.32 / 81.8286.94 / 87.69 / 85.8273.96 / 75.52 / 73.9672.38 / 71.78 / 71.9078.73 / 78.58 / 78.02
Qwen-3.8-MaxDisabled13.38 / 14.24 / 14.4848.48 / 46.97 / 42.937.09 / 6.72 / 4.8550.00 / 48.44 / 51.0424.45 / 25.06 / 24.5728.68 / 28.28 / 27.57
Qwen-3.5-397B-A17BEnabled27.95 / 30.62 / 30.9551.01 / 56.57 / 52.5332.09 / 25.37 / 25.0046.35 / 48.44 / 47.9235.16 / 35.28 / 36.8638.51 / 39.26 / 38.65
Qwen-3.5-397B-A17BDisabled7.14 / 6.95 / 6.6738.38 / 37.37 / 36.871.12 / 0.75 / 1.1234.38 / 35.94 / 33.858.39 / 9.85 / 9.7317.88 / 18.17 / 17.65

The scripts for generating V1 and V2 are available in the prompt_robustness/ directory of our anonymous repository.

We have added the main findings to Section 5.2 and provided the prompt-variant definitions and full results in Appendix C.5.

AuthorsOpenReview ↗

Author Response to Reviewer mGF1 (Part 2)

Weakness 2: Contribution relative to AgentWorldBench

Our contributions primarily concern verification and trajectory construction.

  1. Verification. AgentWorldBench primarily uses reference-grounded LLM-as-a-judge scoring across five dimensions for open-ended observation prediction, and also reports supplementary rule-based verification experiments in its appendix. WorldSimBench uses deterministic code- or rule-based verification for ToolUse, Terminal, Game, and Search, while Web predictions are evaluated by checking components and their values. The distinction therefore concerns the role and coverage of deterministic verification in the main evaluation, rather than its presence or absence.

  2. Long-horizon evaluation. WorldSimBench constructs long textual interaction trajectories with explicitly designed distractors and cross-step dependencies. We first generate diverse story outlines, expand them into step-level descriptions, and then perform function calls and execute the environment at each step to obtain the trajectory. The average trajectory lengths reported in the two papers are shown below, retaining their respective units of steps and turns.

Interaction settingWorldSimBench (steps)AgentWorldBench (turns)
ToolUse / MCP49.224.1
Terminal54.526.5
Search50.315.3

Weakness 3: Broadening model family coverage

We have added Grok-4.6 (xhigh) and Doubao-Seed-2.1-pro (medium), expanding coverage from eight to ten model providers. Both configurations were evaluated on all 3,580 tasks, achieving macro-average scores of 46.18% and 67.69%, respectively. Doubao performs strongly in Game (80.97%), while Grok's Web score (58.33%) exceeds those of several configurations with higher overall rankings, further illustrating differences in domain-specific capabilities across model families.

The table compares the best-performing tested thinking configuration from each provider. Scores are percentages, and the columns follow Table 4.

ModelToolAllGameTerminalWebSearchAvg
GPT-5.578.9582.0983.3382.2980.2981.39
Qwen-3.8-Max75.5286.9484.8573.9672.3878.73
Kimi-K375.2985.0783.3371.8875.3078.17
GLM-5.370.3381.7277.7868.7576.7675.07
Gemini-3.1-Pro-Preview73.1469.0384.8545.3168.7368.21
Doubao-Seed-2.1-pro65.5780.9774.2452.0865.5767.69
DeepSeek-V4-Pro63.2971.6466.1654.6962.6563.69
Claude-Opus-4.752.7158.9674.7552.0864.9660.69
Grok-4.647.5738.4352.0258.3334.5546.18
MiniMax-M2.511.384.8533.3344.7918.2522.52

We used the officially recommended default configurations, with medium reasoning effort for Doubao-Seed-2.1-pro and xhigh reasoning effort for Grok-4.6.

Section 4.1 now notes the two additional providers. Appendix A.2.2 reports the results for a representative model from each provider and the settings used for the two added models.

Question 1: Models used for data construction and potential advantages

Dataset construction primarily used Qwen3.5-Plus and Gemini-3.1-Pro-Preview, with GPT-5.6-sol additionally used for Web verification (Appendix B.8). Gemini-3.1-Pro-Preview, the only construction model also included in the evaluation, does not lead the benchmark.

Its macro-average success rate is 68.21%, below GPT-5.5 (81.39%), Kimi-K3 (78.17%), and GLM-5.3 (75.07%). Thus, participation in dataset construction did not translate into leading performance.

Question 2: Adding a citation to “Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents”

Thank you for recommending this relevant work. Chuang et al. use LLMs to simulate users and tools, track the resulting state, and check whether an agent has completed its task. This reduces the need to build a full database and tool backend, making it easier to create environments for agent evaluation and training.

We have added this work to Section 2 (Related Work) and clarified how it relates to WorldSimBench. Their work evaluates whether an agent completes a task in an LLM-simulated environment. WorldSimBench evaluates how accurately an LLM predicts environment outcomes, using actual environment execution as the reference.

Thanks to the authors for the detailed rebuttal and the additional experiments. My problems got answered and I raised my score.

Reviewer_hLyuOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses one of the most important pain points of the community-- lacking useful data for long trajectory task training. The use of LLM-simulators is one way to address this challenge but the quality of these must be measured.

This paper proposes WorldSimBench that allows on to evaluate the performance of LLM-based simulator performance across really massive set of tasks and documents the performance. Like the Tau-bench, this bench can be a useful and multigenerational bench.

Critical strengths

Providing a very detailed simulation bench to test the quality and pformance of the LLM-simulators for the trajectory trainings.

Critical weaknesses

Why didn't someone do it before? It is excellent one like Job-Bench and ALE.

Questions for authors

Will you maintain this well like the ALE and Tau-bench? with future versions?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer hLyu

Thank you for your encouraging assessment and for recognizing the value and long-term potential of WorldSimBench.

Yes, we plan to maintain WorldSimBench and release future versions. Our maintenance priorities include addressing issues identified over time and gradually expanding the range of interaction environments, including GUI interaction and multimodal settings. We will also maintain versioned datasets and evaluation configurations, with documented changes between releases so that results can be reproduced and compared across versions.

Thank you again for your support and for highlighting the importance of long-term benchmark maintenance.

Thank you for the response. I see other reviews and they have some questions different from mine. I am satisfied with your answers.

Thank you for following up. We’re glad our responses have addressed your concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Sincerely Looking Forward to the Reviewers' Feedback

Dear Reviewers,

Thank you for your careful evaluation and constructive feedback. We appreciate your recognition of WorldSimBench's reproducibility, auditability, and value for evaluating LLM-based environment simulators.

We summarize our responses and additional results below:

  1. We evaluated the Web judge against human annotations on 180 sampled predictions, obtaining 97.78% agreement, with uncertain cases counted as disagreements. (To Reviewer YuW3)

  2. We re-evaluated six non-thinking configurations with sampling settings aligned to their thinking counterparts. The corresponding thinking configurations retained their performance advantage. (To Reviewer YuW3)

  3. We extended error attribution to four additional thinking configurations, identifying shared error patterns alongside model- and domain-specific differences. (To Reviewer YuW3)

  4. We clarified the difficulty-stratified random sampling procedure used to select the 50 samples for human evaluation. (To Reviewer YuW3)

  5. We clarified the distinction between endpoint correctness and accumulated drift in free-running simulation, highlighted the five-run stability results in Table 5, and added two prompt variants under which all six tested configurations retained their macro-average ranking. (To Reviewer mGF1)

  6. We clarified how WorldSimBench differs from AgentWorldBench, specified the roles of the models used in data construction, and discussed the related work of Chuang et al. (To Reviewer mGF1)

  7. We added Grok-4.6 and Doubao-Seed-2.1-pro, evaluating each on all 3,580 tasks and expanding coverage from eight to ten model providers. (To Reviewer mGF1)

  8. We outlined our plan to maintain WorldSimBench, address newly identified issues, and gradually expand to GUI interaction and multimodal settings, with versioned datasets and evaluation configurations. (To Reviewer hLyu)

Detailed results and explanations are provided in our point-by-point responses. Supplementary materials are available in our anonymous repository.

We welcome any further questions and would be happy to clarify our responses.

Best regards, The authors

Dear Reviewers,

Please review the authors' rebuttal and participate in the discussion to address any remaining questions before the period closes on Nov 18 AoE.

Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.