- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldSimBench, a benchmark evaluating LLMs as environment simulators across five domains (ToolUse, Terminal, Game, Web, Search). It comprises 3,580 tasks with trajectories up to 199 steps, constructed via code-grounded story simulation to ensure deterministic reference outcomes. The authors evaluate 32 model configurations, finding that provider-recommended reasoning settings outperform non-reasoning settings in all 13 paired comparisons. Additional analyses examine context retention, scaling with model size, and failure patterns.
Critical strengths
- The benchmark comprises 3,580 tasks distributed across 322 environments and 5 interaction types. The trajectory lengths significantly exceed those in prior work (up to 199 steps, compared to typically fewer than 25 steps in the cited baselines).
- The environment and data construction, based on code-driven story simulations, effectively decouple planning from execution. Furthermore, systematic quality control filters out non-deterministic or under-specified instances.
- Regarding evaluation, four out of the five domains employ precise code/rule-based verifiers, avoiding the reliance on LLM-as-a-judge common in most assessments.
- Experimentally, the authors evaluated 32 model configurations and conducted stability checks (5 runs), context truncation experiments, complexity quadrant analysis, and a detailed review of failure modes.
Critical weaknesses
- Unlike other domains, the Web domain uses GPT-5.6-sol for verification. Despite the presence of scoring rubrics, this introduces potential evaluator bias.
- The error category distribution (Figure 7) is derived solely from DeepSeek-V4-Flash-Think-High, which limits the generalizability of the diagnostic conclusions.
- The paired thinking vs. non-thinking comparison simultaneously alters multiple variables, including temperature, top-p, repetition penalty, and reasoning budget.
Questions for authors
- Could you provide an ablation study that isolates the reasoning effect by toggling only the thinking switch while keeping all other sampling parameters (temperature, top-p, repetition penalty) constant?
- Were the 10 human-evaluated samples per environment selected randomly, or were they chosen to represent the difficulty distribution?
- What is the estimated accuracy of the GPT-5.6-sol Web judge relative to human raters?
4 comments
Author Response to Reviewer YuW3 (Part 2)
Weakness 2: Generalizability of the error analysis
The additional analysis shows that some error patterns are shared across models, while their distributions differ. We extended the error-attribution analysis to GLM-5.3, GLM-5.3-Flash, Qwen3.8-Max, and Kimi-K3 in thinking mode. Instruction-following errors are the largest category in ToolUse across all five configurations, whereas error distributions differ substantially on Web. We therefore clarify that Figure 7 describes the error distribution of the analyzed DeepSeek configuration and provide cross-model results rather than generalizing its proportions to all models.
Each cell lists the percentages in the following order: Instruction Following / Similarity Interference / Numeric Error / Missing Evidence / Format Error / Misinterpreted Evidence. Values are separated by slashes, with percent signs omitted.
| Model | ToolUse | Game | Terminal | Web | Search |
|---|---|---|---|---|---|
| DeepSeek-V4-Flash-High (Paper) | 47.9/6.3/8.5/3.0/26.0/8.3 | 37.2/38.2/0.0/16.7/1.0/6.9 | 20.6/0.0/26.5/20.6/14.7/17.6 | 35.4/1.5/1.5/46.2/10.8/4.6 | 4.1/8.1/5.1/43.9/19.4/19.4 |
| GLM-5.3 | 50.9/5.1/10.8/7.8/17.9/7.5 | 33.3/12.5/0.0/2.1/27.1/25.0 | 38.1/14.3/11.9/2.4/19.0/14.3 | 41.67/55.56/0/0/2.78/0 | 24.6/24.1/1.6/7.8/33.0/8.9 |
| GLM-5.3-Flash | 46.5/4.9/13.0/5.2/20.0/10.4 | 76.4/5.5/5.4/0.9/0.9/10.9 | 34.7/12.2/18.4/14.3/12.2/8.2 | 56.86/29.41/0/3.92/9.80/0 | 19.2/27.5/0.8/12.1/31.7/8.7 |
| Qwen3.8-Max | 55.1/4.7/10.2/2.8/20.4/6.8 | 77.1/11.4/0.0/0.0/2.9/8.6 | 30.0/6.7/16.7/3.3/26.7/16.6 | 55.17/44.83/0/0/0/0 | 23.8/24.2/4.4/0.9/30.0/16.7 |
| Kimi-K3 | 54.0/6.3/11.5/5.4/16.1/6.7 | 80.0/10.0/0.0/0.0/2.5/7.5 | 36.4/15.1/9.1/6.1/18.2/15.1 | 75.00/20.00/0/0/5.00/0 | 30.5/26.1/3.5/4.9/25.1/9.9 |
Section 5.6 now explains which error patterns are shared across models and which differ. Appendix D.2 provides the full comparison. Figure 7 still shows the results for the original DeepSeek configuration.
Question 2: Selection of samples for human evaluation
The samples were selected through difficulty-stratified random sampling, accounting for trajectory length and tool complexity.
Our original procedure selected 10 samples from each of ToolUse, Terminal, Game, Web, and Search, for a total of 50 samples. Within each domain, we divided the data into four difficulty strata and randomly selected samples within these strata. We clarify the existing procedure below:
- Terminal, Game, Web, and Search: We divided the range of step counts within each domain into four equal-width intervals.
- ToolUse: We formed four strata by combining short versus long trajectories (
S ≤ 45versusS ≥ 49) with fewer versus more available tools (T ≤ 10versusT ≥ 13).
The sampling code is available in the human_evaluation/ directory of our anonymous repository.
We have clarified how the samples were selected in Section 3.5. Appendix B.9 provides the grouping rules, how samples were allocated across groups, and the random seed.
Thanks to the authors for their detailed rebuttal and the additional experiments. The Web judge audit (Q3) and the stratified sampling clarification (Q2) are convincing and directly address my concerns. The controlled inference ablation (Q1) is a valuable addition; however, because it relies on a single run rather than the repeated evaluations used elsewhere, I recommend reporting variance or explicitly noting this limitation in the final version. Similarly, while the expanded failure analysis across five models (Weakness 2) usefully demonstrates that error profiles are model-dependent, it reinforces my original point that the diagnostic conclusions drawn from a single configuration have limited generalizability. I appreciate the authors’ transparency in presenting these differences.
While the responses to Q2 and Q3 are satisfactory, the core methodological concern regarding the thinking vs. non-thinking comparison (Q1) is only partially addressed due to the lack of multi-run stability data for the ablation. Furthermore, the expanded failure analysis confirms rather than resolves the limited generalizability of the diagnostic claims. Therefore, I maintain my original score.
Author Response to Reviewer YuW3’s Follow-up Comments
Thank you for your follow-up. Below, we clarify what the error analysis shows and report repeated runs of the additional ablation.
1. Purpose and scope of the error analysis
Our error analysis aims to identify each model’s weaknesses. It does not assume that all models have the same error distribution. Using the same error categories makes comparisons possible, while differences in their proportions help identify where each model needs improvement.
We have added four thinking configurations—GLM-5.3, GLM-5.3-Flash, Qwen-3.8-Max, and Kimi-K3—to the analysis in Appendix D.2. The results show both shared weaknesses and differences: instruction-following errors are the largest category in ToolUse for all five analyzed configurations, whereas the Web error distributions vary across models. Figure 7 presents the distribution for the original model; Appendix D.2 shows the cross-model results.
Such differences have also been reported in prior work: on the same HumanEval task set, wrong logical direction accounts for 10.9% of CodeGen-16B’s errors and 35.0% of GPT-4’s errors (Wang et al., 2025, Figure 1).
Our main contribution is a benchmark for evaluating how accurately LLMs predict environment outcomes, using long interaction trajectories constructed from story outlines and actual environment execution. The error analysis provides additional diagnostic information; the benchmark does not depend on error proportions being identical across models.
2. Stability of the additional ablation
The repeated non-thinking evaluations show small run-to-run variation, and all six configurations remain below their corresponding thinking configurations in Table 4.
Table 5 in the original paper reports means and standard deviations over five runs for eight configurations. For the additional ablation, we now report five runs for each of the six non-thinking configurations below.
With thinking disabled, we set temperature and top_p to the values used for the corresponding thinking configuration. As in the original thinking runs, we left repetition_penalty unspecified, using the service default. We kept the task set and inference settings fixed across the five runs.
Across the six configurations and five domains, the standard deviations range from 0.26 to 1.76 percentage points. The five-run macro-average scores are 13.55–50.33 percentage points below the corresponding thinking scores originally reported in Table 4. These results support the performance advantage of the thinking configurations under the tested settings.
Each domain column reports mean ± standard deviation across five runs, in percent. Avg is the equal-weight mean across the five domains; its standard deviation is not reported. Parentheses show the change from the original non-thinking score in Table 4, in percentage points.
| Model | ToolAll (ToolUse) | Terminal | Game | Web | Search | Avg |
|---|---|---|---|---|---|---|
| Qwen3.8-Max | 13.55 ± 1.10 (+0.17) | 46.66 ± 1.22 (−1.82) | 6.42 ± 0.87 (−0.67) | 51.46 ± 1.21 (+1.46) | 23.89 ± 1.28 (−0.56) | 28.40 (−0.28) |
| Qwen3.7-Max | 11.33 ± 0.94 (+0.90) | 42.12 ± 0.94 (+2.22) | 2.98 ± 0.91 (−1.50) | 42.92 ± 1.12 (−1.87) | 8.00 ± 0.85 (−0.52) | 21.47 (−0.15) |
| Qwen3.7-Plus | 8.78 ± 0.26 (+0.11) | 44.14 ± 1.04 (−2.83) | 1.42 ± 0.64 (−0.07) | 31.35 ± 1.52 (+0.10) | 7.64 ± 0.53 (−1.48) | 18.67 (−0.83) |
| Qwen3.6-Plus | 8.23 ± 0.39 (+0.56) | 43.13 ± 1.62 (−1.82) | 1.42 ± 0.44 (−1.57) | 31.04 ± 0.96 (−3.86) | 8.76 ± 0.46 (−1.46) | 18.52 (−1.63) |
| Qwen3.5-397B-A17B | 6.87 ± 0.51 (−0.27) | 35.96 ± 1.61 (−2.42) | 1.49 ± 0.62 (+0.37) | 33.54 ± 1.46 (−0.84) | 9.27 ± 0.91 (+0.88) | 17.43 (−0.46) |
| Kimi-K2.6 | 7.78 ± 0.65 (+0.59) | 26.87 ± 1.17 (−3.43) | 1.57 ± 0.76 (−0.30) | 34.48 ± 1.76 (+2.19) | 9.32 ± 0.47 (−1.75) | 16.00 (−0.54) |
Author Response to Reviewer YuW3 (Part 1)
Thank you for your thoughtful comments on evaluation reliability and experimental controls. We address your concerns point by point below.
Weakness 1 and Question 3: Reliability of the Web judge
GPT-5.6-sol achieved 97.78% agreement with human judgments on 180 sampled predictions.
For each of three models evaluated in thinking mode—DeepSeek-V4-Pro-High, GLM-5.1, and Gemini-3.1-Pro-Preview—we randomly sampled 30 predictions judged correct and 30 judged incorrect by GPT-5.6-sol. Human annotators checked each required component and its value against the reference answer, and we then compared the resulting sample-level judgments with those of the judge.
Among the four disagreements, two were clear judging errors, while the other two were labeled
uncertainby human annotators because of conflicting evidence or rubric ambiguity. We counted these uncertain cases as disagreements.The annotation tool, annotation results, and case analyses are available in the
web_judge_human_audit/directory of our anonymous repository.We have added the audit result to Section 4.1 and documented the sampling protocol, disagreements, and interpretation of the agreement rate in Appendix A.1.
Weakness 3 and Question 1: Controlling sampling parameters in the thinking comparison
The performance advantage of the thinking configurations persists after aligning the sampling settings. The macro-average scores of the six non-thinking configurations decreased by only 0.08–1.29 percentage points. Compared with the corresponding thinking results in Table 4, the thinking configurations still score 13.22–50.13 percentage points higher, consistently outperforming their non-thinking counterparts across all six comparisons.
We re-evaluated five Qwen models and Kimi-K2.6 with thinking disabled, matching
temperatureandtop_pto their respective thinking configurations and leavingrepetition_penaltyat its default, as in the original thinking runs. The table below reports results from a single run.Scores are percentages. Parentheses report the difference from the corresponding “Without thinking” result in Table 4, in percentage points. Avg is the unweighted mean of success rates across the five domains.
We have summarized the results in Section 4.2 and provided the settings, full results, and limitations in Appendix A.2.1.