acceptodds
Under review as a conference paper at ICLR 2027

GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

Abstract

Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills. The tasks check whether models can predict the environment’s response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects. We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with both deterministic gates and a semantic judge. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra achieves the highest task success rate at 49.57%. We find that model accuracy decreases as actions affect a larger portion of the environment. Specifically, when the agent's changes must propagate to dependent objects, task success drops to an average of 5.1% across all 20 models. Although model strengths differ across tasks, 631 tasks remain unsolved by any model, suggesting a significant gap in general LLM capabilities as a world model.

Primary area:
datasets and benchmarks
Keywords:
world model, evaluation, benchmark, simulation, rl environments

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7hCMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GlobeBench, a benchmark for evaluating LLMs as text-based world models: given a multi-turn trajectory P=(τ1,…,τH)P=(\tau_1,\dots,\tau_H) and a final action aa, the model must predict the next environment state g^=fθ(P,a)\hat g = f_\theta(P,a), which is compared against a gold gg obtained by executing aa against a pinned backend (a container digest, or locked dependency hashes) rather than by annotation. Composition. 1,632 items across five domains — Code (526), Enterprise (381), Games (334), Math & Science (198), Dialogue (193). Items are organized by two new taxonomies: four capabilities (State Coherence, Causal Correctness, Interface Fidelity, Constraint Enforcement) and 19 skills (e.g., Read/Write Persistence, Access Control, Schema Conformance, Quota Enforcement, Sycophantic Confirmation). Each item is produced by pairing a domain recipe with one of 62 operators; an operator fixes the action sequence for exactly one primary skill, and can insert filler turns that leave the tested state untouched. Every item is labelled with reach-back (turns between the deciding event and the final action) and impact radius (L0 = no state change; L1/L2 = one / two-or-more named entities change; L3 = at least one unnamed entity changes). Quality control. 12 item-level gates (deterministic checks plus LLM validity/ambiguity/"tell" judges) reject items that are unsolvable, ambiguous, non-reproducible, or detectable from surface cues. Validation combines a 6-annotator human study of 200 items and a two-phase LLM audit of 1,627 items. Grading. A frozen hybrid grader applies deterministic metrics first (exact/structural match, numeric fidelity, error class + referent, order sensitivity, under-/over-erroring); only responses that no metric settles go to a blinded LLM judge (GPT-5.6 Terra, with GPT-5.6 Sol as fallback). Contributions. (i) The benchmark, harness, model outputs, judge records and reproduction instructions; (ii) the capability/skill taxonomies plus the reach-back and impact-radius axes, which aim to attribute a failure to a named mechanism rather than score a predicted observation holistically (as AgentWorldBench and EnvSimBench do); (iii) an empirical study around three RQs — which parts of a predicted state models get wrong and whether open- and closed-weight models fail differently; failure modes that aggregate success hides; and how automated judgments align with human judgments. Validation / findings. 20 models are evaluated zero-shot, one sample per item, with provider refusals scored as failures. GPT-6 Astra leads at 49.57%, followed by Grok 4.6 (28.13%), Claude Opus 5.5 (26.35%) and Qwen 3.8 Max (25.31%); pooled success is 19.68%, and 631 items are solved by no model (87.6% of them L3, 93.7% beyond 40 turns of reach-back). Success falls from 43.49% at L0 to 5.10% at L3, and from 42.28% within 40 turns to 11.44% beyond. Most answered failures (53.21%) report a wrong value. Models score far higher on error-outcome items (53.16%) than normal ones (16.16%). Both specialized world models rank low (Qwen-AgentWorld 14.46%, DWM 5.21%), which the authors read as evidence that world modelling is an emergent capability of general scale rather than something instilled by task-specific training.

Critical strengths
  1. Execution-grounded golds, one mechanism per item, and a deterministic-first grader — with a complete release. This is the paper's strongest reason for acceptance. Because every gold is executed rather than annotated, because each item is built to isolate a single named skill and is filtered by 12 gates that discard items solvable without exercising it, and because the grader settles everything it provably can before invoking an LLM, a failure can be localised to a mechanism instead of merely counted. That is exactly the property prior world-model evaluations (AgentWorldBench, EnvSimBench) lack, and it is what makes the paper's diagnostic findings — wrong-value failures, hidden-state drift, the error/normal gap — credible at all. Combined with the release of model outputs and judge records (so results can be rescored without re-querying models), this makes the benchmark a durable community resource. Consequential because a benchmark paper's value is its measurement instrument, and this instrument is a real advance.
  2. The reach-back / impact-radius decomposition, and the hidden-state finding. Even granting the confounding discussed below, the axes yield a genuinely new and actionable diagnosis: the bottleneck is not rule application but carrying state forward. The evidence is specific and quantitative — on the quorum family, GPT-6 Astra reports the exact capability on 84% of requests versus 2–8% for the strongest peers, while those peers' weights follow correctly from their own (wrong) capability 75–99% of the time; and once a peer gets the state right, its decision is right 61–91% of the time against >99% for Astra. The 631-unsolved-item analysis (85.1% concentrated in six skills) is also a useful map of where to work next. Consequential because it changes what a follow-up training effort should target.
Critical weaknesses
  1. The headline causal claims about impact radius (and reach-back) are not supported by the paper's own within-group analyses. Impact radius, reach-back, primary skill, operator and output length are correlated by construction: the paper states that each of the 62 operators fixes both reach-back and radius, that operator grouping explains η2=81.64%\eta^2 = 81.64\% of the variance in log⁡(1+reach-back)\log(1+\text{reach-back}) (37.41% for skill), and that reach-back correlates with prompt length at Spearman ρ=0.94\rho = 0.94. Three pieces of the paper's own evidence then undercut the causal reading of its most prominent number:
  • The abstract's "task success drops to an average of 5.1%" at L3 rests on 953 of 977 L3 items whose level is declared by the operator, not measured from an execution diff, and on only 24 measured-L3 items. On the 638 items with a measured radius, success shows no trend across levels (43.49%, 49.17%, 36.93%, 51.88%).
  • The pooled L3−L0 gap of −38.38 points reverses within operator to +12.09 points (95% CI +2.70 to +21.62), which the authors correctly identify as Simpson's paradox. A pooled rate and a within-group difference that disagree in sign cannot both describe the effect of radius.
  • The reach-back gap behaves the same way: −30.84 pooled, −25.11 within skill, but +1.76 (−1.57 to 5.17) within operator and −2.20 within skill × radius. Only 21 of 62 operators have items on both sides of 40 turns, and the 720–1,680-turn audits have no short-side comparison at all. Consequential because "we locate where these models fail" is the paper's central claimed contribution, and the abstract asserts the radius effect causally without the caveat that its own Section 3, Section 6.1 and Appendix A.4 supply. As written, a reader can reasonably conclude that breadth of state change causes failure, whereas the paper's data are equally consistent with "one family of synthetic long-horizon audits is hard." Either the abstract must be reframed descriptively, or a controlled comparison within a single operator across a wide radius/reach-back range must be produced.
  1. The primary metric conflates world-model capability with a provider's safety-classifier behaviour. 3,696 of 32,640 model–item decisions (11.3%) have no usable response, and the authors attribute essentially all of them to Anthropic's cyber safeguards — 1,309 of 1,632 items for Claude Fable 5 (80.21%), 1,044 for Claude Opus 5 (63.97%), 675 for Claude Opus 5.5 (41.36%). Scoring these as failures is defensible for deployment questions, but as the primary success metric it makes the leaderboard a joint measurement of simulation fidelity and provider policy, and it biases exactly the comparisons the paper draws: refusal rates vary sharply by domain (Claude Fable 5.1 refuses 374/526 Code items but 11/334 Games items) and by reach-back, so the reported per-domain and per-reach-back profiles are contaminated for four of twenty models. The paper is transparent about this and reports the alternative numbers (Opus 5.5 would score 44.93%, Fable 5.1 36.62%), which mitigates but does not resolve the issue — the abstract, Table 1 and the "no model solves even half" framing all use the refusal-inclusive metric. Consequential because the leaderboard is a headline deliverable, and because the "closed models pass 23.52% vs. open-weight 15.27%" comparison in RQ1 is not apples-to-apples when one group's denominator includes 19.33% no-response decisions.
Questions for authors
  1. Refusals. If the primary metric excluded refusals (or reported them as a separate "no response" axis), how would the ranking, the per-domain profile, and the claim that no model solves half the benchmark change? Please give the full leaderboard both ways, plus refusal rate by domain and by reach-back band.
  2. Confounding. Given that operator grouping explains 82% of the variance in log⁡(1+reach-back)\log(1+\text{reach-back}), that 953/977 L3 labels are declared rather than measured, and that the L3−L0 gap reverses within operator, what evidence supports the causal reading in the abstract that accuracy "decreases as actions affect a larger portion of the environment"? Can you construct a controlled comparison within a single operator spanning a wide range of reach-back and/or radius (a clarification request, not future work)?
1 comment

Question 1: Refusals

Table 1. Refusals scored as Failures

#ModelRefusal %Acc %
1GPT-6 Astra0.0049.57
2Grok 4.60.0028.13
3Claude Opus 5.541.3626.35
4Qwen 3.8 Max0.0025.31
5GPT-5.50.0025.18
6GPT-6 Sol0.0025.00
7GPT-5.6 Sol0.0023.65
8Qwen 3.8 Flash Next0.0023.22
9Gemini 3.7 Flash0.0021.75
10Claude Fable 5.140.9321.63
11Claude Opus 4.80.0020.04
12GPT-6 Luna0.0017.77
13Claude Opus 563.9715.93
14GLM 5.30.0014.77
15Qwen-AgentWorld 35B (WM)0.0014.46
16Kimi K30.0013.97
17Inkling0.0011.89
18Claude Fable 580.217.29
19Digital World Model (WM)0.005.21
20Laguna S2.10.002.45

Table 2. Refusals excluded

#ModelRefusal %Acc %Δrank
1GPT-6 Astra0.0049.570
2Claude Opus 5.541.3645.07+1
3Claude Opus 563.9744.52+10
4Claude Fable 5.140.9340.02+6
5Claude Fable 580.2136.84+13
6Grok 4.60.0028.13-4
7Qwen 3.8 Max0.0025.31-3
8GPT-5.50.0025.18-3
9GPT-6 Sol0.0025.00-3
10GPT-5.6 Sol0.0023.65-3
11Qwen 3.8 Flash Next0.0023.24-3
12Gemini 3.7 Flash0.0021.75-3
13Claude Opus 4.80.0020.04-2
14GPT-6 Luna0.0017.77-2
15GLM 5.30.0014.77-1
16Qwen-AgentWorld 35B (WM)0.0014.46-1
17Kimi K30.0013.97-1
18Inkling0.0011.89-1
19Digital World Model (WM)0.005.210
20Laguna S2.10.002.450

Table 3. Refusal percentage per domain

DomainFable 5Opus 5Opus 5.5Fable 5.1Items
code97.72%89.16%56.08%71.10%526
enterprise78.74%51.97%39.63%36.22%381
game44.91%25.75%15.87%3.29%334
dialogue89.64%77.72%45.08%31.61%193
math & science86.87%71.21%44.95%42.42%198
Total1632

Table 4. Refusal percentage per reach-back band

Reach-back (turns)Fable 5Opus 5Opus 5.5Fable 5.1Items
1-20071.07%46.99%38.87%39.66%764
201-40078.46%65.64%25.64%30.26%195
401-60081.46%70.20%17.22%21.19%151
601-80094.89%84.67%53.28%45.26%137
801-100096.90%96.12%62.79%55.04%129
1001-120096.24%87.97%56.39%50.38%133
1201-140063.16%84.21%10.53%42.11%19
1401-160097.78%86.67%82.22%73.33%45
1601-1800100.00%96.30%85.19%74.07%27
No reach-back (Pure reads)75.00%40.62%34.38%40.62%32
Total1632

Table 5. Refusal percentage per impact-radius

Impact radiusFable 5Opus 5Opus 5.5Fable 5.1Items
L069.80%41.21%36.41%42.45%563
L196.97%75.76%48.48%42.42%33
L238.98%35.59%28.81%25.42%59
L388.13%78.40%44.73%40.94%977
Total1632

Question 2: Confounding

We thank the reviewer for this question.

Impact radius is the reach of the state change an item is scored on. At L0L_0 the change does not touch any entity (entity is a trackable object in the environment's state), so the model only has to predict the returned output, at L1L_1 it touches one named entity, at L2L_2 two or more named entities, and at L3L_3 it reaches at least one entity it isn’t supposed to change. For 638 items we measured the radius directly from the execution diff.

Update: We have now measured the 983 audit items too, by replaying the trajectory of each one through its executable backend and recording how far it reaches. With these items measured, they no longer sit together in L3L_3 as "declared" and they spread across L0L_0, L1L_1, L2L_2 and L3L_3.

On the measured levels, accuracy still falls as impact radius grows. The before and after values are shown in the table below. All values are success percentages.

VersionL0L_0L1L_1L2L_2L3L_3L0→L3L_0 \to L_3
Table 8, Appendix Section A.4 (covers a subset of 638 items)43.527.127.55.1−38.4-38.4
Entire set of 1632 items32.522.97.64.7−27.8-27.8

We will replace the Section 6.1 numbers and Figure 3 along with the appendix in the camera ready submission with changes also made to Github and HuggingFace.

To address the concern that operator grouping drives the result, we also computed averaged accuracy within each operator family first, then across families with equal weight so large families don’t dominate the overall number. The resulting model ranking matches the base ranking with a Spearman correlation of 0.95.

Reviewer_7m2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed GlobeBench, a benchmark for evaluating text-based world models as environment simulators. It covers next-state prediction tasks across multiple domains, with items constructed by the authors or adapted from agent traces and existing environments. The evaluation is organized by capabilities and skills, and covers a large set of both closed and open-weight models. The experiment results show current LLMs are still far from being reliable general-purpose world simulators.

Critical strengths
  • The strongest part of the paper is the benchmark construction and validation. The gold next states are execution-derived from fixed backends rather than loosely annotated, and the authors further combine quality gates, human validation, and a hybrid grader.
  • The evaluation of LLMs is also comprehensive. The analysis also reveals where current models break, such as on long reach-back or larger impact-radius cases, beyond what was originally defined.
Critical weaknesses
  • One gap is that the benchmark is composed of next-state prediction tasks, while underweighting compounding errors over multiple steps. It is not clear to me how well GlobeBench's single-step prediction correlates with downstream utility.
  • The majority of the benchmark is built from the authors' own recipes and skill taxonomy. It is not clear to me whether the reported failure modes are artifacts of the authors' task design. The paper discusses prior work, but there is limited experimental comparison showing this benchmark captures something meaningfully different from earlier work.
Questions for authors
  • Do the results lead to different conclusions about model rankings or failure modes compared to prior work, or is the main contribution just a new evaluation taxonomy?
  • What is the main takeaway beyond showing current LLMs are not strong world models yet? If the benchmark is meant to be useful for RL training, how should we think about single-step prediction quality vs. actual multi-step simulation quality?
1 comment

Question 1: Main contribution and Prior work

Our benchmark goes beyond a taxonomy of skills. Each task isolates one skill, and its gold answer comes from the actual state reached after execution. This lets us identify which parts of a model's predicted state are wrong, rather than scoring the entire response as correct or incorrect. We can therefore study failure patterns that earlier evaluations could not distinguish. We restate our contributions below:

Benchmarks such as AgentWorldBench [1] and EnvSimBench [2] score the entire predicted observation, making it difficult to tell which capability failed. GlobeBench tests one of 19 skills in each task and records reach-back and impact radius, so we can identify which capability was being tested and where the prediction went wrong.

Our benchmark tests for concrete values and schemas independently, unlike existing benchmarks that utilize LLM judges to evaluate ground truth states. This makes our evaluation deterministic and more reproducible.

631 of 1,632 items are solved by no model, and only 4 by all 20. The top model's lead comes almost entirely from items the others never solve (GPT-6 Astra passes 33.5% of the hardest subset, versus a 2.2% median for the other 19), which a single leaderboard number can't capture.

Some directions that prior world-model evaluations do not report:

  • Dependency propagation: models predict local values but miss consequences for dependent and unnamed entities.
  • Long-horizon state tracking: accuracy falls to 11.44% once the state-setting event is more than 40 turns back.
  • Persistent rule enforcement: models honor the final action but drop rules set earlier, as in the exhausted-quota example. Access Control sits near the floor.
  • Training and rewards need to penalize plausible-but-wrong states, not just implausible ones.
  • Execution-grounded, per-skill evaluation: measuring any of the above requires skill-isolated, grounded grading rather than a holistic score, which is why the taxonomy is the means rather than the contribution.

The table below lists what GlobeBench measures and how it compares with the closest prior benchmarks, AgentWorldBench and EnvSimBench.

CapabilityGlobeBenchAgentWorldBenchEnvSimBench
Gold answer is the executed next state, not written by a human or a model✓✗✓
Each item isolates a single skill, rather than just scoring the final state.✓✗✗
Impact radius axis: how far an action's effects reach✓✗✗
Reach-back axis: how far back the deciding event is✓✗✗
Multiple domains in one benchmark✓✓✓
Grader checked against human expert judgments✓✗✗

Question 2: Takeaways and Single-step prediction

Section 2 describes method papers for simulating environments with world models [3], but there are no reliable benchmarks for measuring how well those simulators actually perform. Existing efforts such as Qwen-AgentWorld [2] cover far fewer capabilities, while GlobeBench isolates 19 skills and traces every failure back to the specific capability the simulator is missing. The takeaway is not simply that today’s LLMs are weak as world models. It is that GlobeBench identifies exactly which capabilities future models need to improve.

We evaluate single-step predictions deliberately. Scoring one step at a time gives us control over the state for each prediction and lets us verify every step in the trace against gold. Generating a whole trace in one shot would let errors compound and make it hard to tell where and why a simulation went wrong.

References:

[1] Zuo, Yuxin, et al. "Qwen-AgentWorld: Language World Models for General Agents." arXiv preprint arXiv:2606.24597 (2026).

[2] Liu, Yi, et al. "EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation." arXiv preprint arXiv:2605.07247 (2026).

[3] Deshpande, Darshan. "Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL." arXiv preprint arXiv:2607.16204 (2026).

Reviewer_2ZQSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GlobeBench, a benchmark for evaluating text-based world models as environment simulators. It contains 1,632 tasks across five domains and 19 simulation skills, with ground-truth responses obtained from executable environments. The authors evaluate 20 models using a combination of deterministic checks and LLM-based judging, and analyze their performance across different skills, impact radius, and history lengths.

Critical strengths
  1. I find the benchmark design quite useful. Evaluating world models through actual environment transitions rather than just plausible text generation is important for agent research. The 19 skills provide a helpful way to examine different simulation capabilities, and the executable backends make the evaluation more convincing.

  2. I also appreciate the detailed experimental analysis and the authors' response to my concerns. In particular, re-measuring the impact radius across the benchmark addresses an important issue in the original evaluation. The additional grading analysis and ranking comparisons are also helpful. I am more confident in the overall benchmark results after reading the rebuttal.

Critical weaknesses

Although the updated impact-radius analysis is much clearer, I think the interpretation could still be more careful. Impact radius, history length, and task families are closely related in this benchmark, so it remains difficult to separate their individual effects. I would prefer to see these results presented mainly as diagnostic observations rather than independent explanations of model failures.

I appreciate the authors' detailed response and the effort to address the original concerns. The additional analyses have improved my impression of the paper, and I am willing to raise my rating from 2 to 3 (Weak Acceptance). I believe GlobeBench can be a useful benchmark for future world-model research.

Questions for authors

None.

1 comment

Question 1: Impact-radius validity and controlled comparisons

We thank the reviewer for this question. It points to an ambiguity in how we presented the L3 label.

Impact radius is the reach of the state change an item is scored on. At L0L_0 the change does not touch any entity (entity is a trackable object in the environment's state), so the model only has to predict the returned output, at L1L_1 it touches one named entity, at L2L_2 two or more named entities, and at L3L_3 it reaches at least one entity it isn’t supposed to change. For 638 items we measured the radius directly from the execution diff.

Update: We have now measured the 983 audit items too, by replaying the trajectory of each one through its executable backend and recording how far it reaches. With these items measured, they no longer sit together in L3L_3 as "declared" and they spread across L0L_0, L1L_1, L2L_2 and L3L_3.

On the measured levels, accuracy still falls as impact radius grows. The before and after values for the model-set mean are shown in the table below. All values are success percentages.

VersionL0L_0L1L_1L2L_2L3L_3L0→L3L_0 \to L_3
Table 8, Appendix Section A.4 (covers a subset of 638 items)43.527.127.55.1−38.4-38.4
Entire set of 1632 items32.522.97.64.7−27.8-27.8

We will replace the Section 6.1 numbers and Figure 3 along with the appendix in the camera ready submission. The re-measured impact radius will be updated on HuggingFace and GitHub.

Question 2: Human validation of the final response grader.

We use deterministic rules to evaluate about 85% of answered responses and their verdicts align with the LLM judge evaluations. Only the remaining 15% which are ambiguous rely on the LLM judge for the final verdict.

We also conducted a study of 10 L3L_3 difficult items. The human scores match the wm-response-judge/4 (Appendix Section B.4) on GPT-6 Astra predictions in 95% of passing attempts and 100% of failing attempts.

We manually reviewed the 392 items scoring below 4 (5 point scale) in the human evaluation and found most of them to be noise or false positives.

Question 3: Robustness to output granularity and task-family composition.

The overall ranking is consistent across the different variants. We list them here:

  • We re-scored every response with grading each field separately instead of marking the whole response right or wrong, and the Spearman rank correlation between that ranking and the base (binary) ranking is 1.00.
  • We did not include an entity-level correctness metric because it is not defined for the lower levels: L0L_0 changes no entity, and L1L_1 changes a single named one, so we cannot partially score them. That leaves only L2L_2 and L3L_3, with the Spearman rank correlation of 0.586
  • We also averaged accuracy within each operator family first, then across families with equal weight so large families don’t dominate the overall number. The resulting ranking matches the base ranking with a Spearman rank correlation of 0.95.
  • We removed the synthetic audit families (families are synthetic in construction but each item is still executed in a real backend) and rescored the remaining items which, resulted in a Spearman rank correlation of 0.97.

GPT-6 Astra stays on top in all three.

Comments on the paper

1 comment

As the deadline for the rebuttal phase approaches, we would be happy to address any additional concerns about the paper. We look forward to your engagement and thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.