- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces GlobeBench, a benchmark for evaluating LLMs as text-based world models: given a multi-turn trajectory and a final action , the model must predict the next environment state , which is compared against a gold obtained by executing against a pinned backend (a container digest, or locked dependency hashes) rather than by annotation. Composition. 1,632 items across five domains — Code (526), Enterprise (381), Games (334), Math & Science (198), Dialogue (193). Items are organized by two new taxonomies: four capabilities (State Coherence, Causal Correctness, Interface Fidelity, Constraint Enforcement) and 19 skills (e.g., Read/Write Persistence, Access Control, Schema Conformance, Quota Enforcement, Sycophantic Confirmation). Each item is produced by pairing a domain recipe with one of 62 operators; an operator fixes the action sequence for exactly one primary skill, and can insert filler turns that leave the tested state untouched. Every item is labelled with reach-back (turns between the deciding event and the final action) and impact radius (L0 = no state change; L1/L2 = one / two-or-more named entities change; L3 = at least one unnamed entity changes). Quality control. 12 item-level gates (deterministic checks plus LLM validity/ambiguity/"tell" judges) reject items that are unsolvable, ambiguous, non-reproducible, or detectable from surface cues. Validation combines a 6-annotator human study of 200 items and a two-phase LLM audit of 1,627 items. Grading. A frozen hybrid grader applies deterministic metrics first (exact/structural match, numeric fidelity, error class + referent, order sensitivity, under-/over-erroring); only responses that no metric settles go to a blinded LLM judge (GPT-5.6 Terra, with GPT-5.6 Sol as fallback). Contributions. (i) The benchmark, harness, model outputs, judge records and reproduction instructions; (ii) the capability/skill taxonomies plus the reach-back and impact-radius axes, which aim to attribute a failure to a named mechanism rather than score a predicted observation holistically (as AgentWorldBench and EnvSimBench do); (iii) an empirical study around three RQs — which parts of a predicted state models get wrong and whether open- and closed-weight models fail differently; failure modes that aggregate success hides; and how automated judgments align with human judgments. Validation / findings. 20 models are evaluated zero-shot, one sample per item, with provider refusals scored as failures. GPT-6 Astra leads at 49.57%, followed by Grok 4.6 (28.13%), Claude Opus 5.5 (26.35%) and Qwen 3.8 Max (25.31%); pooled success is 19.68%, and 631 items are solved by no model (87.6% of them L3, 93.7% beyond 40 turns of reach-back). Success falls from 43.49% at L0 to 5.10% at L3, and from 42.28% within 40 turns to 11.44% beyond. Most answered failures (53.21%) report a wrong value. Models score far higher on error-outcome items (53.16%) than normal ones (16.16%). Both specialized world models rank low (Qwen-AgentWorld 14.46%, DWM 5.21%), which the authors read as evidence that world modelling is an emergent capability of general scale rather than something instilled by task-specific training.
Critical strengths
- Execution-grounded golds, one mechanism per item, and a deterministic-first grader — with a complete release. This is the paper's strongest reason for acceptance. Because every gold is executed rather than annotated, because each item is built to isolate a single named skill and is filtered by 12 gates that discard items solvable without exercising it, and because the grader settles everything it provably can before invoking an LLM, a failure can be localised to a mechanism instead of merely counted. That is exactly the property prior world-model evaluations (AgentWorldBench, EnvSimBench) lack, and it is what makes the paper's diagnostic findings — wrong-value failures, hidden-state drift, the error/normal gap — credible at all. Combined with the release of model outputs and judge records (so results can be rescored without re-querying models), this makes the benchmark a durable community resource. Consequential because a benchmark paper's value is its measurement instrument, and this instrument is a real advance.
- The reach-back / impact-radius decomposition, and the hidden-state finding. Even granting the confounding discussed below, the axes yield a genuinely new and actionable diagnosis: the bottleneck is not rule application but carrying state forward. The evidence is specific and quantitative — on the quorum family, GPT-6 Astra reports the exact capability on 84% of requests versus 2–8% for the strongest peers, while those peers' weights follow correctly from their own (wrong) capability 75–99% of the time; and once a peer gets the state right, its decision is right 61–91% of the time against >99% for Astra. The 631-unsolved-item analysis (85.1% concentrated in six skills) is also a useful map of where to work next. Consequential because it changes what a follow-up training effort should target.
Critical weaknesses
- The headline causal claims about impact radius (and reach-back) are not supported by the paper's own within-group analyses. Impact radius, reach-back, primary skill, operator and output length are correlated by construction: the paper states that each of the 62 operators fixes both reach-back and radius, that operator grouping explains of the variance in (37.41% for skill), and that reach-back correlates with prompt length at Spearman . Three pieces of the paper's own evidence then undercut the causal reading of its most prominent number:
- The abstract's "task success drops to an average of 5.1%" at L3 rests on 953 of 977 L3 items whose level is declared by the operator, not measured from an execution diff, and on only 24 measured-L3 items. On the 638 items with a measured radius, success shows no trend across levels (43.49%, 49.17%, 36.93%, 51.88%).
- The pooled L3−L0 gap of −38.38 points reverses within operator to +12.09 points (95% CI +2.70 to +21.62), which the authors correctly identify as Simpson's paradox. A pooled rate and a within-group difference that disagree in sign cannot both describe the effect of radius.
- The reach-back gap behaves the same way: −30.84 pooled, −25.11 within skill, but +1.76 (−1.57 to 5.17) within operator and −2.20 within skill × radius. Only 21 of 62 operators have items on both sides of 40 turns, and the 720–1,680-turn audits have no short-side comparison at all. Consequential because "we locate where these models fail" is the paper's central claimed contribution, and the abstract asserts the radius effect causally without the caveat that its own Section 3, Section 6.1 and Appendix A.4 supply. As written, a reader can reasonably conclude that breadth of state change causes failure, whereas the paper's data are equally consistent with "one family of synthetic long-horizon audits is hard." Either the abstract must be reframed descriptively, or a controlled comparison within a single operator across a wide radius/reach-back range must be produced.
- The primary metric conflates world-model capability with a provider's safety-classifier behaviour. 3,696 of 32,640 model–item decisions (11.3%) have no usable response, and the authors attribute essentially all of them to Anthropic's cyber safeguards — 1,309 of 1,632 items for Claude Fable 5 (80.21%), 1,044 for Claude Opus 5 (63.97%), 675 for Claude Opus 5.5 (41.36%). Scoring these as failures is defensible for deployment questions, but as the primary success metric it makes the leaderboard a joint measurement of simulation fidelity and provider policy, and it biases exactly the comparisons the paper draws: refusal rates vary sharply by domain (Claude Fable 5.1 refuses 374/526 Code items but 11/334 Games items) and by reach-back, so the reported per-domain and per-reach-back profiles are contaminated for four of twenty models. The paper is transparent about this and reports the alternative numbers (Opus 5.5 would score 44.93%, Fable 5.1 36.62%), which mitigates but does not resolve the issue — the abstract, Table 1 and the "no model solves even half" framing all use the refusal-inclusive metric. Consequential because the leaderboard is a headline deliverable, and because the "closed models pass 23.52% vs. open-weight 15.27%" comparison in RQ1 is not apples-to-apples when one group's denominator includes 19.33% no-response decisions.
Questions for authors
- Refusals. If the primary metric excluded refusals (or reported them as a separate "no response" axis), how would the ranking, the per-domain profile, and the claim that no model solves half the benchmark change? Please give the full leaderboard both ways, plus refusal rate by domain and by reach-back band.
- Confounding. Given that operator grouping explains 82% of the variance in , that 953/977 L3 labels are declared rather than measured, and that the L3−L0 gap reverses within operator, what evidence supports the causal reading in the abstract that accuracy "decreases as actions affect a larger portion of the environment"? Can you construct a controlled comparison within a single operator spanning a wide range of reach-back and/or radius (a clarification request, not future work)?
Question 1: Refusals
Table 1. Refusals scored as Failures
Table 2. Refusals excluded
Table 3. Refusal percentage per domain
Table 4. Refusal percentage per reach-back band
Table 5. Refusal percentage per impact-radius
Question 2: Confounding
We thank the reviewer for this question.
Impact radius is the reach of the state change an item is scored on. At L0 the change does not touch any entity (entity is a trackable object in the environment's state), so the model only has to predict the returned output, at L1 it touches one named entity, at L2 two or more named entities, and at L3 it reaches at least one entity it isn’t supposed to change. For 638 items we measured the radius directly from the execution diff.
Update: We have now measured the 983 audit items too, by replaying the trajectory of each one through its executable backend and recording how far it reaches. With these items measured, they no longer sit together in L3 as "declared" and they spread across L0, L1, L2 and L3.
On the measured levels, accuracy still falls as impact radius grows. The before and after values are shown in the table below. All values are success percentages.
We will replace the Section 6.1 numbers and Figure 3 along with the appendix in the camera ready submission with changes also made to Github and HuggingFace.
To address the concern that operator grouping drives the result, we also computed averaged accuracy within each operator family first, then across families with equal weight so large families don’t dominate the overall number. The resulting model ranking matches the base ranking with a Spearman correlation of 0.95.