acceptodds
Under review as a conference paper at ICLR 2027

CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds

Abstract

The growing use of language models for forecasting motivates a broader evaluation of their world-model capabilities. Such evaluation must go beyond predictive accuracy to assess hidden-state inference and intervention prediction. Real-world forecasting benchmarks offer limited access to hidden-state ground truth and controlled experiments for evaluating intervention predictions. We introduce CivWorldBench, a benchmark built on a multi-agent strategy game engine with recorded internal states, controlled observations, and paired intervention rollouts. Its 1,900 decision points from 475 worlds support three task levels: association forecasting, belief inference, and intervention prediction. Controlled observations and reference predictors support attributable evaluation of capability gaps, with an expert harness and empirical ceiling estimates contextualizing scores. Stored trajectories make intermediate predictions verifiable, while agent-play experiments test whether world-model capabilities are usable for decision-making. The main panel comprises eleven frontier language models spanning different capability levels. Composite benchmark scores correlate with real-world forecasting leaderboards on matched model panels, more closely than with a general-capability index, supporting its use as a proxy for real-world forecasting capability. Strong association forecasts do not consistently extend to belief inference or intervention prediction, and effect magnitudes remain poorly calibrated despite informative direction predictions. Process verification reveals errors missed by final-answer evaluation, and structured reasoning improves intervention prediction for the strongest model in the process study. Agent-play experiments identify useful assisted plans and further motivate evaluating decision-relevant capabilities beyond association forecasting. These findings identify hidden-state inference, intermediate-state prediction, and effect calibration as priorities for world-model training.

Primary area:
datasets and benchmarks
Keywords:
World Models, Benchmark, Forecasting, Large Language Models, Multi-agent Simulation, Causal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZwPzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CivWorldBench, a structured benchmark designed to evaluate LLMs’ world-model capabilities. It is built on a multi-agent strategy game engine, i.e., Freeciv, with recorded internal states, controlled observations, and paired intervention rollouts. The benchmark consists of 475 simulated worlds, 1,900 decision points, and 18,499 prompts, and it defines the composite Sandbox World Forecasting Index (SWFI) aggregated across task levels and settings. Specifically, it covers three hierarchical task levels: L1 association forecasting, L2 belief inference, and L3 intervention prediction, and with three evaluation settings, including OMNI ID, EGO, and OMNI OOD. The authors evaluate 11 frontier LLMs and presents some key empirical findings, e.g., composite benchmark scores correlate with real-world forecasting leaderboards, and there exists model deficiencies in intermediate-state prediction, hidden-state inference, and intervention effect calibration.

Critical strengths
  1. The authors introduce CivWorldBench, a unified multi-agent benchmark built upon the Freeciv game engine capable of logging internal hidden parameters. It covers association, hidden-state inference, and counterfactual intervention with verifiable ground truth. Unlike most existing benchmarks, which lack ground truth for hidden internal mechanisms and only record final outcomes, this benchmark captures latent engine states and paired intervention rollouts. This design supports process verification and attributable diagnosis of prediction failures rather than just evaluating final-answer accuracy, allowing researchers to conduct paired controlled experiments.
  2. The paper conducts an empirical assessment that connects benchmark metrics to both external real-world forecasting performance and in-simulation agent decision utility. It validates SWFI on independent real-world forecasting leaderboards and carries out agent-play studies, showing that its composite SWFI index exhibits strong correlation with human forecasting leaderboards and serves as a valid proxy for real-world predictive capability. This structural design effectively separates observational pattern-matching from underlying causal and strategic understanding, reinforcing that the benchmark captures practically relevant capabilities instead of only game-specific pattern-matching.
Critical weaknesses
  1. The multi-agent dynamics are fully generated by the Freeciv game engine and its modified built-in AI agents. Even though the benchmark’s generalization and OOD settings change rulesets, map layouts and difficulty levels, the authors do not test transfer across different simulators. This means we cannot verify if these findings apply to other simulators or real human social systems. The high benchmark performance therefore cannot definitively guarantee equivalent real-world performance. In addition, all agents are Freeciv bots with rigid strategies, lacking flexible deception or other typical behaviors in general multi agent social environments. This means observed model failure modes may not transfer to scenarios with LLM opponents, limiting the benchmark’s generalizability for social world modelling.
  2. The composite index SWFI assigns equal weights across the three task levels (L1 Association, L2 Belief, L3 Intervention) and across settings (OMNI-ID, EGO, OMNI-OOD). This uniform distribution is not empirically calibrated based on downstream decision utility. As a single scalar metric, SWFI collapses multi-dimensional model performance and masks cross-task capability imbalances, simplifying world-model capacity and obscuring critical skill tradeoffs. Additionally, the paper does not systematically evaluate prompt sensitivity for L2 and L3 tasks. Observed performance differences may stem from prompt design rather than inherent world-model capabilities of LLMs.
Questions for authors
  1. The benchmark mixes three evaluation settings: OMNI ID, EGO, OMNI OOD. Could you clarify what exact distribution shift differentiates OMNI OOD from the basic OMNI ID set?
  2. When computing SWFI, all component metrics and settings are assigned equal fixed weights that are pre-specified prior to model evaluation. Did you conduct ablation experiments to examine the sensitivity of SWFI rank orderings to alternative plausible weighting schemes over these metrics and settings?
  3. Your results show that structured reasoning helps strong models but hurts weak ones. Have you identified what causes weaker models to perform worse under these structured output protocols? Can prompt calibration fix negative Capture, or do structural internal state flaws dominate?
  4. You should include the harness into your evaluation, e.g., codex, claude code, and dsh, which can sometime increase the performance significantly. Please add more results.
2 comments
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer ZwPz,

We hope this message finds you well. Thank you again for your detailed review and suggestions. As the discussion period nears its end, we would like to kindly remind you of our response and briefly point to the results we added at your request, all in the revised PDF (App. P, changes in blue):

  • Generator strata (W1.1): the ranking agrees with the full-set ranking within each of 15 generator strata (τ ≥ 0.67; App. P.2).
  • SWFI weighting (W2.1, Q2): across 20,000 random weightings, the ranking stays close to the submitted one (median Kendall τ = 0.89; App. P.3).
  • Prompt sensitivity (W2.2): with a paraphrased and a reordered prompt, only 5 of 110 paired tests exclude zero, about the chance level (App. P.4).
  • Harnesses (Q4): we ran Codex and DeepSeek Harness, two of the harnesses you suggested. They raise Flash-Next's SWFI from 14.7 to 35.7 and 39.4, mostly because the model answers with its own factual forecast at hand; Capture stays negative under every harness (App. P.5).

We would be grateful to know whether these results address your concerns. If any point remains unclear, we are glad to clarify it before the discussion closes.

Best regards,

Authors of Submission 39690

Thank you for your responses together with new supplementary results. Most of my original concerns have been addressed, and I have updated my rating accordingly.

Reviewer_oXnGOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a benchmark to evaluate how well language models on a multi-agent strategy game. The models are evaluate on how well they can predict future state of the game, infer the hidden value of other players' traits and game settings, and predict the consequences of an intervention. For each evaluation aspect, the benchmark provides reference solutions to compare models with, and proposes appropriate metrics. The paper also includes sanity checks as well as insights on how each metric of the benchmark relate to other metrics or other benchmarks.

Critical strengths

It is clear that meaningful effort is put both on the paper and the benchmark, and the contribution goes beyond designing some tasks and reporting the performance of some models on it. I praise the authors for the high-quality work.

Here I briefly mention the aspects that I particularly appreciated:

  • Coverage: The benchmark spans a wide range of configurations and horizons. It provides 475 worlds and 1,900 decision points.
  • Evaluated Aspects: The benchmark goes beyond predicting a certain quantity or even into the future. Testing the ability to inferr the parameters for other player's traits and the result of interventions gives insights on the model's deeper understandings of the world.
  • References: Multiple references are designed to make sense of the model performance numbers. These baselines allow understanding where the models can improve the most.
  • Metrics: The metrics used to report the performance of the models in some tasks are non-trivial and appear to be well-thought.
  • Validity Checks: In Section 5, Insights 1 works as a nice sanity check and insight 3 is reassuring to see the benchmark correlates with other forecasting benchmarks more than the general benchmark suites.
  • Insights: I liked how different reasoning structures, including code, is tested (Section 6, insight 3), examining sign and magnitude of predictions (Figure 4), and relating model's utility as world model with its L2 and L3 performance (Section 7)
Critical weaknesses

I mainly have two concerns about the current version: clarity of the main text, and statistical significance of the evaluations.

Statistical significance. In all evaluations of language models, there is some stochasticity due to the positive sampling temperature. It appears to me that the performance numbers in this benchmark have an extra source of noise due to game engine mechanics. I suggest the authors to include significant discussions on this. I see Figure 6 has shaded areas for the confidence interval and there is a brief discussion in Section G.5. I might have missed some results in the appendix, but many reported numbers don't have confidence intervals or standard errors. I understand this might make tables and figures crowded, but the range of possible errors could be mentioned. As a technical capability of the benchmark, I wonder if it supports multiple game engine runs to reduce the engine-side noise and how computationally expensive they are.

Main text clarity. This is fairly minor. I find the main text to be too dense sometimes. Some terms are only defined in a concise vague way or not clearly defined at all. In all cases, I could clarify them with the supplementary material. I understand the space constraints, but I prefer having less results in main text that are more easily understandable. Some examples: "intermediate-state skill over the trend reference" in line 399, " final-answer reference" in line 403, cells, and anchors.

Questions for authors

Please see weakness.

Reviewer_serCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CivWorldBench. It ls a benchmark for evaluating language world models in multi-agent environments built on Freeciv. It has three levels of capability evaluation, e.g. association forecasting (L1), belieft inference (L2), and intervention prediction (L3). The benchmark contains 475 simulated worlds and 1,900 decision points, with recorded internal states and controlled intervention rollouts providing verifiable ground truth.

The paper evaluates 11 language models across different observation settings and find that strong forecasting performance does not necessarily translate into accurate hidden-state inference or intervention prediction.

Critical strengths

Comprehensive and verifiable benchmark design. The benchmark covers forecasting, hidden-state inference, and intervention prediction within the same environment. Having access to the internal game states and paired intervention rollouts allows for a more detailed evaluation beyond just final prediction accuracy. The different observation settings and reference predictors also make the evaluation more convincing.

The experiments show that good forecasting performance does not necessarily mean a model is also good at hidden-state inference or intervention prediction. The intermediate-state analysis is also interesting, as models can get the final prediction right even when their intermediate predictions are wrong.

Critical weaknesses

The contribution of world modeling to downstream decision-making is unclear. The full-game improvements are not statistically significant (Appendix N.5), and individual world-model assistance shows no clear gains over self-review. The reported 2.36-point improvement comes from adding more candidate plans and selecting them based on actual rollouts. It is therefore unclear whether the gains come from better world modeling or simply from additional planning and computation.

Questions for authors

same as weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.