- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CivWorldBench, a structured benchmark designed to evaluate LLMs’ world-model capabilities. It is built on a multi-agent strategy game engine, i.e., Freeciv, with recorded internal states, controlled observations, and paired intervention rollouts. The benchmark consists of 475 simulated worlds, 1,900 decision points, and 18,499 prompts, and it defines the composite Sandbox World Forecasting Index (SWFI) aggregated across task levels and settings. Specifically, it covers three hierarchical task levels: L1 association forecasting, L2 belief inference, and L3 intervention prediction, and with three evaluation settings, including OMNI ID, EGO, and OMNI OOD. The authors evaluate 11 frontier LLMs and presents some key empirical findings, e.g., composite benchmark scores correlate with real-world forecasting leaderboards, and there exists model deficiencies in intermediate-state prediction, hidden-state inference, and intervention effect calibration.
Critical strengths
- The authors introduce CivWorldBench, a unified multi-agent benchmark built upon the Freeciv game engine capable of logging internal hidden parameters. It covers association, hidden-state inference, and counterfactual intervention with verifiable ground truth. Unlike most existing benchmarks, which lack ground truth for hidden internal mechanisms and only record final outcomes, this benchmark captures latent engine states and paired intervention rollouts. This design supports process verification and attributable diagnosis of prediction failures rather than just evaluating final-answer accuracy, allowing researchers to conduct paired controlled experiments.
- The paper conducts an empirical assessment that connects benchmark metrics to both external real-world forecasting performance and in-simulation agent decision utility. It validates SWFI on independent real-world forecasting leaderboards and carries out agent-play studies, showing that its composite SWFI index exhibits strong correlation with human forecasting leaderboards and serves as a valid proxy for real-world predictive capability. This structural design effectively separates observational pattern-matching from underlying causal and strategic understanding, reinforcing that the benchmark captures practically relevant capabilities instead of only game-specific pattern-matching.
Critical weaknesses
- The multi-agent dynamics are fully generated by the Freeciv game engine and its modified built-in AI agents. Even though the benchmark’s generalization and OOD settings change rulesets, map layouts and difficulty levels, the authors do not test transfer across different simulators. This means we cannot verify if these findings apply to other simulators or real human social systems. The high benchmark performance therefore cannot definitively guarantee equivalent real-world performance. In addition, all agents are Freeciv bots with rigid strategies, lacking flexible deception or other typical behaviors in general multi agent social environments. This means observed model failure modes may not transfer to scenarios with LLM opponents, limiting the benchmark’s generalizability for social world modelling.
- The composite index SWFI assigns equal weights across the three task levels (L1 Association, L2 Belief, L3 Intervention) and across settings (OMNI-ID, EGO, OMNI-OOD). This uniform distribution is not empirically calibrated based on downstream decision utility. As a single scalar metric, SWFI collapses multi-dimensional model performance and masks cross-task capability imbalances, simplifying world-model capacity and obscuring critical skill tradeoffs. Additionally, the paper does not systematically evaluate prompt sensitivity for L2 and L3 tasks. Observed performance differences may stem from prompt design rather than inherent world-model capabilities of LLMs.
Questions for authors
- The benchmark mixes three evaluation settings: OMNI ID, EGO, OMNI OOD. Could you clarify what exact distribution shift differentiates OMNI OOD from the basic OMNI ID set?
- When computing SWFI, all component metrics and settings are assigned equal fixed weights that are pre-specified prior to model evaluation. Did you conduct ablation experiments to examine the sensitivity of SWFI rank orderings to alternative plausible weighting schemes over these metrics and settings?
- Your results show that structured reasoning helps strong models but hurts weak ones. Have you identified what causes weaker models to perform worse under these structured output protocols? Can prompt calibration fix negative Capture, or do structural internal state flaws dominate?
- You should include the harness into your evaluation, e.g., codex, claude code, and dsh, which can sometime increase the performance significantly. Please add more results.
2 comments
Thank you for your responses together with new supplementary results. Most of my original concerns have been addressed, and I have updated my rating accordingly.
Follow-up on Our Response
Dear Reviewer ZwPz,
We hope this message finds you well. Thank you again for your detailed review and suggestions. As the discussion period nears its end, we would like to kindly remind you of our response and briefly point to the results we added at your request, all in the revised PDF (App. P, changes in blue):
We would be grateful to know whether these results address your concerns. If any point remains unclear, we are glad to clarify it before the discussion closes.
Best regards,
Authors of Submission 39690