- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces DECE, a unified benchmark of ten Eurogames for evaluating LLM agents. Its main contribution is a shared evaluation framework that compares complete model–harness systems across games while also considering token usage and cost. Experiments on 1,032 matches show broadly consistent cross-game rankings, alongside game- and harness-specific differences.
Critical strengths
The paper evaluates several recent models, which is a notable strength.
Critical weaknesses
Here are several weaknesses:
- The paper does not sufficiently motivate why benchmarking LLM agents on Eurogames is an important research problem. More importantly, despite the substantial experimental effort, the benchmark yields few novel or substantive insights beyond model rankings and general performance.
- The overall writing and presentation need substantial improvement.
Questions for authors
Please refer to weaknesses.
4 comments
Thanks! Response (2/2): Findings Beyond Rankings and Presentation
W1 — What does the paper establish beyond model rankings?
The submitted experiments provide three findings with concrete implications for agent evaluation:
-
Cross-game strength is informative, but game-specific evaluation remains necessary. In the leave-one-game-out analysis, rankings constructed from nine games achieve 83.9% mean pair-order accuracy on the held-out game, with mean Kendall τb = 0.664. Together with the observed local reversals, this quantifies both the consistency and the limits of aggregate rankings. Researchers can assess how well a configuration’s competitive standing carries across rule systems, rather than assuming that its overall rank describes every game.
-
Harness choice can substantially affect the performance attributed to a model. In the submitted matched Splendor experiment, Qwen3.8-Max achieves 33.3% PWS with OpenCode and 91.7% with Qwen Code, with all six matched scenarios favoring Qwen Code. Setup seeds, focal seats, and opponent configurations are fixed within each scenario. This supports a concrete evaluation implication: the execution harness must be specified and assessed as part of the agent configuration. The paper reports uncertainty and the local scope of this comparison; the inference concerns the complete harness configuration.
-
Resource criteria change which configurations are competitive choices. The token-based and dollar-based Pareto frontiers identify different candidate configurations, and membership varies across games. Thus, token consumption, monetary cost, and competitive performance provide distinct information for configuration selection. These resource comparisons concern recorded consumption associated with retained completed matches, as specified in the resource-accounting appendix.
These findings establish cross-game consistency, substantial local sensitivity to harness choice, and game-dependent performance–resource trade-offs. The shared experimental framework makes these aspects jointly inspectable and reproducible.
W2 — Writing and presentation
In the consolidated manuscript revision, we will clarify the research motivation in the Introduction, foreground the above empirical takeaways in the Results and Conclusion, and make the distinction between measured configuration effects and their possible underlying mechanisms explicit. These explanations are provided here so they can be assessed before that revision is submitted. The evidence supporting these points is already contained in the submitted experiments.
We would appreciate clarification of any specific experimental protocol, analysis, or conclusion underlying the soundness assessment, so that we can address that concern directly.
Thank you for the rebuttal and clarifications. However, my main concern about the significance of the contribution remains unresolved. While Eurogames involve resource competition and long-term planning, the paper does not sufficiently establish what evaluating LLMs in this setting reveals beyond their performance on these particular games.
In particular, observing that LLMs struggle with such tasks is not, by itself, a sufficiently informative finding without a clearer account of the underlying limitations and their relevance beyond the benchmark. The rebuttal does not convincingly explain how these results could guide subsequent research or improvements in more general agent capabilities (E.g., post-training). My concern is therefore not with games as an evaluation setting, but with the limited actionable insight provided by this particular benchmark.
For these reasons, I will maintain my original score. Thank you again for the thoughtful response.
What DECE makes measurable
Thank you for asking how DECE could guide agent development. In the introduction, we model strategic interactions in Eurogames as partially observable stochastic games (POSGs). DECE provides a platform for studying how a model and its harness handle these interactions by recording what the agent observed, which actions were available, and what it chose.
The case study in Appendix H.2 compares the same model, Qwen3.8-Max, under two harnesses. With Qwen Code, it made greater use of early discounts than with OpenCode in five of six matched Splendor scenarios. In 7 Wonders, it made fewer free builds but also saw fewer opportunities to build for free.
These observations suggest a question for both harness design and post-training: what should an agent remember and do as its options change? With the model fixed, researchers could compare a harness that summarizes past actions with one that also tracks which plans remain viable. For post-training, they could annotate replay segments with short explanations of what changed and why it matters for the next decision, then use those examples to teach the model to produce its own summaries. Held-out games could test whether this improves play under the same harness and token budget.
Thanks! Response (1/2): Motivation for Evaluating Agents on Eurogames
Thank you for your feedback. We appreciate the opportunity to clarify the motivation for DECE and the findings beyond aggregate rankings supported by the submitted experiments.
W1 — Why evaluate LLM agents on Eurogames?
Our motivation is to evaluate whether LLM agents can sustain effective decisions as earlier choices shape later opportunities and other players change the available options. This requires both model reasoning and execution support for maintaining relevant information across turns.
This direction has concrete research precedents. BALROG (ICLR 2025) evaluates agentic reasoning through interactive games, while LMGame-Bench (ICLR 2026) examines perception, memory, and reasoning through modular execution support. GAMA-Bench (ICLR 2025) studies multi-agent decision-making, and LudoBench (ICLR 2026) uses strategy tabletop games to assess situated perception, rule integration, and short-horizon optimization.
Eurogames offer controlled settings where persistent investments, resource competition, and delayed scoring connect decisions across time. In Splendor, early development cards change the costs of later purchases; in Azul, placements constrain subsequent scoring opportunities. Explicit rules and terminal scoring make outcomes measurable and support reproducible comparisons.
DECE examines complete native agent configurations across ten rule systems. A common participant roster supports cross-game comparisons, while matched harness experiments assess how execution choices affect performance under fixed scenarios, seats, and opponents. These properties connect the choice of environments directly to the empirical questions addressed by the submitted experiments.
References: