acceptodds
Under review as a conference paper at ICLR 2027

DECE: A Unified Arena for LLM Agents across Diverse Eurogames

Abstract

Interactive environments offer a practical way to evaluate large language model (LLM) agents, and competitive games extend this evaluation to decisions shaped by competing goals and other agents' choices. Yet existing evaluations confined to individual games or execution frameworks leave unclear which competitive advantages persist across settings. We introduce (iverse urogame ompetitive nvironments), a unified arena spanning ten Eurogames with shared interfaces and evaluation protocols. These games combine lasting strategic commitments, competition over shared resources, and multiple routes to victory. Our primary study evaluates 12 model–harness configurations, spanning nine models and four execution harnesses, in 480 three-player matches across ten games. The main and supplementary studies comprise 1,032 completed matches in total. These comparisons reveal broad cross-game ranking agreement alongside game-specific strengths. Matched studies show that the same model can perform differently across execution harnesses, adding another source of variation to these profiles. These performance advantages do not consistently translate into better token or cost efficiency, making resource use an additional consideration when comparing configurations. Together, the findings support evaluating complete agent configurations by both their competitive performance across games and the resources needed to achieve it.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, competitive evaluation, game-based benchmarks, Eurogames, execution harnesses, resource efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g8FWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DECE, a unified benchmark of ten Eurogames for evaluating LLM agents. Its main contribution is a shared evaluation framework that compares complete model–harness systems across games while also considering token usage and cost. Experiments on 1,032 matches show broadly consistent cross-game rankings, alongside game- and harness-specific differences.

Critical strengths

The paper evaluates several recent models, which is a notable strength.

Critical weaknesses

Here are several weaknesses:

  1. The paper does not sufficiently motivate why benchmarking LLM agents on Eurogames is an important research problem. More importantly, despite the substantial experimental effort, the benchmark yields few novel or substantive insights beyond model rankings and general performance.
  1. The overall writing and presentation need substantial improvement.
Questions for authors

Please refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Thanks! Response (1/2): Motivation for Evaluating Agents on Eurogames

Thank you for your feedback. We appreciate the opportunity to clarify the motivation for DECE and the findings beyond aggregate rankings supported by the submitted experiments.

W1 — Why evaluate LLM agents on Eurogames?

Our motivation is to evaluate whether LLM agents can sustain effective decisions as earlier choices shape later opportunities and other players change the available options. This requires both model reasoning and execution support for maintaining relevant information across turns.

This direction has concrete research precedents. BALROG (ICLR 2025) evaluates agentic reasoning through interactive games, while LMGame-Bench (ICLR 2026) examines perception, memory, and reasoning through modular execution support. GAMA-Bench (ICLR 2025) studies multi-agent decision-making, and LudoBench (ICLR 2026) uses strategy tabletop games to assess situated perception, rule integration, and short-horizon optimization.

Eurogames offer controlled settings where persistent investments, resource competition, and delayed scoring connect decisions across time. In Splendor, early development cards change the costs of later purchases; in Azul, placements constrain subsequent scoring opportunities. Explicit rules and terminal scoring make outcomes measurable and support reproducible comparisons.

DECE examines complete native agent configurations across ten rule systems. A common participant roster supports cross-game comparisons, while matched harness experiments assess how execution choices affect performance under fixed scenarios, seats, and opponents. These properties connect the choice of environments directly to the empirical questions addressed by the submitted experiments.

References:

AuthorsOpenReview ↗

Thanks! Response (2/2): Findings Beyond Rankings and Presentation

W1 — What does the paper establish beyond model rankings?

The submitted experiments provide three findings with concrete implications for agent evaluation:

  1. Cross-game strength is informative, but game-specific evaluation remains necessary. In the leave-one-game-out analysis, rankings constructed from nine games achieve 83.9% mean pair-order accuracy on the held-out game, with mean Kendall τb = 0.664. Together with the observed local reversals, this quantifies both the consistency and the limits of aggregate rankings. Researchers can assess how well a configuration’s competitive standing carries across rule systems, rather than assuming that its overall rank describes every game.

  2. Harness choice can substantially affect the performance attributed to a model. In the submitted matched Splendor experiment, Qwen3.8-Max achieves 33.3% PWS with OpenCode and 91.7% with Qwen Code, with all six matched scenarios favoring Qwen Code. Setup seeds, focal seats, and opponent configurations are fixed within each scenario. This supports a concrete evaluation implication: the execution harness must be specified and assessed as part of the agent configuration. The paper reports uncertainty and the local scope of this comparison; the inference concerns the complete harness configuration.

  3. Resource criteria change which configurations are competitive choices. The token-based and dollar-based Pareto frontiers identify different candidate configurations, and membership varies across games. Thus, token consumption, monetary cost, and competitive performance provide distinct information for configuration selection. These resource comparisons concern recorded consumption associated with retained completed matches, as specified in the resource-accounting appendix.

These findings establish cross-game consistency, substantial local sensitivity to harness choice, and game-dependent performance–resource trade-offs. The shared experimental framework makes these aspects jointly inspectable and reproducible.

W2 — Writing and presentation

In the consolidated manuscript revision, we will clarify the research motivation in the Introduction, foreground the above empirical takeaways in the Results and Conclusion, and make the distinction between measured configuration effects and their possible underlying mechanisms explicit. These explanations are provided here so they can be assessed before that revision is submitted. The evidence supporting these points is already contained in the submitted experiments.

We would appreciate clarification of any specific experimental protocol, analysis, or conclusion underlying the soundness assessment, so that we can address that concern directly.

Thank you for the rebuttal and clarifications. However, my main concern about the significance of the contribution remains unresolved. While Eurogames involve resource competition and long-term planning, the paper does not sufficiently establish what evaluating LLMs in this setting reveals beyond their performance on these particular games.

In particular, observing that LLMs struggle with such tasks is not, by itself, a sufficiently informative finding without a clearer account of the underlying limitations and their relevance beyond the benchmark. The rebuttal does not convincingly explain how these results could guide subsequent research or improvements in more general agent capabilities (E.g., post-training). My concern is therefore not with games as an evaluation setting, but with the limited actionable insight provided by this particular benchmark.

For these reasons, I will maintain my original score. Thank you again for the thoughtful response.

AuthorsOpenReview ↗

What DECE makes measurable

Thank you for asking how DECE could guide agent development. In the introduction, we model strategic interactions in Eurogames as partially observable stochastic games (POSGs). DECE provides a platform for studying how a model and its harness handle these interactions by recording what the agent observed, which actions were available, and what it chose.

The case study in Appendix H.2 compares the same model, Qwen3.8-Max, under two harnesses. With Qwen Code, it made greater use of early discounts than with OpenCode in five of six matched Splendor scenarios. In 7 Wonders, it made fewer free builds but also saw fewer opportunities to build for free.

These observations suggest a question for both harness design and post-training: what should an agent remember and do as its options change? With the model fixed, researchers could compare a harness that summarizes past actions with one that also tracks which plans remain viable. For post-training, they could annotate replay segments with short explanations of what changed and why it matters for the next decision, then use those examples to teach the model to produce its own summaries. Held-out games could test whether this improves play under the same harness and token budget.

Reviewer_9qEkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DECE provides a unified arena for evaluating complete LLM agent configurations across ten Eurogames. Its main study compares 12 model–harness configurations, spanning nine models and four harnesses, in 480 three-player matches. Supplementary studies examine player counts, non-LLM opponents, and matched harness comparisons. The paper reports cross-game ranking agreement alongside local performance differences and analyzes performance relative to token consumption and API-equivalent cost. The matched studies fix setup seeds, focal seats, and opponent configurations to estimate local effects of changing the execution harness.

Critical strengths
  • A substantial and carefully organized evaluation platform: The common game interface, shared participant roster, seat-balanced protocol, and performance–resource reporting support systematic comparison of complete agent configurations. Game-level profiles complement aggregate rankings.
  • Careful treatment of harness evidence: The separate matched studies control setup seeds, focal seats, and opponent configurations. Scenario-level uncertainty estimates and explicit discussion of multiplicity and local scope make the evidence more interpretable than unpaired leaderboard differences.
Critical weaknesses
  • The added evaluation value is not sufficiently established: GameBench already studies multiple games and scaffolding, while PTCG-Bench examines harness components and performance–cost trade-offs. DECE's broader Eurogame coverage and native-harness integration are useful, but the paper does not sufficiently demonstrate which important evaluation gap this combination closes or which consequential assessment becomes possible beyond those precedents. A benchmark contribution need not introduce a new algorithm; my concern is the significance of the additional capability demonstrated here.
  • The empirical increment remains modest: Cross-game agreement, local advantages, and resource trade-offs provide useful measurements, but yield limited new guidance about agent capability or configuration selection. The matched harness studies cover two games and two harnesses, with six or nine scenarios per model–game comparison. The authors appropriately acknowledge this scope and leave component and equal-budget analyses to future work. This preserves the validity of their local findings, but limits the broader insight supporting acceptance.
Questions for authors
  1. Which existing result or supported research use case most clearly demonstrates an important evaluation capability not adequately covered by GameBench or PTCG-Bench? How does DECE's integration change what researchers can meaningfully assess?
  2. Beyond establishing local harness differences, which empirical finding should most materially change a reader's expectation or configuration-selection decision? How broadly does the current evidence support that implication?
5 comments
AuthorsOpenReview ↗

Thanks! Response (1/3): Eurogames and Evaluation Design

W1/Q1 — What additional assessment does DECE support?

DECE uses ten Eurogames to examine whether an agent configuration’s strengths carry across different rule systems. These games are well suited to studying sustained decisions: players invest in future opportunities, compete for resources, and revise plans as opponents act. They share these demands but implement them through different rules, including permanent discounts, spatial commitments, resource conversion, and delayed scoring.

Relative to GameBench and PTCG-Bench, DECE extends the assessment in three concrete ways.

1. Comparing complete native agent systems. GameBench compares models with CoT and RAP scaffolding. PTCG-Bench evaluates backbones under a shared ReAct harness and ablates harness components. DECE compares complete model–harness configurations with their native tools and context management. This supports choosing between execution systems that researchers can actually run. The separate matched studies hold setup seeds, focal seats, and opponent configurations fixed when changing the harness. For example, the submitted Qwen-Max/Splendor comparison reports a local 58.3-point PWS advantage for Qwen Code over OpenCode, with uncertainty and multiplicity discussed in Section 4.3.

2. Assessing the scope of a configuration’s advantage. GameBench already demonstrates game-dependent scaffold performance, and PTCG-Bench examines ranking consistency across deck archetypes within PTCG. DECE examines ranking consistency across distinct rule systems and the strategic settings where advantages concentrate. Section 4.2 tests rankings formed from nine games against the held-out game, obtaining 83.9% mean pair-order accuracy. Table 4 then shows that Qwen-Max/Qwen Code’s descriptive advantage over OpenCode averages 29.7 PWS points across four games with prominent spatial planning and public current holdings, compared with zero across the other six. Game-deletion checks assess whether this pattern depends on one game. This distinguishes broad competitiveness from advantages concentrated in particular settings.

3. Comparing resource efficiency across games. GameBench does not report measured inference costs, while PTCG-Bench analyzes performance–cost trade-offs within its game. DECE evaluates per-game token and dollar Pareto frontiers, with uncertainty, for the same configuration roster. Researchers can therefore examine whether an efficient choice remains attractive when either the game or the resource measure changes. For example, Kimi-K2.7-Code/OpenCode appears on six token frontiers but no dollar frontier: low token consumption does not necessarily imply low cost under the fixed pricing schedule. These comparisons use recorded consumption associated with retained completed matches.

These assessments are demonstrated in the submitted paper. Responses (2/3) and (3/3) present the additional matched results, trajectory analyses, and held-out configuration-selection analysis, together with their uncertainty and scope.

AuthorsOpenReview ↗

Thanks! Response (2/3): Follow-up Evidence for Target-Game Validation

W2/Q2 — Which result should change a configuration choice?

A configuration that leads in one game should be validated in the target game before it is adopted. The completed follow-up shows why: OpenCode is supported in Ark Nova, and Azul does not reproduce Qwen Code's original advantage. Cross-game rankings identify strong candidates; direct comparisons in the target game test whether a switch is supported.

1. Submitted results: broad competitiveness alongside concentrated advantages. Cross-game rankings identify competitive configurations, yet configuration differences vary across settings. In Table 4, Qwen3.8-Max/Qwen Code exceeds OpenCode by an average of 29.7 PWS points across four games with prominent spatial planning and public current holdings, compared with zero across the other six. These are descriptive comparisons under the main cohort's opponent samples. The separate matched Splendor study then reports a 58.3-point advantage for Qwen Code under fixed scenarios and opponent configurations (Section 4.3).

2. Follow-up: testing two preferences suggested by the original profiles. The primary cohort favors Qwen Code in Azul and OpenCode in Ark Nova. We tested both preferences in 72 additional matches: 18 paired scenarios per game, with six scenarios per focal seat. Each pair compares Qwen3.8-Max/Qwen Code (QC) with Qwen3.8-Max/OpenCode (OC), holding the setup seed, focal seat, and opponent configurations fixed. The opponents are Qwen3.8-Flash/OC and Qwen3.8-Max/OC.

GameOriginal profile: QC − OCFollow-up: QC − OC95% interval
Azul+33.3 pp−2.8 pp[−33.3, +27.8] pp
Ark Nova−33.3 pp−25.0 pp[−44.4, −2.8] pp

PWS is the fraction of opponents outranked, with half credit for ties. Intervals use the planned 5,000-resample scenario bootstrap. Original-profile differences use the main cohort's opponent mix, while follow-up differences use the two fixed opponent configurations, so the two columns differ in opponent population in both games. The follow-up uses native execution budgets and is reported separately from the submitted 1,032-match dataset.

Ark Nova supports the original OC preference. OC scores 55.6% PWS versus QC's 30.6%, performing better in eight paired scenarios, worse in two, and tying in eight. The 95% interval for QC − OC lies below zero.

Azul does not reproduce the original QC advantage. QC scores 63.9% PWS versus OC's 66.7%. The interval is wide, so we do not claim equivalence.

The planned difference between the two game effects is +22.2 pp, with a 95% interval of [−13.9, +58.3] pp, which includes zero.

Execution scope. The follow-up includes an Ark Nova rule correction, four restarts of unfinished matches, and two opponent-session resets; completed outcomes were retained. The model alias has no verified immutable snapshot, and token use was not equalized.

What this changes for a reader. The original profiles suggested game-specific harness preferences; the follow-up tests whether those preferences receive support in paired comparisons against two fixed opponent configurations. Under the recorded follow-up conditions, the results favor OC in Ark Nova but provide no clear support for choosing QC in Azul. Researchers can use cross-game profiles to identify promising configurations, then test a proposed switch through matched comparisons in the target setting. DECE supports this workflow across multiple rule systems while retaining each configuration’s native execution setup.

How broadly this holds. The follow-up tests one model (Qwen3.8-Max) with two harnesses in two games, and matched coverage now spans four games, with 18 scenarios in each new game compared with six or nine in the submitted studies. The implication is procedural rather than a ranking of harnesses: it does not depend on QC or OC being generally better, only on advantages not transferring automatically across games.

AuthorsOpenReview ↗

Thanks! Response (3/3): Selection Rules and Behavioral Evidence

W2/Q2 — What do the records add about configuration selection and behavior?

Two findings bear on how a performance profile should be used. Despite strong cross-game ranking agreement, neither tested selection rule established a reliable held-out gain over the generalist, and greater reuse of early investments did not consistently accompany better play. These analyses complement the target-setting comparisons in Response (2/3).

1. Testing whether a selection rule delivers better subsequent outcomes. The submitted 83.9% held-out pair-order agreement and mean Kendall τ-b of 0.664 show that cross-game rankings agree. Whether that agreement improves a target-game choice is a separate empirical question, so we tested it.

For each target game, the generalist and three-configuration shortlist use only the other nine games. Target matches are split by whole match into calibration and test sets, with six appearances per configuration in each half. Calibration selects the highest-PWS candidate whose mean recorded cost does not exceed the generalist's. We also evaluate selection without this cost cap. Test outcomes and costs do not enter selection.

Selection ruleHeld-out PWS gain over generalistHeld-out mean-cost ratio
Calibration cost cap+2.9 pp [−7.9, +16.1]0.99 [0.53, 1.01]
Performance only+9.2 pp [−10.5, +24.5]1.83 [0.75, 2.17]

The exploratory 95% intervals resample whole matches within game and partition and refit generalist selection, shortlisting, and calibration choice. Repeated splits assess sensitivity rather than add independent samples. Both PWS-gain intervals include zero, with six calibration appearances per configuration.

Harmonies shows the risk of selecting on a small calibration sample. Calibration favors Sol/OC over the nine-game generalist, Sol/Codex, by 25.0 pp, but the selected configuration scores 16.7 pp lower on test matches.

2. Examining whether stronger outcomes accompany greater use of earlier investments. We analyzed all 120 matches from the two submitted matched studies, covering all eight model–game comparisons and keeping each study's opponents separate. In Splendor, we measured permanent discounts acquired in own turns 1–10 and their use in purchases during turns 11–20. In 7 Wonders, we tracked Age-I prerequisites and the later free-chain constructions that directly use them.

The following comparisons show how these behavioral measures relate to competitive outcomes:

ComparisonObserved PWS patternInvestment and subsequent use
Qwen3.8-Max / SplendorQC exceeds OC by 58.3 ppQC acquires 0.50 more early discount cards [0.17, 0.83] and obtains 2.61 more subsequent gem discounts attributed to those cards [0.67, 5.17].
Qwen3.8-Max / 7 WondersFour scenarios retain the same PWS despite reduced chain use, and another improves.QC makes 1.33 fewer direct free-chain constructions from early prerequisites [−1.83, −0.67].
Qwen3.8-Flash / 7 Wonders, bridge studyQC achieves a PWS gain.The gain is not accompanied by a consistent increase in early-chain use.

Numerical behavioral differences are QC minus OC; brackets show exploratory 95% intervals. The measures retain their game-specific definitions.

In Splendor, stronger outcomes accompany both greater early investment and greater subsequent use. The 7 Wonders comparisons do not show the same relationship. Greater reuse therefore does not by itself indicate better play.

The trajectories let researchers test whether a proposed behavioral explanation fits the outcomes in each game, rather than assuming it transfers. These measures exclude final score, and no behavioral sign test survives Holm adjustment.

Scope. The selection analysis covers ten target games with six calibration appearances per configuration; the behavioral analysis covers the 120 matches of the two submitted matched studies.

Reviewer_AjXhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a unified environment for evaluating complete LLM agent configurations across ten Eurogames. The primary experiment compares twelve configurations in 480 three-player matches. Additional player-count settings, paired comparisons of agent harnesses, and non-LLM baselines bring the evaluation to 1,032 completed matches. The paper studies cross-game rankings, the effects of harness choices, and the relationship between performance and resource use.

Critical strengths
  1. A useful evaluation resource with a coherent experimental design. The ten environments cover different forms of sustained investment, spatial decisions, and scoring. A common participant pool and balanced seats make cross-game comparisons easier to interpret. The paper evaluates complete agent configurations explicitly, and the paired harness experiments hold scenarios, seats, and opponents fixed. Together, these choices make the resource useful beyond reporting another leaderboard.

  2. The empirical evidence is inspectable and generally interpreted at the right scope. The supplementary final rankings support reconstruction of the main PWS and per-game Elo tables, as well as the reported leave-one-game-out ranking agreement. The paper also reports uncertainty and qualifications for its smaller paired experiments. For example, it distinguishes the unadjusted and multiple-comparison-adjusted evidence for Qwen-Max on Splendor. These details support the empirical contribution, while leaving the broader model-by-harness conclusions appropriately local to the tested settings.

Critical weaknesses
  1. The frontier measures resources associated with retained completed matches, which may differ from the cost of obtaining them. Appendix E.1 explicitly excludes replaced match IDs from resource accounting. The supplementary ledger has final_attempt_index greater than one for 325 of the 480 primary matches, but the field alone does not explain how these attempts relate to failures, continuations, or replaced records. Because the frontier is intended to inform system selection, the practical effect of this accounting choice matters. Please explain the recovery protocol and whether including omitted expenditure would change the ordering. Where full costs cannot be recovered, the main figure should clearly identify its conditional scope. The available evidence does not establish performance-dependent selection or an error in the reported conditional cost means.

  2. The scope of validation against the stated game rules is less clear than the validation of result summaries. The appendix documents rule variants, and the supplement contains test materials. For complex games such as Ark Nova, Brass, and Wingspan, a concise account of existing checks on key mechanics, hidden information, and terminal scoring would make the benchmark easier to trust and reuse. Agreement between an implementation's records and its tables cannot by itself establish rule fidelity. This is a request to describe the existing validation coverage; no particular rule violation has been identified.

Questions for authors
  1. What does final_attempt_index count, and what triggers continuation or replacement? Are these decisions independent of game performance, and can you quantify the effect of omitted attempts on total cost or the frontier?
  2. Which independent rule cases or reference implementations have been used to validate the engines, especially hidden-information handling and complex terminal scoring? Please summarize the existing coverage and known deviations.
3 comments
AuthorsOpenReview ↗

Thanks! Response (1/2): Recovery Protocol and Resource Accounting

W1/Q1 — Recovery protocol and resource accounting

Thank you for distinguishing the reported conditional costs from the expenditure required to obtain the completed cohort. We clarify the execution bookkeeping and accounting scope below.

final_attempt_index is the runner’s outer execution-attempt index, not a count of completed games or a token-cost multiplier. Although 325 of the 480 primary matches have an index greater than one, multiple attempt databases can contain continuations of the same match ID and committed game state.

Recovery handles execution interruptions, timeouts, API throttling, and context or session failures. The model APIs impose request-per-minute (RPM) and token-per-minute (TPM) limits; recovery uses backoff and continuation from saved progress. Depending on checkpoint compatibility, it continues a saved match or starts a new match ID for the same scheduled game, seed, participants, and seats. Continuation reuses the native harness session when available. Some recovery paths retain the board state but create a new session, with recovery context supplied in handoff cases; we therefore distinguish board-state continuation from native-session resumption in the recovery documentation.

The scheduler does not use scores or rankings to trigger recovery, and successfully completed scheduled matches are skipped. These are execution-based recovery rules; failure incidence can nevertheless vary across configurations and game durations.

Observed continuation usage is already included in the reported costs. Appendix E.1 aggregates native session counters, deduplicated native responses, or controller usage associated with each retained completed match ID. It therefore covers recorded continuations, rather than only the final worker invocation. The submitted native-record reconciliation incorporated 2,723 additional responses and deduplicated copied records. These additions are already reflected in the reported resource ledger.

The following table summarizes the retained-ID reconciliation already included in the submitted ledger:

Check on the retained-match native-usage reconciliationResult
Additional native responses included2,723
Additional input-plus-output tokens included372,239,423
Additional API-equivalent cost included by the retained-ID reconciliation$85.86
Configuration-by-game mean costs changed43
Games with unchanged point-estimate Elo–dollar-cost Pareto membership10/10
Maximum membership-frequency change in the archived 5,000-resample analysis1.9 percentage points

As a check within retained IDs, that reconciliation added $85.86 in API-equivalent consumption while leaving point-estimate dollar-cost Pareto membership unchanged in all ten games. This result concerns the retained-ID reconciliation only. Expenditure under replaced IDs and unlogged requests remains excluded, and its effect on the full acquisition-cost frontier has not been established.

The unchanged result concerns dollar-frontier membership: full cost rankings change in Century and Harmonies, and the token frontier gains GPT-5.6-Sol/Codex in Century.

To clarify the accounting scope here, and in the forthcoming consolidated manuscript revision:

Costs report recorded API-equivalent consumption associated with retained completed match IDs, including observed continuations. Replaced IDs and unlogged requests are excluded. These frontiers do not estimate the total expenditure required to obtain the completed cohort.

We also report here the endpoint and model-string variation found for the original-cohort DeepSeek Flash/DSh participant, including an unversioned alias and an FP8-labelled preset. The following records cover its 36 retained original-cohort appearances:

Retained appearancesEndpointRecorded model string
21dashscope.aliyuncs.com/compatible-mode/v1deepseek-v4-flash-0731
14api.deepseek.comdeepseek-v4-flash
1openrouter.ai/api/v1deepseek/deepseek-v4-flash-0731@preset/agentbga-dsv4flash-baidu-fp8

All 36 record DSh and reasoning effort max. The 84 expansion appearances use deepseek/deepseek-v4-flash-0731 through OpenRouter, also with DSh/max. An identical underlying snapshot across these routes is not verified. The reported costs consistently use the paper’s fixed API-equivalent price schedule, rather than endpoint-specific invoices.

AuthorsOpenReview ↗

Thanks! Response (2/2): Validation Against Game Rules

W2/Q2 — Validation against game rules

Rule-focused tests were already included in the submitted implementation, alongside trajectory and result-consistency checks. The targeted validation log records 93 passing tests, and we verified that the cited test files match the submitted package. For the three games highlighted by the reviewer, the existing coverage includes:

GameExisting validation and representative cases
Ark NovaA 27-requirement crosswalk connects the rulebook, Glossary, and FAQ to implementation and tests. Cases cover private hands, pending choices, action and project resolution, and terminal scoring, including applying final conservation points before victory-point calculation.
Brass: BirminghamTests cover hidden opponent hands, resource connectivity and consumption, double-rail construction, loans, liquidation, and the canal-to-rail transition. A constructed scoring case specifies 5 industry points plus 3 link points; separate cases check terminal tie-breaking.
WingspanTests cover private hands and bonus information, public-event projections, food and egg payments, reaction timing, round-goal ties, bonus-card thresholds, and terminal tie-breaking. Bonus-card cases use explicit boundary tables.

The cited scoring cases use explicit expected outcomes, rather than obtaining their expected answers from the scoring function under test. Hidden-information checks inspect seat observations and public-event projections. The approach uses constructed rule cases; no separate reference engine was used. Execution coverage across all 170 Wingspan bird cards is reported separately from validation of individual rules and interactions.

We have added a concise appendix table linking the existing checks to their rule sources, expected outcomes, and test files. This makes rule-case validation, hidden-information checks, component execution, and trajectory consistency separately identifiable.

The appendix also distinguishes intentional deviations from selected game settings. Century exposes merchant hands publicly, an information-scope deviation. Selected settings include Ark Nova’s Map 0 and base-game content, Wingspan’s green goal board, and Harmonies Board A without nature spirits. These declarations now appear alongside the validation summary.

Thank you for the detailed responses about how interrupted matches are handled and how individual rules were tested. I am keeping my rating at 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

First Revision: Summary of Updates

We have uploaded a revised manuscript incorporating the clarifications and additional analyses discussed in our responses. The main updates are:

  • Recovery and costs (AjXh): Appendix D.7 clarifies recovery procedures; Appendix E.1 and the Figure 4 caption clarify resource-accounting scope.
  • Rule validation (AjXh): Appendix A.2 and Table 6 summarize existing rule-focused tests and validation coverage.
  • Motivation and findings (9qEk, g8FW): The Introduction, Section 2, and main-text discussion clarify the evaluation scope and empirical takeaways.
  • Additional evidence (9qEk, g8FW): Section 4.3 and Appendix G.4 report the 72-match follow-up; Appendix H provides behavioral and held-out configuration-selection analyses.

The follow-up is reported separately from the original dataset, and the original primary rankings and resource frontiers remain unchanged. Thank you for the helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.