- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EuroArena, focusing on long-horizon planning in the two-player board game Splendor Duel under resource competition and interference from an opponent. The authors recruited twelve volunteers from a board game club to play against six models using a common recurrent memory mechanism. They retained thirty complete game trajectories, five per model; the human players won all thirty games. The authors construct resource dependency graphs (RDG) to trace how gems, reserved cards, and permanent discounts support subsequent purchases. They combine RDG analysis, trajectory forking with desk reshuffling, and manually verified error cases to examine behavioral differences between humans and models. Under the evaluated configurations and against these opponents, the models can generate and revise explicit plans and propose alternatives, but still overlook possible opponent responses and incorrectly integrate state information. The authors further argue that human actions more often support later goals, whereas models focus more narrowly on the nearest subgoal.
Critical strengths
-
The game mechanics are closely aligned with long-term decision-making under competition. Players must weigh immediate points against the benefits of permanent discounts for later purchases, while responding to opponents who compete for resources, reserve target cards, and change purchasing opportunities. Complete games played by skilled humans provide a valuable behavioral reference for examining these decisions.
-
The trajectory analysis provides concrete evidence beyond game outcomes. RDGs link earlier resource acquisition to later purchases and have the potential to characterize patterns of resource reuse. Table 2 presents specific errors, including miscounting gems, failing to account for existing permanent discounts, and overlooking an opponent's ability to reserve a target card, together with their consequences. These cases can be checked against the game states and rules, and the authors report verification by at least two experts. Table 3 also demonstrates the models' ability to revise plans and propose conditional alternatives, extending the analysis beyond their failures.
-
The forking experiments test whether models can capitalize on positions established by humans. In Figures 5–6, the authors start from midgame states with similar progress toward winning, reshuffle the market, and have identically configured models take over both sides. They use DeepSeek-V4-Pro and GPT-5.6-Terra as the continuation models. Aggregating the results in Figure 6 separately for each continuation model, the side taking over the human position wins eight games and loses one in both cases. This shows that, in the selected positions, the resources, discounts, and other advantages accumulated by humans can still benefit the models taking over, even when the humans are no longer playing.
Critical weaknesses
-
The manuscript has a noticeable formatting issue. Equation (1) extends beyond the right boundary of the text into the page margin. The authors should check and correct the formatting throughout the manuscript.
-
Equation (1) is inconsistent with Figure 4. According to Equation (1), if the entire game consists of a single stage, all connections fall within that stage, so the number of cross-stage connections should be zero. However, Figure 4 shows substantial nonzero counts at K=1. The authors should verify and explain this discrepancy; see Q1.
-
The causes of some basic errors are not sufficiently analyzed. In Table 2, Claude-Opus-5 overlooks the possibility that the opponent could reserve the card it intends to buy, while GPT-5.6-Terra fails to account for its existing permanent discounts. Appendix A.2 states that the models receive the full rules. The displayed system prompt also explicitly mentions card reservation and discounts and instructs the models to base their decisions on both the current state and memory. Why such basic errors persist despite these instructions warrants further analysis. Using the actual inputs and outputs from the decisions where these errors occur, the authors should discuss whether the problems arise from how rules and states are presented or understood, how memory is maintained, or a failure to use available information when making decisions; see Q2.
-
The experimental scale is small, and the metrics in Table 1 are insufficiently explained. Each model plays only five games, making estimates of model performance sensitive to individual games. In addition, Table 1 does not clearly define the metrics or explain how they are calculated. The authors should provide these details; see Q3.
-
The cost of evaluating new models and the comparability of results remain limitations. The main evaluation relies on skilled human opponents. Researchers following this setup would need to organize additional human–model games, and differences in opponents' skill levels and strategies across evaluation runs could affect comparisons between models. Although the appendix includes model self-play experiments, the paper does not explain how these could support a reusable automated evaluation procedure that yields comparable results for new models; see Q4.
Questions for authors
-
Q1 - Equation and figure: According to Equation (1), the number of cross-stage connections should be zero at K=1, but Figure 4 shows nonzero values. Please verify and explain this inconsistency.
-
Q2 - Causes of basic errors: Given that the authors report providing the full rules and that the prompt explicitly mentions card reservation and discounts, why do the models in Table 2 still overlook the opponent's ability to reserve a target card and fail to account for their own existing permanent discounts? Please provide the complete inputs for these two decisions, including the actual rule text, current game state, legal options, and memory from the preceding decision, along with the model outputs. These records would help clarify whether the rules or states were presented unclearly, the models misunderstood them, memory organization and updates caused information to be omitted, or the models had the relevant information but failed to use it in their decisions.
-
Q3 - Metric definitions in Table 1: How are "Highest Winning Condition Completion(%)" and "Total Token Discounts" calculated? Please clarify their definitions and calculation procedures.
-
Q4 - Evaluating new models: Would researchers evaluating new models still need to recruit skilled human players? If so, how can the human opponents' familiarity with the game and level of expertise be kept comparable to those in this study? Are reusable automated opponents and an evaluation procedure provided?