acceptodds
Withdrawn from ICLR 2027

EuroArena: Do Large Language Models Excel at Long-Horizon Planning under Competition?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Large language models (LLMs) have demonstrated strong performance on complex tasks, such as coding and mathematics, but can they plan effectively while other agents are planning against them? Adversarial interaction introduces a distinct challenge: a plan must account not only for the consequences of one's own actions, but also for opponents who can contest resources and disrupt its execution, which is largely overlooked in current benchmarks. We introduce , a benchmark featuring two-player Eurogames that tests long-horizon planning through competition over shared resources, investment in future gains, and adaptation to opponent's actions. Across more than 400 hours of Splendor Duel gameplay against skilled human players, we find that, surprisingly, six strong LLMs struggle to match human expertise, despite demonstrating the ability to formulate concrete plans, revise them after feedback, and propose alternatives for possible future states. To investigate this gap, we introduce resource dependency graphs to trace how earlier decisions support later purchases, and leverage this with human-verified trajectory analysis. We find a recurring contrast: LLMs prioritize reaching the next subgoal, whereas humans more often take actions that support later subgoals and make better use of accumulated resources. Models also overlook adverse opponent responses and make reasoning errors when integrating multiple information sources. These findings reveal a gap between generating and revising explicit plans and making decisions that sustain progress toward long-term goals under competition.

Primary area:
datasets and benchmarks
Keywords:
Long-Horizon Decision Making, Planning, Large Language Model, Benchmark, Euro Game

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HnABOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EuroArena, focusing on long-horizon planning in the two-player board game Splendor Duel under resource competition and interference from an opponent. The authors recruited twelve volunteers from a board game club to play against six models using a common recurrent memory mechanism. They retained thirty complete game trajectories, five per model; the human players won all thirty games. The authors construct resource dependency graphs (RDG) to trace how gems, reserved cards, and permanent discounts support subsequent purchases. They combine RDG analysis, trajectory forking with desk reshuffling, and manually verified error cases to examine behavioral differences between humans and models. Under the evaluated configurations and against these opponents, the models can generate and revise explicit plans and propose alternatives, but still overlook possible opponent responses and incorrectly integrate state information. The authors further argue that human actions more often support later goals, whereas models focus more narrowly on the nearest subgoal.

Critical strengths
  • The game mechanics are closely aligned with long-term decision-making under competition. Players must weigh immediate points against the benefits of permanent discounts for later purchases, while responding to opponents who compete for resources, reserve target cards, and change purchasing opportunities. Complete games played by skilled humans provide a valuable behavioral reference for examining these decisions.

  • The trajectory analysis provides concrete evidence beyond game outcomes. RDGs link earlier resource acquisition to later purchases and have the potential to characterize patterns of resource reuse. Table 2 presents specific errors, including miscounting gems, failing to account for existing permanent discounts, and overlooking an opponent's ability to reserve a target card, together with their consequences. These cases can be checked against the game states and rules, and the authors report verification by at least two experts. Table 3 also demonstrates the models' ability to revise plans and propose conditional alternatives, extending the analysis beyond their failures.

  • The forking experiments test whether models can capitalize on positions established by humans. In Figures 5–6, the authors start from midgame states with similar progress toward winning, reshuffle the market, and have identically configured models take over both sides. They use DeepSeek-V4-Pro and GPT-5.6-Terra as the continuation models. Aggregating the results in Figure 6 separately for each continuation model, the side taking over the human position wins eight games and loses one in both cases. This shows that, in the selected positions, the resources, discounts, and other advantages accumulated by humans can still benefit the models taking over, even when the humans are no longer playing.

Critical weaknesses
  • The manuscript has a noticeable formatting issue. Equation (1) extends beyond the right boundary of the text into the page margin. The authors should check and correct the formatting throughout the manuscript.

  • Equation (1) is inconsistent with Figure 4. According to Equation (1), if the entire game consists of a single stage, all connections fall within that stage, so the number of cross-stage connections should be zero. However, Figure 4 shows substantial nonzero counts at K=1. The authors should verify and explain this discrepancy; see Q1.

  • The causes of some basic errors are not sufficiently analyzed. In Table 2, Claude-Opus-5 overlooks the possibility that the opponent could reserve the card it intends to buy, while GPT-5.6-Terra fails to account for its existing permanent discounts. Appendix A.2 states that the models receive the full rules. The displayed system prompt also explicitly mentions card reservation and discounts and instructs the models to base their decisions on both the current state and memory. Why such basic errors persist despite these instructions warrants further analysis. Using the actual inputs and outputs from the decisions where these errors occur, the authors should discuss whether the problems arise from how rules and states are presented or understood, how memory is maintained, or a failure to use available information when making decisions; see Q2.

  • The experimental scale is small, and the metrics in Table 1 are insufficiently explained. Each model plays only five games, making estimates of model performance sensitive to individual games. In addition, Table 1 does not clearly define the metrics or explain how they are calculated. The authors should provide these details; see Q3.

  • The cost of evaluating new models and the comparability of results remain limitations. The main evaluation relies on skilled human opponents. Researchers following this setup would need to organize additional human–model games, and differences in opponents' skill levels and strategies across evaluation runs could affect comparisons between models. Although the appendix includes model self-play experiments, the paper does not explain how these could support a reusable automated evaluation procedure that yields comparable results for new models; see Q4.

Questions for authors
  • Q1 - Equation and figure: According to Equation (1), the number of cross-stage connections should be zero at K=1, but Figure 4 shows nonzero values. Please verify and explain this inconsistency.

  • Q2 - Causes of basic errors: Given that the authors report providing the full rules and that the prompt explicitly mentions card reservation and discounts, why do the models in Table 2 still overlook the opponent's ability to reserve a target card and fail to account for their own existing permanent discounts? Please provide the complete inputs for these two decisions, including the actual rule text, current game state, legal options, and memory from the preceding decision, along with the model outputs. These records would help clarify whether the rules or states were presented unclearly, the models misunderstood them, memory organization and updates caused information to be omitted, or the models had the relevant information but failed to use it in their decisions.

  • Q3 - Metric definitions in Table 1: How are "Highest Winning Condition Completion(%)" and "Total Token Discounts" calculated? Please clarify their definitions and calculation procedures.

  • Q4 - Evaluating new models: Would researchers evaluating new models still need to recruit skilled human players? If so, how can the human opponents' familiarity with the game and level of expertise be kept comparable to those in this study? Are reusable automated opponents and an evaluation procedure provided?

Reviewer_uQtUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce EuroArena, a five-game benchmark for long-horizon planning. Experiments are done across 6 models and 30 human versus LLM games, but only for Splendor Duel (Splendor). They claim as contributions the whole five game benchmark, analysis over resource dependency graphs that link early and late purchase decisions, and human-verified trajectory analysis that demonstrate humans win every game played (Splendor) due to the tendency of LLMs to pursue immediate gains rather than playing for late game advantages. They also document reasoning mistakes and provide examples of replanning.

Critical strengths

I think the analysis of what was run is good. I like that for Splendor, the LLMs are tested against human players on top of the self-play examples and the forked continuation is good to demonstrate how the LLMs might act in advantageous situations they wouldn't have otherwise been able to arrive in.

Critical weaknesses

A number of major weaknesses but I'd say the worst two offenders are:

1.) Claiming a 5 game benchmark but only running models over 1 game. As it stands, the other 4 games have no justification or reason to be in the paper.

2.) The experiments don't really distinguish completing explanations for why the LLMs perform worse than the humans. Between standard hallucination or rule processing errors, there's no real control establishing where strategic weakness comes into play after this. Evaluation only one game and five human matches per model also makes it uncertain how reliable the derived insights are.

Questions for authors

1.) can the graph metrics distinguish actual genuine foresight versus, for example, knowing certain Splendor strategies from online guides or advice?

2.) Why were the other four environments included if no experiments or analysis have been done on them?

3.) What distinguishes EuroArena from prior competitive planning work such as AucArena, CivRealm, SPIN-Bench, etc

Reviewer_hpadOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark, EuroArena, for evaluating the long-horizon competitive planning capabilities of LLMs, and runs the eval against skilled humans. The human contestants won all 30 games played, indicating a substantial gap in performance and revealing highlighting weaknesses in model reasoning/planning, especially updating plans based on new information. To diagnose the models' planning failures, the authors conduct three analyses:

  1. Resource Dependency Graphs (RDGs), which partition decision traces into K stages to count cross-stage dependencies, used as evidence that LLMs form long-horizon plans but are myopic in their execution.
  2. A mid-game forking experiment from 50–60% game completion with reshuffled card markets.
  3. An LLM-assisted, human-verified qualitative taxonomy of error modes.
Critical strengths
  1. Human evaluation: 30 full games at max reasoning effort against 12 experienced club players establishes a clean 0/30 baseline across six frontier models, and the recurrent-memory JSON schema logs each model's multi-horizon plans and opponent predictions at every turn without confounding model play with external search. This surfaces the empirical gap between coherent written plans and selected actions.

  2. Mid-game trajectory forking with reshuffling separates position quality from late-game execution. Selecting checkpoints at ~50% progression where human and LLM win-condition progress is matched, reshuffling the visible market cards, then having identical LLMs finish both sides is an interesting test that does yield some insight and supports the hypothesis that humans had better supporting plans.

Critical weaknesses
  1. Undelivered benchmark framing and total lack of reproducibility: The submission frames its central contribution as "EuroArena: a benchmark of five two-player Euro games", yet only a single game (Splendor Duel) is implemented or evaluated. The other four games exist only as 3-paragraph rule summaries in Appendix A.1. There are no links to code, environments, game logs, or models, and no reproducible reference bot. I acknowledge the limitations note disclosing the single-game scope, and a deep single-game study of Splendor Duel would be a legitimate contribution on its own. But even on that narrower scope, the manuscript omits a Reproducibility Statement, includes no repository or data link, and depends entirely on recruited human volunteers with no automated reference opponent (such as MCTS or a heuristic bot). As submitted, no external researcher can run or verify any part of EuroArena, which directly undermines the main contribution.

  2. Sample size and lack of LLM vs LLM play. The entire paper is based on 30 game trajectories, 5 per model. I'm inclined to agree with the qualitative analysis: yes LLMs suffer from a number of shortcomings that Splendor Duel highlights, but I have very little confidence in the relative model rankings. We don't know how strong the humans were, some models may have faced notably weaker/stronger competition. Having LLM vs LLM play would have provided much more insight into relative model strength on this task.

  3. Lack of ablations. The trajectory forking experiment is the closest thing to an ablation in the paper. While the prompt and JSON structure looks very reasonable, I can't be confident something hasn't slipped through the cracks. Some kind of ablations either over the prompt, memory structure, or game play would provide insight and make these findings more trustworthy.

  4. RDG just adds lower-half edges whenever an earlier discount card reduces the cost of a later purchase. A zero-lookahead greedy bot that buys cheap level-1 discount cards early will generate dense cross-stage RDG curves, yet no heuristic baseline is tested. More importantly, Table 1 + Figure 4 directly contradict 3.1's thesis that cross-stage RDG engines drive competitive play: GPT-6-Astra builds a human-sized discount engine and has a near-human Figure 4 RDG curve, yet loses 0/5 and finishes second-worst in win-condition progress (57.0%)—well behind Claude-Opus-5 (84.0% with 10.2 discounts) and Kimi-K3 (73.0% with 7.0 discounts). Overall, I don't have a lot of confidence in the RDG analysis being truly reflective of model planning.

Questions for authors
  1. Will you release the Splendor Duel engine, RDG tooling, and the 30 game trajectories, together with an automated opponent baseline (e.g., MCTS or a heuristic bot)? Do executable environments exist for the four other, or will you reframe the paper as a Splendor Duel study?
  2. How can we be confident that the RDG reflects true planning and what explains Astra's performance despite having the closest similarity to human gameplay in terms of cross-stage connections?
  3. Each of the 12 volunteers ranked the six models by preference before play, with 75.0% ranking GPT-6-Astra first. How were volunteers assigned to models, and how many games did each volunteer play?

Minor issues:

The abstract and introduction claim "more than 400 hours" of gameplay, whereas Table 4 sums to 214:02:38 of model turn elapsed time plus an estimated 148:40:00 of human time, summing to 362.7 hours total. Please reconcile.

Please define the formula for Highest Winning Condition Completion (%), state whether ± in Table 1 and the shaded bands in Figure 4 denote SD, SE, or CI, report actual <re_plan> and <diverse> frequencies from Appendix A.2 in §3.4, remove the duplicated Limitations paragraph and unfinished clause ("Though carefully designed and,") at l. 602–616, and fix "Appendix 4" -> Appendix B (l. 205) and minor typos ("can they plan" l. 035, "latter" l. 160/189, "desk" for "deck" l. 264/268, "targe" Fig. 2, "a agentic" l. 478, "an three" l. 628).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.