acceptodds
Withdrawn from ICLR 2027

FantasyClash: Benchmarking Strategic Reasoning in Fantasy Sports

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An intelligent agent that operates among its peers must reason strategically: it must model how its counterparts will respond to its own action. Existing evaluations of strategic reasoning in LLM agents rely on stylized matrix games, board and card games, or dialogue-based negotiation, which lack the complex dynamics that govern the uncertainty real-world agents tackle. We present FantasyClash, a benchmark for strategic reasoning under real-world uncertainty. The benchmark is built on the popular fantasy sports format, where agents (1) bid for athletes in an auction, (2) field weekly lineups from their pools, and (3) bet on their teams' performance relative to others. Crucially, the outcomes are determined by the corresponding athletes' performances in real games, which imbues the game with high-dimensional, real-world dynamics. Evaluating 8 frontier LLMs, we find that agents vary substantially in their innate ability to play the game. We further write detailed descriptions of 7 strategies and verify them as effective prompts for improving an agent's performance. We empirically identify intransitive cycles among a subset of strategy-prompted agents, illustrating strategic depth where success depends on inferring counterparts' strategies. Furthermore, we find that agents rarely act on what their opponents do, but some agents exploit an opponent's strategy once they see its full text. Finally, we characterize the relative strengths of the 8 agents through a large-scale evaluation with an Elo rating system.

Primary area:
datasets and benchmarks
Keywords:
strategic reasoning, multi-agent, benchmark, LLM agents, game theory, fantasy sports, opponent modeling, intransitivity, Elo rating

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GDRbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to evaluate strategic reasoning in Large Language Model agents under real-world uncertainty. Towards this, the paper introduces Fantasyclash, a multi-agent fantasy hockey environment in which agents 1) research historical player statistics and news through database queries, 2) acquire players through an auction and select weekly lineups, and 3) bet on their lineups’ performance relative to other agents. The outcomes are determined by players’ performances in historical NHL games. Consequently, agents must handle both uncertainty about player performance and competition over player acquisition and betting returns.

For experiments, the paper evaluates eight LLMs and three deterministic baselines. The authors introduce seven strategy prompts and find that five improve performance in their main strategy experiment. They further document cyclic performance relationships among three strategies, declining returns as more agents adopt the same strategy, and limited spontaneous use of opponents’ past behavior. Disclosing an opponent’s strategy enables exploitation in some cases. Finally, the paper uses a Plackett–Luce model to summarize agent performance on an Elo scale across different pools of competitors.

Critical strengths
  1. The environment brings together two meaningfully different sources of difficulty. Agents must predict an external process they cannot control, while making decisions whose consequences depend on other agents. The combination of player acquisition, lineup selection, and betting creates a richer decision problem than evaluating predictions or individual actions in isolation. Moreover, performance is measured through realized game outcomes rather than an LLM judge’s assessment of whether an explanation sounds strategic.

  2. The paper goes beyond reporting an aggregate leaderboard. The strategy interventions, cyclic matchups, and opponent-disclosure experiments provide useful ways to examine why agents succeed or fail. The authors also report unfavorable findings, including strategies that reduce performance and opponent-tracking prompts that do not produce a clear performance improvement. These analyses are more informative than presenting only the best-performing model or prompt.

  3. The experimental environment is documented in substantial detail. The appendices describe the prompts, data filtering, anonymization, temporal cutoffs, and configurable game settings. This makes important aspects of the evaluation inspectable and provides a useful foundation for subsequent controlled experiments.

Critical weaknesses
  1. (Major weakness) The experiments do not sufficiently distinguish strategic reasoning from the other capabilities required to play the game.

The paper claims to evaluate strategic reasoning, but performance also depends on statistical forecasting, database querying, arithmetic, and budget management.

  • Appendix A.3 documents frequent failures in SQL queries, budget management, and interpretation of fantasy points. These errors can substantially affect performance independently of strategic reasoning.
  • The existing deterministic baselines do not isolate this issue, as they differ from LLM agents in forecasting capabilities and access to information.

Consequently, an agent may outperform another simply because it is better at data analysis or execution, rather than reasoning about opponents. A controlled evaluation holding forecasting and execution capabilities fixed, while varying only opponent-aware decision-making, would be needed to establish that the benchmark measures strategic reasoning.

  1. (Major weakness) The strategy experiments demonstrate that LLMs can follow effective strategies, but not necessarily that they can develop or select them.

The authors provide seven predefined strategy prompts and show that some improve performance. However, this largely measures whether agents can execute strategies designed by the researchers.

  • For instance, Decoy, Tax Collector, and Loyalist explicitly specify how agents should behave in particular competitive situations. Their effectiveness does not establish that LLMs can independently discover these strategies.
  • Similarly, the cyclic dominance in Table 2 may arise entirely from the predefined rules, without agents reasoning about opponents.

Therefore, the experiments demonstrate strategic interactions in the environment, but provide limited evidence of strategic reasoning by the LLMs themselves. A comparison with deterministic implementations of the same strategies, together with experiments requiring agents to select or adapt strategies independently, would help establish the additional contribution of LLM reasoning.

  1. (Major weakness) The conclusions about agents' ability to reason about opponents are not sufficiently supported by the evaluation.

Section 5.4 reports that only 119 out of 14,099 action explanations mention an opponent, and only one uses an opponent's past behavior. However, this does not necessarily imply that agents fail to reason about opponents.

  • An agent may consider an opponent's behavior without explicitly mentioning it in its explanation. Conversely, mentioning an opponent does not establish that this information influenced its decisions.
  • The opponent-tracking intervention does not produce a clear performance improvement, but this alone cannot distinguish a failure to understand opponents from a failure to act effectively on that understanding.
  • The strategy-disclosure experiment is interesting, but Table 3 provides only two model–season cases, limiting the generality of its conclusions.

A more direct experiment would compare agents' decisions with and without information about opponents, holding other information fixed. This would better establish whether agents actually use opponent information and whether doing so improves their decisions.

  1. (Minor weakness) The aggregate agent rankings may be misleading given the paper's own evidence of opponent-dependent performance.

The paper demonstrates that relative performance depends on the opponents and their strategies. However, its Plackett–Luce evaluation summarizes performance using a single rating per agent.

While such rankings are useful descriptively, they may depend substantially on the composition of opponent pools. The paper should examine whether the rankings remain stable under different opponent distributions and report matchup-level performance alongside the aggregate ratings.

Questions for authors

Questions For Authors

(Major) How can the authors establish that FantasyClash measures strategic reasoning rather than forecasting, querying, or execution capabilities? Can they provide controlled comparisons holding these other capabilities fixed?

(Major) How do LLM agents compare with deterministic implementations of the same predefined strategies? Can agents independently discover, select, or adapt strategies without being explicitly instructed to follow them?

(Major) Why should the frequency of opponent mentions in action explanations be considered evidence of opponent reasoning? Can the authors directly measure how agents' decisions change when opponent information is provided or withheld?

(Minor) How sensitive are the Plackett–Luce rankings to the composition of opponent pools? Would the rankings remain consistent under a common opponent distribution?

1 comment

General Response

We would like to thank all of the reviewers for your time reviewing the paper. Given the initial reviews, we have decided to revise and resubmit, and hope to incorporate your invaluable comments in further revisions. We sincerely appreciate your thoughtful comments and effort spent on reviewing the paper.

Nonetheless, we would like to take the chance to re-emphasize the main focus of FantasyClash, and why we believe the work is novel among a plethora of seemingly similar effort. The main capability we aim to assess in agents is at the intersection of both (a) strategic behaviours concerning one's opponents, and (2) real-world uncertainties, stemming from the need to bet on real world sports outcomes. From this view, while each of the component is individually valuable and therefore has accumulated a substantial literature, we do not view them as comparable benchmarks.

Again, we would like to sincerely thank you for the time you have put in reviewing the paper.

Reviewer_ZqT7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FantasyClash, a simulation of a sports market (NHL) where agents can bid on players, set team lineups, and bet on match results. 8 frontier LLMs, 3 rule-based baselines are evaluated. The authors also show that rewriting various strategies as prompts can affect outcomes.

Critical strengths
  1. The benchmark is interesting and provides an extensible testbed for strategic reasoning.
  2. Experiments are extensive and evaluate different aspects that could have influence on the strategies of agents.
Critical weaknesses
  1. On the experimental design: two aspects of the game are simultaneously evaluated for each agent: ranking (among many teams) and gain (from betting). The NetGain, as defined in Section 3.1, is directly computable from each team's LineupPoints and stakes. This setting is unrealistic since one can bet on specific games.

  2. On novelty: there are existing benchmarks on evaluating strategic reasoning in sports markets, such as https://www.gr.inc/papers/KellyBenchPaper.pdf, https://arxiv.org/pdf/2607.24573. The first one evaluates agents's betting strategies directly on real-world sports markets.

Questions for authors
  1. The language models see anonymized player names such as P1234 in the inputs. How much does this anonymization change the experimental results, comparing to using a. real player names and b. realistic synthetic names?
  2. What is the token budget for each of the models? How many output tokens does each run generate?
  3. How is a match's result decided? Is there randomness in each game's result?
1 comment

General Response

We would like to thank all of the reviewers for your time reviewing the paper. Given the initial reviews, we have decided to revise and resubmit, and hope to incorporate your invaluable comments in further revisions. We sincerely appreciate your thoughtful comments and effort spent on reviewing the paper.

Nonetheless, we would like to take the chance to re-emphasize the main focus of FantasyClash, and why we believe the work is novel among a plethora of seemingly similar effort. The main capability we aim to assess in agents is at the intersection of both (a) strategic behaviours concerning one's opponents, and (2) real-world uncertainties, stemming from the need to bet on real world sports outcomes. From this view, while each of the component is individually valuable and therefore has accumulated a substantial literature, we do not view them as comparable benchmarks.

Again, we would like to sincerely thank you for the time you have put in reviewing the paper.

Reviewer_QAKLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

How do LLMs perform in the task of strategic reasoning? The paper suggests that present methods focus on stylized games that lack the complex dynamics and uncertainty found in real-world environments. Towards addressing this, the paper presents FantasyClash, a benchmark built on fantasy sports where agents bid for athletes, field weekly lineups, and bet on performance. The paper evaluates several frontier LLMs on the benchmark. Further the paper suggests multiple inference-time strategies to improve performance.

Critical strengths
  • This is an extremely important area. AI-vs-AI-vs-humans is a scenario that is very important and soon pervasive.
  • Evaluating frontier LLM performance and identifying weaknesses and suggesting simple methods (e.g. inference time) to improve performance are natural and correct methods to approach the problem space.
Critical weaknesses
  • This is an extremely active area of research and the authors miss several crucial background work. For instance,

https://arxiv.org/abs/2503.12349

set an elaborate benchmark for strategic reasoning, almost two years ago. Multiple NeurIPS competitions last year (2025) explored game playing agents, including MindGames and PokeAgent challenges. ARC-AGI3 is another well known industry benchmark. I wouldn't think FantasyClash is anywhere more advanced than all these efforts. These papers and efforts also identify how frontier LLMs perform on games and their weaknesses and strengths w.r.t depth of strategic reasoning, interactivity levels and cooperative vs competitive settings. There are substantial discoveries of frontier LLMs performance across these works. 

Further, several papers identify training time methods to improve game playing performance (see https://yunfeixie233.github.io/MEMO/ for a recent paper; ICML 2026) and even post-training methods (see https://arxiv.org/abs/2605.12519 for a recent NuerIPS 2026 paper).

I would strongly recommend the authors due some careful literature review. The present submission is far from the completeness and detail of understanding already known on this area.

Questions for authors

see "weakness" section above.

1 comment

General Response

We would like to thank all of the reviewers for your time reviewing the paper. Given the initial reviews, we have decided to revise and resubmit, and hope to incorporate your invaluable comments in further revisions. We sincerely appreciate your thoughtful comments and effort spent on reviewing the paper.

Nonetheless, we would like to take the chance to re-emphasize the main focus of FantasyClash, and why we believe the work is novel among a plethora of seemingly similar effort. The main capability we aim to assess in agents is at the intersection of both (a) strategic behaviours concerning one's opponents, and (2) real-world uncertainties, stemming from the need to bet on real world sports outcomes. From this view, while each of the component is individually valuable and therefore has accumulated a substantial literature, we do not view them as comparable benchmarks.

Again, we would like to sincerely thank you for the time you have put in reviewing the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.