- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper aims to evaluate strategic reasoning in Large Language Model agents under real-world uncertainty. Towards this, the paper introduces Fantasyclash, a multi-agent fantasy hockey environment in which agents 1) research historical player statistics and news through database queries, 2) acquire players through an auction and select weekly lineups, and 3) bet on their lineups’ performance relative to other agents. The outcomes are determined by players’ performances in historical NHL games. Consequently, agents must handle both uncertainty about player performance and competition over player acquisition and betting returns.
For experiments, the paper evaluates eight LLMs and three deterministic baselines. The authors introduce seven strategy prompts and find that five improve performance in their main strategy experiment. They further document cyclic performance relationships among three strategies, declining returns as more agents adopt the same strategy, and limited spontaneous use of opponents’ past behavior. Disclosing an opponent’s strategy enables exploitation in some cases. Finally, the paper uses a Plackett–Luce model to summarize agent performance on an Elo scale across different pools of competitors.
Critical strengths
-
The environment brings together two meaningfully different sources of difficulty. Agents must predict an external process they cannot control, while making decisions whose consequences depend on other agents. The combination of player acquisition, lineup selection, and betting creates a richer decision problem than evaluating predictions or individual actions in isolation. Moreover, performance is measured through realized game outcomes rather than an LLM judge’s assessment of whether an explanation sounds strategic.
-
The paper goes beyond reporting an aggregate leaderboard. The strategy interventions, cyclic matchups, and opponent-disclosure experiments provide useful ways to examine why agents succeed or fail. The authors also report unfavorable findings, including strategies that reduce performance and opponent-tracking prompts that do not produce a clear performance improvement. These analyses are more informative than presenting only the best-performing model or prompt.
-
The experimental environment is documented in substantial detail. The appendices describe the prompts, data filtering, anonymization, temporal cutoffs, and configurable game settings. This makes important aspects of the evaluation inspectable and provides a useful foundation for subsequent controlled experiments.
Critical weaknesses
- (Major weakness) The experiments do not sufficiently distinguish strategic reasoning from the other capabilities required to play the game.
The paper claims to evaluate strategic reasoning, but performance also depends on statistical forecasting, database querying, arithmetic, and budget management.
- Appendix A.3 documents frequent failures in SQL queries, budget management, and interpretation of fantasy points. These errors can substantially affect performance independently of strategic reasoning.
- The existing deterministic baselines do not isolate this issue, as they differ from LLM agents in forecasting capabilities and access to information.
Consequently, an agent may outperform another simply because it is better at data analysis or execution, rather than reasoning about opponents. A controlled evaluation holding forecasting and execution capabilities fixed, while varying only opponent-aware decision-making, would be needed to establish that the benchmark measures strategic reasoning.
- (Major weakness) The strategy experiments demonstrate that LLMs can follow effective strategies, but not necessarily that they can develop or select them.
The authors provide seven predefined strategy prompts and show that some improve performance. However, this largely measures whether agents can execute strategies designed by the researchers.
- For instance, Decoy, Tax Collector, and Loyalist explicitly specify how agents should behave in particular competitive situations. Their effectiveness does not establish that LLMs can independently discover these strategies.
- Similarly, the cyclic dominance in Table 2 may arise entirely from the predefined rules, without agents reasoning about opponents.
Therefore, the experiments demonstrate strategic interactions in the environment, but provide limited evidence of strategic reasoning by the LLMs themselves. A comparison with deterministic implementations of the same strategies, together with experiments requiring agents to select or adapt strategies independently, would help establish the additional contribution of LLM reasoning.
- (Major weakness) The conclusions about agents' ability to reason about opponents are not sufficiently supported by the evaluation.
Section 5.4 reports that only 119 out of 14,099 action explanations mention an opponent, and only one uses an opponent's past behavior. However, this does not necessarily imply that agents fail to reason about opponents.
- An agent may consider an opponent's behavior without explicitly mentioning it in its explanation. Conversely, mentioning an opponent does not establish that this information influenced its decisions.
- The opponent-tracking intervention does not produce a clear performance improvement, but this alone cannot distinguish a failure to understand opponents from a failure to act effectively on that understanding.
- The strategy-disclosure experiment is interesting, but Table 3 provides only two model–season cases, limiting the generality of its conclusions.
A more direct experiment would compare agents' decisions with and without information about opponents, holding other information fixed. This would better establish whether agents actually use opponent information and whether doing so improves their decisions.
- (Minor weakness) The aggregate agent rankings may be misleading given the paper's own evidence of opponent-dependent performance.
The paper demonstrates that relative performance depends on the opponents and their strategies. However, its Plackett–Luce evaluation summarizes performance using a single rating per agent.
While such rankings are useful descriptively, they may depend substantially on the composition of opponent pools. The paper should examine whether the rankings remain stable under different opponent distributions and report matchup-level performance alongside the aggregate ratings.
Questions for authors
Questions For Authors
(Major) How can the authors establish that FantasyClash measures strategic reasoning rather than forecasting, querying, or execution capabilities? Can they provide controlled comparisons holding these other capabilities fixed?
(Major) How do LLM agents compare with deterministic implementations of the same predefined strategies? Can agents independently discover, select, or adapt strategies without being explicitly instructed to follow them?
(Major) Why should the frequency of opponent mentions in action explanations be considered evidence of opponent reasoning? Can the authors directly measure how agents' decisions change when opponent information is provided or withheld?
(Minor) How sensitive are the Plackett–Luce rankings to the composition of opponent pools? Would the rankings remain consistent under a common opponent distribution?
General Response
We would like to thank all of the reviewers for your time reviewing the paper. Given the initial reviews, we have decided to revise and resubmit, and hope to incorporate your invaluable comments in further revisions. We sincerely appreciate your thoughtful comments and effort spent on reviewing the paper.
Nonetheless, we would like to take the chance to re-emphasize the main focus of FantasyClash, and why we believe the work is novel among a plethora of seemingly similar effort. The main capability we aim to assess in agents is at the intersection of both (a) strategic behaviours concerning one's opponents, and (2) real-world uncertainties, stemming from the need to bet on real world sports outcomes. From this view, while each of the component is individually valuable and therefore has accumulated a substantial literature, we do not view them as comparable benchmarks.
Again, we would like to sincerely thank you for the time you have put in reviewing the paper.