acceptodds
Under review as a conference paper at ICLR 2027

SpireBench: Evaluating Long-Horizon Strategic Play of LLMs with Slay the Spire

Abstract

Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success. Across 20 LLMs at the game’s highest difficulty, frontier models beat heuristic baselines and the average human player but remain far behind experts, and the gap opens as the run lengthens: the strongest models win the first act as often as experts, then fall behind. What separates strong models from weak is how well they play each fight; all of them misjudge risk over a run, preserving health rather than spending it to grow stronger. A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge of the game, and models do not improve from the memories they write across runs, which are often wrong and overly cautious. Our harness is the first to give a model the complete original game, and we compare it with existing harnesses. Overall, SpireBench offers an unsaturated long-horizon benchmark with tools to locate where and why agents fail.

Primary area:
datasets and benchmarks
Keywords:
agentic, llm, benchmark, reasoning, long-horizon, video game

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FBfyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpireBench, a benchmark for long-horizon decision-making in Slay the Spire. A mod reads the state of the original game, and the harness presents it as text and executes the model's tool calls. Models thus play the full game, from combat and card selection to route planning. The authors evaluate 20 models as the Ironclad at Ascension 20 (A20), with human players and rule-based policies as references, and examine the effects of reskinning, memory mechanisms, and interaction design. Strong models show competent tactical play, but their gap relative to human experts widens as runs progress. Cross-run memory also fails to yield consistent gains.

Critical strengths
  • The full game connects decisions across multiple time horizons. Models must plan actions within a turn while considering how card choices, resource use, and route selection affect the rest of the run. Early choices have lasting consequences, making this a useful setting for testing whether local competence translates into successful play over an entire run.

  • The controlled comparisons examine several factors behind performance. Beyond comparing models, the paper varies game terminology, memory handling, and interaction design. These experiments help assess how performance depends on prior game knowledge, accumulated experience, and the harness itself, adding explanatory value to the model rankings.

  • The behavioral analyses provide concrete evidence for understanding failures. The authors examine combat logs, resource choices, prediction calibration, and memory contents, with data from ordinary players and experts as references. These analyses connect final scores to specific behaviors and identify questions that future work can test.

Critical weaknesses
  • The main results average ten runs without comparing models at each position in the run sequence. Table 1 reports performance averaged over ten runs. Cross-run memory is enabled by default, so later runs can in principle draw on experience from more preceding games; the memory available at the start of each run differs. One model might initially perform poorly and then improve with experience, while another might gain little from memory and perform similarly throughout. They could nevertheless receive the same average score, obscuring these differences. Figure 18 already compares the first five and last five runs and includes a no-memory control on the same seeds, but it does not show model performance aligned by individual run position and seed. Reporting these results would help reveal differences hidden by the averages.

  • Interpreting performance requires distinguishing rule understanding from long-horizon decision-making. Poor decisions may reflect a misunderstanding of the rules or action effects, or a failure to plan over the long term despite understanding them. For example, Figure 3(a) shows that models are more inclined to heal at rest sites. Interpreting this behavior requires knowing whether they understand what card upgrades do, that upgrades last for the rest of the run, and how healing at act transitions works. A brief discussion would be helpful; see Q1 for the specific questions.

  • Consolidating self-written notes at the end of a run may limit retrospective analysis. In Figure 6, consolidation receives the outcome, run notes, and existing memory bank. The importance of an action may become apparent only after a loss, when the player looks back over the run, so the model may not have recorded it at the time. Consolidating these notes alone may therefore leave out the information needed to reconsider earlier decisions and discover mistakes that went unnoticed during play. This could be one reason for the limited gains from cross-run memory. I suggest a brief discussion; see Q4.

Questions for authors
  • Q1 · Rules and inputs: Appendix C.2 states that models receive a compendium of rules and items. Could the authors show the prompts the models actually receive, including how the rules, current state, history, and memories are organized and presented? For example, when choosing between upgrading a card and healing, how are upgrade effects, their duration, and healing at act transitions explained? Where does this information appear, and how much space does it occupy in the input? Are there experiments that help distinguish poor performance due to a failure to understand or use the relevant rules from poor long-term decisions despite understanding them?

  • Q2 · Conversation resets between floors: Are models explicitly told that a new conversation will begin on the next floor and which information will be retained? Without this knowledge, a model might not record a useful strategy discovered on the current floor, such as a particular card combination, and lose access to it on the next floor. Please clarify how this is communicated in the prompts.

  • Q3 · History management in Continuous: The Continuous setting in Table 2 keeps one conversation throughout a run. Is the full conversation history retained verbatim, and what happens when it exceeds the context window?

  • Q4 · Information for reflection and the quality of the resulting lessons: Beyond the inputs listed in Figure 6, can the model revisit the full game trajectory? The purpose would be to support the kind of retrospective analysis human players perform: reconsidering whether an earlier choice contributed to a later loss, such as passing up an important card, losing unnecessary HP in a fight that could have been completed without damage and thus reducing the margin for error later, or spending a turn's energy on the wrong cards. Appendix C.3 already asks models to summarize what went well and what went wrong. However, models and human players alike may struggle to recognize the importance of a decision at the time and only realize after a loss that they should have chosen differently. The model may therefore never have recorded the action, leaving subsequent consolidation unable to revisit that part of the run. Does this limit the retrospective analysis that consolidation can support? I suggest sharing cases in which models derive advanced strategies or techniques through reflection and apply them in later runs. Such examples would help readers assess how much useful insight agents can derive from their own play: whether they learn lessons that guide later decisions, rather than merely recount events of limited practical value, such as "I played Strike in this run, then had too little HP and was killed by an enemy." I would welcome a discussion of two separate possibilities: first, whether access to the full trajectory, with the agent deciding whether and how to use it, could address the information limits of self-written notes; second, whether explicitly asking the agent to review the run, analyze the causes of failure, and consider alternative choices could improve what it learns. The first concerns access to information; the second concerns guidance of the agent's workflow.

  • Q5 · Memory controls and legend clarity: Table 24 includes a Keep notes condition for XML notes. Why is there no corresponding condition for Compact notes? The legend in Figure 18 should clarify the form of notes used within each run and how they are handled between runs. For example, I assume "No memory" still uses XML notes within a run but does not pass memory to the next run. Clarifying distinctions like this would make the figure easier to interpret.

  • Q6 · Performance changes in the memory experiments: In Figure 18, performance changes in different directions across models between the first five and last five runs; some strong models do not improve and even decline. I suggest a brief discussion of possible reasons to help readers interpret these patterns.

  • Q7 · The effect of single-action interaction: Single action allows the model to receive feedback from the environment after each step before choosing its next action. Intuitively, this should help performance, but Table 2 does not show a consistent advantage. I suggest briefly discussing possible reasons.

  • Q8 · Motivation for By index: In Table 2, By index replaces selection by name with selection by number. This otherwise seems similar to plain Single action. Why was this additional setting included? Was there a particular consideration behind it?

  • Q9 · The only Heart kill: Which seed and which run in the sequence produced Sol 5.6's Heart kill? Was it the tenth (final) run? I assume it used the default XML notes and cross-run memory configuration, right? I suggest describing the important decisions in this successful run and comparing them with the performance and decisions of other strong models, such as Fable and Astra, on the same seed. Where did those models fail? This would help readers understand how the successful run unfolded.

  • Q10 · Content unlocks: Did all runs use the same set of unlocked cards, relics, and other content? My understanding is that the available pool can expand as more games are played. If it changed across runs, the differences between runs would reflect more than random seeds and accumulated experience. Please clarify this aspect of the setup.

  • Q11 · Quitting and reloading: In the original game, players can restart the current combat by quitting and reloading. Are agents allowed to use this feature deliberately to replay a fight and try different decisions?

6 comments
AuthorsOpenReview ↗

Response 1 - Rules and inputs; Conversation resets

We thank the reviewer for their positive feedback on the paper, and address their questions below.

W2 / Q1. Rules and inputs.

Appendix B.6 ("What the model receives", p. 28) now details the prompts used.

The principle we use for our prompts is to tell the model the same details about the game that a human is presented with, and try not to hand-hold the model and bias it towards a particular solution space. This may leave some details about the game unclear; if and when this happens, we expect the model to witness how the game or harness works by interacting with the environment and then to record what it learnt in its memories for future reference. For rest sites, this means the prompt only offers the choice between resting and upgrading a card, and shows how many hit points the rest would heal. We do not explicitly tell the model that card upgrades persist for the rest of the run. If this is not clear to the model, we expect it to note that down for future reference. It does see the effect of each upgrade in its system prompt, so it decides whether to upgrade knowing what it will get from the upgrade. In practice, the models almost never doubt that a rest-site upgrade persists: across the 160 annotated runs, only gpt-oss-120b twice takes one to last a single fight (Appendix I.2, p. 74). The recurring error is the reverse: treating an upgrade made during a fight, whose card text says it lasts "for the rest of combat", as permanent. Qwen3.6 even carries this belief in its memory bank from seed to seed.

Do the models know these rules? We examined the models' reasoning traces (new Appendix I.2, I.3, pp. 74, 76). The models rarely misstate what an upgrade does: only 6% of such statements are wrong. The end-of-act heal is different: only five models ever take it into account at a rest site (Appendix I.3, p. 76), only the four strongest record it correctly, and ten models never record it (Appendix E.4, p. 54).

Rule understanding against long-term decisions. We separate three kinds of error.

  1. Misreading the turn. On turns where the enemy intents shown determine the HP the player will have at the start of their next turn exactly, the weakest models still mispredict it 31% of the time and the four strongest 0.5% (Section 4).
  2. Wrong beliefs about the game. On general rules (of the turn, the shop and debuff timing, most of which the system prompt states), the weakest models are wrong in 36–48% of their statements and the four strongest in 3–4%. The strongest models' remaining errors are mostly about knowledge a player builds up over many runs: what an enemy will do (15% of their statements wrong) and what an item does (10%) (new table of rule errors by tier, Appendix I.2, p. 74).
  3. Misjudging the decision when the facts are right. Claude Fable 5.1 and Claude Opus 5 estimate what named upcoming fights will cost and weigh the end-of-act heal, yet the late-act rest-site decisions that weigh it, most of them theirs, still end in a heal anyway in 21 of 32 (Appendix I.3, p. 76; Appendix E.4, p. 54).

Weak models already fail at the first two. Strong models rarely do, except on knowledge that has to be learnt over many runs, so their gap to expert players lies both there and in their judgement. This is now discussed in Appendix I.2 (p. 76), and the rest-site case in Section 4 (p. 6).

Q2. Conversation resets. At the end of a floor, we prompt the model to write notes to take into the next floor. The models were not explicitly told that a new conversation will begin. They are told, though, that notes carry forward. The save_note tool description says "Save a note about something you learned during this floor and want to remember either for future floors in this run, or future runs. [...] These notes persist for the rest of this run, and will be used to update your permanent memories at the end of the run.", and the end-of-floor prompt says "Reflect on this floor. If you learned anything worth remembering for later floors or future runs, call save_note to record it. You may call save_note multiple times in a single response—one call per insight. Write concise, distilled insights—not a log of what happened." We have added this to Appendix B.6 (p. 28). In practice the models usually record what they work out: they write down 88% of their discoveries about how the game works (Claude Fable 5.1 every time), though the weakest models do so in a third of cases or fewer (Appendix I.4, p. 77). Writing a rule down does not keep it, though: after discovering a rule for themselves, models state it wrongly again, mostly in a later run (new table of rules observed but not kept, Appendix I.4, p. 77).

AuthorsOpenReview ↗

Response 2 - Learning during evaluation, and memory consolidation methods

W1/Q6. Learning along the chain of runs. We agree; a mean over ten runs can hide different learning trajectories: two models could reach the same mean, one starting poorly and improving as its memory bank grows, the other playing at a constant level. The submission's Fig. 18 (now Fig. 20, p. 87) compares runs 1–5 with runs 6–10, which cannot separate these for one model; this is why its arrows point in different directions (Q6; discussed beside it, p. 84). The two halves are different seeds, and five seeds per half are too few to compare within one model. To address this, the revised paper now separates the trajectories in three ways. None shows a model improving as its bank grows, though the third shows that the finished bank after all ten runs of building memories helps a little.

  1. Run by run. New evaluation (Fig. 14, p. 60; App. F, p. 58): we show every model's ten runs in the order played, one column per seed. Half the models (10 of 20) perform better over runs 6–10 than over runs 1–5, and half the reverse (mean change −0.5 floors). What varies is mostly the seed's intrinsic difficulty, not the run index. Models agree which seeds are hard (mean final floor ranges from 9.2 on seed 4, where all of the four strongest models die by floor 7, to 25.6 on seed 8).
  2. Against the same seed without memory. New evaluation (Fig. 21, p. 88, Table 36; App. J.3): each main run with a growing memory bank is compared against the same model on the same seed without cross-run memory. This removes both the model's competency and seed's difficulty, and with it most of the noise in Fig. 20, so learning would show as a difference that grows along the sequence of runs. It does not: averaged over runs the difference is −0.06 acts cleaner for consolidated memories, +0.03 for notes kept verbatim, and +0.01 for the compact summary. The fitted change per run is within ±0.02 acts for every method (p ≥ 0.29; 9 models).
  3. The learnt bank, deployed on new seeds. New experiment (App. J.5, Table 40, p. 91): each model's final bank from the ten main seeds is frozen and given to the same model on ten new seeds, against the same seeds run without a bank. This separates what the memory bank holds from the experience of writing it, and has more statistical power since all ten runs use the full bank. We find memory banks do hold something useful: +2.1 floors over 80 paired runs (6 of 8 models higher; p=0.062; 95% interval −0.2 to +4.3), 9 p.p. more runs that clear Act 1, +0.11 mean acts cleared; a similar size to the gain from removing the thematic reskin (+1.6 floors, +11 p.p. Act 1, +0.14 acts; Table 38, p. 90). At ten seeds per model this falls just short of conventional significance, but every estimate points to a benefit.

Still, some strong models decline because errors accumulate in the banks, remaining uncorrected; models whose banks hold more errors clear fewer acts (Spearman ρ=−0.67), and the banks lean towards caution (App. H.6, E.4, pp. 69, 54).

Q5. Memory controls. Each experimental arm is defined by whether and how it keeps records between floors within a run and between runs (App. B.3, "Notes and memory", p. 23, and new Table 4, p. 24; modes renamed for clarity):

  • XML Notes: keyed notes presented to the model in XML format within the run, each kept until the model deletes it;. Nothing passes to the next run, so each run starts with a blank slate.
  • XML Notes + Keep Notes: the same XML note-taking within a run, and every note passes to the next run verbatim unless the model deletes it; there is no end-of-run consolidation.
  • XML Notes + Memo, the main-table configuration: the same XML notes; at the end of the run the model chooses which lessons to save into a separate list of memories, and only that list passes on.
  • Compact Notes: one prose summary (compactify) instead of a list of notes, rewritten by the model at the end of every floor; nothing passing between runs.
  • Compact Notes + Memo: the same summary within the run, plus a second summary rewritten at the end of each run which passes between runs.

There is no compact counterpart of + Keep Notes (one summary rewritten every floor and carried across runs). We thank the reviewer for the suggestion; we don't expect adding it would change our findings, but will consider it for completeness.

We also added a No notes or memory arm, which writes nothing within a run (entirely losing history on the conversation reset between floors), so that the impact of within-run notes can be measured. Against it, XML notes add 0.09 acts (95% interval −0.07 to +0.30) and −0.1 floors, and the compact summary is similar (8 models); adding a between-run memory bank to either kind of note-taking does not change the act clear rate (App. J.3, Table 35, p. 85). We have updated the captions of Figs. 18 and 20 (pp. 81, 87) (Figs. 16 and 18 in the initial submission) to say what each memory arm keeps within a run and what it passes to the next.

AuthorsOpenReview ↗

Response 3 - Reflection and continuous conversations

W3 / Q4. Information for reflection. At the end of a run the model sees the full conversation for the floor where the run ended and receives the result (win/loss), each act's full map with its route plan and the path taken, a one-line record of every floor (encounter, turns, change in HP and gold, cards, relics and potions gained or skipped) equivalent to what is shown to a human in the run history page, all its run notes and its existing memories (Appendix B.6, "End of the run", p. 42). It does not receive the turn-by-turn trajectory from previous floors.

The new Appendix I.5–I.7 (pp. 78–79) examines what the models do with this:

  • Consolidation is not confined to the notes. In all but one of the 160 runs read, the model saves at least one lesson that appears in none of its notes, for example one drawn from the floor record.
  • The limit is the diagnosis. A routing mistake is named as a cause in 75 runs, a card wrongly taken in 40 and a card wrongly skipped in 28, but the models blame the final fight in 87 runs where the annotating model, reading the same record, does so in 27; in 67 runs the model's account contradicts its own floor record, and in 84 it misses a large problem the record shows, such as dying with its gold unspent (Appendix I.6, p. 78).
  • Lessons that are applied. A memory from an earlier run is cited as the reason for a decision in 7% to 11% of calls. Where it could be checked, a cited memory was correct more than ten times as often as it was false, but when present false memories persist: GLM-5.3 carries from seed to seed the wrong rule that the heal between acts is 55% of maximum HP instead of 75% (Appendix I.4, p. 77). Appendix I.7 (p. 79) follows one seed in detail: Claude Opus 5 cites its memories 91 times in that run and, on the strength of lessons from earlier deaths, routes past every harder fight in the third act, which costs it the run due to failing to scale its strength further. Appendix B.6 ("The next run", p. 42) shows examples of the techniques models write down.

We have not tested giving the agent the full turn-by-turn trajectory for all previous floors in the run, because for the runs with the most to learn from it would not fit in the model's context window. We measured the full record (every message, tool call and tool result, without the model's reasoning) of each of the 200 main-table runs: the median run is 121k tokens, the 90th percentile 676k, and the longest 3.2M (Claude Opus 5), three times its 1M-token window. The record exceeds the model's own window in 28 of the 200 runs, and in 64 if the available reasoning text is included. These are mostly the longest runs of the strongest models: 7 of Opus 5's ten runs, 5 of Claude Fable 5.1's, 2 of GPT-6 Astra's, and GPT-5.6 Sol's run that killed the Heart (1.2M tokens against a 922k window). Weaker models die sooner and so have shorter records, but they also have smaller windows: Gemma 4 E4B (32k) would overflow in 7 runs. These figures leave out the system prompt and memory bank, which are repeated (from the KV cache) on each floor. Nonetheless, this could still be explored by chunking the turn-by-turn history and having the model read it with the hindsight of knowing how the run ended.

But our evidence suggests access to information is not the only constraint — in some cases the models already misread the concise floor summary we provide them with: their accounts of a loss contradict it in 42% of runs and miss a large problem it shows in 53% (Appendix I.6, p. 78). With regards to guidance, the consolidation prompt already asks the model to consider what went well, where it went wrong and what it wishes it had known. We have now added both points, and the information the summary leaves out, to the Limitations (Section 5, Limitations, p. 10).

Q3. History in Continuous. The history is kept as sent, with no summarization. If it outgrows the model's context window, the oldest turns are dropped and a note tells the model how many were removed (now stated in Appendix C.6, p. 46; the trimming is described in Appendix B.5, p. 24). This happened on 12% of the Continuous arm's calls (809 of 6,655), from three models: gpt-oss-120b (35% of its calls) and Nemotron 3 Nano Omni (38%), in every run, and Qwen3.5 122B-A10B (6%, in five runs). With one conversation per floor, the same two models are truncated on 1% and 20% of their calls in the main runs (Appendix B.5, p. 24).

Reviewer_3kzVOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce SpireBench to measure the ability of LLMs to perform risk-reward trade-offs using the deck building card game Slay the Spire. They claim as contributions: A harness for the game, evaluation over 20 LLMs at the game's hardest difficulty, results from human expert play, knowledge ablations via significant entity and term replacement, and an investigation of the model's use of memory mechanisms and failure modes. They find that the best models play far too conservatively compared to human players, resulting in early success but a failure to scale to late game.

Critical strengths

The biggest strengths of the paper to me are the analysis around how the models use the memory mechanisms, knowledge ablations, and the analysis showing that LLMs play far too conservatively to find success later. While the former two can be demonstrated in other benchmarks, the latter is the most interesting insight to me. A large part of the difficulty and fun of these types of games is balancing the risk reward trade off: do you put yourself in a strong early position or take early damage to secure the late game? That models tend to seemingly exclusively prefer the former is very interesting to me, especially as I imagine the internet contains many transcripts on Slay the Spire containing optimal strategies for beating the game.

Critical weaknesses

Up front, there is a major ethics issue I am concerned about regarding both the use of a commercial product (Slay the Spire) as well as online recordings (Twitch steamer VODs) that I would strongly recommend rejection unless these are resolved. I did not see any IRB documented in the appendix. I do not have full knowledge of everything this entails legally, but I know twitch streamers have in the past issued DMCA take downs for outside entities hosting their VODs. I would suspect recorded gameplay falls under a similar jurisdiction.

I would also suggest the authors reframe from using terms that are less familiar to a wider, general audience. For example, 'Twitch', 'streamer', "no model builds its deck for the act boss the map shows, and the one deck-building signal is taking area-damage cards": I have the domain familiarity to know what all of these mean but an unfamiliar reader might struggle.

The larger issue, however, is in how the work engages in prior work. The body of work around text-adventure games as long-horizon, agentic tasks is completely absent. SWE-bench, BALROG, SmartPlay, NetHack, and Crafter are cited, but understated in their contributions in the introduction. BALROG and Dynamic cheatsheet largely already explore the two questions the authors position as 'open' in the introduction of "does a model rely on stored knowledge or on reasoning, and can it store and effectively use long-term memories? How is it impacted by stochastic?" TALES (Cui et al., 2025) explicitly exposes models using prior knowledge of games in its reasoning traces as well.

Questions for authors

1.) Please address the ethical concern

2.) Is it possible that the tendency to prefer early, safer play was due to the system prompt? I didn't see it included in the appendix.

3.) Similarly for the memory, without more information about the prompt used its difficult to establish whether the results are due to small wordings in the prompt or genuine inabilities of the model

4.) Please address the issues regarding the related work.

5 comments
AuthorsOpenReview ↗

Response to ethics concerns

Q1. Ethics concern

The reviewer's ethics concern appears to stem from a misunderstanding of our data. We do not use, host, or redistribute any Twitch VODs or other videos owned by Twitch streamers. Our expert baselines are aggregate statistics (win rates and floors reached) computed from run logs and aggregate summaries that the players themselves have shared publicly with the Slay the Spire community for the purpose of analysis. The run logs we use are not a full action-by-action record of a run; they are a JSON summary file, written by Slay the Spire itself and viewable in its in-game "Run history" screen, containing the player's current and maximum HP, gold, cards and relics acquired, and cards and relics skipped, for each floor of the run.

We use these logs only to compute per-act win rates, floors reached, survival and HP per floor, and obtain them from three places. First, we use aggregate, anonymised, human telemetry released publicly by the game's developer, Mega Crit. Second, for three streamers (vmService, panacea108, cheg0) we use data shared on a Slay the Spire data repository under the Apache 2.0 license. This GitHub repository asserts the data is shared with the permission of the streamers. We have no reason to doubt the veracity of this statement—there is no way of obtaining the run log files unless the streamer in question willfully provides them themself. Third, we use run log data downloaded from Baalorlord's personal website. He chooses to run a script which automatically uploads each run log file to his website—something not needed for viewers to enjoy his video content and done only so that members of the community can analyse his run data. Baalorlord has previously shared video analysis of his run data, and invited viewers to download his run log data themselves to do their own analysis [1], [2], [3]. As our usage of these run log files is for analysis of Baalorlord's play performance, it is entirely consistent with the stated purpose for which the run log files have been publicly distributed.

Our appendix matter at time of submission included evaluation of three other streamers—XecnaR, Lifecoach, and TerrenceMHS. These are headline summaries for their highest record win rate shared by the streamers themselves which we have not verified. Without run logs for these players we could not compute per-act win rates, floors reached or HP per floor, so they were not in the main paper. We have now decided this additional analysis is not sufficiently interesting to include in the paper as it is too limited and concerns the highest performance attained by humans, rather than average performance (not directly comparable to our LLM runs). Thus we have chosen to remove this section from the appendix.

With regards to the use of a commercial game: benchmarks built on commercial games are established practice: Atari through the Arcade Learning Environment, StarCraft II, and Minecraft. Orak, a recent benchmark we compare against, itself uses Slay the Spire along with 11 other commercially available games. A game as rich and complex as Slay the Spire cannot be built solely for a research paper, which is why commercial games have their place within ML research. We play legally purchased copies of Slay the Spire through a modding interface built into the game and in accordance with their content policy, redistribute no game assets, and require anyone running the benchmark to purchase their own copy.

With regards to the institutional review board (IRB) approval, the reviewer is correct that we do not have approval from a formal IRB investigation for this study, as it is not applicable. We actually did originally seek IRB approval in connection with our work because we were considering collecting data on human students who would blind-play the same StS seeds as the LLMs. Had we recruited human subjects we would certainly have needed an ethics approval, but since we elected to proceed with the study without recruiting human subjects we were advised we did not need an ethics review.

We have now added an ethics statement to the paper (between Discussion and References), covering all of the above.

  • [1] "[Show #1031 (2022-07-29)] 200 Heart Kills - a retrospective! Slay the Spire", Baalorlord Unedited, 0:31:24-0:32:00.
  • [2] "[Show #1136 (2023-01-20)] 400 wins in 2022: What did we learn? Slay the Spire and Monster Train", Baalorlord Unedited, 0:12:01-0:12:50.
  • [3] "[Show #1328 (2023-12-01)] Mastery Challenge Retrospective: Looking at numbers from this year", Baalorlord Unedited, 2:12:04-2:12:20.

thanks for clarifying, the licensing information resolves many of my concerns there.

Thank you for the quick response; we are glad to hear it.

Reviewer_aFJPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper tackles the problem of evaluating and understanding LLMs on game "Slay the Spire" (StS)
    • The game is a single-player turn-based strategy card-based game where the player needs to climb a tower, collect cards, fight enemies and improve the deck.
    • Particularly, the challenge presented by the game for LLMs is the ability of balancing risk, making decisions under uncertainty, and trading-off short-term rewards for long-term gains.
  • Towards this goal, the paper presents a novel StS harness involving: specific system prompt, plan-then-act mechanism, and persistent read/write memory.
  • The paper evaluates various LLMs (ranging from Astra/Opus to smaller local models) using the proposed harness at the game in the most challenging setting.
  • Furthermore, the paper presents various analysis, such as highlighting LLM-strategies (e.g., more cautious actions), influence of memory, and influence of prior knowledge.
Critical strengths
  1. Interesting Analysis: The paper presents some interesting strategies that the LLMs use that contrasts human counter-parts (Fig. 3), such as models playing more cautiously (determined by number of fights against 'elites')
  2. Discussion on what findings may or may not transfer: I appreciate the authors writing Section 5.1 on findings they find consistent or inconsistent with related works.
Critical weaknesses

Major concerns

1. Narrow/specific scope; relevance to broader community unclear

  • I believe the paper examines LLMs under a very narrow scope:
    • Slay the Spire (StS) game - specific difficulty level
    • Their proposed harness, evaluated on that game
  • This raises some concerns for me (as someone who had not heard of this game prior to reading the paper)
    • Reading is often difficult to understand (e.g., Introduction mentions 'mega crit', 'A20 Ironclad'), since it is grounded in game-specific terminology
    • The analysis largely anthropomorphizes the LLM models and evaluates it's abilities specifically for the StS game (e.g., whether they use HP as a resource). Are these trends reflective of general game-playing abilities (e.g., like Orak), or StS-specific abilities?
    • If StS-specific abilities, are these reflective of broader game-playing patterns?
    • What are general take-aways from the ananlysis, that translates to domains/games outside of StS?
  • Overall, my biggest concern is the narrow scope of the paper's analysis, where take-aways beyond StS game playing mechanisms remain unclear.

2. Memory bank

  • The authors claim as a core contribution (L93) the memory bank mechanism, allowing the agent to read/write notes across runs to a persistent data store.
  • This however does not appear to indicate clear quantitative gains - highlighted by the authors in L419.
  • A more surprising bit is that the agent does not appear to learn fundamental game mechanics (e.g., HP carried to end keeps only a quarter of its value; L319), which is exploited by human games.

Minor concerns

3. Missing Analysis: Why do models lag behind experts?

  • While I appreciate the analysis of how the LLM models within the proposed harness play the game, what remains unclear is why there is a gap between LLM models and human players.
  • Specifically, what explains the gap - Long-context reasoning? Lack of memory? Long-range planning? State tracking?
  • As a result, although the gap is thoroughly quantified, the paper provides limited technical insights on what challenges the community should tackle to address the gaps.
Questions for authors

Clarification requests:

  1. (wrt Concern 1) What are general take-aways from the analysis, that provides insights to a non-StS game player?
  2. (wrt Concern 2) If the memory bank shows no benefits in improving success rates, what benefits does the proposed harness gain from this?

Suggestions for future work:

  1. Expanding the scope of the analysis to non-StS games
  2. Extending analysis to address/remark on general LLM capabilities
3 comments
AuthorsOpenReview ↗

Response 1 - Take-away findings

We thank the reviewer for their feedback on the paper, which we have taken on board and used to improve the presentation of our work.

W1, Q1. Take-aways.

The findings below apply for agents in general; evaluating agents on Slay the Spire provided a suitable way of measuring them. They are now set out in the Discussion (Section 5, the paragraph two, p. 10).

  1. Agents under-invest when the payoff is delayed, even when they know the rule. Given a resource that can be kept or spent on growth that pays off later, the models hoard it even to their future detriment. In Slay the Spire, this resource is health: in the first act the models heal at 57–75% of rest sites, where experts heal at 9%. Choosing to heal comes with the cost of skipping a card upgrade. A card upgrade can save several hit points per fight, accumulating over the run to become more valuable than the immediate heal. Knowing the rule does not change the choice: the four strongest models correctly state that most lost health is restored at the end of each act, yet the late-act rest-site decisions that weigh this rule, most of them Claude Fable 5.1's and Claude Opus 5's, still end in a heal in 21 of 32 cases (Section 4, p. 6; Appendix E.4, p. 54). The same choice faces any agent managing a budget of compute, money or time; this is similar to BALROG's knowing–doing gap, here measured on one decision against human reference data.

  2. Self-written memory does not compound without a check on what is stored. Along a chain of ten runs, memory does not reliably improve play while it is being written (now with Claude Opus 5 at low effort added, a new No notes or memory arm and a new analysis by position in the chain): none of three mechanisms that carry it from one run to the next beats the same runs without memory (90 paired runs over 9 models), none improves as the bank grows, while notes kept within a run raise acts cleared by about half, from 0.19 to 0.28 per run (8 models; Table 35, p. 85; Figure 21, p. 88). Reusing the finished bank on ten new seeds (new experiment) gives a small improvement: about two more floors and nine more points of runs that clear Act 1, with a similar gain in floors whoever wrote the bank (Table 40, p. 91), of a similar size to the gain from removing the thematic reskin (Table 3, p. 9). The final memory banks show why: 17% of memories are false, advice to be cautious outnumbers advice to take risks seven to one, and only 5 of 20 models ever delete one (30 deletions in 200 runs; Appendix H.6, p. 69). Errors accumulate and existing biases are reinforced. Any agent that keeps its own notes over a long task with one delayed, noisy outcome, as most open-ended deployments have, is in this regime; reported memory gains come where something checks what is stored (as in some of the work discussed in the 'Memory across episodes' related work section).

  3. Scores on a familiar task include knowledge the model brought with it, even when every rule is supplied. Renaming everything in the game while keeping every mechanic and stated rule costs 9 points of first-act win rate and 1.3 floors (11 models; Table 3, p. 9), concentrated in a few models: DeepSeek V4.1 Flash falls from 0.80 acts to 0.30, and GPT-5.6 Luna from 0.60 to 0.10 (Section 4, p. 6; Table 39, p. 91). Unfamiliar words are not the cause, since nonsense names cost least (0.03 acts, against 0.09 for the thematic reskin on the same seven models; Table 38, p. 90). Beyond StS, benchmarks built on well-known tasks mix reasoning with recall, by an amount that differs by model; our renaming control measures it directly.

  4. Checks on single steps overstate competence on long tasks, and so does the agent's own confidence. Every model plays at least 87% of its turns without an invalid action, and even the very worst model wins 78% of fights. Yet every model's successful decision making at different scales falls to below 50% somewhere between a single fight and a whole run (Table 18, p. 67). The models' forecasts follow the same pattern: forecasts of the next turn are accurate and track skill, but the four models that never clear an act give themselves a 55–76% chance of surviving it, about the same as the four strongest (59–74%). For evaluating agents, step-level metrics such as valid tool calls are not evidence of long-horizon competence, and the agent's self-assessment cannot stand in for measuring it due to poor self-knowledge calibration.

On whether the benchmark measures something general: mean acts cleared tracks the Artificial Analysis Intelligence Index (Pearson r = 0.82), but the other public evaluations predict only 50% of its variance for a held-out model, against 85% for Humanity's Last Exam (Section 4, "Relation to public LLM benchmarks", p. 8).

AuthorsOpenReview ↗

Response 2 - Scope, Jargon, Memory

W1. Scope

While it would be ideal to evaluate a large number of models on a large range of tasks, and inspect their behaviour in detail for each one, this simply isn't feasible for a single research paper to cover. It is especially hard when the tasks in question are long. In this work, we chose to investigate a large number of models on one particular (but expansive) task in a lot of detail. It lets us hold the task fixed while varying the model, the harness, the memory mechanism, and the model's prior knowledge, and compare against human play on the same decisions. The fact the game is very well established means there is readily available data from thousands of human players, and from expert players who have each published dozens to hundreds of their runs at the hardest difficulty, against which model performance can be compared. In our opinion, this approach has value alongside other approaches, and can provide conclusions which generalize to other domains. We now state this design choice in the Introduction (the paragraph before the list of contributions, p. 1, L50).

W1. Jargon

We have revised the main text to remove unnecessary game-specific terminology and to explain the rest where it first appears: enemy intents, elite fights and relics (Section 2, Background, p. 3, L111); the character played, the keys that open the final act and the final boss (Section 3, Evaluation protocol, p. 5, L267); and the highest difficulty, Ascension 20 (A20), in the Introduction (p. 2, L82). Players' slang such as "hallway fight" and "lethal" is replaced in Section 4 (p. 6), and "Twitch streamers" are now "expert players".

W1. Anthropomorphization

With regards to "HP as a resource", this is a common refrain said by strategic players across board games, role-playing games (e.g. Dungeons and Dragons), and video games alike. Amateur human players have an intrinsic tendency to view the health of their character as something they need to preserve for its own sake (possibly due to loss aversion), whereas more strategic (and professional) game players often view their character's health as a resource more like other in-game resources such as gold. While viewing HP as a resource is an abstract and strategic perspective important for this game, it is also something which is common across many games and not specific to Slay the Spire. We were not able to identify any parts of the paper which anthropomorphize the model any more than existing terminology does through analogy (e.g. "think", "memory", etc.), and would welcome pointers to any problematic phrasing we have missed.

W2, Q2. Memory bank

We do not claim that the memory bank improves performance; the contribution is the measurement of it, and we have revised the contribution list to say so. Cross-run memory is the capability under test: it lets us ask whether an agent improves from its own experience, against a no-memory control on the same seeds. We find no consistent or significant improvement from memory consolidation within the chains, and, in a new experiment, a small improvement when a finished bank is reused on new seeds (about two more floors and nine more points of Act 1 clears, of a similar size to the gain from removing the thematic reskin); the paper now shows why (point 2 of our take-aways above).

The null result is not because the models ignore their memories. They cite a memory as the reason for a decision in 7% of calls on the second run and 11% on the tenth, a rise seen in 15 of 16 models (Appendix I.4, p. 76). Nor is it that they never see the rules: they often work a rule out from what happens in front of them and later state it wrongly again. For example, gpt-oss-120b observes five times that its Block is gone at the start of its turn, yet states that Block lasts into the next turn in eight of its ten runs; Claude Haiku 4.5 works out on its first run that the numbers shown on its cards already include its debuffs, then applies them a second time on the next (Appendix I.4, p. 76, new Table 30, p. 77).

The end-of-act heal the reviewer pointed to is the clearest case. We find only the four strongest models ever record the rule correctly, ten models never record it, and the models that know it still do not act on it (Appendix E.4, p. 54).

AuthorsOpenReview ↗

Response 3 - Why the gap between LLMs and human experts

W3. Why do models lag behind experts? The revised paper adds this analysis (Section 4, "How the models spend a run", p. 6, "Strong models fight better", p. 8, and "How the models reason", p. 9; Appendix E, p. 50, "Strategy and play quality"; Appendix I, p. 73, "Reading the reasoning"). Taking the reviewer's candidates in turn:

  • State tracking and use of rules separate strong models from weak ones, but not strong models from experts. On turns where the displayed information determines the next HP exactly, the bottom tier mispredicts it 31% of the time and the top four models 0.5%; the top four never miss the opportunity to kill an enemy with the cards available in hand, where the bottom tier misses 11.7% (Section 4, p. 6; Table 15, p. 57). What the strongest models still get wrong is knowledge built up over many runs rather than read once: what an enemy will do next (15% of their statements about it are wrong, against 34% for the bottom tier) and what a relic, potion or power does (10% against 51%) (new table of rule errors by tier, Appendix I.2, p. 74).
  • Long context is not the limit: one conversation for the whole run loses 0.06 acts and costs more (Table 2, p. 6). With our default of one conversation per floor, the context rarely fills: the oldest turns of a floor had to be dropped on 475 of the 37,306 calls of the main runs (1.3%), all from the three models with the smallest windows (Gemma 4 E4B with 32k tokens, 321 of its 820 calls; Nemotron 3 Nano Omni and gpt-oss-120b with 128k, 139 and 15 calls), and never for the other 17 models (Appendix B.5, p. 24).
  • Lack of memory is not the limit: we provide memory consolidation infrastructure to the model and do not limit its capacity.
  • Long-range planning under risk is where the gap to experts lies. In the first act the models fight 0.9–1.4 elite enemies (harder fights which yield more rewards) against 2.6 for experts, and heal at 57–75% of rest sites against 9%. Human runs that play the first act as the models do clear the second act in 15% of cases, against 38% for runs that take more risk (Table 14, p. 54). The reasoning shows that this is not for lack of knowing the rules: Claude Fable 5.1 and Claude Opus 5 estimate what a named upcoming fight will cost and take into account that the end of the act restores most missing health, yet still choose to heal at most of these rest sites, giving up the upgrade (Appendix I.3, p. 76; Appendix E, p. 50).
  • The first-act route explains about a fifth of the gap to experts (Appendix E.2, p. 52). The fights are one candidate for another cause, and there the gap grows along the run. In the first three fights, when every deck is still close to the starter deck, the top four models lose 2.0 HP per ordinary fight against the experts' 1.0, so the difference is in how the turn is played; by the second act the gap is about 6 HP per fight and by the third 7–11, as the experts' decks and relics pull ahead (Appendix E.1, p. 50, new Figure 11, p. 56).
  • New: single decisions, taken out of the run (Section 4, "Single decisions fail at the same horizon", p. 8; Appendix L, p. 95; Appendix M, p. 106). In 126 authored puzzles with a known best answer, accuracy falls with the horizon of the decision for all 17 models tested: 81% where the answer is one turn, 66% where it is a plan across a fight, and 52% where the decision pays after the fight. In one puzzle, waiting three turns is certain to leave the player with more health, yet 31 of 34 plays kill by the second turn, so the under-investment is not only a response to uncertainty. On decisions replayed from the experts' own winning runs, shown the state the expert saw, the models rest at 19–60% of the rest sites where the expert instead upgraded a card.

For the ML community, this points to three things to work on: value estimates that look further ahead and sharpen as the outcome approaches, systems or training to ensure memory is checked against outcomes and pruned if found to be incorrect, and decisions that follow from the rules the agent can already state (the four strongest models record the end-of-act heal correctly, and Claude Fable 5.1 and Claude Opus 5 weigh it at late-act rest sites, yet mostly choose to heal anyway). The Discussion now names all three (Section 5, the end of its first paragraph, p. 10).

Suggestions for future work. We thank you for the suggestion and agree that extending the study to other games would be a natural fit for follow-up work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additions since the original submission: an analysis of the models' reasoning (Appendix I) and two diagnostic benchmarks (Appendices L and M)

We thank all the reviewers again for their time and feedback on the paper.

We have been working hard at extending our analysis substantially since the original submission. Beyond the changes described in our individual responses, the revised paper has three large additions: an analysis of what the models write while they play (Appendix I), and two diagnostic benchmarks, played through the same harness, that evaluate the models on single decisions instead of a whole run (Appendices L and M). These bear on questions from all three reviews: why the models fall behind experts (aFJP, W3), whether a poor decision reflects a misunderstood rule or poor long-horizon judgement (FBfy, W2/Q1), and whether the cautious play comes from our setup (3kzV, Q2). They are summarised in Section 4 ("Strong models fight better" and "Single decisions fail at the same horizon", p. 8; "How the models reason", p. 9).

Reading the models' reasoning (Appendix I, p. 73). A separate model (Claude Opus 5.5, not one of those evaluated) read all ten runs of 16 models and tagged their reasoning, notes and memories against fixed categories, quoting the model for every tag (53,456 quotes).

  • Stronger models reason from more correct facts: the four strongest misstate rules far less often than the weakest (6–11% against up to 66%; Appendix I.2, p. 74).
  • The caution is deliberate, and not prompted: Claude Fable 5.1 and Opus 5 project their health into a named upcoming fight and still rest, and a model declining a risk cites its current health or a remembered lesson, almost never the prompt (Appendix I.3, p. 76).
  • Memories are used but not kept correct: a memory is cited in 7% of calls on the second run and 11% on the tenth, yet models state wrongly a rule they had worked out for themselves (Table 31, p. 78).
  • The models diagnose a loss poorly: they blame the final fight in 87 of 157 defeats, where the annotator, reading the same record, does so in 27 (Appendix I.6, p. 78).

Puzzle benchmark (Appendix L, p. 95). 126 newly-authored one-room puzzles, each with a known best answer, classified by the time horizon the decision impacts. Models play each one under the real names and our reskin (without notes or memories).

  1. Accuracy falls with the horizon of the decision, for every one of the 17 models tested: 81% where the answer is the play of one turn, 66% where it is a plan across a fight, and 52% where the decision pays after the fight (Table 45, p. 99).

  2. The models decline a delayed payoff even when it is certain. Where the fight is won either way and the question is what is carried out of it, accuracy is 29%. In one such puzzle, waiting three turns is certain to leave the player with more health, and 31 of 34 plays kill by the second turn (Appendix L.5, p. 101). Nothing in that puzzle is hidden or uncertain, so the under-investment we report in runs is not only a response to uncertainty, nor to a rule the models do not know.

  3. Puzzle accuracy orders the models almost as their runs do (Spearman ρ = 0.92 with mean floor; Appendix L.6, p. 103), yet solving puzzles is not enough: Gemini 3.8 Flash answers 88% of them and reaches mean floor 22.9, while Claude Fable 5.1 answers 84% and reaches 36.2. This may be explained by their performance against the act boss, a long fight of a kind no puzzle contains: Gemini reaches the first boss in all ten of its runs and beats it in five, where Fable beats it in all nine it reaches (Appendix E.5, p. 56). Our new findings suggest that Gemini builds its deck against heuristics it has learnt from pretraining, but does not know how to pilot such a deck.

  4. The cost of the reskin grows with the horizon: 2, 4, and 8 points (Table 47, p. 100).

Expert decisions benchmark (Appendix M, p. 106). Decisions from the expert players' winning runs, each shown to the model as the state the expert saw, and scored by whether the model makes the expert's choice. Unlike the comparison over runs, this compares models with human play on the same states. The items are a stratified draw of 900 decisions. The best model makes the expert's choice on 64%, twelve of the 15 on fewer than half, and uniform random choice on 26% (Table 50, p. 107). At the rest sites where the expert upgraded a card, the models rest instead on 19–60%.

Overall. These additions confirm the conclusions of the submission, that the models are poor at long-horizon decisions and at taking informed risks, and resolve much more finely where and why they fail. The models are competent where the outcome is a turn away and get worse the further ahead in the future the outcome of a decision applies. They pass up a delayed payoff even when it is certain; on the same states as expert players they take the safe option where the experts invest; and their reasoning shows the caution is chosen, not prompted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.