- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SpireBench, a benchmark for long-horizon decision-making in Slay the Spire. A mod reads the state of the original game, and the harness presents it as text and executes the model's tool calls. Models thus play the full game, from combat and card selection to route planning. The authors evaluate 20 models as the Ironclad at Ascension 20 (A20), with human players and rule-based policies as references, and examine the effects of reskinning, memory mechanisms, and interaction design. Strong models show competent tactical play, but their gap relative to human experts widens as runs progress. Cross-run memory also fails to yield consistent gains.
Critical strengths
-
The full game connects decisions across multiple time horizons. Models must plan actions within a turn while considering how card choices, resource use, and route selection affect the rest of the run. Early choices have lasting consequences, making this a useful setting for testing whether local competence translates into successful play over an entire run.
-
The controlled comparisons examine several factors behind performance. Beyond comparing models, the paper varies game terminology, memory handling, and interaction design. These experiments help assess how performance depends on prior game knowledge, accumulated experience, and the harness itself, adding explanatory value to the model rankings.
-
The behavioral analyses provide concrete evidence for understanding failures. The authors examine combat logs, resource choices, prediction calibration, and memory contents, with data from ordinary players and experts as references. These analyses connect final scores to specific behaviors and identify questions that future work can test.
Critical weaknesses
-
The main results average ten runs without comparing models at each position in the run sequence. Table 1 reports performance averaged over ten runs. Cross-run memory is enabled by default, so later runs can in principle draw on experience from more preceding games; the memory available at the start of each run differs. One model might initially perform poorly and then improve with experience, while another might gain little from memory and perform similarly throughout. They could nevertheless receive the same average score, obscuring these differences. Figure 18 already compares the first five and last five runs and includes a no-memory control on the same seeds, but it does not show model performance aligned by individual run position and seed. Reporting these results would help reveal differences hidden by the averages.
-
Interpreting performance requires distinguishing rule understanding from long-horizon decision-making. Poor decisions may reflect a misunderstanding of the rules or action effects, or a failure to plan over the long term despite understanding them. For example, Figure 3(a) shows that models are more inclined to heal at rest sites. Interpreting this behavior requires knowing whether they understand what card upgrades do, that upgrades last for the rest of the run, and how healing at act transitions works. A brief discussion would be helpful; see Q1 for the specific questions.
-
Consolidating self-written notes at the end of a run may limit retrospective analysis. In Figure 6, consolidation receives the outcome, run notes, and existing memory bank. The importance of an action may become apparent only after a loss, when the player looks back over the run, so the model may not have recorded it at the time. Consolidating these notes alone may therefore leave out the information needed to reconsider earlier decisions and discover mistakes that went unnoticed during play. This could be one reason for the limited gains from cross-run memory. I suggest a brief discussion; see Q4.
Questions for authors
-
Q1 · Rules and inputs: Appendix C.2 states that models receive a compendium of rules and items. Could the authors show the prompts the models actually receive, including how the rules, current state, history, and memories are organized and presented? For example, when choosing between upgrading a card and healing, how are upgrade effects, their duration, and healing at act transitions explained? Where does this information appear, and how much space does it occupy in the input? Are there experiments that help distinguish poor performance due to a failure to understand or use the relevant rules from poor long-term decisions despite understanding them?
-
Q2 · Conversation resets between floors: Are models explicitly told that a new conversation will begin on the next floor and which information will be retained? Without this knowledge, a model might not record a useful strategy discovered on the current floor, such as a particular card combination, and lose access to it on the next floor. Please clarify how this is communicated in the prompts.
-
Q3 · History management in Continuous: The Continuous setting in Table 2 keeps one conversation throughout a run. Is the full conversation history retained verbatim, and what happens when it exceeds the context window?
-
Q4 · Information for reflection and the quality of the resulting lessons: Beyond the inputs listed in Figure 6, can the model revisit the full game trajectory? The purpose would be to support the kind of retrospective analysis human players perform: reconsidering whether an earlier choice contributed to a later loss, such as passing up an important card, losing unnecessary HP in a fight that could have been completed without damage and thus reducing the margin for error later, or spending a turn's energy on the wrong cards. Appendix C.3 already asks models to summarize what went well and what went wrong. However, models and human players alike may struggle to recognize the importance of a decision at the time and only realize after a loss that they should have chosen differently. The model may therefore never have recorded the action, leaving subsequent consolidation unable to revisit that part of the run. Does this limit the retrospective analysis that consolidation can support? I suggest sharing cases in which models derive advanced strategies or techniques through reflection and apply them in later runs. Such examples would help readers assess how much useful insight agents can derive from their own play: whether they learn lessons that guide later decisions, rather than merely recount events of limited practical value, such as "I played Strike in this run, then had too little HP and was killed by an enemy." I would welcome a discussion of two separate possibilities: first, whether access to the full trajectory, with the agent deciding whether and how to use it, could address the information limits of self-written notes; second, whether explicitly asking the agent to review the run, analyze the causes of failure, and consider alternative choices could improve what it learns. The first concerns access to information; the second concerns guidance of the agent's workflow.
-
Q5 · Memory controls and legend clarity: Table 24 includes a Keep notes condition for XML notes. Why is there no corresponding condition for Compact notes? The legend in Figure 18 should clarify the form of notes used within each run and how they are handled between runs. For example, I assume "No memory" still uses XML notes within a run but does not pass memory to the next run. Clarifying distinctions like this would make the figure easier to interpret.
-
Q6 · Performance changes in the memory experiments: In Figure 18, performance changes in different directions across models between the first five and last five runs; some strong models do not improve and even decline. I suggest a brief discussion of possible reasons to help readers interpret these patterns.
-
Q7 · The effect of single-action interaction: Single action allows the model to receive feedback from the environment after each step before choosing its next action. Intuitively, this should help performance, but Table 2 does not show a consistent advantage. I suggest briefly discussing possible reasons.
-
Q8 · Motivation for By index: In Table 2, By index replaces selection by name with selection by number. This otherwise seems similar to plain Single action. Why was this additional setting included? Was there a particular consideration behind it?
-
Q9 · The only Heart kill: Which seed and which run in the sequence produced Sol 5.6's Heart kill? Was it the tenth (final) run? I assume it used the default XML notes and cross-run memory configuration, right? I suggest describing the important decisions in this successful run and comparing them with the performance and decisions of other strong models, such as Fable and Astra, on the same seed. Where did those models fail? This would help readers understand how the successful run unfolded.
-
Q10 · Content unlocks: Did all runs use the same set of unlocked cards, relics, and other content? My understanding is that the available pool can expand as more games are played. If it changed across runs, the differences between runs would reflect more than random seeds and accumulated experience. Please clarify this aspect of the setup.
-
Q11 · Quitting and reloading: In the original game, players can restart the current combat by quitting and reloading. Are agents allowed to use this feature deliberately to replay a fight and try different decisions?
6 comments
Response 2 - Learning during evaluation, and memory consolidation methods
W1/Q6. Learning along the chain of runs. We agree; a mean over ten runs can hide different learning trajectories: two models could reach the same mean, one starting poorly and improving as its memory bank grows, the other playing at a constant level. The submission's Fig. 18 (now Fig. 20, p. 87) compares runs 1–5 with runs 6–10, which cannot separate these for one model; this is why its arrows point in different directions (Q6; discussed beside it, p. 84). The two halves are different seeds, and five seeds per half are too few to compare within one model. To address this, the revised paper now separates the trajectories in three ways. None shows a model improving as its bank grows, though the third shows that the finished bank after all ten runs of building memories helps a little.
- Run by run. New evaluation (Fig. 14, p. 60; App. F, p. 58): we show every model's ten runs in the order played, one column per seed. Half the models (10 of 20) perform better over runs 6–10 than over runs 1–5, and half the reverse (mean change −0.5 floors). What varies is mostly the seed's intrinsic difficulty, not the run index. Models agree which seeds are hard (mean final floor ranges from 9.2 on seed 4, where all of the four strongest models die by floor 7, to 25.6 on seed 8).
- Against the same seed without memory. New evaluation (Fig. 21, p. 88, Table 36; App. J.3): each main run with a growing memory bank is compared against the same model on the same seed without cross-run memory. This removes both the model's competency and seed's difficulty, and with it most of the noise in Fig. 20, so learning would show as a difference that grows along the sequence of runs. It does not: averaged over runs the difference is −0.06 acts cleaner for consolidated memories, +0.03 for notes kept verbatim, and +0.01 for the compact summary. The fitted change per run is within ±0.02 acts for every method (p ≥ 0.29; 9 models).
- The learnt bank, deployed on new seeds. New experiment (App. J.5, Table 40, p. 91): each model's final bank from the ten main seeds is frozen and given to the same model on ten new seeds, against the same seeds run without a bank. This separates what the memory bank holds from the experience of writing it, and has more statistical power since all ten runs use the full bank. We find memory banks do hold something useful: +2.1 floors over 80 paired runs (6 of 8 models higher; p=0.062; 95% interval −0.2 to +4.3), 9 p.p. more runs that clear Act 1, +0.11 mean acts cleared; a similar size to the gain from removing the thematic reskin (+1.6 floors, +11 p.p. Act 1, +0.14 acts; Table 38, p. 90). At ten seeds per model this falls just short of conventional significance, but every estimate points to a benefit.
Still, some strong models decline because errors accumulate in the banks, remaining uncorrected; models whose banks hold more errors clear fewer acts (Spearman ρ=−0.67), and the banks lean towards caution (App. H.6, E.4, pp. 69, 54).
Q5. Memory controls. Each experimental arm is defined by whether and how it keeps records between floors within a run and between runs (App. B.3, "Notes and memory", p. 23, and new Table 4, p. 24; modes renamed for clarity):
- XML Notes: keyed notes presented to the model in XML format within the run, each kept until the model deletes it;. Nothing passes to the next run, so each run starts with a blank slate.
- XML Notes + Keep Notes: the same XML note-taking within a run, and every note passes to the next run verbatim unless the model deletes it; there is no end-of-run consolidation.
- XML Notes + Memo, the main-table configuration: the same XML notes; at the end of the run the model chooses which lessons to save into a separate list of memories, and only that list passes on.
- Compact Notes: one prose summary (compactify) instead of a list of notes, rewritten by the model at the end of every floor; nothing passing between runs.
- Compact Notes + Memo: the same summary within the run, plus a second summary rewritten at the end of each run which passes between runs.
There is no compact counterpart of + Keep Notes (one summary rewritten every floor and carried across runs). We thank the reviewer for the suggestion; we don't expect adding it would change our findings, but will consider it for completeness.
We also added a No notes or memory arm, which writes nothing within a run (entirely losing history on the conversation reset between floors), so that the impact of within-run notes can be measured. Against it, XML notes add 0.09 acts (95% interval −0.07 to +0.30) and −0.1 floors, and the compact summary is similar (8 models); adding a between-run memory bank to either kind of note-taking does not change the act clear rate (App. J.3, Table 35, p. 85). We have updated the captions of Figs. 18 and 20 (pp. 81, 87) (Figs. 16 and 18 in the initial submission) to say what each memory arm keeps within a run and what it passes to the next.
Response 3 - Reflection and continuous conversations
W3 / Q4. Information for reflection. At the end of a run the model sees the full conversation for the floor where the run ended and receives the result (win/loss), each act's full map with its route plan and the path taken, a one-line record of every floor (encounter, turns, change in HP and gold, cards, relics and potions gained or skipped) equivalent to what is shown to a human in the run history page, all its run notes and its existing memories (Appendix B.6, "End of the run", p. 42). It does not receive the turn-by-turn trajectory from previous floors.
The new Appendix I.5–I.7 (pp. 78–79) examines what the models do with this:
- Consolidation is not confined to the notes. In all but one of the 160 runs read, the model saves at least one lesson that appears in none of its notes, for example one drawn from the floor record.
- The limit is the diagnosis. A routing mistake is named as a cause in 75 runs, a card wrongly taken in 40 and a card wrongly skipped in 28, but the models blame the final fight in 87 runs where the annotating model, reading the same record, does so in 27; in 67 runs the model's account contradicts its own floor record, and in 84 it misses a large problem the record shows, such as dying with its gold unspent (Appendix I.6, p. 78).
- Lessons that are applied. A memory from an earlier run is cited as the reason for a decision in 7% to 11% of calls. Where it could be checked, a cited memory was correct more than ten times as often as it was false, but when present false memories persist: GLM-5.3 carries from seed to seed the wrong rule that the heal between acts is 55% of maximum HP instead of 75% (Appendix I.4, p. 77). Appendix I.7 (p. 79) follows one seed in detail: Claude Opus 5 cites its memories 91 times in that run and, on the strength of lessons from earlier deaths, routes past every harder fight in the third act, which costs it the run due to failing to scale its strength further. Appendix B.6 ("The next run", p. 42) shows examples of the techniques models write down.
We have not tested giving the agent the full turn-by-turn trajectory for all previous floors in the run, because for the runs with the most to learn from it would not fit in the model's context window. We measured the full record (every message, tool call and tool result, without the model's reasoning) of each of the 200 main-table runs: the median run is 121k tokens, the 90th percentile 676k, and the longest 3.2M (Claude Opus 5), three times its 1M-token window. The record exceeds the model's own window in 28 of the 200 runs, and in 64 if the available reasoning text is included. These are mostly the longest runs of the strongest models: 7 of Opus 5's ten runs, 5 of Claude Fable 5.1's, 2 of GPT-6 Astra's, and GPT-5.6 Sol's run that killed the Heart (1.2M tokens against a 922k window). Weaker models die sooner and so have shorter records, but they also have smaller windows: Gemma 4 E4B (32k) would overflow in 7 runs. These figures leave out the system prompt and memory bank, which are repeated (from the KV cache) on each floor. Nonetheless, this could still be explored by chunking the turn-by-turn history and having the model read it with the hindsight of knowing how the run ended.
But our evidence suggests access to information is not the only constraint — in some cases the models already misread the concise floor summary we provide them with: their accounts of a loss contradict it in 42% of runs and miss a large problem it shows in 53% (Appendix I.6, p. 78). With regards to guidance, the consolidation prompt already asks the model to consider what went well, where it went wrong and what it wishes it had known. We have now added both points, and the information the summary leaves out, to the Limitations (Section 5, Limitations, p. 10).
Q3. History in Continuous. The history is kept as sent, with no summarization. If it outgrows the model's context window, the oldest turns are dropped and a note tells the model how many were removed (now stated in Appendix C.6, p. 46; the trimming is described in Appendix B.5, p. 24). This happened on 12% of the Continuous arm's calls (809 of 6,655), from three models: gpt-oss-120b (35% of its calls) and Nemotron 3 Nano Omni (38%), in every run, and Qwen3.5 122B-A10B (6%, in five runs). With one conversation per floor, the same two models are truncated on 1% and 20% of their calls in the main runs (Appendix B.5, p. 24).
Response 1 - Rules and inputs; Conversation resets
We thank the reviewer for their positive feedback on the paper, and address their questions below.
W2 / Q1. Rules and inputs.
Appendix B.6 ("What the model receives", p. 28) now details the prompts used.
The principle we use for our prompts is to tell the model the same details about the game that a human is presented with, and try not to hand-hold the model and bias it towards a particular solution space. This may leave some details about the game unclear; if and when this happens, we expect the model to witness how the game or harness works by interacting with the environment and then to record what it learnt in its memories for future reference. For rest sites, this means the prompt only offers the choice between resting and upgrading a card, and shows how many hit points the rest would heal. We do not explicitly tell the model that card upgrades persist for the rest of the run. If this is not clear to the model, we expect it to note that down for future reference. It does see the effect of each upgrade in its system prompt, so it decides whether to upgrade knowing what it will get from the upgrade. In practice, the models almost never doubt that a rest-site upgrade persists: across the 160 annotated runs, only gpt-oss-120b twice takes one to last a single fight (Appendix I.2, p. 74). The recurring error is the reverse: treating an upgrade made during a fight, whose card text says it lasts "for the rest of combat", as permanent. Qwen3.6 even carries this belief in its memory bank from seed to seed.
Do the models know these rules? We examined the models' reasoning traces (new Appendix I.2, I.3, pp. 74, 76). The models rarely misstate what an upgrade does: only 6% of such statements are wrong. The end-of-act heal is different: only five models ever take it into account at a rest site (Appendix I.3, p. 76), only the four strongest record it correctly, and ten models never record it (Appendix E.4, p. 54).
Rule understanding against long-term decisions. We separate three kinds of error.
Weak models already fail at the first two. Strong models rarely do, except on knowledge that has to be learnt over many runs, so their gap to expert players lies both there and in their judgement. This is now discussed in Appendix I.2 (p. 76), and the rest-site case in Section 4 (p. 6).
Q2. Conversation resets. At the end of a floor, we prompt the model to write notes to take into the next floor. The models were not explicitly told that a new conversation will begin. They are told, though, that notes carry forward. The save_note tool description says "Save a note about something you learned during this floor and want to remember either for future floors in this run, or future runs. [...] These notes persist for the rest of this run, and will be used to update your permanent memories at the end of the run.", and the end-of-floor prompt says "Reflect on this floor. If you learned anything worth remembering for later floors or future runs, call
save_noteto record it. You may callsave_notemultiple times in a single response—one call per insight. Write concise, distilled insights—not a log of what happened." We have added this to Appendix B.6 (p. 28). In practice the models usually record what they work out: they write down 88% of their discoveries about how the game works (Claude Fable 5.1 every time), though the weakest models do so in a third of cases or fewer (Appendix I.4, p. 77). Writing a rule down does not keep it, though: after discovering a rule for themselves, models state it wrongly again, mostly in a later run (new table of rules observed but not kept, Appendix I.4, p. 77).