acceptodds
Under review as a conference paper at ICLR 2027

Are LLM Agents Long-Termist? Evaluating Long-Horizon Resource Optimization in Roguelike Environments

Abstract

In real-world long-horizon tasks, agents must complete immediate objectives while deciding which resources to retain, discard, upgrade and combine, because these choices shape their future options. Existing benchmarks largely assess task progression and final success, leaving ongoing resource management underexamined. We study Long-Horizon Resource Optimization as the ability to sustain progress by constructing, using and revising limited resources as task demands evolve, balancing immediate gains against future performance. The delayed and interacting effects of resource choices make this difficult to assess: poor outcomes may result from either poor resource management or ineffective use of current resources. We introduce Long Game Bench, a unified benchmark spanning Slay the Spire 2, Balatro, and Luck be a Landlord through a common interface for agent interaction. In these roguelike games, early choices constrain later options, and adding a resource can weaken an existing configuration. To examine the two sources of failure, the benchmark measures overall progress, execution with current resources, and the frequency and diversity of active resource interventions. We also compare models under a shared high-quality configuration and with active management disabled to probe the roles of resource management and execution. We evaluate 6 frontier LLMs and 4 coding-agent harnesses across over 1,000 complete game trajectories. Agents often make local progress but fail to reliably convert it into successful runs. Providing models with the same high-quality configuration narrows the performance gap, reducing the standard deviation from 29.4% to 13.8% and raises the average win rate from 17% to 74%, while disabling active management reduces it from 17% to 6%. These findings highlight the importance of resource management for long-horizon performance. These findings suggest that resource management contribute substantially to long-horizon performance.

Primary area:
datasets and benchmarks
Keywords:
Evaluation, Benchmarking, Datasets & Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_p8JbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper connects LLMs to several roguelike games to explore the ability of LLMs to optimize for long term objectives. The paper explores both providing the LLMs with a decision harness as well as using the LLM providers own coding harnesses. The paper draws links between management actions and long term optimization.

Critical strengths
  • It was interesting to see how different models and in different harnesses performed on these type of games. The paper is a good starting point for research questions about long term optimization in LLMs.
Critical weaknesses
  • Several issues with the presentation. Metrics are not clearly defined (or at all) or are explained in terms of game mechanics without clearly defining the game mechanics. Overall the paper seems to assume the reader has familiarity with the games. Figure 2 has clear issues with the buy and remove icons being swapped and overall being confusing. Figure 4 is also very confusing with weird axes scales. The appendix writing style is low quality and seems to contain some sections that feel like they are verbatim LLM output. Several data points such as progression in Figure 1 are needlessly presented with different scales making progression across games hard to compare. Many citations also are not the published versions of works or are referenced not using the lead authors name when they should be (eg using github handles instead of known names). The research questions and findings seem disconnected and the paper does not clearly connect them.
  • Several issues with the methodology. The human baseline uses an expert human but the LLMs are playing the games blind. Either the human should be blind or the LLMs should be given in context examples of successful and unsuccessful trajectories. It makes sense in this case that any strong learning agent would not be able to perform as well globally as locally as it is more likely to have encountered similar local situations during training. Many of the results are confounded by aspects that cannot be controlled for such as what training data/environments closed models use, however its is known that recent post trained models are trained on long term objectives with reinforcement learning and the opportunity to try and identify the effect of this on long term optimization by controlling for base vs post trained models or different reasoning levels is missed. Several claims seem to be overstated or are actually attributes of the environment rather than the LLMs. For example, finding 2 and 6 seem more likely to be a property of the environment rather than the LLMs and any optimal agent would likely have similar results.
Questions for authors
  • How strong is a simple RL baseline of these games? Could such models trained with short term and long term objectives not be more useful as a baseline of short and long term oracles?
  • Are baseline boss-success rates computed over all started runs or only runs reaching each boss? How does the comparison with deck-forced runs isolate resource management from earlier combat failures and resource attrition?
  • Why is the resources not considered part of the state? Is there some mechanism that prevents this from being the case?
7 comments
AuthorsOpenReview ↗

Game Mechanics (1/5)

Dear reviewer p8Jb:

Thank you for recognizing our model comparisons and its value as a starting point for long-term optimization research. We have carefully prepared responses to address your concerns.

W1.1: Insufficient Explanation of Game Mechanics

Thank you for raising this concern. We will clarify the descriptions of the game mechanics already included in the paper and provide a more detailed introduction to each of the three games.

Existing Overview of the Game Mechanics. Figure 1 already provides an intuitive abstraction of the game mechanics: players start with an initial set of resources and must sequentially face increasingly difficult challenges. As they progress, they continuously acquire, skip, remove, and upgrade components in their resource pools. Different components may exhibit complementary, synergistic, or conflicting relationships. Ultimately, game performance depends on whether players can continuously manage and optimize their resources to construct a synergistic build capable of meeting increasingly demanding challenges.

Detailed Introductions to the Three Games.

  • Slay the Spire 2: Players explore a branching map, progressing through successive floors by engaging in combat, encountering events, and visiting shops. They strengthen their builds by acquiring, upgrading, and removing cards, as well as collecting relics and potions, with the ultimate goal of defeating the Bosses at each stage. The core gameplay revolves around constructing and continuously optimizing a resource system consisting of cards, relics, potions, and other components.

  • Balatro: Players form different poker hands by combining playing cards and enhance their scoring capabilities using Jokers, Tarot cards, Planet cards, and other components. At each stage, they must meet an increasingly high score target within a limited number of hands and discards. The core gameplay revolves around managing resources such as the deck, Jokers, consumables, and money to construct a synergistic system that can consistently generate high scores.

  • Luck be a Landlord: Players generate coins through a slot machine to pay increasingly expensive rent. In each round, they select from a randomly offered set of symbols to add to the slot machine, and use items to enhance symbol effects and modify the reward mechanisms. The core gameplay revolves around constructing and continuously optimizing a resource system consisting of symbols and items to sustain long-term coin generation.

Thank you again for raising this concern. In the revised manuscript, we will add a detailed introduction to the game mechanics of all three games in the Appendix.

AuthorsOpenReview ↗

Evaluation Metrics (2/5)

W1.2: The problem of unclear indicator definitions.

Thank you for raising this question. The detailed definition and calculation of the indicators are shown in the table below. We will optimize them in the revised version.

Primary DimensionSecondary DimensionWhat It MeasuresComputation
Overall Progress—How far the agent advances throughout the gameUses game-native progression metrics: cumulative Floor in STS2, Ante in Balatro, and Rent in LBaL.
ExecutionEfficiencyHow effectively the current resource system converts available resources into useful outcomesE=1N∑i=1NGiRi\displaystyle E=\frac{1}{N}\sum_{i=1}^{N}\frac{G_i}{R_i}. NN: number of local tasks; GiG_i: total effective output obtained in the ii-th task; RiR_i: amount of resources consumed in the ii-th task.
ExecutionTask CostThe action cost required to complete local tasksC=1N∑i=1NAi\displaystyle C=\frac{1}{N}\sum_{i=1}^{N}A_i. NN: number of local tasks; AiA_i: number of actions required to complete the ii-th task.
ExecutionBurstThe peak output capacity of the current resource systemB=1N∑i=1N(13∑j=13Pi,(j))\displaystyle B=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{3}\sum_{j=1}^{3}P_{i,(j)}\right). NN: number of local tasks; Pi,(j)P_{i,(j)}: the jj-th highest single-action output in the ii-th task, with outputs sorted in descending order.
Active ManagementIntervention FrequencyHow frequently the agent proactively adjusts its resource configurationIfreq=NinterventionNavailable decision\displaystyle I_{\mathrm{freq}}=\frac{N_{\mathrm{intervention}}}{N_{\text{available decision}}}. NinterventionN_{\mathrm{intervention}}: number of resource decision points at which the agent performs an intervention; Navailable decisionN_{\text{available decision}}: total number of available resource decision points.
Active ManagementManagement DiversityThe diversity of resource management mechanisms used by the agentIcoverage=Nused management typesNavailable management types\displaystyle I_{\mathrm{coverage}}=\frac{N_{\text{used management types}}}{N_{\text{available management types}}}. Nused management typesN_{\text{used management types}}: number of distinct management action types used in a trajectory; Navailable management typesN_{\text{available management types}}: total number of available management action types (6 in STS2, 8 in Balatro, and 4 in LBaL).

The following table summarizes the game-specific definitions of the variables used in the evaluation metrics above:

VariableSTS2BalatroLBaL
GiG_iTotal damage dealt in a combatTotal score obtained in a blindTotal gold obtained in a rent stage
RiR_iTotal energy spent in a combatNumber of hands played in a blindNumber of spins performed in a rent stage
AiA_iNumber of combat turnsNumber of hands played per blindNumber of spins per rent payment
Pi,(j)P_{i,(j)}jj-th highest single-turn damagejj-th highest single-hand scorejj-th highest single-spin reward

For Execution, the three sub-metrics are first computed as trajectory-level raw scores for each individual trajectory. We then average these raw scores across all complete trajectories for each model. Each sub-metric is subsequently normalized by dividing by its 95th percentile P95P95, calculated over all trajectories within the corresponding game environment; Task Cost is inversely normalized. The final Execution score is the equally weighted average of the three normalized sub-metrics.

For Active Management, the two sub-metrics are first computed for each individual trajectory and combined into a trajectory-level score using: ABM = 07˙∗ifreq+03˙∗icoverage0\.7*ifreq + 0\.3*icoverage×100. We then average the trajectory-level scores across all complete trajectories for each model. Finally, the model-level score is normalized by the P95 of trajectory-level ABM scores across all trajectories within the corresponding game environment, yielding the Manage score.

AuthorsOpenReview ↗

Research Questions and Presentation (3/5)

W1.3: The research questions and findings seem disconnected and the paper does not clearly connect them.

The three research questions in our paper are directly addressed by the reported findings, but we agree that their correspondence could be made clearer. The mapping is as follows:

  • RQ1: Can current LLM agents effectively perform Long-Horizon Resource Optimization? Addressed by Findings 1 and 2, which evaluate agents' overall long-horizon performance and reveal the gap between task progression and successful completion.

  • RQ2: What capabilities affect agents' Long-Horizon Resource Optimization? Addressed by Findings 5 and 6, which use controlled ablations to examine the respective roles of in-combat execution and active resource management.

  • RQ3: What behavioral patterns do agents exhibit? Addressed by Findings 3 and 4, together with the Case Study in Section 3.5, which analyze agents' resource-management behaviors, differences across agent configurations, and the contrast between human and agent decision-making over long horizons.

We will make these connections more explicit in the revised version to help readers follow how the experimental findings answer the research questions.

W1.4: Other Presentation Issues (Appendix, Figures, Data Scales, and Citations)

11 Appendix and Figures

Thank you very much for pointing out these issues. In the revised version, we will revisit both figures, correct the Buy and Remove icon assignments in Figure 2, and adjust the axis presentation in Figure 4 to improve clarity and readability. We will also carefully review and polish the Appendix to improve its overall writing quality.

22 Data Scales

For cross-game progression, we did report each game's native progression metrics and will improve this presentation in the revised version. We also report normalized average progression in the Prog. Avg column of Table 1, which expresses progress as a percentage of each game's win threshold.

33 Citations

Thank you very much for pointing out the issues with citation formats and publication versions. For engineering reference projects such as BalatroBot, BalatroLLM, and sts2-cli, which do not have corresponding academic papers, we used their GitHub handles as citation identifiers in the original manuscript. In the revised version, we will verify the project authors' information and standardize these citations. We will also comprehensively review the cited versions and prioritize conference or journal versions for formally published works. In particular, we will update the citations for CivRealm, Voyager, PTCG-Bench, TRIP-Bench, UltraHorizon, and DeepPlanning accordingly.

We thank you again for these helpful suggestions, which have helped us identify areas for improvement in figure presentation, writing quality, and citation practices. We will carefully address each of these issues in the revised version.

Reviewer_PhAaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates LLM agents in Slay the Spire 2, Balatro, and Luck be a Landlord, focusing on decisions that acquire, remove, upgrade, and combine resources over a run. It compares six models and four agent harnesses, analyzes trajectories, and intervenes by supplying strong pre-boss resource states or disabling selected optimization actions. The main interpretation is that resource management is a major source of failure and that shared high-quality resources reduce performance differences between models.

Critical strengths
  1. The environments make resource management consequential. Decisions can change the value of later actions through synergy, deck composition, and accumulated resources. This provides a concrete setting for evaluating how agents manage future opportunities, and the unified interfaces could support further study.

  2. The paper attempts interventions to explain performance. Supplying shared resource states and disabling selected actions are useful starting points for testing explanations of full-run results. They provide more direct evidence about the effects of available resources and actions than behavior-count correlations alone.

Critical weaknesses
  1. The headline dispersion statistic does not clearly measure a gap between models under the same conditions. Pooling the eighteen rounded baseline values in Figure 3a across six models and three bosses gives a sample standard deviation of 29.358 percentage points; the corresponding deck-forced values give 13.843. These reproduce the abstract's 29.4 and 13.8, but mix model variation with boss difficulty. The adjacent mean improvement from approximately 17% to 74% concerns Act 3. Within that boss condition, the standard deviation across models is approximately 13.38 to 12.77. Averaging each model over bosses first produces a different contraction, approximately 18.13 to 5.28, so there is evidence of reduced dispersion under an alternative estimand. The paper should specify which quantity supports its claim and use consistent aggregation. These calculations use rounded figure entries; exact data and formulas are needed. Figure 3a also gives GPT-5.6-Sol an Act-3 baseline of 33 and increment of 40, implying 73%, while Section 3.4 reports 66%.

  2. The shared-state experiment changes several determinants of success at once. Section 3.4 supplies the complete pre-boss state from a winning reference trajectory, including the deck, relics, HP, and potions, and evaluates only the boss fight. This establishes the benefit of that resource state. It does not distinguish better long-term management from an easier immediate fight, greater tolerance of execution errors, or a ceiling effect. Interpretation also depends on whether the deck-free success rate uses all started runs or only those reaching the boss. Similarly, disabling skip, remove, and upgrade removes ways to strengthen a deck; the resulting loss demonstrates the usefulness of those operations but does not by itself identify a general deficit in long-term planning. The main claims need either controls that separate these explanations or a narrower account of the total intervention effect.

Questions for authors
  1. What are the unrounded inputs, aggregation units, and formula for the 29.4-to-13.8 claim? Please align them with the reported Act-3 means and reconcile GPT-5.6-Sol's 66% versus the figure's implied 73%.
  2. What are the denominators and attempt counts for each model, boss, and mode? In particular, does deck-free performance include runs that never reach the boss, and how are unfinished runs handled?
  3. Which existing controls distinguish resource-management quality from the immediate difficulty of combat under the supplied state? If none do, what conclusion about the intervention can be supported without the stronger capability attribution?
4 comments
AuthorsOpenReview ↗

Raw Data and Standard Deviation (1/2)

Dear reviewer PhAa:

Thank you for recognizing our evaluation setting and intervention analyses for understanding agent decisions. We have carefully prepared responses to address your concerns.

W1.1 & Q1.1 & Q2.1: Inconsistency between the aggregation used for the standard deviation and that used for the mean win rate; raw data and calculation formulas.

Thank you for this careful observation. You are correct that the standard deviation was calculated over a broader range of conditions than the mean win rate. This is because the two statistics serve different purposes in our analysis. To measure the improvement in the overall win rate, we focus on the final Boss, whose win rate reflects the agent's overall completion performance. However, when comparing the performance gaps among models under the baseline and deck-forced settings, the results on Bosses 1 and 2 are also important, as they provide additional evidence of the models' overall performance across different stages of the game.

The raw data under the baseline and deck-forced settings are provided below.

The game configurations (character–seed pairs) used in this ablation study were: Silent–LFLFWUJ8KS, Defect–LFLFWUJ8KS, and Ironclad–83P4DLYGSE.

Baseline raw data W=numberofwinningtrajectories;N=numberofcompletetrajectories;WR=winrateW = number of winning trajectories; N = number of complete trajectories; WR = win rate:

ModelBoss1 W/NWRBoss2 W/NWRBoss3 W/NWR
claude-fable-58/989%6/967%3/933%
gpt-5.6-sol8/989%5/956%3/933%
deepseek-v4-pro4/757%2/729%1/714%
kimi-k36/967%5/956%1/911%
claude-opus-4-88/1080%1/1010%1/1010%
glm-5.33/933%1/911%0/90%

Deck-forced raw data:

ModelBoss1 W/NWRBoss2 W/NWRBoss3 W/NWR
claude-fable-515/15100%15/15100%10/1567%
gpt-5.6-sol15/15100%13/1587%11/1573%
deepseek-v4-pro15/15100%15/15100%13/1587%
kimi-k315/15100%14/1593%9/1560%
claude-opus-4-814/1593%13/1587%14/1593%
glm-5.315/15100%11/1573%10/1567%

Standard deviation calculation:

The original values of 29.4 and 13.8 reported in the paper were calculated using the rounded win rates shown in the figure. The calculation is as follows:

For the baseline setting:

Pbaseline=89+67+33+89+56+33+⋯+11+018≈41.3889\displaystyle P_{\text{baseline}} = \frac{89+67+33+89+56+33+\cdots+11+0}{18} \approx 41.3889

SDbaseline=117∑i=118(Pi−41.3889)2≈29.3581≈29.4\displaystyle SD_{\text{baseline}} = \sqrt{\frac{1}{17}\sum_{i=1}^{18}(P_i-41.3889)^2} \approx 29.3581 \approx \mathbf{29.4}

For the deck-forced setting:

Pdeck-forced=100+100+66+100+87+73+⋯+73+6718≈87.7222\displaystyle P_{\text{deck-forced}} = \frac{100+100+66+100+87+73+\cdots+73+67}{18} \approx 87.7222

SDdeck-forced=117∑i=118(Pi−87.7222)2≈13.8428≈13.8\displaystyle SD_{\text{deck-forced}} = \sqrt{\frac{1}{17}\sum_{i=1}^{18}(P_i-87.7222)^2} \approx 13.8428 \approx \mathbf{13.8}

The change values shown in the figure are rounded values calculated from the raw data. Therefore, adding the displayed change to a rounded baseline value may occasionally not reproduce the displayed value under the deck-forced setting. The standard deviations should instead be calculated directly from the raw win rates, without intermediate rounding. Using the unrounded win rates derived from the raw counts, we obtain:

For the baseline setting:

Pbaseline=(8/9+6/9+3/9+8/9+5/9+3/9+⋯+1/9+0/9)18×100=41.3580\displaystyle P_{\text{baseline}} = \frac{(8/9+6/9+3/9+8/9+5/9+3/9+\cdots+1/9+0/9)}{18} \times 100 =41.3580

SDbaseline=117∑i=118(Pi−41.3580)2≈29.2468≈29.2\displaystyle SD_{\text{baseline}} = \sqrt{\frac{1}{17}\sum_{i=1}^{18}(P_i-41.3580)^2} \approx 29.2468 \approx \mathbf{29.2}

For the deck-forced setting:

Pdeck-forced=(15/15+15/15+10/15+15/15+13/15+11/15+⋯+11/15+10/15)18×100≈87.7778\displaystyle P_{\text{deck-forced}} = \frac{(15/15+15/15+10/15+15/15+13/15+11/15+\cdots+11/15+10/15)}{18} \times 100 \approx 87.7778

SDdeck-forced=117∑i=118(Pi−87.7778)2≈13.7674≈13.8\displaystyle SD_{\text{deck-forced}} = \sqrt{\frac{1}{17}\sum_{i=1}^{18}(P_i-87.7778)^2} \approx 13.7674 \approx \mathbf{13.8}

We will align these values in the revised version and provide the raw data and calculation procedure in the appendix.

AuthorsOpenReview ↗

Trajectory Criteria and Ablation Design (2/2)

W1.2 & Q1.2 & Q2.1: Trajectory inclusion criteria for deck-free evaluation and the incorrect Act-3 win rate reported for GPT-5.6-Sol.

For the baseline deck-free settings, Boss-level win rates are calculated using all trajectories that were started and subsequently completed. If a trajectory ends in death before reaching a particular Boss, it is counted as a failure for that Boss and all subsequent Bosses: it contributes zero to the number of wins but remains in the denominator.

We sincerely apologize for the incorrect Act-3 win rate reported for GPT-5.6-Sol under the deck-forced setting, and greatly appreciate you bringing this issue to our attention. We will correct this value in the revised version.

W2 & Q3: Experimental Setup and Objectives of the Deck-Forced and Deck-Greedy Ablations

Deck-forced. The purpose of the shared-state experiment in Section 3.4 is precisely to separate long-term resource management from immediate combat execution, thereby evaluating LLM agents' ability to execute combat decisions when provided with identical resource conditions. Specifically, we replay a winning reference trajectory up to the state immediately before a Boss fight, and provide all models with exactly the same pre-boss state—the outcome of long-term resource management—including the deck, relics, HP, and potions. This state represents an actual resource configuration obtained by a strong model through gameplay. The only difference lies in how each model independently makes decisions and uses the available resources eg˙,˙selectingandplayingcardsandusingpotionse\.g\., selecting and playing cards and using potions to defeat the Boss. The experimental results show that when models are given a better resource configuration than the one they constructed themselves, their boss-fight success rates increase substantially. This corresponds to Finding 5 in the paper.

Deck-greedy. The purpose of this experiment is to demonstrate, at the mechanistic level, the necessity of active management for long-term resource optimization, rather than to claim that current agents generally lack long-horizon planning capabilities. This corresponds to Finding 6 in the paper. The deck-free win rate uses all started and completed trajectories as its denominator.

We hope our responses have adequately addressed your concerns and clarified the contributions of our work. We look forward to your further feedback and discussion.

Thank you for the detailed responses. The table clears up the discrepancy for GPT-5.6-Sol at Boss 3. I also appreciate the clarification that the reported standard deviation pools results across all three bosses.

My main reservation remains Finding 5. A baseline run can end before the agent reaches the boss. In the deck-forced condition, the agent starts at the boss with resources from a winning trajectory. The shared starting state is useful for comparing combat performance, but the gain over baseline leaves open how much of the difference comes specifically from long-term resource management. That attribution was central to my original assessment, so my rating remains.

Dear Reviewer PhAa,

Thank you for your follow-up and for acknowledging our previous clarifications. To further address your concern regarding the attribution of performance improvements to resource management, we conducted two additional analyses that provide complementary evidence for Finding 5. The results show that (1) deck-forced retains its advantage under a consistent conditional Boss win-rate definition, and (2) following successful resource management decisions improves average game progression by 7.3 floors, while preserving the full gameplay process.

1. Recomputed Conditional Boss Win Rates

Deck-forced retains its advantage even after excluding early deaths from the baseline Boss win-rate denominator. Specifically, deck-forced achieves higher win rates in 14 of 18 model–Boss combinations, with one tie.

We recalculated the baseline win rates using only trajectories that actually reached the corresponding Boss, making the comparison conditional on reaching the Boss encounter.

ModelBoss 1 BaselineBoss 1 Deck-forcedΔ (pp)Boss 2 BaselineBoss 2 Deck-forcedΔ (pp)Boss 3 BaselineBoss 3 Deck-forcedΔ (pp)
GPT-5.6-Sol89% (8/9)100% (15/15)+1171% (5/7)87% (13/15)+1560% (3/5)73% (11/15)+13
Claude-Fable-589% (8/9)100% (15/15)+1186% (6/7)100% (15/15)+1450% (3/6)67% (10/15)+17
Claude-Opus-4-880% (8/10)93% (14/15)+1325% (1/4)87% (13/15)+62100% (1/1)93% (14/15)−7
DeepSeek-V4-Pro57% (4/7)100% (15/15)+43100% (2/2)100% (15/15)050% (1/2)87% (13/15)+37
Kimi-K386% (6/7)100% (15/15)+14100% (5/5)93% (14/15)−720% (1/5)60% (9/15)+40
GLM-5.333% (3/9)100% (15/15)+6750% (1/2)73% (11/15)+230%67% (10/15)+67

These results reinforce our original observation that agents can achieve substantially better Boss-fight outcomes when provided with high-quality pre-Boss resource states, even after accounting for differences in early survival.

2. Additional Deck-Guided Ablation

Better resource management decisions enable weaker agents to progress further in the full game. To further validate this finding, we conducted an additional deck-guided ablation.

Unlike deck-forced, deck-guided starts from Floor 1. Agents independently execute all battles, while non-combat decisions (path selection, card rewards, shops, campfires, and events) follow those from a successful reference trajectory. Thus, early combat encounters and survival risks are preserved.

We evaluate four weaker models on model–character combinations with baseline average progression below 40 floors.

ModelSilent BaselineSilent Deck-guidedΔDefect BaselineDefect Deck-guidedΔIronclad BaselineIronclad Deck-guidedΔOverall BaselineOverall Deck-guidedΔ
DeepSeek-V4-Pro17.0 (n=2)48.0 (n=5)+31.037.7 (n=3)38.8 (n=5)+1.127.5 (n=2)37.0 (n=5)+9.528.9 (n=7)41.3 (n=15)+12.4
Kimi-K337.7 (n=3)45.0 (n=5)+7.348.0 (n=3)— (≥40)—16.0 (n=3)19.8 (n=5)+3.826.8 (n=6)32.4 (n=10)+5.6
Claude-Opus-4-821.0 (n=3)35.6 (n=5)+14.629.3 (n=3)25.0 (n=5)−4.330.5 (n=4)18.4 (n=5)−12.127.3 (n=10)26.3 (n=15)−1.0
GLM-5.317.0 (n=3)33.0 (n=5)+16.017.0 (n=3)34.0 (n=5)+17.033.3 (n=3)33.6 (n=5)+0.322.4 (n=9)33.5 (n=15)+11.1
Overall—————————26.2 (n=32)33.5 (n=55)+7.3

Across the evaluated runs, average progression increases from 26.2 to 33.5 floors (+7.3) under deck-guided, despite agents independently executing all combat encounters.

Together, these experiments provide complementary evidence for Finding 5: Weak Models Win with Strong Builds. Deck-forced shows that weaker agents can defeat the final Boss when provided with strong pre-Boss resource states, while deck-guided shows that following successful build decisions can improve their progression throughout the full game.

We hope this additional evidence addresses your remaining concern and would appreciate reconsideration of the original rating.

Reviewer_5FndOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongGameBench, a benchmark for evaluating long-horizon resource management in LLM agents through three roguelike games. The authors evaluate six LLMs with four coding-agent implementations over more than 1,000 trajectories, using metrics that capture game progression, execution, and resource management. They also include ablations with strong predefined builds and restricted deck refinement to better isolate where agents fail. The results show that current agents can make reasonable short-term progress, but often struggle to maintain and improve effective resource configurations over long trajectories.

Critical strengths
  1. The paper evaluates fully autonomous decision-making in challenging long-horizon games, including both combat and non-combat actions. Unlike prior benchmarks that often rely on rule-based policies for non-combat decisions, it allows agents to manage the entire gameplay process.
  2. The paper provides a comprehensive evaluation across multiple LLMs and agent frameworks. The extensive experiments, including ablation studies and behavioral analyses, offer useful insights into agents’ long-horizon decision-making capabilities and suggest potential directions for improving existing agent harnesses on these tasks.
Critical weaknesses
  1. The novelty is somewhat limited. Prior work such as Orak [1] already studies long-horizon decision-making in games that overlap with those used in this benchmark. While LongGameBench provides a more automated interface and relies less on hand-designed heuristics, I do not think these differences are substantial enough. The paper would benefit from a more detailed discussion of how LongGameBench differs from Orak, together with direct experimental comparisons under comparable settings to demonstrate whether the proposed benchmark leads to meaningfully different findings or exposes additional failure modes.
  2. The benchmark is relatively small, covering only three games, which raises concerns about how well the conclusions generalize beyond these specific environments. More importantly, the paper does not provide a clear methodology for scaling LongGameBench to a broader set of games. It would strengthen the work to discuss which parts of the benchmark construction are game-specific, whether the pipeline can be systematically extended to new environments, and whether the proposed evaluation metrics can be defined in a game-agnostic way rather than relying on manually designed notions of progression or resource management for each game.

[1] Park, Dongmin, et al. "Orak: A foundational benchmark for training and evaluating llm agents on diverse video games." International Conference on Learning Representations. Vol. 2026. 2026

Questions for authors
  1. How does this paper differ from Orak beyond the more automated interface and reduced reliance on hand-designed heuristics? Could the authors provide a direct comparison under comparable settings and clarify whether this paper reveals failure modes or conclusions that Orak does not?
  2. How scalable is the benchmark construction pipeline beyond the three selected games? Which components are game-specific, and can the proposed evaluation metrics be defined in a more game-agnostic way so that this paper can be extended to substantially more environments without significant manual redesign?
3 comments
AuthorsOpenReview ↗

Novelty and Comparison with Orak (1/3)

Dear reviewer 5Fnd:

Thank you for recognizing the insights and contributions from our systematic evaluation work. We have carefully prepared the following responses to address your concerns.

W1 & Q1: The novelty is somewhat limited.

Thank you for raising this concern. Overall, Orak aims to evaluate the diverse gaming capabilities of LLMs, whereas LongGameBench focuses on the specific capability of long-horizon resource optimization, studying how agents autonomously construct, manage, and optimize resource systems to sustain long-term task performance. To address your concern, we compare the two works from three perspectives: research objectives, evaluation settings, and research findings, and will further clarify these distinctions in the revised version.

  • Research Objectives. Orak aims to evaluate the diverse gaming capabilities of LLMs across seven dimensions: rule following, logical reasoning, spatial reasoning, long-text understanding, long-term planning, error handling, and odds handling. These seven dimensions are not evaluated independently. Instead, the paper ultimately uses game progression as an aggregate metric to measure overall model performance. In contrast, we focus on a more specific capability that has not yet been sufficiently studied in isolation: Long-Horizon Resource Optimization, the ability to construct effective resource systems and sustain long-term task performance. Furthermore, the dimension most closely related to our work in Orak is Long-term Planning, which is defined as "The extent to which strategic planning is required" and categorized according to whether more than three consecutive actions need to be planned. This dimension primarily demonstrates control over the LLM's planning process by introducing a dedicated Planning module into a custom harness, rather than independently measuring the LLM's ability to autonomously perform long-horizon resource planning and management under native interaction conditions. Therefore, although both works involve long-horizon decision-making, they differ substantially in their overall research objectives, the capabilities they focus on, and how these capabilities are evaluated.

  • Evaluation Settings. The two benchmarks also differ substantially in their evaluation settings, as summarized below:

ComparisonOrakLongGameBench
Game Version12
Action SpaceThe LLM is only responsible for combat decisions and card reward selection; map route selection, shop interactions, potion usage, and other decisions are handled automatically by rule-based scripts.All in-game decision points are delegated to the agent for autonomous decision-making.
Characters1 character5 characters
Engineering InterfaceUses a Mod for communication between the game and the external evaluation framework.Supports headless execution and parallel runs, facilitating large-scale evaluation.
  • Evaluation Metrics and Research Findings. Orak ultimately measures model performance using game progression. Its research primarily focuses on revealing differences in the overall gaming capabilities of LLMs across different games, and investigating the effects of factors such as Reflection, Planning, input modality, and Fine-tuning on game performance. In contrast, LongGameBench establishes a more fine-grained evaluation framework to analyze agents' long-horizon resource optimization capabilities from different perspectives: Overall Progress measures overall task progression, Execution measures the agent's ability to use its current resources to complete local challenges, and Active Management measures the frequency and diversity of the agent's proactive resource adjustments. Through our experiments, we reveal an issue that Orak does not independently investigate: the performance bottleneck of LLM agents in long-horizon resource optimization lies in their ability to continuously construct and refine resource systems, rather than in their ability to utilize existing resources.
AuthorsOpenReview ↗

Benchmark Scale and Extensibility (2/3)

W2.1 & Q2.1: The benchmark covers a limited number of games, and its construction and scalability are insufficiently explained.

Thank you for raising concerns about the scale of LongGameBench and the extensibility of its game construction pipeline. We clarify both the rationale behind our game selection and how the benchmark can be extended to new environments.

  • Regarding the benchmark scale, our goal is not to maximize the number of games covered, but rather to carefully select environments that can effectively capture Long-Horizon Resource Optimization. Specifically, a suitable environment should satisfy the following criteria: 11 it contains a resource system that agents must construct, continuously optimize, and maintain; 22 early decisions have significant long-term consequences beyond immediate gains; 33 success cannot be achieved through resource accumulation alone, but requires agents to proactively acquire, remove, upgrade, and reorganize resources, sometimes sacrificing short-term benefits; 44 task difficulty increases progressively; and 55 the environment is not dominated by other factors such as real-time execution and supports reproducible evaluation with fixed seeds. The three games included in our benchmark were carefully selected from nearly twenty similar candidate games based on these criteria. Furthermore, prior work has also shown that benchmark effectiveness depends on how well the environment design matches the capabilities being evaluated. For example, Crafter uses a single open-world survival environment, yet evaluates agents' exploration, resource acquisition, and long-horizon reasoning capabilities through its rich game mechanics and achievement system.

  • Regarding the extension pipeline, LongGameBench can be extended to additional games, provided that the new environments satisfy all the criteria described above. The extension process mainly consists of two parts:

  1. Environment Integration: This involves implementing the communication protocols and interfaces between the game and the external evaluation framework, designing interaction and control mechanisms, and supporting experiment management and trajectory recording.

  2. Game-specific Adaptation: This involves adapting the state and resource representations, action space, and evaluation metric computation to the mechanics of each game. These components require game-specific customization.

Therefore, the selection of the three games in LongGameBench reflects a deliberate design based on explicit criteria for evaluating a specific capability, rather than a limitation imposed simply by the number of available games. We will further clarify the game selection criteria and the extensibility of the benchmark in the revised version.

AuthorsOpenReview ↗

Evaluation Metrics (3/3)

W2.2 & Q2.2: Generalizability and extensibility of the evaluation metrics.

Thank you for this valuable suggestion. We agree that evaluation metrics should be as generalizable across games as possible. In fact, LongGameBench already abstracts a unified evaluation framework at three levels: Overall Progress, Execution, and Active Management. Although different games instantiate these dimensions using their respective native metrics, the evaluation objectives, dimension definitions, and computation procedures remain consistent across environments. The specific definitions are summarized below.

Primary DimensionSecondary DimensionWhat It MeasuresComputation
Overall Progress—How far the agent advances throughout the gameUses game-native progression metrics: cumulative Floor in STS2, Ante in Balatro, and Rent in LBaL.
ExecutionEfficiencyHow effectively the current resource system converts available resources into useful outcomesE=1N∑i=1NGiRi\displaystyle E=\frac{1}{N}\sum_{i=1}^{N}\frac{G_i}{R_i}. NN: number of local tasks; GiG_i: total effective output obtained in the ii-th task; RiR_i: amount of resources consumed in the ii-th task.
ExecutionTask CostThe action cost required to complete local tasksC=1N∑i=1NAi\displaystyle C=\frac{1}{N}\sum_{i=1}^{N}A_i. NN: number of local tasks; AiA_i: number of actions required to complete the ii-th task.
ExecutionBurstThe peak output capacity of the current resource systemB=1N∑i=1N(13∑j=13Pi,(j))\displaystyle B=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{3}\sum_{j=1}^{3}P_{i,(j)}\right). NN: number of local tasks; Pi,(j)P_{i,(j)}: the jj-th highest single-action output in the ii-th task, with outputs sorted in descending order.
Active ManagementIntervention FrequencyHow frequently the agent proactively adjusts its resource configurationIfreq=NinterventionNavailable decision\displaystyle I_{\mathrm{freq}}=\frac{N_{\mathrm{intervention}}}{N_{\text{available decision}}}. NinterventionN_{\mathrm{intervention}}: number of resource decision points at which the agent performs an intervention; Navailable decisionN_{\text{available decision}}: total number of available resource decision points.
Active ManagementManagement DiversityThe diversity of resource management mechanisms used by the agentIcoverage=Nused management typesNavailable management types\displaystyle I_{\mathrm{coverage}}=\frac{N_{\text{used management types}}}{N_{\text{available management types}}}. Nused management typesN_{\text{used management types}}: number of distinct management action types used in a trajectory; Navailable management typesN_{\text{available management types}}: total number of available management action types (6 in STS2, 8 in Balatro, and 4 in LBaL).

The following table summarizes the game-specific definitions of the variables used in the evaluation metrics above:

VariableSTS2BalatroLBaL
GiG_iTotal damage dealt in a combatTotal score obtained in a blindTotal gold obtained in a rent stage
RiR_iTotal energy spent in a combatNumber of hands played in a blindNumber of spins performed in a rent stage
AiA_iNumber of combat turnsNumber of hands played per blindNumber of spins per rent payment
Pi,(j)P_{i,(j)}jj-th highest single-turn damagejj-th highest single-hand scorejj-th highest single-spin reward

For Execution, the three sub-metrics are first computed as trajectory-level raw scores for each individual trajectory. We then average these raw scores across all complete trajectories for each model. Each sub-metric is subsequently normalized by dividing by its 95th percentile P95P95, calculated over all trajectories within the corresponding game environment; Task Cost is inversely normalized. The final Execution score is the equally weighted average of the three normalized sub-metrics.

For Active Management, the two sub-metrics are first computed for each individual trajectory and combined into a trajectory-level score using: ABM = 07˙∗ifreq+03˙∗icoverage0\.7*ifreq + 0\.3*icoverage×100. We then average the trajectory-level scores across all complete trajectories for each model. Finally, the model-level score is normalized by the P95 of trajectory-level ABM scores across all trajectories within the corresponding game environment, yielding the Manage score.

We hope our responses have adequately addressed your concerns and clarified the contributions of our work. We look forward to your further feedback and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.