acceptodds
Under review as a conference paper at ICLR 2027

SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning

Abstract

Frontier LLM agents have shown to be capable of solving increasingly complex tasks for which humans have measurable solutions. This begs the pertinent question of whether LLM-based agents can go beyond what humans have already solved. The ability to develop sophisticated strategies to tackle consequential problems becomes paramount as well trodden human developed solutions become insufficient for problems for which we lack context or enough training data. We study agents' capability of such strategy formation through the communal practice of video game speedrunning. In speedrunning, practitioners compete to find the fastest way to complete a video game under certain conditions, and in so doing uncovering interesting unorthodox play styles that require a through understanding and mastery of the underlying game mechanics. We introduce **SPEEDRUNBENCH**, a benchmark that evaluates frontier LLM agents across 9 different games. To perform well in this benchmark, agents must repeatedly improve their strategy, reflect on their performance, exploit their gained knowledge, and reason across a long-horizon of actions to improve on an increasingly difficult problem: being faster than themselves and everyone else. Our experiments show that while frontier agents approach human world records in simple platformer games, they remain behind human performance on longer more complex games under practical budgets. These results suggest that SPEEDRUNBENCH is a useful testbed for studying agents' strategy formation capabilities as well as being a saturation-resistant evaluation measure, as there is almost always a faster completion time waiting to be discovered.

Primary area:
datasets and benchmarks
Keywords:
speedrun, video game, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pE61OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SpeedrunBench, a benchmark for evaluating LLM agents through video-game speedrunning rather than simple task completion. The motivation is that completion-based game benchmarks may saturate as models improve, while speedrunning provides a continuously optimizable objective that can continue distinguishing strong agents. The benchmark includes nine games across platforming, racing, RPG, strategy, and puzzle genres, and evaluates three settings: Online, Offline-Scratch, and Offline-Seed. Results show that frontier agents can approach human records on short platformers but remain substantially slower on longer and more complex games. One interesting finding is that agents often do better starting from scratch than improving an existing but suboptimal solution. However, on short platformers, giving agents many more scorer evaluations can largely close this gap.

Critical strengths
  1. Continued optimization after successful completion is a compelling benchmark design choice. Unlike binary success metrics, speedrunning provides a naturally graded objective that can continue differentiating strong agents even after they all solve the task.

  2. The seed-vs-scratch result reveals an interesting failure mode of current agents. A valid but suboptimal seed can anchor the agent so strongly that starting from scratch produces much better trajectories, highlighting difficulty in escaping an existing solution basin and performing global revision.

Critical weaknesses
  1. The Offline setting is essentially black-box optimization over a long discrete action sequence, yet the paper lacks non-LLM search baselines. This makes it difficult to determine whether improvements come from LLM-specific strategy formation or generic search under a large evaluation budget. Simple baselines such as random mutation with elitism, hill climbing, evolutionary search, beam search, or MCTS where applicable would help isolate the value of LLM-based proposal generation.

  2. The paper’s claims about “strategy formation” and especially “discovery” are stronger than the evidence presented. Most experiments show that agents can reduce frame count under repeated feedback, but this does not by itself demonstrate qualitatively new strategy discovery. The paper provides little analysis of whether agents change routes, invent techniques absent from the seed, or discover new mechanics/exploits. As presented, the evidence more directly supports iterative trajectory optimization.

  3. The experimental protocol is not specified sufficiently for reproducibility, and uncertainty is largely absent. Important details of the environment-agent interface remain unclear, including what feedback is returned after each trace, how progress is represented, whether intermediate states are available, and how traces are encoded and edited. In addition, results are mostly reported as point estimates without repeated-run statistics, despite stochastic LLM behavior. Fig. 8 further complicates interpretation by selecting the better of offline-scratch and Offline-seed.

Questions for authors
  1. How are human RTA records converted into frame counts across different platforms, emulators, frame rates, loading behavior, and timing conventions?

  2. Sec. 5.1 appears numerically inconsistent. The agent can evaluate up to 200 traces per turn, while Fig. 6b spans roughly 100 turns, suggesting at most ~20K evaluations, yet the text reports 220K evaluated traces over ~114 hours. What exactly counts as an “evaluated trace,” and how is 220K obtained?

  3. Please clarify the full experimental protocol: agent harness and prompts, available tools, persistent context/memory across turns, precise inputs/outputs in each setting, model/API versions, decoding parameters, and number of independent runs/seeds.

2 comments

We thank the reviewer for the thoughtful review.

W1: "…the paper lacks non-LLM search baselines. This makes it difficult to determine whether improvements come from LLM-specific strategy formation or generic search under a large evaluation budget."

SpeedrunBench is designed to compare LLM agents with one another and to measure how far each remains from the human world record under a fixed interaction budget. It does not claim that LLM agents outperform dedicated search algorithms. A non-LLM baseline would address a different question, how well a game can be optimized by any means. We will state this scope explicitly. The BFS solver behind the Astray seed (Fig. 9) remains as a non-LLM reference point.

W2: "The paper’s claims about “strategy formation” and especially “discovery” are stronger than the evidence presented."

We agree that “discovery” should be stated precisely, and will clarify that it refers to agents finding faster routes from sparse feedback. The evidence for strategy formation lies in how agents reach the goal. Without any reference trajectory, scratch agents construct death-free routes that require 3.4–4.6 times fewer frames than the seeded results, and they do so with distinct strategies, such as placing one jump per obstacle or jumping at a regular rhythm. Seeded agents instead retain the seed's route and its deaths (7–11, against 10 in the seed), which shows that a working but slow route constrains the strategies agents form.

W3/Q3: "Important details of the environment-agent interface remain unclear…" / "…results are mostly reported as point estimates without repeated-run statistics, despite stochastic LLM behavior."

After each turn, the agent receives whether the goal was reached, the frame count, the furthest progress, the number of deaths and a hash of the final state. It retains its session across turns, and only the scorer is withheld. OFFLINE configurations request temperature 0.3. Where an API does not accept the parameter, as for Claude Opus and GPT-5.6 Sol, the provider's default applies. The Reproducibility Statement will list the exact settings per model. We repeated the Table 2 SuperTux comparison with two independent runs per model.

modelOFFLINE-SCRATCH (run 1 / 2)OFFLINE-SEED (run 1 / 2)
Gemini 3.7 Flash2,180 / 2,1309,454 / 9,205
GLM 5.31,961 / 1,9748,661 / 9,027
DeepSeek-V4-Flash2,725 / no clear9,203 / 9,278
Kimi K3no clear / no clear9,254 / 8,306
DeepSeek-V4-Prono clear / no clear9,227 / 9,332

Q1: "How are human RTA records converted into frame counts across different platforms, emulators, frame rates, loading behavior, and timing conventions?"

Each record is converted at the game's frame rate over the segment the task scores: 60 frames per second for SuperTux 1-1 (26.98 s) and Super Mario Land 1-1 (37.6 s, from power-on), the flagpole touch for Super Mario Bros. 1-1 (18.59 s), the single-lap record for Mario Kart 64 (10.87 s) with the menu prefix removed, and 59.73 frames per second for Pokémon (11:17 to the first badge). SuperTuxKart is compared in seconds.

Q2: "What exactly counts as an “evaluated trace,” and how is 220K obtained?"

We clarify that the description of the experimental setup for the Super Mario Land run shown in Figure 6b was not subject to the per-turn limit of 200 evaluations. Each evaluated trace corresponds to a single candidate trajectory assessed by the scorer. We will clarify both points in the revised manuscript.

Thank you for the clarifications and additional experiments. I appreciate the authors' efforts.

However, my main concern remains unresolved. It is still unclear to me whether the performance improvements come from LLM reasoning or primarily from test-time search with a large number of scorer evaluations.

I therefore maintain my original rating.

Reviewer_k19tOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SPEEDRUNBENCH, an evaluation benchmark designed to assess the capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs) in video game speedrunning. The core premise is that moving beyond binary or milestone-based game completion to fine-grained temporal optimization (measured in environment frames) establishes a continuous, saturation-resistant testbed for evaluating long-horizon reasoning, strategic planning, and iterative optimization.

The benchmark spans nine games across six genres (platformers, RPGs, racing, puzzle, and strategy), deliberately pairing popular commercial titles (e.g., Super Mario Bros, Pokémon, Mario Kart 64) with open-source equivalents (e.g., SuperTux, Tuxemon, SuperTuxKart) to investigate pretraining memorization effects. The evaluation framework comprises three operating regimes:

  1. ONLINE: An interactive vision-language agent setting where models observe screenshots frame-by-frame (or in short action chunks) to predict subsequent controller inputs.
  2. OFFLINE-SCRATCH: A text-only trajectory optimization setting where models attempt to synthesize a complete frame-indexed action sequence from start to goal without an initial demonstration.
  3. OFFLINE-SEED: A text-only trajectory editing setting where models iteratively propose edits to a suboptimal baseline trace within an auto-research feedback loop based on execution feedback.

The authors evaluate ten frontier foundation models (four proprietary: Claude Opus 5, GPT-5.6-sol, Grok 4.6, Gemini 3.7 Flash; six open-weight: Kimi-K3, GLM 5.3, GLM 5.2, DeepSeek-V4-Pro, DeepSeek-V4-Flash, Inkling-Small). Their primary empirical claims are:

  • Speedrunning remains highly challenging for current frontier agents, which remain significantly behind human baselines (averaging 316%–432% of human world records).
  • In the seeded offline setting, models exhibit an "anchoring" effect, struggling to escape local optima around suboptimal demonstrations, whereas models clearing from scratch find routes 3.8×–4.9× faster.
  • Expanding per-turn evaluation budgets (allowing an internal scorer up to 200 rollouts per turn) substantially closes the gap to human records on short 2D levels (e.g., DeepSeek-V4-Pro reaching within 7% of the record on SuperTux 1-1).
Critical strengths
  1. Compelling Framing Around Saturation-Resistant Optimization: Reframing game benchmarking from binary task completion toward continuous temporal optimization (minimizing frame counts) addresses a recognized vulnerability in modern LLM evaluations: the rapid saturation of static pass/fail benchmarks. By demanding fine-grained sequential decision-making over long horizons, speedrunning provides a conceptually durable dimension for evaluating agent optimization capabilities.
  2. Paired Benchmark Design Probing Pretraining Memorization: Systematically pairing widely documented commercial titles (e.g., Pokémon, Super Mario Bros) with structurally matched open-source counterparts (e.g., Tuxemon, SuperTux) is a well-conceived methodology for investigating whether agent performance is driven by memorized walkthrough data or generalized strategic reasoning.
Critical weaknesses
  1. Methodological Baseline Invalidation (TAS vs. RTA) and Lack of Non-LLM Search Baselines:

    • Apples-to-Oranges Human Comparison: In the OFFLINE settings (particularly in Section 5.1, where models utilize an in-turn scorer executing thousands of rollbacks and candidate evaluations over 100+ hours), the agent is operating as a Tool-Assisted Speedrun (TAS) optimizer. However, the authors anchor their comparative evaluation against human Real-Time Attack (RTA) world records. Human RTA speedrunners play continuously in real time with human reaction constraints and zero rollbacks. Benchmarking an offline iterative search process against human RTA records rather than verified human TAS frame records (e.g., from TASVideos) invalidates claims regarding how close models are to human performance.
    • Unproven LLM Agency (Missing Non-Neural Baselines): The paper attributes trajectory refinements to LLM "reasoning," "exploration," and "discovery." However, the authors provide no comparison against standard trajectory optimization or stochastic search baselines (e.g., random frame mutation, genetic algorithms, Cross-Entropy Method, or MCTS) under the same rollout budget. In a deterministic emulator, simple random frame perturbations can often eliminate idle frames or discover minor optimizations. Without non-neural baselines, there is no evidence that LLMs are engaging in strategic reasoning rather than acting as expensive stochastic mutators. (Indeed, the anonymous repository includes a MutationEditor that was explicitly left unbenchmarked).
  2. Severe Reproducibility Deficits, Incomplete Artifacts, and Internal Inconsistencies:

    • Incomplete Release and Missing Reproducibility Statement: The manuscript contains no formal Reproducibility Statement. Furthermore, while an anonymous repository is provided for Tuxemon, the harnesses, environment wrappers, prompt templates, and seed files for the remaining eight games or non-Claude agents are completely omitted. Crucial hyperparameters (temperatures, top_p, exact system prompts, output schemas, API version hashes) are absent, rendering independent replication impossible.
    • Absence of Limitations: The submission contains neither a dedicated Limitations section nor any discussion of limitations, computational expenditure, token consumption, or failure modes, directly omitting a mandatory conference component.
    • Pervasive Contradictions and Reporting Errors:
      • Arithmetic Error in Section 5.1: The text claims that on Super Mario Land 1-1, GLM 5.2 reached near-record performance after "220K evaluated traces" across an experiment defined as 100 turns (Figure 6b) or 200 turns with "up to 200 traces per turn." A budget of 100–200 turns at 200 evaluations/turn mathematically bounds the rollouts to 20,000–40,000 traces; the origin of the 220K figure is unexplained and mathematically contradictory.
      • Contradictory Definition of OFFLINE-SCRATCH: Section 3.1 states that Exploration is tested "especially in an OFFLINE-SCRATCH setup where given seed gameplay trajectories are not necessarily fast gameplays," directly contradicting Section 3.2, which defines OFFLINE-SCRATCH as synthesizing trajectories "from scratch i.e., without a reference trace."
      • Mismatch in Table 2: Section 4.4 states that "All seven OFFLINE-SEED runs end within 12% of the seed, between 8,412 and 9,428 frames," whereas Table 2 lists 9 runs total (5 for SuperTux, 4 for Super Mario Land), and the 8,412–9,428 frame range applies exclusively to SuperTux.
      • Contradictory Contamination Evidence (§5.3): The authors claim open-source Tuxemon is an unexposed control, yet models improved more on Tuxemon (5.9%–6.9%) than on Pokémon (<1%), and the rank correlation was statistically insignificant (−0.18-0.18). This empirical result contradicts the hypothesis that pretraining contamination makes commercial games easier to optimize.
Questions for authors
  1. TAS vs. RTA Baseline Selection: Why were human Real-Time Attack (RTA) records chosen as the comparative baseline for the OFFLINE optimization setup rather than verified human Tool-Assisted Speedrun (TAS) records (e.g., from TASVideos.org)? For Super Mario Bros and Super Mario Land, where exact TAS frame counts are well-documented, how do the agent runs compare directly against these true tool-assisted limits?
  2. Non-Neural Search Baselines: Have the authors evaluated a non-LLM baseline—such as random frame mutation, genetic search, or simulated annealing (e.g., using the MutationEditor present in your Tuxemon codebase)—under the identical evaluation budget? Can the authors provide empirical evidence that the LLM's edits outperform simple stochastic local search?
  3. Clarification of Experimental Arithmetic in Section 5.1: Could the authors clarify the rollout count on page 8? The text states that GLM 5.2 evaluated "220K evaluated traces" on Super Mario Land 1-1, but the setup specifies 100 turns (Figure 6b) with up to 200 evaluations per turn (which totals at most 20,000 evaluations). How was the 220K number accumulated, and what were the exact computational and financial costs of this run?
  4. Resolution of Conceptual Contradiction in OFFLINE-SCRATCH: Does OFFLINE-SCRATCH provide a seed trajectory or not? Please reconcile the statement in Section 3.1 ("where given seed gameplay trajectories are not necessarily fast") with the formal definition in Section 3.2 ("without a reference trace").
  5. Release Plan for the Remaining Eight Environments: Will the execution harnesses, prompt templates, controller schemas, and seed files for the other eight games (Super Mario Bros, Super Mario Land, Mario Kart 64, Pokémon, SuperTux, SuperTuxKart, Astray, Civilization I) be released publicly in the supplementary material or upon publication to ensure benchmark reproducibility?
2 comments

We thank the reviewer for the careful reading of the paper.

W1/Q1: "…Benchmarking an offline iterative search process against human RTA records rather than verified human TAS frame records (e.g., from TASVideos) invalidates claims regarding how close models are to human performance."

Our objective is to measure how far LLM-based agents remain from the human world record, whether that record was set as a real-time attack (RTA) or with tool assistance (TAS). For simplicity and consistency across games, we use the RTA records published on speedrun.com (https://www.speedrun.com/), except in Section 5.2, whose near-optimal seeds are TAS trajectories from TASVideos.org. We do not claim that the agents' setting matches human play.

W1/Q2: "Without non-neural baselines, there is no evidence that LLMs are engaging in strategic reasoning rather than acting as expensive stochastic mutators."

SpeedrunBench compares LLM agents with one another and with the human world record, and does not claim that LLM edits outperform stochastic search. We will revise wording that attributes gains to "reasoning" or "discovery". A non-LLM baseline would address a different question, how well a game can be optimized by any means. We will state this scope explicitly. The BFS solver behind the Astray seed (Fig. 9) remains as a non-LLM reference point.

W2/Q5: "Crucial hyperparameters (temperatures, top_p, exact system prompts, output schemas, API version hashes) are absent…"

Code, prompts and seeds for the open source games (SuperTux, Tuxemon and SuperTuxKart) are available at https://anonymous.4open.science/r/speedrunbench-FD02/. OFFLINE configurations request temperature 0.3. Where an API does not accept the parameter, as for Claude Opus and GPT-5.6 Sol, the provider's default applies.

W2/Q3: "…the origin of the 220K figure is unexplained and mathematically contradictory."

We clarify that the description of the experimental setup for the Super Mario Land run shown in Figure 6b was not subject to the per-turn limit of 200 evaluations. Each evaluated trace corresponds to a single candidate trajectory assessed by the scorer. We will clarify both points in the revised manuscript.

W2/Q4: "Does OFFLINE-SCRATCH provide a seed trajectory or not?"

We thank the reviewer for pointing this out. OFFLINE-SCRATCH involves no seed, and we will correct Section 3.1.

W2: "…Table 2 lists 9 runs total (5 for SuperTux, 4 for Super Mario Land), and the 8,412–9,428 frame range applies exclusively to SuperTux."

The text should read nine, and the 8,412–9,428 range applies to SuperTux only. We also repeated the Table 2 SuperTux comparison with two independent runs per model.

modelOFFLINE-SCRATCH (run 1 / 2)OFFLINE-SEED (run 1 / 2)
Gemini 3.7 Flash2,180 / 2,1309,454 / 9,205
GLM 5.31,961 / 1,9748,661 / 9,027
DeepSeek-V4-Flash2,725 / no clear9,203 / 9,278
Kimi K3no clear / no clear9,254 / 8,306
DeepSeek-V4-Prono clear / no clear9,227 / 9,332

W2: "This empirical result contradicts the hypothesis that pretraining contamination makes commercial games easier to optimize."

We agree and revise this claim. The paired comparison shows no advantage on the familiar game: the largest improvement is 6.9% on Tuxemon (GLM 5.3), whereas no model improves on Pokémon by more than 1%, and per-model results are uncorrelated (ρ = −0.18, n = 10, not significant).

Thank you for the response, I will not change my score.

Reviewer_fqoyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SpeedrunBench, a benchmark designed to evaluate LLM agents based on how quickly they can complete video games rather than simply whether they can finish them. The authors argue that traditional game-based benchmarks mainly focus on task completion, which may not adequately measure an agent’s ability to develop new strategies, explore alternative solutions, and continuously improve its performance. They advocate speedrunning as a more saturation-resistant evaluation approach because even after an agent successfully completes a game, there is still room to improve its completion time. The benchmark includes nine games across different genres, including platformers, racing games, RPGs, puzzles, and strategy games. It introduces three evaluation settings: Online, where agents interact with games through screenshots and actions; Offline-Scratch, where agents construct complete action sequences from scratch; and Offline-Seed, where agents iteratively modify an existing successful trajectory to improve its completion time. The authors evaluate more than ten frontier LLM agents, including Claude Opus 5, GPT-5.6-sol, Grok 4.6, DeepSeek V4, Gemini 3.7 Flash, GLM, and Kimi K3. I think the main contribution is the benchmark design, particularly its emphasis on evaluating strategic exploration, iterative optimization, and long-horizon reasoning beyond basic task completion. For validation, the authors compare agent performance against human speedrunning world records across the nine games. The results show that even the strongest agent, Claude Opus 5, averages approximately 316% of the human world-record completion time, while the evaluated models range from 316% to 432%. No agent reaches a human world record under the standard evaluation budget, and performance remains particularly limited on longer and more complex games. Interestingly, the authors find that providing an existing successful trajectory does not necessarily lead to better results. For example, agents improve the seeded SuperTux trajectory by less than 12% and achieve less than 1% improvement on Pokémon without additional evaluation resources. However, when agents are allowed to evaluate up to 200 candidate trajectories per iteration, DeepSeek-V4-Pro comes within 7% of the SuperTux world record, while GLM 5.2 comes within 3% of the Super Mario Land record. These results suggest that access to additional evaluation feedback and computational resources can substantially influence optimization performance. The authors also compare Pokémon with the less familiar Tuxemon and report a slightly negative rank correlation of −0.18 across ten models, suggesting that performance on popular games does not necessarily transfer to less familiar environments. Taken together, the paper supports the position that speedrunning provides a challenging evaluation setting for studying LLM agents’ ability to discover strategies, improve existing solutions, and maintain long-horizon coherence, while highlighting the substantial gap between current agent capabilities and human-level speedrunning performance.

Critical strengths

First, I think one of the strongest aspects of this paper is its benchmark design and the clear motivation for evaluating LLM agents beyond simple task completion. I agree with the authors that successfully finishing a game does not necessarily mean an agent has developed strong strategic reasoning or optimization capabilities. Speedrunning provides an interesting alternative because agents must continuously improve their solutions rather than simply reach a predefined goal. I particularly appreciate the three evaluation settings, Online, Offline-Scratch, and Offline-Seed, which allow the authors to examine different capabilities, including real-time decision-making, independent strategy discovery, and iterative trajectory optimization. The inclusion of nine games across different genres also makes the benchmark more interesting than evaluations focused on a single game or task type. I believe this is a meaningful contribution because it introduces a useful evaluation perspective that could help researchers better understand the limitations of current LLM agents, particularly in long-horizon planning and self-improvement.

Second, I appreciate the extensive experimental evaluation and the interesting findings about how different evaluation settings and computational budgets affect agent performance. The authors evaluate more than ten frontier LLM agents and compare their performance against human speedrunning world records, which provides a clear and interpretable reference point. One particularly interesting observation is that providing agents with a successful reference trajectory does not always improve their performance, while allowing additional candidate evaluations can lead to substantial improvements. For example, DeepSeek-V4-Pro approaches within 7% of the SuperTux world record, and GLM 5.2 comes within 3% of the Super Mario Land record when additional scoring opportunities are available. I also appreciate the comparison between Pokémon and Tuxemon, which provides useful evidence that performance on familiar games may not generalize to less familiar environments. These experiments support the authors’ argument that speedrunning can reveal meaningful differences in agents’ strategic capabilities that traditional completion-based benchmarks might overlook. I consider this experimental evidence an important strength because it demonstrates both the practical value of the benchmark and its potential to guide future research on agent reasoning and optimization.

Critical weaknesses

First, my main concern is whether the experimental results are statistically reliable enough to support the paper’s conclusions about agent performance and strategic reasoning. Although the authors evaluate more than ten frontier LLM agents, the reported world-record ratios, ranging from 316% to 432% in Figure 1, and the comparisons between Offline-Seed and Offline-Scratch in Table 2 are presented without sufficient information about repeated runs, random seeds, confidence intervals, or performance variance. In addition, while the authors use a fixed number of optimization iterations, the computational budgets are not clearly normalized in terms of token consumption or wall-clock time across different models. This makes it difficult to determine whether the observed performance differences reflect actual reasoning capabilities or differences in computational resources and search efficiency. I am particularly concerned that allowing up to 200 candidate evaluations per iteration leads to dramatic improvements, such as DeepSeek-V4-Pro improving from 9,428 to 1,847 frames on SuperTux. Without comparisons against simpler search or optimization baselines under matched budgets, it is unclear how much of this improvement comes from the agent’s strategic reasoning rather than repeated trial and error. Similarly, the Pokémon-versus-Tuxemon comparison reports a rank correlation of −0.18 across ten models, but this is relatively weak evidence for conclusions about pretraining contamination without statistical significance testing or uncertainty estimates. I think these issues are consequential because a benchmark needs to provide reliable and interpretable comparisons before it can serve as a meaningful reference for future research. I would strongly encourage the authors to include multiple runs with different random seeds, confidence intervals, matched computational budgets, and comparisons with non-LLM optimization baselines.

Second, I am concerned about the benchmark’s reproducibility, potential pretraining contamination, and the consistency of its evaluation settings. Although the authors include nine games across different genres, several titles, including Super Mario Bros., Super Mario Land, Pokémon, and Mario Kart 64, have extensive publicly documented speedrunning strategies and trajectories that may have appeared in LLM pretraining data. As a result, strong performance on these games may partly reflect memorized strategies rather than independent exploration or strategy discovery. The Pokémon-versus-Tuxemon comparison is a useful initial attempt to investigate this issue, but the two games also differ in mechanics, trajectory complexity, and optimization difficulty, making it difficult to isolate the effect of pretraining exposure. I also think the paper needs clearer documentation of the benchmark implementation, including action-trace encoding, emulator and game versions, determinism controls, and the selection of human world records and their corresponding speedrunning categories. Another concern is the dependence on particular reference trajectories in Offline-Seed. For example, Table 2 uses a 9,498-frame SuperTux trajectory generated by GPT-5.6-sol as the common starting point, which may introduce bias toward the strategies or action patterns of that model. The substantial performance differences between Offline-Scratch and Offline-Seed further suggest that seed quality can strongly influence the results. These concerns matter because the value of a new benchmark depends heavily on whether independent researchers can reproduce its results and make fair comparisons across different agents. I would like to see more complete implementation details, clearly documented world-record categories and seed provenance, and experiments using multiple or model-independent reference trajectories. Addressing these issues would substantially strengthen the benchmark’s credibility and its potential value to the research community.

Questions for authors

I have several questions that could materially affect my assessment of the paper. First, could the authors clarify whether the results reported in Figure 1, Figure 8, and Table 2 are based on multiple independent runs or single evaluations? In particular, could you provide the number of random seeds and confidence intervals for the reported 316%–432% world-record ratios? Since the performance differences between some models, such as Claude Opus 5 (316%) and GLM 5.3 (333%), are relatively small, I would like to understand whether the model rankings remain stable across repeated experiments. Second, how were the computational budgets controlled across different agents? Although the authors use 200 optimization iterations and allow up to 200 candidate evaluations per iteration in some experiments, it is unclear whether the comparisons account for differences in token consumption, wall-clock time, or computational cost. Could the authors clarify whether the substantial improvements observed with additional scoring opportunities reflect stronger strategic reasoning or simply greater search resources? Relatedly, have you compared the agents against simpler optimization approaches under matched evaluation budgets? Third, I am concerned about the dependence on reference trajectories in Offline-Seed. Since Table 2 uses a 9,498-frame trajectory generated by GPT-5.6-sol as the common starting point for SuperTux, could this introduce an advantage for that particular model? Have the authors evaluated whether the results remain consistent when using trajectories generated by other models or model-independent reference solutions? Fourth, could the authors provide more implementation details to support reproducibility, including the emulator and ROM versions, action-trace encoding, random seed controls, input polling rates, and save-state verification? I would also appreciate clarification on how the human world records were selected and whether the corresponding speedrunning categories, such as any-percent or glitchless, are consistently matched to the benchmark tasks. These details are important because differences in game configurations or speedrunning categories could substantially change the reported completion-time ratios. Fifth, regarding the Pokémon-versus-Tuxemon comparison, could the authors provide a statistical significance test or bootstrap confidence interval for the reported rank correlation of −0.18 across ten models? I would also like to understand how the authors distinguish potential pretraining contamination from differences in game mechanics, task difficulty, and trajectory complexity. Finally, in the near-optimal Super Mario Bros. experiment, GPT-5.6-sol improves an 18,834-frame reference trajectory by three frames after 200 iterations. Could the authors clarify whether this improvement comes from discovering a meaningful route optimization or game mechanic, or whether it is simply a small timing adjustment? This distinction matters because the paper emphasizes strategy discovery and iterative self-improvement rather than low-level action refinement alone. Convincing answers to these questions, particularly regarding statistical reliability, seed sensitivity, computational fairness, and the interpretation of pretraining contamination, could significantly improve my assessment. As separate suggestions for future work, I would encourage the authors to include additional unfamiliar games, establish standardized computational budgets, and release a reproducible benchmark environment with fixed emulator configurations and reference trajectories.

1 comment

We thank the reviewer for the detailed and constructive assessment.

W1/Q1:

"…repeated runs, random seeds, confidence intervals, or performance variance." / "…whether the model rankings remain stable across repeated experiments."

We repeated the Table 2 SuperTux comparison with two independent runs per model.

modelOFFLINE-SCRATCH (run 1 / 2)OFFLINE-SEED (run 1 / 2)
Gemini 3.7 Flash2,180 / 2,1309,454 / 9,205
GLM 5.31,961 / 1,9748,661 / 9,027
DeepSeek-V4-Flash2,725 / no clear9,203 / 9,278
Kimi K3no clear / no clear9,254 / 8,306
DeepSeek-V4-Prono clear / no clear9,227 / 9,332

Every OFFLINE-SCRATCH clear requires 3.4–4.6 times fewer frames than the corresponding OFFLINE-SEED result, and the two runs of Gemini 3.7 Flash and of GLM 5.3 agree to within 2.5%. We will state that Figure 1 and Figure 8 are based on single runs.

W1/Q2:

"…not clearly normalized in terms of token consumption or wall-clock time…"

Token counts are neither observable for proprietary models nor comparable across tokenizers, so we match agents on scorer calls: one per turn, at most 200 per run. Section 5.1 relaxes this budget and is reported separately.

W1/Q2:

"…how much of this improvement comes from the agent's strategic reasoning rather than repeated trial and error."

The two cannot be fully disentangled. Trial and error alone does not yield a first clear: with the same budget, GLM 5.3 and Gemini 3.7 Flash clear from scratch in both re-runs, whereas Kimi K3 and DeepSeek-V4-Pro clear in neither. After a route is found, gains are dominated by trial and error, ending in one-frame timing adjustments. We therefore present Section 5.1 as additional search rather than stronger reasoning, and make no claim that LLM edits outperform non-LLM search.

W2/Q3:

"…could this introduce an advantage for that particular model?"

We find no such advantage. In Figure 5a, GPT-5.6-sol's OFFLINE-SEED result (9,306 frames) is slower than those of most other models. The seed does anchor the route: seeded bests retain 7–11 deaths (the seed has 10), whereas scratch clears have none. We will add a model-independent seed.

W2/Q5:

"…memorized strategies…" / "…significance test…for the reported rank correlation of −0.18…"

We revise this claim. The paired comparison shows no advantage on the familiar game: the largest improvement is 6.9% on Tuxemon (GLM 5.3), whereas no model improves on Pokémon by more than 1%, and per-model results are uncorrelated (ρ = −0.18, n = 10, not significant). The data therefore do not support a contamination advantage, although differences in mechanics prevent ruling it out.

W2/Q4:

"…emulator and ROM versions, action-trace encoding, random seed controls, input polling rates, and save-state verification?"

On SuperTux, trajectories map step indices to key events and replay bit-identically at 60 steps per second. The other games use each title's native frame-indexed input format. Agents receive only a terminal summary per candidate, never intermediate states or savestates. Code, prompts and seeds for the open source games are available at this anonymized repo https://anonymous.4open.science/r/speedrunbench-FD02/. Each reference is the any% record for the task's scope.

Q6:

"…a meaningful route optimization or game mechanic, or…simply a small timing adjustment?"

It is a small timing adjustment. The improved trajectory differs from the reference in a single input, a jump before the final axe held for two frames instead of three, and the route is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.