- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SpeedrunBench, a benchmark for evaluating LLM agents through video-game speedrunning rather than simple task completion. The motivation is that completion-based game benchmarks may saturate as models improve, while speedrunning provides a continuously optimizable objective that can continue distinguishing strong agents. The benchmark includes nine games across platforming, racing, RPG, strategy, and puzzle genres, and evaluates three settings: Online, Offline-Scratch, and Offline-Seed. Results show that frontier agents can approach human records on short platformers but remain substantially slower on longer and more complex games. One interesting finding is that agents often do better starting from scratch than improving an existing but suboptimal solution. However, on short platformers, giving agents many more scorer evaluations can largely close this gap.
Critical strengths
-
Continued optimization after successful completion is a compelling benchmark design choice. Unlike binary success metrics, speedrunning provides a naturally graded objective that can continue differentiating strong agents even after they all solve the task.
-
The seed-vs-scratch result reveals an interesting failure mode of current agents. A valid but suboptimal seed can anchor the agent so strongly that starting from scratch produces much better trajectories, highlighting difficulty in escaping an existing solution basin and performing global revision.
Critical weaknesses
-
The Offline setting is essentially black-box optimization over a long discrete action sequence, yet the paper lacks non-LLM search baselines. This makes it difficult to determine whether improvements come from LLM-specific strategy formation or generic search under a large evaluation budget. Simple baselines such as random mutation with elitism, hill climbing, evolutionary search, beam search, or MCTS where applicable would help isolate the value of LLM-based proposal generation.
-
The paper’s claims about “strategy formation” and especially “discovery” are stronger than the evidence presented. Most experiments show that agents can reduce frame count under repeated feedback, but this does not by itself demonstrate qualitatively new strategy discovery. The paper provides little analysis of whether agents change routes, invent techniques absent from the seed, or discover new mechanics/exploits. As presented, the evidence more directly supports iterative trajectory optimization.
-
The experimental protocol is not specified sufficiently for reproducibility, and uncertainty is largely absent. Important details of the environment-agent interface remain unclear, including what feedback is returned after each trace, how progress is represented, whether intermediate states are available, and how traces are encoded and edited. In addition, results are mostly reported as point estimates without repeated-run statistics, despite stochastic LLM behavior. Fig. 8 further complicates interpretation by selecting the better of offline-scratch and Offline-seed.
Questions for authors
-
How are human RTA records converted into frame counts across different platforms, emulators, frame rates, loading behavior, and timing conventions?
-
Sec. 5.1 appears numerically inconsistent. The agent can evaluate up to 200 traces per turn, while Fig. 6b spans roughly 100 turns, suggesting at most ~20K evaluations, yet the text reports 220K evaluated traces over ~114 hours. What exactly counts as an “evaluated trace,” and how is 220K obtained?
-
Please clarify the full experimental protocol: agent harness and prompts, available tools, persistent context/memory across turns, precise inputs/outputs in each setting, model/API versions, decoding parameters, and number of independent runs/seeds.
2 comments
Thank you for the clarifications and additional experiments. I appreciate the authors' efforts.
However, my main concern remains unresolved. It is still unclear to me whether the performance improvements come from LLM reasoning or primarily from test-time search with a large number of scorer evaluations.
I therefore maintain my original rating.
We thank the reviewer for the thoughtful review.
SpeedrunBench is designed to compare LLM agents with one another and to measure how far each remains from the human world record under a fixed interaction budget. It does not claim that LLM agents outperform dedicated search algorithms. A non-LLM baseline would address a different question, how well a game can be optimized by any means. We will state this scope explicitly. The BFS solver behind the Astray seed (Fig. 9) remains as a non-LLM reference point.
We agree that “discovery” should be stated precisely, and will clarify that it refers to agents finding faster routes from sparse feedback. The evidence for strategy formation lies in how agents reach the goal. Without any reference trajectory, scratch agents construct death-free routes that require 3.4–4.6 times fewer frames than the seeded results, and they do so with distinct strategies, such as placing one jump per obstacle or jumping at a regular rhythm. Seeded agents instead retain the seed's route and its deaths (7–11, against 10 in the seed), which shows that a working but slow route constrains the strategies agents form.
After each turn, the agent receives whether the goal was reached, the frame count, the furthest progress, the number of deaths and a hash of the final state. It retains its session across turns, and only the scorer is withheld. OFFLINE configurations request temperature 0.3. Where an API does not accept the parameter, as for Claude Opus and GPT-5.6 Sol, the provider's default applies. The Reproducibility Statement will list the exact settings per model. We repeated the Table 2 SuperTux comparison with two independent runs per model.
Each record is converted at the game's frame rate over the segment the task scores: 60 frames per second for SuperTux 1-1 (26.98 s) and Super Mario Land 1-1 (37.6 s, from power-on), the flagpole touch for Super Mario Bros. 1-1 (18.59 s), the single-lap record for Mario Kart 64 (10.87 s) with the menu prefix removed, and 59.73 frames per second for Pokémon (11:17 to the first badge). SuperTuxKart is compared in seconds.
We clarify that the description of the experimental setup for the Super Mario Land run shown in Figure 6b was not subject to the per-turn limit of 200 evaluations. Each evaluated trace corresponds to a single candidate trajectory assessed by the scorer. We will clarify both points in the revised manuscript.