acceptodds
Withdrawn from ICLR 2027

PlayScriptBench: Benchmarking Vision-Language Models for Executable Game-Playing Program Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in vision-language models (VLMs) have endowed multimodal agents with increasingly strong visual perception and long-horizon reasoning capabilities, making VLM-based autonomous game playing feasible. Existing game benchmarks primarily evaluate whether an agent can select appropriate actions from the current visual state across different scenarios and games. However, most of them adopt a step-by-step interaction paradigm where the current visual observation is sent to the agent before each action. In practical deployment, this paradigm introduces two fundamental limitations: the game loop must wait for model inference, making real-time interaction impractical, and repeated VLM calls incur substantial cost. Therefore, we propose PlayScriptBench, a script-based game benchmark that evaluates whether VLMs can synthesize an executable game-playing program from game states. This setting jointly evaluates visual grounding, understanding of game dynamics, long-horizon strategic reasoning, and executable program synthesis, providing a more comprehensive assessment of game-playing agents. To support more tractable evaluation for current general-purpose VLMs, we further introduce a four-stage pipeline — Explore, Perception, API Construction, and Strategy Composition — that decomposes controller synthesis into game exploration, state perception, interface construction, and strategic control, enabling agents to progressively construct an executable program. Experiments on 16 games spanning 4 genres show that even strong VLMs struggle on our benchmark, revealing a substantial gap between passive visual game understanding and executable closed-loop game control. These results establish script-based controller synthesis as a complementary direction for benchmarking VLM-based game-playing agents.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, game playing agent, program synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MgazOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Playscriptbench, a benchmark of 16 different HTML games in which VLMs must write a standalone game-playing controller from pixels instead of predicting one action per frame. The controller synthesis is split into four stages (Explore, Perception, API Construction, Strategy Composition). The authors evaluate 13 VLMs. Experiments show that even strong models struggle to turn visual understanding into effective closed-loop control.

Critical strengths
  • The paper studies a very relevant setting: asking VLMs to write a reusable game-playing controller instead of predicting one action per frame.
  • The benchmark is a solid engineering effort: 16 HTML5 games across 4 genres, different rendering engines.
  • Splitting the task into perception, API and strategy artifacts, each scored on its own, allows fine-grained diagnosis of failures.
Critical weaknesses
  • I question whether the proposed evaluation protocol really defines a new problem. In my view, calling the VLM at every frame and the proposed setting are two different methods for the same problem. Calling the VLM at every frame gives finer control but costs latency. The proposed approach has a high initial compute cost and lower latency afterwards. It would be more interesting to frame this as a general problem where the call interval ranges from 1 (most existing work) to infinity (the proposed setting). Methods could then be compared more precisely depending on the objective (score, latency, etc.).
  • The paper mixes up the benchmark and the proposed baseline. Sections 4.1, 4.2 and 4.3 describe the benchmark. Section 4.4 describes a baseline harness that divides the problem into four stages, but that is a methodological choice. This is more than a writing issue: the choice of baseline needs to be motivated, and alternatives should be considered. It matters because if these stages are treated as part of the benchmark, we appear to evaluate only the VLM, while the design of the harness and the details of each stage have a substantial impact.
  • The tables report min-max normalized scores, which leads to several issues:
    • There is no fixed reference point, such as human performance, a random policy or a hand-written controller.
    • It amplifies noise. For example, Chrome Dino raw scores range only from 44.3 to 50.3 (Table 5), and rescaling them to 0–1 wrongly suggests large, significant differences.
    • It makes the benchmark harder to adopt. If a new model gets a better score, should it become the new maximum? Then the same model would be reported with different, non-comparable numbers across papers. This is highly problematic.
  • The key comparison against the frame-by-frame baseline in Table 3 is unfair for several reasons:
    • The proposed method uses oracle perception, while the VLM baseline must read the screen itself. The authors acknowledge this (l. 485), but only with an unclear statement: "this gap reflects amortization under non-matched state access".
    • The VLM baseline is not allowed any reasoning. This detail is important but is hidden in the appendix (l. 970).
    • It covers only a single model, even though these experiments are meant to justify the relevance of the setting, which is the core claim of the paper.
  • The evaluation results are weak. Perception and API code are evaluated for 13 models (Obs. in Table 1). However, every Stra. and Joint score in Table 1 runs the controller on the oracle perceive() and GameAPI, not on the model's own code. I ask the authors to confirm this reading, since the explanations are hard to follow. Stage 3 run with the model's own Stage 1–2 code is reported for only two models. The benchmark's core claim, that a controller can be built from pixels alone, is therefore barely tested. Moreover, in the "w/o Oracle" ablation (Table 2), GPT-5.6 Luna collapses to 0.013 and Qwen 3.6 to 0.052. Scores this close to 0 undermine the claim that the proposed baseline can really build a controller from pixels alone.
Questions for authors

Questions:

  • The paper states that 10 seeds are used throughout, but A.6.4 lists only 3 seeds ([42, 123, 456]). The reported means are all multiples of 1/3, which suggests 3 seeds. Could the authors clarify?
  • The Table 2 caption says: "Per. and API. are averaged directly across 16 games, so they differ from the genre-averaged Overall Obs. in Table 1." However, each of the 4 genres contains exactly 4 games, so the two averages should be mathematically equal. In practice they differ a lot: for GPT-5.6 Luna, Table 2 gives Obs. ≈ 0.685, while Table 1 reports 0.53. Is this a bug, or am I misunderstanding something?

Suggestions:

  • Explain the missing entries ("—") in Tables 5 to 8. The captions only say that no run was completed, without giving a reason, and there is no obvious reason why some models lack scores. These missing entries may also affect the averages and the min-max bounds.
  • The paper seems to have been written in a rush, as shown by the missing conclusion and the inconsistencies listed above. I recommend that the authors carefully proofread the paper and add a conclusion that summarizes the findings and discusses limitations.
  • Some missing references:
    • AutoHarness (Lou et al., ICLR 2026)
    • Learning Game-Playing Agents with Generative Code Optimization (Kuang et al. ICML 2025 PRAL workshop)
Reviewer_RdKLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PlayScriptBench, a benchmark of 16 HTML5 games for generating executable game-playing controllers. A four-stage pipeline produces exploration summaries, perception functions, state APIs, and strategy scripts. The evaluation covers 13 VLMs using perception/API accuracy, oracle-conditioned gameplay scores, and a composite Joint metric. It includes pipeline ablations and a resource comparison with a per-action VLM baseline.

Critical strengths
  1. Separate perception/API artifacts and interchangeable reference components offer a useful framework for locating failures. The oracle-versus-own-perception ablation illustrates the importance of this distinction.
Critical weaknesses
  1. Narrow evaluation with limited demonstrated capability coverage. The benchmark evaluates game-specific controller synthesis on 16 fixed HTML5 games with predefined state schemas and interfaces. It does not test generalization across visual changes or game mechanics, or distinguish general reasoning from implementing familiar game-specific heuristics. The paper therefore establishes a specialized automation scenario, but does not adequately substantiate its claimed coverage of visual grounding, dynamics understanding, and long-horizon reasoning (§§3–4; A.6).

  2. The motivation is weakly substantiated. Avoiding repeated VLM calls motivates a possible deployment choice, but does not by itself establish why this is an informative capability benchmark. The paper does not characterize when controller synthesis offers a useful trade-off between inference cost, flexibility, and reliability. Moreover, Table 3 compares oracle-assisted scripts against a screenshot-based baseline, so the claimed performance–efficiency advantage is not demonstrated under comparable conditions (§6.4).

  3. Limited substantive insight beyond expected pipeline effects. The main observations—that additional visual information helps, perception errors propagate, executable code can implement poor strategies, and reusable scripts amortize inference—are largely predictable. The qualitative failure taxonomy does not quantify their prevalence or establish whether they arise from model limitations or the restricted synthesis protocol (A.6–A.8). The stronger interpretations are unsupported: a nonsignificant correlation does not establish independent capabilities, and the claim that perception is not the main bottleneck conflicts with the substantial degradation after removing oracle perception.

  4. The primary metrics do not measure the advertised end-to-end task. Main strategy scores use oracle perception and APIs, while Joint multiplies offline observation accuracy by oracle-conditioned gameplay rather than measuring integrated execution. This surrogate is not validated as a measure of actual visual-controller performance. End-to-end results are provided for only two models and fall to 0.013/0.052, leaving the central task insufficiently characterized across the evaluated models (Tables 1–2; A.3.3).

  5. Material inconsistencies undermine confidence in the reported results. Table 2 implies GPT-5.6 Luna’s Obs. is 0.6845, versus 0.53 in Table 1; the genre-averaging explanation cannot account for this when every genre contains four games. The supplied Snake implementation awards points in multiples of ten, yet Table 6 reports noninteger means over supposedly ten runs. These discrepancies require clarification of the actual runs and aggregation procedure before the quantitative conclusions can be trusted.

  6. Reference validity and reproducibility are insufficiently established. The main text and A.3.2 disagree on whether API correctness is evaluated using oracle states or model-generated states, changing what the metric measures. The supplied Snake and Doodle Jump reference perception routines also return constant-zero scores despite specifications requiring the current score. Without the shared evaluation harness, graders, and per-seed logs, it is unclear which implementations produced the reported results or whether the reference components support the claimed failure attribution.

Questions for authors
  1. Protocol clarification: Which input states were used for the reported API scores? Which reference implementations were used and how were they validated, given the constant-zero scores in the supplied Snake and Doodle Jump perception routines?

  2. Validation of existing claims: What are the complete end-to-end results across all 13 models, and does the performance comparison with the in-loop baseline hold under matched state access and interaction budgets?

Reviewer_JnjQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Recent advances in vision-language models (VLMs) have endowed multimodal agents with increasingly strong visual perception and long-horizon reasoning capabilities, making VLM-based autonomous game playing feasible. Existing game benchmarks primarily evaluate whether an agent can select appropriate actions from the current visual state across different scenarios and games. However, most of them adopt a step-by-step interaction paradigm where the current visual observation is sent to the agent before each action. In practical deployment, this paradigm introduces two fundamental limitations: the game loop must wait for model inference, making real-time interaction impractical, and repeated VLM calls incur substantial cost.

Therefore, we propose PlayScriptBench, a script-based game benchmark that evaluates whether VLMs can synthesize an executable game-playing program from game states. This setting jointly evaluates visual grounding, understanding of game dynamics, long-horizon strategic reasoning, and executable program synthesis, providing a more comprehensive assessment of game-playing agents.

To support more tractable evaluation for current general-purpose VLMs, we further introduce a four-stage pipeline — Explore, Perception, API Construction, and Strategy Composition — that decomposes controller synthesis into game exploration, state perception, interface construction, and strategic control, enabling agents to progressively construct an executable program.

Experiments show that even strong VLMs struggle on our benchmark, revealing a substantial gap between passive visual game understanding and executable closed-loop game control. These results establish script-based controller synthesis as a complementary direction for benchmarking VLM-based game-playing agents.

Critical strengths
  1. A novel evaluation paradigm that replaces per-frame inference with program synthesis. The paper requires VLMs to synthesize a standalone, executable controller capable of running autonomously at native frame rates, thereby decoupling inference overhead from runtime gameplay into an offline development phase. This setting not only effectively assesses a model's ability to compress visual understanding into persistent algorithmic logic, but also opens up a highly valuable and complementary benchmarking direction for embodied AI and autonomous game agents.

  2. A decoupled diagnostic framework that resolves the attribution black-box problem. In end-to-end program synthesis, evaluation often suffers from severe attribution ambiguity—namely, whether a gameplay failure stems from perceptual errors or flawed strategic reasoning. The paper thoroughly decouples visual grounding from strategic reasoning through a four-stage pipeline and a pluggable Oracle–VLM Probe mechanism, empirically demonstrating a substantial capability gap and decoupling between visual perception and strategic planning in frontier VLMs.

Critical weaknesses
  1. The core selling point of the paper is to evaluate the ability of VLMs to synthesize autonomous, executable controllers. However, ablation experiments show that once the human-provided perfect perception ground truth is removed and the models must use their own synthesized perception and API modules, the gameplay performance of all frontier models collapses catastrophically. This exposes the insufficient practical viability of the current end-to-end "hand-crafted CV code to control" paradigm in real-world, unassisted scenarios.

  2. The paper pre-hardcodes detailed human-designed templates for each game across different stages, meaning that models are essentially "filling in the blanks" within human-crafted frameworks, which substantially reduces the difficulty of autonomous task exploration. Furthermore, it is difficult to generalize this approach to genuinely complex or open-world environments, limiting the representativeness of this benchmark as an evaluation tool for general-purpose agents.

Questions for authors
  1. The ablation experiments show that without Oracle assistance (w/o Oracle), the true closed-loop scores of all models collapse almost entirely to zero. If the agent were allowed several rounds of self-correcting code in the sandbox, could the end-to-end performance recover to a viable level? Has this been tested?

  2. In Stage 1 and Stage 2, the paper hardcodes and injects highly detailed GameState data structure definitions, tolerance thresholds, and GameAPI method signatures. If these pre-defined templates were removed—providing only game rules and raw screenshots to let the VLM discover key states and define interfaces entirely autonomously—to what extent would the model's perception and strategy synthesis performance degrade?

  3. It is suggested to explore allowing agents to invoke open-source visual tools (such as Grounding DINO, SAM, or lightweight detectors) within their scripts, which may better reflect real-world code synthesis and control development practices in industry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.