- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Playscriptbench, a benchmark of 16 different HTML games in which VLMs must write a standalone game-playing controller from pixels instead of predicting one action per frame. The controller synthesis is split into four stages (Explore, Perception, API Construction, Strategy Composition). The authors evaluate 13 VLMs. Experiments show that even strong models struggle to turn visual understanding into effective closed-loop control.
Critical strengths
- The paper studies a very relevant setting: asking VLMs to write a reusable game-playing controller instead of predicting one action per frame.
- The benchmark is a solid engineering effort: 16 HTML5 games across 4 genres, different rendering engines.
- Splitting the task into perception, API and strategy artifacts, each scored on its own, allows fine-grained diagnosis of failures.
Critical weaknesses
- I question whether the proposed evaluation protocol really defines a new problem. In my view, calling the VLM at every frame and the proposed setting are two different methods for the same problem. Calling the VLM at every frame gives finer control but costs latency. The proposed approach has a high initial compute cost and lower latency afterwards. It would be more interesting to frame this as a general problem where the call interval ranges from 1 (most existing work) to infinity (the proposed setting). Methods could then be compared more precisely depending on the objective (score, latency, etc.).
- The paper mixes up the benchmark and the proposed baseline. Sections 4.1, 4.2 and 4.3 describe the benchmark. Section 4.4 describes a baseline harness that divides the problem into four stages, but that is a methodological choice. This is more than a writing issue: the choice of baseline needs to be motivated, and alternatives should be considered. It matters because if these stages are treated as part of the benchmark, we appear to evaluate only the VLM, while the design of the harness and the details of each stage have a substantial impact.
- The tables report min-max normalized scores, which leads to several issues:
- There is no fixed reference point, such as human performance, a random policy or a hand-written controller.
- It amplifies noise. For example, Chrome Dino raw scores range only from 44.3 to 50.3 (Table 5), and rescaling them to 0–1 wrongly suggests large, significant differences.
- It makes the benchmark harder to adopt. If a new model gets a better score, should it become the new maximum? Then the same model would be reported with different, non-comparable numbers across papers. This is highly problematic.
- The key comparison against the frame-by-frame baseline in Table 3 is unfair for several reasons:
- The proposed method uses oracle perception, while the VLM baseline must read the screen itself. The authors acknowledge this (l. 485), but only with an unclear statement: "this gap reflects amortization under non-matched state access".
- The VLM baseline is not allowed any reasoning. This detail is important but is hidden in the appendix (l. 970).
- It covers only a single model, even though these experiments are meant to justify the relevance of the setting, which is the core claim of the paper.
- The evaluation results are weak. Perception and API code are evaluated for 13 models (Obs. in Table 1). However, every Stra. and Joint score in Table 1 runs the controller on the oracle perceive() and GameAPI, not on the model's own code. I ask the authors to confirm this reading, since the explanations are hard to follow. Stage 3 run with the model's own Stage 1–2 code is reported for only two models. The benchmark's core claim, that a controller can be built from pixels alone, is therefore barely tested. Moreover, in the "w/o Oracle" ablation (Table 2), GPT-5.6 Luna collapses to 0.013 and Qwen 3.6 to 0.052. Scores this close to 0 undermine the claim that the proposed baseline can really build a controller from pixels alone.
Questions for authors
Questions:
- The paper states that 10 seeds are used throughout, but A.6.4 lists only 3 seeds ([42, 123, 456]). The reported means are all multiples of 1/3, which suggests 3 seeds. Could the authors clarify?
- The Table 2 caption says: "Per. and API. are averaged directly across 16 games, so they differ from the genre-averaged Overall Obs. in Table 1." However, each of the 4 genres contains exactly 4 games, so the two averages should be mathematically equal. In practice they differ a lot: for GPT-5.6 Luna, Table 2 gives Obs. ≈ 0.685, while Table 1 reports 0.53. Is this a bug, or am I misunderstanding something?
Suggestions:
- Explain the missing entries ("—") in Tables 5 to 8. The captions only say that no run was completed, without giving a reason, and there is no obvious reason why some models lack scores. These missing entries may also affect the averages and the min-max bounds.
- The paper seems to have been written in a rush, as shown by the missing conclusion and the inconsistencies listed above. I recommend that the authors carefully proofread the paper and add a conclusion that summarizes the findings and discusses limitations.
- Some missing references:
- AutoHarness (Lou et al., ICLR 2026)
- Learning Game-Playing Agents with Generative Code Optimization (Kuang et al. ICML 2025 PRAL workshop)