- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PLAYSUITE, a large-scale benchmark for interactive visual intelligence covering 5,734 video games. It provides a unified keyboard-and-mouse interaction framework, an HPC-oriented batched execution system, and a milestone-based automated evaluation protocol using a Video-LLM judge. The authors evaluate 14 open-weight VLM/CUA/VLA models and show that current models make meaningful progress on only a limited fraction of the games.
Critical strengths
- Large-scale and diverse benchmark. PLAYSUITE contains 5,734 environments spanning multiple technical stacks, including Pygame, HTML5, Godot, and Unity. This is more than a simple increase in benchmark size: supporting such heterogeneous games requires unified handling of launching, input, pausing, video recording, and evaluation. Therefore, the work addresses several practical infrastructure challenges for large-scale closed-loop agent evaluation.
- Effective systems-level optimization. The paper introduces an architecture in which multiple game workers share a single vLLM inference worker. In the Qwen3-VL-8B/H100 experiment, using eight concurrent workers increases throughput from 6.7 to 42.4 evaluations per GPU-hour. This systems contribution is practically useful and potentially reusable for future large-scale agent benchmarks.
Critical weaknesses
- The reliability validation of the Video-LLM judge is insufficient for fine-grained leaderboard comparisons. The authors use Qwen3.5-9B as the default judge. Although its agreement with human annotators within one progress level reaches 90.4%, and the gap to larger judges is relatively small, this is still a fairly loose criterion. Since many models on the final leaderboard differ by only a small margin in average score, the current validation does not fully establish that the judge is sufficiently reliable for stable fine-grained model ranking. It would be useful to directly verify whether the model ranking is preserved under human evaluation. In addition, although the paper evaluates several alternative judge models, they are all relatively small-scale models; including stronger judge models would further strengthen the credibility of the evaluation protocol.
- The representativeness of the 200-game subset is unclear. The experiments show that a sample of around 200 games can approximately recover the overall model ranking, but the paper does not clearly explain how the final fixed 200-game subset used for controlled analyses is selected. It is also unclear whether this subset is representative of the full benchmark in terms of genre, difficulty, interaction type, or game engine. As a result, the subsequent ablation results may be sensitive to the particular subset chosen.
Questions for authors
- Is the leaderboard ranking preserved under human evaluation or stronger Video-LLM judges, especially for models with only small score differences? This would help assess whether the current judge is reliable enough for fine-grained model ranking.
- How was the fixed 200-game subset selected for the controlled analyses, and how representative is it of the full benchmark in terms of genre, difficulty, interaction type, and game engine?
We thank the reviewer for their thoughtful feedback and constructive suggestions.
Q1. Reliability of Model Rankings
Based on your actionable feedback, and as detailed in our Shared Response, we strengthened our evaluation through human verification of game-specific rubrics, additional human evaluations of gameplay progress, and comparisons with stronger Video-LLM judges. These results show that the 27B thinking judge achieves substantially better agreement with human assessments while remaining practical for large-scale evaluation, supporting its adoption in our revised evaluation. We further examine whether the revised judge preserves model rankings:
Together, these analyses show that the overall leaderboard is stable under the revised judging protocol and that the adopted judge better reflects human-assessed progress differences. We will include the detailed comparisons and uncertainty analyses in the appendix.
Q2. Details on the Itch 200-Game Subset
We clarify that the 200-game subset was selected through uniform random sampling from the 5,630 itch.io games, independently of model performance. We assessed its representativeness by comparing of game characteristics, difficulty proxies, and model rankings against the full itch.io corpus:
These results suggest that the subset broadly reflects the full itch.io corpus and preserves its overall model ranking. The largest compositional difference is slightly fewer Adventure and more Puzzle games. We will include the sampling procedure, game manifest, and distribution comparisons in the appendix.