acceptodds
Under review as a conference paper at ICLR 2027

PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

Abstract

Recent advances in multimodal foundation models yield strong performance on static perception and reasoning benchmarks, yet largely overlook a central aspect of intelligence: acting competently in dynamic environments over extended time horizons. We introduce PlaySuite, a large-scale benchmark for evaluating interactive visual intelligence across more than 5K open-source video games curated from PyWeek and itch.io. Spanning diverse genres and engines, including Pygame, HTML5, Godot, and Unity, these independent games are largely out-of-distribution for current models, reducing the likelihood that success can be achieved by retrieving memorized walkthroughs or web-scale training artifacts. To enable scalable evaluation across heterogeneous titles, we develop a unified closed-loop interaction framework optimized for HPC clusters alongside a Video-LLM-as-a-judge protocol that maps observable gameplay milestones to standardized progress levels. We evaluate fourteen recent open models spanning vision-language models, computer-use agents, and vision-language-action models. Our results reveal a considerable perception-action gap: despite strong reasoning capabilities, current models struggle to maintain sustained progress and exhibit recurring failures in spatial grounding, action execution, and self-correction. PlaySuite provides a reproducible and extensible testbed for measuring progress from visual perception to goal-directed interaction, and a foundation for developing models that can act, adapt, and generalize in dynamic visual environments.

Primary area:
datasets and benchmarks
Keywords:
Multimodal agents, Interactive visual intelligence, Benchmarking in video game environments

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LxgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PLAYSUITE, a large-scale benchmark for interactive visual intelligence covering 5,734 video games. It provides a unified keyboard-and-mouse interaction framework, an HPC-oriented batched execution system, and a milestone-based automated evaluation protocol using a Video-LLM judge. The authors evaluate 14 open-weight VLM/CUA/VLA models and show that current models make meaningful progress on only a limited fraction of the games.

Critical strengths
  1. Large-scale and diverse benchmark. PLAYSUITE contains 5,734 environments spanning multiple technical stacks, including Pygame, HTML5, Godot, and Unity. This is more than a simple increase in benchmark size: supporting such heterogeneous games requires unified handling of launching, input, pausing, video recording, and evaluation. Therefore, the work addresses several practical infrastructure challenges for large-scale closed-loop agent evaluation.
  2. Effective systems-level optimization. The paper introduces an architecture in which multiple game workers share a single vLLM inference worker. In the Qwen3-VL-8B/H100 experiment, using eight concurrent workers increases throughput from 6.7 to 42.4 evaluations per GPU-hour. This systems contribution is practically useful and potentially reusable for future large-scale agent benchmarks.
Critical weaknesses
  1. The reliability validation of the Video-LLM judge is insufficient for fine-grained leaderboard comparisons. The authors use Qwen3.5-9B as the default judge. Although its agreement with human annotators within one progress level reaches 90.4%, and the gap to larger judges is relatively small, this is still a fairly loose criterion. Since many models on the final leaderboard differ by only a small margin in average score, the current validation does not fully establish that the judge is sufficiently reliable for stable fine-grained model ranking. It would be useful to directly verify whether the model ranking is preserved under human evaluation. In addition, although the paper evaluates several alternative judge models, they are all relatively small-scale models; including stronger judge models would further strengthen the credibility of the evaluation protocol.
  2. The representativeness of the 200-game subset is unclear. The experiments show that a sample of around 200 games can approximately recover the overall model ranking, but the paper does not clearly explain how the final fixed 200-game subset used for controlled analyses is selected. It is also unclear whether this subset is representative of the full benchmark in terms of genre, difficulty, interaction type, or game engine. As a result, the subsequent ablation results may be sensitive to the particular subset chosen.
Questions for authors
  1. Is the leaderboard ranking preserved under human evaluation or stronger Video-LLM judges, especially for models with only small score differences? This would help assess whether the current judge is reliable enough for fine-grained model ranking.
  2. How was the fixed 200-game subset selected for the controlled analyses, and how representative is it of the full benchmark in terms of genre, difficulty, interaction type, and game engine?
1 comment

We thank the reviewer for their thoughtful feedback and constructive suggestions.

Q1. Reliability of Model Rankings

Based on your actionable feedback, and as detailed in our Shared Response, we strengthened our evaluation through human verification of game-specific rubrics, additional human evaluations of gameplay progress, and comparisons with stronger Video-LLM judges. These results show that the 27B thinking judge achieves substantially better agreement with human assessments while remaining practical for large-scale evaluation, supporting its adoption in our revised evaluation. We further examine whether the revised judge preserves model rankings:

  • Agreement with human pairwise assessments. We compare the judge's progress scores with human assessments of different models playing the same games (269 pairs of gameplay recordings). For pairs where humans assign different progress levels, the revised 27B thinking judge agrees on which run performed better in 67% of comparisons, disagrees in only 3%, and assigns equal progress levels in the remaining 29%. When the judge assigns different progress levels to the two runs, its ordering agrees with humans 95% of the time.
  • Leaderboard stability. We re-evaluated all 14 models across both corpora (k=0 and k=3) using the 27B judge with and without thinking. Rankings remain nearly identical (Spearman ρ = 0.996 on itch.io and 0.982 on PyWeek), with only 1 and 3 of 91 model pairs changing order, respectively. These reversals involve closely matched models, and none is confidently ranked in opposite directions by the two judges.
  • Stability across judge sizes. We further compared 9B and 27B thinking judges on the itch.io-200 and PyWeek subsets. Rankings remain strongly correlated (ρ = 0.92), preserving the top model and bottom two. All 11 pairwise reversals occur among the 20 closest model pairs (median score difference: 0.06 progress levels), and none is confidently ranked in opposite directions. The revised judge also largely preserves the submitted 9B judge's overall model ranking (ρ = 0.94), while agreeing substantially better with human assessments of individual runs (67% vs. 39%).
  • Closely matched models. Using 95% game-bootstrap intervals on the revised leaderboard, 88 of 91 model pairs on itch.io are separated. The remaining three pairs differ by at most 0.023 progress levels, indicating that ranking uncertainty is limited to models with nearly identical performance.

Together, these analyses show that the overall leaderboard is stable under the revised judging protocol and that the adopted judge better reflects human-assessed progress differences. We will include the detailed comparisons and uncertainty analyses in the appendix.

Q2. Details on the Itch 200-Game Subset

We clarify that the 200-game subset was selected through uniform random sampling from the 5,630 itch.io games, independently of model performance. We assessed its representativeness by comparing of game characteristics, difficulty proxies, and model rankings against the full itch.io corpus:

  • Game characteristics. The subset has broadly similar distributions of genres, game engines, interaction type distributions. For example, Adventure, Puzzle, and Action games account for 32.5%, 26.0%, and 20.5% of the subset, compared with 38.8%, 22.6%, and 17.0% of the full corpus, respectively. Game engines are similarly represented (Unity: 27.0% vs. 26.4%; Godot: 11.0% vs. 8.8%; Bitsy: 9.5% vs. 8.0%), as are interaction types (keyboard and mouse: 57% vs. 63%; mouse-only: 24% vs. 20%; keyboard-only: 19% vs. 16%).
  • Game difficulty. Since intrinsic difficulty is not directly annotated, we examine average progress achieved by the 14 models and the Partial+ rate of random inputs as proxies. Both are comparable to the full corpus (mean progress: 0.941 vs. 0.985; random-input Partial+: 17.0% vs. 16.7%), suggesting that the subset is not unusually easy or difficult for the evaluated models.
  • Model rankings. The subset closely reproduces the full-corpus leaderboard (Spearman ρ = 0.981). This agreement is also typical of random 200-game subsets (median ρ = 0.972), supporting its use for controlled comparisons.

These results suggest that the subset broadly reflects the full itch.io corpus and preserves its overall model ranking. The largest compositional difference is slightly fewer Adventure and more Puzzle games. We will include the sampling procedure, game manifest, and distribution comparisons in the appendix.

Reviewer_Rhh7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. Motivation: This paper studies interactive visual intelligence: whether multimodal models can go beyond understanding static images and actually perceive, plan, act, and adapt in dynamic environments. Existing game benchmarks typically cover only a small number of games, making it difficult to evaluate broad generalization.
  2. PlaySuite Benchmark: The paper introduces PlaySuite, a large-scale benchmark containing 5,734 open-source video games from PyWeek and itch.io, spanning diverse genres, visual styles, mechanics, and engines. This is over 100× more games than prior multi-game benchmarks, and the diversity is intended to reduce reliance on memorized game-specific knowledge.
  3. Interactive Evaluation: PlaySuite provides a unified closed-loop interaction framework in which models observe game frames, reason about the current state, execute keyboard/mouse actions, and then receive the updated game state. Because different games lack a common reward function, the authors use a Video-LLM-as-a-judge to identify gameplay milestones and map progress into five levels, from None (0) to Completed (4).
  4. Key Findings: Experiments on 14 open models, including VLMs, computer-use agents, and vision-language-action models, reveal a clear perception–action gap: models with strong visual reasoning still make meaningful progress in only a minority of games. Common failures include spatial grounding errors, incorrect control selection, repetitive actions, poor self-correction, and ineffective use of temporal history. Interestingly, adding more interaction history does not consistently help and can even reduce performance.
Critical strengths
  1. Large-scale and diverse benchmark: PlaySuite covers 5,734 games across different genres, engines, and visual styles, providing a much stronger test of cross-game generalization than benchmarks with only a few environments.
  2. True closed-loop evaluation: Models must continuously observe, reason, act, and adapt based on environmental feedback, rather than simply answering questions about static screenshots or prerecorded videos.
  3. Scalable and unified evaluation: The standardized keyboard/mouse interface and Video-LLM judge make it possible to evaluate different model families across thousands of heterogeneous games without designing a separate reward function for every game.
Critical weaknesses
  1. Reliance on an LLM judge: Gameplay progress is determined by a Video-LLM rather than ground-truth game states or rewards, which may introduce evaluation errors despite the authors’ human validation.
  2. Progress metric is relatively coarse: The five-level progress score captures observable milestones but does not directly evaluate strategy efficiency, reasoning quality, or mastery of game mechanics.
  3. Limited evaluation setting: The benchmark focuses on zero-shot gameplay and removes inference latency from game dynamics, so it does not evaluate learning/adaptation through fine-tuning or RL, nor real-time reaction ability.
Questions for authors
  1. Reliability of the LLM Judge: Since gameplay progress is evaluated using a Video-LLM judge rather than ground-truth game states or rewards, how reliable is the automated evaluation across different game genres and visual styles? Could the authors provide more details on human-judge agreement and the types of evaluation errors observed?

  2. Effectiveness of the Progress Metric: The five-level progress metric provides a unified way to evaluate gameplay across diverse games, but it may overlook differences in action efficiency and strategic reasoning. How well does this metric reflect actual gameplay competence? Could models achieve similar progress scores despite substantial differences in their strategies or action efficiency?

  3. Impact of Evaluation Settings: The benchmark focuses on zero-shot gameplay and excludes inference latency from game dynamics. How might these settings affect the observed performance and model rankings? In particular, would the main findings remain consistent under real-time interaction constraints?

2 comments

We thank the reviewer for their thoughtful feedback and constructive suggestions.

Q1. Reliability of the Video-LLM Judge Across Games

Per reviewer request, and as already detailed in our Shared Response, we have strengthened the evaluation through human verification of game-specific rubrics, additional human evaluations of gameplay progress, and stronger Video-LLM judges. The revised 27B thinking judge achieves substantially better agreement with human assessments, providing stronger evidence for the reliability of our progress judgments.

To further characterize the judge's reliability, we examine its errors across the 26 human-labelled games (280 human–judge comparisons):

  • Across corpora: exact agreement on progress levels is 77% on itch.io games and 82% on PyWeek games.
  • Across genres: for genres with at least 35 labels, exact agreement ranges from 76% (for Action) to 86% (for Adventure). Agreement is lower for games identified as puzzles (67%) or management games (71%).
  • Across visual styles: Exact agreement is 86% for basic 2D/text-based games, 81% for illustrated 2D, 76% for 3D, and 74% for pixel art. The revised 27B thinking judge improves agreement over the original judge across all four categories.
  • Remaining Partial+ disagreements: these (n=20) concentrate on whether the player earned a progress unit. All 15 false-negative and all 5 false-positive Partial+ comparisons disagree with the annotators on this question. In 11 of the 15 misses, annotators' responses map the run Partial, while the judge rated it None or Minimal, suggesting that most misses involve difficulty recognizing early milestone completion.
  • Player-control judgments: agreement with human answers on whether the player’s inputs visibly control gameplay increases from 65% for the non-thinking 27B judge to 83% with thinking enabled. This distinction matters because visible game activity alone does not establish player control. This distinction is important for separating progress attributable to the agent's actions from game activity that occurs independently of the agent.

This analysis will help characterize where the judge performs reliably and where additional ambiguity remains. We will include these agreement and error statistics in the appendix.

Q2. Effectiveness of the Progress Metric

The reviewer is quite right that models reaching the same milestone may differ substantially in their strategies and action efficiency, especially across the thousands of heterogeneous games included in PlaySuite. To assess whether the Progress Score reflects meaningful gameplay competence, we evaluated simple reference policies using the same game-specific rubrics and revised judge (see oADm W2). Idle and repeated fixed-action policies achieve Partial+ in only 0.0% and 2.9% of runs, respectively, while random valid inputs and keyboard mashing achieve 16.6% and 18.4%, compared with 28.7% on average across the 14 foundation models. These results suggest that the metric distinguishes purposeful gameplay from undirected actions.

However, evaluating these differences requires a more detailed analysis of action trajectories, including how actions contribute to progress and whether alternative strategies would have been more effective. Such analyses generally require additional game-specific knowledge and, in some cases, access to internal game states or rewards, as in [a] and [b]. Our current Progress Score therefore captures what models achieve within a fixed interaction budget, establishing a common foundation upon which more fine-grained evaluations of strategy, efficiency, and gameplay competence can be developed. We will clarify this distinction in the revised paper.

[a] Park, Dongmin, et al. “Orak: A foundational benchmark for training and evaluating llm agents on diverse video games.” International Conference on Learning Representations. Vol. 2026. 2026.

[b] Paglieri, Davide, et al. “Balrog: Benchmarking agentic llm and vlm reasoning on games.” International Conference on Learning Representations. Vol. 2025. 2025.

Q3. Impact of Evaluation Settings

We clarify that PlaySuite evaluates models at a fixed action frequency, pausing game time during inference. This allows us to compare their gameplay capabilities independently of inference speed. Under real-time constraints, slower models would have less time to react to changes in the game, potentially reducing their performance and changing model rankings. We therefore do not claim that the current rankings would transfer unchanged to real-time interaction. We will better stress this in the limitations.

We do believe our main finding is broader: current foundation models struggle to make meaningful progress across diverse, unseen games, even when inference latency is removed as a constraint. Real-time evaluation would introduce an additional challenge rather than eliminate the difficulties observed in our setting.

Finally, zero-shot refers to the absence of game-specific training or fine-tuning; models can still select their actions based on updated visual observations after each interaction. We will clarify these distinctions and the limitations of our evaluation setting in the revised paper.

Reviewer_oADmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PlaySuite, a benchmark for interactive visual intelligence spanning 5,734 open-source games. The framework supports large-scale closed-loop evaluation through unified keyboard and mouse interfaces, isolated game execution, and shared batched inference. A video-language model maps gameplay progress onto a five-level scale. The paper evaluates 14 open models and analyzes the effects of interaction history, action frequency, and evaluation scale, finding that current models achieve meaningful progress in only a minority of games.

Critical strengths
  • Large-scale environments and practically valuable infrastructure. Unified interfaces across game engines, combined with isolated execution and shared inference, improve throughput and provide a useful resource for studying cross-game generalization. The number and diversity of games substantially broaden the evaluation scope.

  • Useful analysis of evaluation methodology. Beyond reporting model rankings, the paper validates agreement between automated and human assessments and examines how game sample size affects ranking stability. These analyses help characterize the cost and reliability of large-scale interactive evaluation.

Critical weaknesses
  • Validation of the automated judge is insufficient to establish the reliability of key metrics. Human validation covers 60 videos, which is limited relative to thousands of heterogeneous games. Agreement “within one level” is also permissive: misclassifying Minimal as Partial directly changes the reported Partial+ success rate. Exact agreement, a level-wise confusion matrix, and precision and recall at the Partial+ threshold should be reported.

  • Reference baselines are missing for interpreting low scores. The results primarily compare foundation models, leaving unclear how much they outperform random actions, simple keyboard policies, or repeated fixed actions. Since a score increase qualifies as Partial progress, simple actions may produce progress in some games. Conversely, low scores may reflect failures to start gameplay or discover the controls.

  • Cross-game comparability of the unified Progress Score requires further validation. The same level may represent substantially different difficulty and progress across games. Entering a new area, earning a few points, and completing an intermediate objective do not necessarily require comparable capabilities. Although the paper clarifies that these scores are not completion percentages, averaging them still implicitly treats the intervals between levels as equal. Full level distributions, alternative aggregation methods, and genre-balanced rankings should be reported to assess whether the main conclusions depend on the scoring scheme and game composition.

Questions for authors

Please refer to the weakness part.

1 comment

We thank the reviewer for their thoughtful assessment and constructive suggestions.

W1. Validation of Automated Judge

To strengthen our validation, we conducted a separate human evaluation of 182 gameplay recordings, in addition to the 60 recordings evaluated in our original submission. The new panel includes 280 human ratings, and we additionally evaluated stronger Video-LLM judges using a revised rubric-based protocol. Beyond within-one-level agreement, we now report exact agreement, five-level confusion matrices, and precision/recall at the Partial+ threshold.

As detailed in our Shared Response, the 27B thinking judge improves exact agreement with human assessments from 47% to 78%, while Partial+ precision improves from 47% to 86% relative to the original 9B judge. False-positive Partial+ judgments decrease from 32 to 5, while recall also improves slightly (64% to 73%).

This improvement is particularly important for our benchmark: higher Partial+ precision means that when the judge reports meaningful gameplay progress, that judgment is substantially more likely to agree with human assessment. The reduction in false positives therefore addresses the reviewer's concern that models might receive credit for progress they did not achieve. We adopt the stronger judge for the revised leaderboard and report the confusion matrix in the appendix.

W2. Reference Baselines for Interpreting Model Performance

We agree that simple reference policies are important for interpreting the progress achieved by foundation models. To establish a clearer reference point, we have added several non-learning policies: idle, repeated fixed actions, random valid inputs, and keyboard mashing. These policies are evaluated under the same games and interaction budgets as the foundation models, using the revised 27B thinking judge. We compare these policies along two dimensions: whether they successfully initiate gameplay, and whether they achieve meaningful progress (Partial+) according to the game-specific rubrics.

PolicyReaches gameplayPartial+
Idle17.7%0.0%
Repeated fixed actions33.9%2.9%
Random valid inputs58.7%16.6%
Keyboard mashing50.5%18.4%
Foundation models (14), no history76.4%28.7%

For the random-input and keyboard-mashing baselines, we evaluate one seed across the benchmark and two additional seeds on the 304 games of itch-200 and PyWeek. Across seeds, Partial+ ranges from 15.5% to 15.9% for random inputs and from 14.5% to 18.5% for keyboard mashing.

Results: As shown above, undirected policies can achieve some progress, with random inputs and keyboard mashing reaching Partial+ in 16.6% and 18.4% of games, respectively. Foundation models outperform these baselines overall, reaching gameplay in 76.4% of runs and achieving Partial+ in 28.7% (13.2–46.5% across models). We will include this comparison in the main benchmark results, with policy definitions, seed variability, and per-model comparisons in the appendix.

W3. Cross-Game Comparability and Progress Score Aggregation

The reviewer is right that progress levels may represent different degrees of difficulty or achievement across games. We now recognize that averaging these levels assumes equal numerical spacing between them. To examine whether this affects our conclusions, we have extended our analysis to include:

  • First, we examined the full progress-level distributions. These show how frequently models reach each of the five levels. Across all 14 models, without history, 46.7% of runs achieve None, 24.6% Minimal, 16.6% Partial, 7.7% Substantial, and 4.3% Completed. We will include the per-model distributions in the appendix.
  • Second, we compared alternative ways of ranking models. Partial+ rate counts how often a model achieves at least Partial progress. We also ranked models by how many other models each beats in paired within-game comparisons. Both produce the same model ordering as mean progress. Thus, the reported ordering is preserved when we replace mean progress with these alternative ranking methods.
  • Third, we tested dependence on genre composition. We calculated each model’s mean within each genre and then weighted genres equally, so larger genres no longer dominate the overall score. This changes only positions 3–5, occupied by UI-TARS, EvoCUA, and Qwen3-VL; their genre-weighted means differ by at most 0.002 levels. Agreement with the original ranking remains high (Kendall τ = 0.93). Gemma-4-26B-A4B and Qwen3-Omni-30B-A3B remain first and second, and the bottom-three group is unchanged.

These results support two concrete conclusions: limited progress is evident without averaging the ordinal scores, and the broad model ordering persists across the tested ranking methods. The exact ordering of the three closely performing models depends on genre weighting, so we will qualify those comparisons.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Shared Response

We thank all three reviewers for their thoughtful feedback and constructive suggestions. In this shared response, we address their common questions regarding the reliability of our automated evaluation, including human validation, and stronger Video-LLM judges.

Revised judging protocol. The reviewers (oADm, Rhh7, Lxgt) wonder whether the automated judge can reliably assess gameplay progress. To improve the judge’s reliability, we revised the judging protocol to use game-specific rubrics that describe the progress unit, milestone and five-minute goal that a player should achieve. The judge uses these criteria together with the gameplay recording and timestamped action log to distinguish player achievements from passive score changes, opponents’ achievements, and misleading end screens. We believe this makes the criteria for awarding progress more explicit and the judgments easier to inspect.

Human verification of rubrics. To further examine whether these criteria are appropriate across different games (oADm, Rhh7), four human annotators reviewed rubrics for 100 games from the itch-200 subset. Annotators assessed whether each rubric was clear and consistent with the game's objectives, or whether any material criteria needed to be changed. Rubrics were rated as clear and appropriate for their respective games 90.4% of the time.

Expanded human validation. The reviewers (oADm, Lxgt) also highlight the importance of a larger human evaluation and more precise agreement measures. We therefore expanded our validation to 182 held-out gameplay recordings from 26 games, with nine annotators providing 280 progress ratings. Of these, 98 recordings received two independent annotations. Annotators were blinded to model identities and judge predictions. They assessed progress using the game-specific rubrics. Human–human exact agreement reached 90%, with quadratic-weighted κ = 0.84, indicating that independent annotators can apply the progress criteria consistently.

Stronger judge comparison. To address the requests for more detailed reliability measures and stronger judges (oADm, Rhh7, Lxgt), we evaluated the original judge alongside several rubric-based configurations. In addition to within-one-level agreement, we now report exact agreement and precision/recall at the Partial+ threshold.

JudgeExact ↑Within one ↑Partial+ precision ↑Partial+ recall ↑
Original 9B milestone47%92%47%64%
Rubric 9B52%74%35%91%
Rubric 9B (thinking)71%89%57%80%
Rubric 27B59%90%58%91%
Rubric 27B (thinking)78%96%86%73%

For a fair comparison between judges, the following table uses the same 259 human–judge comparisons with valid outputs across the evaluated judge configurations, drawn from the full panel of 280 human annotations.

The rubric-based 27B thinking judge improves exact agreement from 47% to 78% and Partial+ precision from 47% to 86% compared with the original judge. False-positive Partial+ judgments decrease from 32 to 5, while recall also improves slightly (64% to 73%).

Together, these results strengthen our evaluation in three complementary ways: human verification of the game-specific rubrics, more consistent progress ratings from independent human annotators, and improved agreement between the judge and human assessments. We will reflect these findings throughout the revised paper, updating the evaluation protocol, human validation, and benchmark results and analyses accordingly.

AuthorsOpenReview ↗

Shared Response (Part 2)

Following the improvements to our judging protocol and expanded human validation described in our first Shared Response, we recomputed the main results using the stronger 27B thinking judge, including the full leaderboard, genre analysis, subsampling experiments, and other evaluation settings.

Revised Table 2: Main Leaderboard

Mean Progress Score with (k=3) and without (k=0) observation history, and k=0 Partial+ rate.

Modelitch k=3itch k=0Partial+PyWeek k=3PyWeek k=0Partial+
VLMs
Qwen3-Omni-30B-A3B0.891.5646.7%0.801.2235.9%
Gemma-4-26B-A4B1.991.3138.7%1.491.0532.0%
Qwen2.5-VL-7B0.431.2336.8%0.431.0933.0%
Qwen3-VL-8B0.931.2336.5%0.801.1939.8%
Gemma-4-E4B1.001.0632.0%0.771.0132.4%
Qwen2.5-Omni-7B0.230.8724.9%0.370.9223.3%
InternVL2.5-8B0.240.7020.0%0.250.7419.4%
Gemma-3-12B0.450.6919.7%0.500.6818.4%
CUAs
UI-TARS-1.5-7B0.991.2436.9%0.831.0129.4%
EvoCUA-8B1.240.9727.9%0.991.0229.1%
OpenCUA-7B0.880.9526.5%0.860.9725.5%
GUI-Owl-1.5-8B0.740.8223.5%0.640.7722.5%
VLAs
OS-Atlas-Pro-7B0.380.6718.6%0.240.436.8%
ShowUI-2B0.280.4713.3%0.210.339.7%

Main findings. The revised leaderboard supports the paper's central finding that current models struggle to make meaningful progress across diverse, unseen games. Even the strongest model reaches Partial+ in only 46.7% of itch.io games, averaging 28.7% across models. Qwen3-Omni leads without history, while Gemma-4-26B performs best with history. Observation history generally reduces performance, although individual models benefit from it. CUAs remain competitive with similarly sized VLMs, while the two evaluated VLAs perform comparatively poorly without history.

Revised Table 3: Progress by Genre

Mean Progress Score and Partial+ rate across itch.io genres with at least 60 games (k=0).

GenreNMeanPartial+
Adventure2,1841.16935.2%
Simulation5061.10634.0%
Racing720.93126.8%
Strategy2230.91325.4%
Puzzle1,2710.90627.5%
Survival1710.87022.3%
Shooter2050.73015.8%
Action9580.70417.7%

Genre findings. The genre ordering changes from the submission: Adventure now ranks highest and Puzzle falls near the middle, while Shooter and Action rank lowest. To investigate this change, we compared both judges against human assessments. Across 77 human annotations from seven Adventure games, exact agreement improves from 48% to 87%, while recall of human-rated Partial+ progress increases from 17% to 75%. We will update the genre-level conclusions in the revised paper.

AuthorsOpenReview ↗

Shared Response (Part 3)

Updated Analyses of Evaluation Settings

We re-evaluated our experiments on observation history, prompting strategies, and action frequency using the revised judge. The table reports paired changes in mean Progress Score, with 95% game-bootstrap confidence intervals.

ComparisonPyWeek Δ [95% CI]itch-200 Δ [95% CI]
k=0 → 3−0.235 [−0.284, −0.188]−0.171 [−0.218, −0.126]
k=3 → 5−0.026 [−0.060, +0.009]−0.065 [−0.100, −0.029]
k=3 → 10−0.036 [−0.075, +0.003]−0.070 [−0.110, −0.033]
k=5 → 10−0.010 [−0.046, +0.029]−0.009 [−0.042, +0.024]
Strategy field−0.055 [−0.110, +0.001]−0.048 [−0.090, −0.003]
Few-shot examples−0.080 [−0.139, −0.022]−0.042 [−0.087, +0.009]
3 Hz vs. 1/3 Hz, k=0+0.202 [+0.138, +0.271]—
3 Hz vs. 1/3 Hz, k=3+0.177 [+0.117, +0.240]—

Effects of evaluation settings. The main findings remain consistent: additional history generally reduces performance, explicit strategy prompting and few-shot examples do not consistently improve progress, and higher action frequency improves performance on PyWeek. One detail changes: increasing history from five to ten observations no longer produces a meaningful additional decrease in either corpus. We will revise this analysis accordingly.

Subsampling stability. With 200 randomly sampled games, the median Spearman correlation with the full leaderboard is ρ = 0.982 (compared with 0.974 in the submission). However, the exact top-three set is recovered in only 47.5% of 200-game samples, reflecting the near-tie between the third- and fourth-ranked models. This supports the use of smaller subsets for broad model comparisons, while emphasizing the need for larger samples when distinguishing closely matched models.

Summary. Taken together with the expanded human validation and judge comparisons in our first Shared Response, these results provide stronger evidence for the reliability of PlaySuite's main findings. The revised evaluation preserves the central conclusions on limited gameplay progress, broad model rankings, history effects, and sampling stability, while correcting the genre-level interpretation. We will incorporate these results and corresponding qualifications throughout the revised manuscript.

We thank the reviewers for their feedback and constructive suggestions, which have helped us strengthen the evaluation and improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.