acceptodds
Withdrawn from ICLR 2027

LetsPlay: A Gameplay Trajectory Dataset for Visual Game Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Playing games through screenshots requires agents to interpret visual observations, remember past events, and execute actions toward a task objective. Training agents to acquire these capabilities requires gameplay trajectories that capture both what the agent observes and how it interacts with the environment. However, existing game benchmarks often rely on textual state descriptions or game-specific actions, while many visual game environments provide no gameplay trajectories for training. To address this, we introduce LetsPlay, a gameplay trajectory dataset for training visual game agents. The dataset aligns screenshots with recorded device actions using a unified action schema and further annotates each action with its reasoning, immediate goal, and expected outcome. In total, LetsPlay covers 25 games, comprising 2,036 gameplay trajectories and 315K action steps. Building on LetsPlay, we develop PlayAgent, a multimodal agent for long-horizon gameplay. The agent conditions its decisions on recent actions and gameplay context. Extensive experiments demonstrate the usefulness of LetsPlay for training visual game agents and the effectiveness of PlayAgent in completing game tasks.

Primary area:
datasets and benchmarks
Keywords:
Gameplay Trajectories, Visual Game Agents, Dataset Curation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mKTpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Let'sPlay, a gameplay trajectory dataset covering 25 games, 2,036 trajectories, and 315K action steps. Each step contains visual observations, device actions, and language annotations. The authors also develop PlayAgent by fine tuning a VLM with history windows, curriculum learning, and action weighted loss. Experiments show that PlayAgent achieves a 36.0% task success rate across 125 tasks.

Critical strengths
  1. Let'sPlay provides gameplay trajectories with visual observations, device actions, and language annotations across 25 games. This could be a useful resource for training visual game agents.
  2. PlayAgent substantially improves over its base model, showing the value of gameplay trajectory supervision.
Critical weaknesses
  1. Limited novelty. The proposed method mainly combines standard sft, curriculum learning, and weighted loss. The technical contribution is relatively simple, with limited methodological novelty.
  2. Insufficient experimental analysis.The paper lacks ablation studies for its key components, making it difficult to identify what drives the performance gains. The improvement over the strongest baseline is also modest, and generalization to unseen games is not evaluated.
  3. The paper lacks visualizations of the dataset, model architecture, and gameplay examples.
Questions for authors

Can the authors provide ablation studies to clarify the individual contributions of language annotations, curriculum learning, action loss weighting, and history windows? How well does PlayAgent generalize to unseen games that are not included in the training dataset?

Reviewer_oA8nOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a gameplay trajectory dataset across 25 games and 5 genres, with action annotations. It proposes a unified action representation. Besides expert model-based trajectories, human-based trajectories are also incporated. Furthermore, this paper proposes practical techniques to improve supervised fine-tuning on the proposed dataset. Results show that a Qwen3-8B model, after training on the proposed dataset, achieves competitive performance against the strong Qwen-3.8-Max model.

Critical strengths

The dataset covers a diverse range of games and genres. It provides a unified action representation and includes action annotations, a feature that is very useful but absent in comparable datasets. The action representation is general and well-defined. The proposed techniques are well-motivated and address practical domain challenges.

Critical weaknesses

The effectiveness of the proposed curriculum learning is not explicitly verified. It is unclear whether this method improves final performance or learning efficiency.

Questions for authors
  1. Could the authors provide an ablation study regarding the curriculum learning component, i.e., a "PLAYAGENT without curriculum" baseline?
  2. What proportion of the data was collected from human players?
Reviewer_WsT9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is a visual game dataset paper. The authors just collect a small-scale dataset with screenshots and annotate each action with its reasoning, immediate goal, and expected outcome. The data contribution is trivial, and there are no insights in the experimental results.

Critical strengths
  1. The paper introduces a unified gameplay trajectory dataset covering 25 games, with screenshots, executable actions, and language annotations, which may be useful for research on visual game agents.

  2. The evaluation covers multiple game genres and compares against several strong proprietary and open-source models, providing an initial assessment of the dataset’s utility.

Critical weaknesses
  1. The empirical evidence does not convincingly establish the effectiveness of the proposed method. PLAYAGENT achieves only a marginal improvement over the strongest reported baseline (36.0% vs. 33.6%), without sufficient ablations to isolate the contributions of trajectory-length curriculum, action-class loss weighting, and language-based annotations. The evaluation metric—counting a task as successful if any of three attempts succeeds—also obscures per-episode reliability.

  2. The work is limited in scale, with only 2,036 trajectories across 25 games. Combined with the relatively trivial methodological contributions—standard supervised fine-tuning, curriculum learning, and action-weighted loss—the experimental results are insufficient to establish a significant advance.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.