acceptodds
Under review as a conference paper at ICLR 2027

Learning Agentic Vision-Language Game Agents from Trajectory-Derived Supervision

Abstract

Vision-language game agents are typically trained to predict the next action from gameplay demonstrations in a supervised fine-tuning (SFT) manner. While effective for learning short-horizon behavior, this training view underuses the process structure already present in trajectories, including how plans persist across multiple steps, how actions are carried out under a plan, and how execution outcomes signal when a new plan is needed. Gameplay trajectories contain richer supervision than action labels alone. When organized into plan-conditioned segments, they naturally specify how an agent should form a plan, act under that plan over time, and track whether execution is still aligned with it. In this paper, we propose a trajectory-to-process supervision framework for learning plan-act-verify vision-language game agents from gameplay trajectories. Unlike conventional CoT-before-action training, where plans mainly serve as intermediate prefixes for action prediction, our formulation explicitly trains the model to carry out a plan across multiple steps and to generate the next plan when the current one succeeds, fails, or is no longer valid. The resulting agent moves beyond direct acting toward closed-loop gameplay, using trajectory-derived process annotations as supervision for agentic behavior. Across Tetris, Sokoban, and MiniGrid, our method consistently improves over direct action prediction, step-wise plan-as-CoT, and open-loop plan-chunk baselines, and transfers across Qwen3-VL-8B, InternVL3-8B, and Gemma3-12B backbones.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Agents, Imitation Learning, Hierarchical Planning, Plan-Conditioned Policies, Game Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fKinOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a trajectory supervision framework for training vision language game agents to plan, act, and verify their execution. The method segments gameplay trajectories into local plans and constructs supervision signals for action prediction and execution status verification. Experiments on Tetris, Sokoban, and MiniGrid show improvements over direct action prediction, Plan-as-CoT, and Plan-Chunk baselines across multiple vision language backbones.

Critical strengths
  1. The paper presents a straightforward way to extract planning and verification supervision from existing gameplay trajectories.
  2. The method outperforms several baselines across three games and multiple model backbones. The ablation studies also provide useful evidence for the proposed training strategy.
Critical weaknesses
  1. Limited novelty.The main approach largely combines existing ideas in hierarchical planning, imitation learning, and execution verification. The technical contribution is relatively incremental, with limited new insights beyond restructuring trajectory data for supervised fine tuning.
  2. Limited evaluation complexity and generalization.The experiments focus on three relatively simple and structured games. It remains unclear whether the proposed method can generalize to a broader range of games, which deserves further investigation.
Questions for authors

What is the key technical distinction between the proposed framework and existing hierarchical imitation learning or plan conditioned policy learning methods? Can the authors provide evidence that the method works in more complex environments?

1 comment

The authors did not submit a rebuttal, so I'm going to keep my original score unchanged.

Reviewer_Q1xEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a trajectory-to-process supervision method that extends game trajectories from conventional step-by-step observation-action supervision to process supervision comprising planning, plan-conditioned execution, and verification. The paper segments expert trajectories according to local plans and constructs alternative-plan, alternative-action, and incompatible-plan samples, enabling the model to learn when to appropriately maintain the current plan, how to perform failure recognition, and how to conduct dynamic replanning. A distinctive design is that, during training, selective loss masking is applied to artificially constructed erroneous actions, so that only the corresponding state and verification signals are trained, rather than training the model to learn the erroneous actions. Experiments on Tetris, Sokoban, and MiniGrid show that the method outperforms Direct Action, Plan-as-CoT, and Plan-Chunk, and can transfer to multiple VLM backbones. The authors also conduct ablation studies validating the effects of various types of process supervision and verifier-controlled replanning.

Critical strengths
  1. The design of negative-sample supervision and selective loss masking is somewhat innovative. By constructing failure or plan-invalidation samples through alternative-action and incompatible-plan approaches, the authors train the model to learn failure recognition and replanning conditions. At the same time, to prevent the model from learning erroneous actions, they mask the loss on action tokens in these samples and retain only supervision over state and verification signals. This design is reasonable, and ablation experiments also demonstrate its impact on performance.

  2. The paper constructs process-level supervision from existing expert trajectories; this idea is practical and the experimental validation is complete. The paper extends conventional game trajectories into training data that include plan-conditioned execution, failure recognition, and plan-invalidation judgment, and validates the approach on multiple game environments and VLM backbones. It also conducts various ablation experiments, forming a relatively complete chain of evidence that supports the view that process supervision helps final performance.

Critical weaknesses
  1. The paper's core novelty is relatively limited; the current work is more like a supervised recombination of existing ideas than a new agentic learning formulation. Persistent subgoal/plan-conditioned execution is already closely related to hierarchical imitation and planner-executor approaches, and verification-driven replanning is also related to existing work such as progress monitoring, verifier-based VLA, and closed-loop execution. The main contribution of this paper is to unify these existing ideas into a single VLM through trajectory-derived supervision, combined with counterfactual sample construction and selective loss masking. Although the latter is a interesting design, I still think the overall novelty is insufficient.

  2. The supervision recipe proposed in the paper relies heavily on environment structure, and its generality has not been fully demonstrated. The current method depends on environment-specific solvers, symbolic states, and hand-designed rules to define segments, plans, alternative actions, and verification labels. This means that the paper has so far only verified that “this scheme is effective in structured games where process labels can be programmatically constructed,” but has not verified that “this scheme can learn agentic behavior from general gameplay trajectories.” This limits the value of this innovation relative to existing hierarchical / verifier-based methods.

Questions for authors
  1. Please clarify the essential differences between this paper and hierarchical imitation / goal-conditioned imitation, planner-executor style policies, and verifier/progress-aware replanning methods. Aside from unifying these components into a single VLM and training it via trajectory-derived supervision, does this paper introduce any new algorithmic capability or training principle? This will greatly influence my judgment of the paper's novelty.

  2. I think that the most valuable design in this paper is primarily selective loss masking, whereas plan-conditioned execution and verification-driven replanning already have considerable related work. Please clearly state what the authors consider to be the core and most innovative technical contribution, and provide a more direct comparison with the closest methods.

  3. The current process supervision relies on solvers, symbolic states, and hand-designed rules. In ordinary gameplay trajectories that lack such structured information, can this information be reliably constructed? Without such information, does the advantage of this method over existing imitation learning / verifier-based approaches still hold?

  4. The main results in the paper appear not to report the means and variances over multiple independent training runs. Please clarify whether the results in Table 1, 3, 4, 5, and 6 come from a single training run; if multi-seed results are available, please provide the corresponding statistics. In Table 6, the performance under 10% and 30% corruption is actually higher than the performance under the 0% setting, so the result variance will directly affect my judgment of whether these differences are actually meaningful.

1 comment

Since the authors have not provided a response to my concerns, I will maintain my original rating.

Reviewer_HuXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a training framework to construct process supervision from game trajectories. The author cut trajectories through task structures, and generate plan label based on segment terminate state, and then annotate four execution states for each step. The same VLM takes the roles of planner and actor-verifier, and decide to continue execution or replan at inference based on the predicted execution status. The training added three augmented sample and loss masking. At Tetris, Sokoban and MiniGrid, the proposed method outperforms Direct Action, Plan-as CoT and Plan-Chunk.

Critical strengths

1.The formulation is clean and motivation is clear. The paper models plan generation, plan execution, and plan termination individually. The plan stays effective across different steps, and the learned verifier decides when to replan. This is closer to real scenario in long-horizon control compared to plan-as-CoT which re-plans every step.

2.The improvement is obvious. On Qwen3-vl-8B, Tetris increases from 17.81 to 62.42 lines, and sokoban raises from 64% to 86% compared to direct action. The method also outperforms plan-as-cot and plan-chunk.

3.Control experiments are effective. The matched multi-task baseline in Table 9 rules out the explanation that that the improvement is just due to more samples. Table 5 uses a fixed checkpoint and only changes replanning strategies to evaluate the effectiveness of predicted status.

Critical weaknesses
  1. There are no comparisons to hindsight relabeling and option termination. Generating plan labels using segment termination state is fundamentally hindsight goal relabling (HER). The four-state verifier corresponds to the termination function in options framework (Sutton et al., 1999; Bacon et al., 2017), and only adding two negative classes. Success detection is also related.

  2. State description and two-image input has confounding factors. Only this method outputs current state description s_j and next state prediction s_{j+1} and there none of the baselines do. In tasks like Tetris, reading out coordinates itself is strong grounding and forward-model supervision, and may have contributed to the majority of benefits. Also, actor-verifier inputs two images and baseline only inupt one image. Table 9 controls the sample count and does not explicitly control output format and input structure.

  3. Some results might weaken the paper claim. For example , In Table 7, Minigrid random cutting is higher than cutting based on task structure. So what takes effect might be the hindsight subgoal conditioning, rather than the task structure in the trajectories. In table 6, the scores are even higher after 10% and 30% corruption (66.61, 65.95 vs 62.42), it may mean that “random plans with correctly derived actions” are themselves equivalent to alternative-plan augmentation.

Questions for authors
  1. What would be the result if we remove s_j and s_{j+1}? What will be the gap be if the same state description is added to the baselines?

  2. What is the core methodological difference of this work compared to hindsight goal relabeling and option termination?

  3. How is the label of corrupted samples generated in Table 6? Why the performance improves after corruption?

  4. Could the authors provide multi-seed mean and variance for the main result?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.