- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors provide a training framework for agentic imagination that allows vision language model-based agents to do model based planning. Within their framework, an agent receives an initial observation and plans K steps into the future, where each step consists of an action and a prediction of the next observation. The predicted observation is used as an input for the next planning step allowing optimistic linear plans as well as counterfactual reasoning over observations.
These actions are chunked when executed and there is no intermediary feedback until the next planning phase. They evaluate this work across 4 environments and test generalization to out of distribution and unseen environment instantiations, and show increased performance relative to the selected baselines under a broad range of conditions. Ablations show that the proposed framework relies heavily on the content of the predicted observations, counterfactual branching helps at critical decision points and long horizon planning helps in situations where early decisions could be detrimental. They also show that RL provides significant improvements beyond purely supervised fine tuning and that it can create an adaptive planning process. This provides a concrete foundation for visual based reasoning in VLM based agentic systems.
Critical strengths
There is a reasonable variety within the environments evaluated, testing various cases of long horizon planning. The paper is written well and easy to follow. The proposed framework is clear and utilizes appropriate SFT and RL methods. This work provides a clear and well structured foundation to approach planning through vision based reasoning in VLM based agentic systems.
Critical weaknesses
The main issue is that the literature surrounding model based planning is quite sophisticated for the environments tested, but their only baselines are their own no-imagine models and proprietary VLMs which may not be well suited for the task. Similar works are mentioned in the related works, but not directly compared.
This work also shares many similarities to Offline RL and Decision Transformers but is not adequately addressed in related work. There is, also a large overlap with belief-state planning for RL, that may provide a more standardized foundation for the planning processes discussed in the paper.
The stochastic nature of text and image generation is not addressed.
The results that are presented do not have error bars and it is unclear if any multi-seed validation was done for the results presented. This makes the results themselves potentially very brittle.
The novelty of this work in relation to prior work is not adequately addressed.
Questions for authors
Why is counterfactual reasoning only being done on alternative actions rather than alternative outcomes per action? Have you considered this as a possible cause of the failure in Pacman with its own predictions, where next observations are non-deterministic given the current action? Letting the model reason through its uncertainty over observations might provide better performance.
Why are stronger baselines, that might have better success in these tasks, like Dreamer, not evaluated for this work? How do more standard deep RL model-based planning baselines compare?
Can you explain some of the failure modes compared to Sol at K >= 10? It seems surprising that Maze and Sokoban have such high success rates for Sol in comparison.
How does this work compare to Transformer based world models that attempt to do a similar reasoning process without explicit image generation?
Is there any variability in the success rates, due to the stochastic nature of both text and image generation? Is there any multi-seed validation done on these environments?
6 comments
[W2] Offline trajectory learning and belief-state planning
Paper: Section 5; Appendix A.3 (Table 7; Figure 9).
Response.
We thank the reviewer for pointing out these connections. We have expanded our discussion and added comparisons with Decision Transformer (DT), Trajectory Transformer (TT), and Behavioral Cloning (BC).
Decision Transformer learns a return-conditioned action policy from offline trajectories, while Trajectory Transformer models trajectory sequences for planning. Agima also learns from offline expert trajectories, but explicitly trains the model to generate visual futures, evaluate their outcomes, and revise its plans based on these imagined observations. We additionally include behavioral cloning to distinguish the benefits of trajectory learning and return conditioning from those of visual imagination.
Table H2. Offline controls with same-checkpoint easy/hard results
| Method | State | Checkpoint | Sok easy/hard (%) | Maze easy/hard (%) |
|---|---|---|---|---|
| DT | Structured | Matched 40k | 5% / 0% | 10% / 0% |
| DT | Structured | Training-end | 56% / 0% | 41% / 0% |
| DT | Latent | Matched 40k | 29% / 0% | 56% / 0% |
| DT | Latent | Training-end | 67% / 0% | 63% / 0% |
| TT | Structured | Matched 40k | 0% / 0% | 0% / 0% |
| TT | Structured | Training-end | 21% / 0% | 41% / 3% |
| TT | Latent | Matched 40k | 0% / 0% | 0% / 0% |
| TT | Latent | Training-end | 21% / 0% | 19% / 0% |
| BC | Structured | Matched 40k | 0% / 0% | 0% / 0% |
| BC | Structured | Training-end | 7% / 0% | 1% / 3% |
| BC | Latent | Matched 40k | 14% / 0% | 35% / 0% |
| BC | Latent | Training-end | 38% / 0% | 47% / 0% |
DT and BC use K=1; TT uses K=5 with search width 1 and horizon K. Matched 40K denotes 40,000 complete expert-episode presentations. Training-end checkpoints follow the respective training budgets or stopping criteria and are not necessarily converged. “—” indicates an unfinished evaluation.
These results highlight Agima's substantially stronger easy-to-hard generalization. At the matched 40K training budget, all DT, TT, and BC baselines achieve 0% success on both hard tasks. Even with additional training, DT reaches 41–67% success on easy tasks after 176K–302K expert-episode presentations, yet still achieves 0% on both hard splits. TT and BC similarly achieve at most 3% hard success. In contrast, Agima achieves 29% on hard Sokoban and 47% on hard Maze2D with only 40K expert-episode presentations.
These results suggest that learning to imitate actions or model offline trajectories does not necessarily lead to generalization on harder planning problems. In contrast, Agima learns to explicitly imagine action consequences and reason over possible futures, enabling substantially stronger generalization with a smaller offline training budget.
Connection to belief-state planning. Agima is also related to belief-state planning, where an agent reasons about states that are not directly observed using its observation and action history. Agima similarly uses predicted visual futures as internal observations to guide planning between real environment interactions. However, unlike classical belief-state methods that maintain a probability distribution over possible states, Agima generates visual predictions without explicitly maintaining a calibrated posterior or performing Bayesian belief updates. We have clarified this distinction in the revised paper.
[W3, W4, Q5] Stochastic variability and multi-seed validation
Paper: Appendix A.5.
Response.
To evaluate the robustness of our results to stochastic inference, we conducted additional evaluations with three different inference seeds.
On Maze2D, Agima achieves 47%, 46%, and 44% success across the three seeds, compared to 10%, 10%, and 12% for No Imagine. Similarly, on Sokoban, Agima achieves 29%, 25%, and 24%, compared to 2%, 0%, and 1% for No Imagine.
The improvements are consistent across all three seeds: Agima achieves an average success rate of 45.7% vs. 10.7% on Maze2D, and 26.0% vs. 1.0% on Sokoban. These results demonstrate that the gains from visual imagination are robust to stochastic inference variability.
[W5] Novelty relative to prior work
Paper: Sections 2, 4, and 5.
Response.
We agree that using world models for planning is not new. Our novelty lies in unifying visual imagination and reasoning within a single omni model, and learning how to use imagination through agentic RL.
First, Agima unifies policy, world modeling, and outcome evaluation in a single omni model. To our knowledge, Agima is the first to train one omni model to carry out the complete agent loop: propose actions, generate their visual consequences, reason over its own predictions to evaluate possible outcomes, and revise its plans accordingly. Unlike approaches that rely on separate dynamics models and planning modules, Agima directly interleaves actions, imagined images, and language reasoning within the same model.
Second, Agima learns how to imagine through agentic RL. Rather than relying on a predefined planning procedure, Agima learns when, where, and how to imagine from task-success rewards. We observe that RL teaches the model to explore alternatives when imagined plans fail, focus branching on critical decisions, and even discover a new planning format that combines different imagination strategies without explicit demonstrations. This demonstrates that the organization of imagination itself can be learned and improved through RL.
Our contribution therefore goes beyond applying world-model planning to an omni model: we train an agent that can generate and reason over its own visual futures, and learn to adapt its imagination strategy to improve decision-making.
[Q1] Alternative outcomes for the same action
Paper: Appendix B; Table 2.
Response. “Same-action outcome sampling” means generating several candidate successor images for the same state and action, then comparing or aggregating them before acting. Under the evaluated greedy-ghost policy, Pacman’s transition is deterministic given the full state and action, so multiple samples probe decoding uncertainty rather than distinct environment outcomes. The Oracle intervention directly isolates successor fidelity: true successors raise success from 37% to 81%, identifying action-conditioned prediction accuracy as the main headroom.
[Q3] Failure modes versus GPT-5.6 Sol at long horizons
Paper: Appendix A.6 (Table 10).
Response.
A limitation of Agima at longer horizons is error accumulation in its imagined visual futures. Unlike text-based reasoning in GPT-5.6 Sol, Agima repeatedly generates visual successors and conditions subsequent planning on these predictions. At K≥10, small errors in predicted states can accumulate without real environment feedback, causing the agent to plan based on increasingly inaccurate observations.
We examine this through an Oracle successor intervention experiment reported in Appendixx A.6, where we replace generated images with ground-truth images while keeping the planning policy fixed. At K=∞, this improves success from 7% to 34% on Sokoban.
This suggests that world-model accuracy is a major bottleneck for Agima at long horizons. Improving long-horizon visual consistency is therefore an important direction for future work.
Response to Reviewer h8Zz
Thank you for recognizing the potential of visual reasoning for agentic planning. Agima makes imagination an adaptive reasoning process within one omni-model agent: it proposes actions, generates their visual consequences, and uses language reasoning to evaluate and compare the resulting plans. Agentic RL further teaches the model how to imagine—when to extend a rollout, reject a plan, or explore an alternative.
Our expanded experiments test this mechanism and its generalization benefits. With the same backbone, expert data, and training budget, visual imagination achieves 29%/47% on hard Sokoban/Maze2D, versus 10%/5% for latent imagination and 2%/10% for action-only planning. New comparisons with DreamerV3, Thinker, I2A, IMPALA, DT, and TT further demonstrate stronger transfer to harder, unseen configurations. Repeated inference evaluations show that the gains persist across sampling seeds. We address each concern below.
[W1, Q2] Task-specific model-based planning baselines
Paper: Appendix A.2 (Table 6; Figure 8).
Response. Thank you for identifying this gap. We have added comparisons with DreamerV3, Thinker, I2A, and IMPALA, covering both model-based planning and model-free RL approaches. The results are summarized below.
Table H1. Online baselines and Agima representation controls
Importantly, we verify our training process by testing that the online baselines have learned the easy tasks before evaluating their generalization to hard levels. We report easy/hard performance from the same checkpoints:
Table H1a. Verified same-checkpoint easy/hard checks
These results highlight Agima's stronger generalization to challenging planning tasks, alongside its potential for greater sample efficiency. On Sokoban, DreamerV3, Thinker, I2A, and IMPALA achieve 90–98% success on easy levels but only 0–2% on hard levels. Similarly, DreamerV3 achieves 99–100% on easy Maze2D but only 7–11% on hard Maze2D. These results show that existing online RL approaches can successfully learn the easier tasks but struggle to generalize to more challenging instances. In contrast, Agima achieves 29% on hard Sokoban and 47% on hard Maze2D, substantially outperforming these baselines despite receiving real observations only every five actions (K=5), compared to every action (K=1) for the online baselines.
Agima also shows promising data efficiency. The online baselines are trained for 10–50M environment transitions to achieve strong easy-task performance, yet their hard-task success remains limited. At earlier checkpoints of approximately 0.44M transitions, all baselines achieve 0% hard success and at most 8% easy success. In comparison, Agima achieves substantially higher hard-task success using only 40K offline expert-episode presentations. These results suggest that learning to plan through visual imagination can enable stronger easy-to-hard generalization with limited training data.