acceptodds
Under review as a conference paper at ICLR 2027

Learning To Plan with Agentic Imagination

Abstract

A single push in Sokoban can make a puzzle unsolvable. Avoiding such errors requires anticipating consequences before acting, which is especially hard when feedback is sparse and actions cannot be undone. We introduce Agima, a training framework that teaches an agent to imagine before it acts. A single omni model both proposes actions and generates images of where they lead, then treats these imagined frames as observations. It can follow a plan forward, test an alternative move, or compare competing plans, all before touching the real environment. Supervised fine-tuning teaches these forms of imagination, and reinforcement learning then optimizes the model's textual planning decisions using only binary task-success rewards. Across four environments spanning Maze2D, Sokoban, Pacman, and PushT, Agima plans in harder or unseen configurations excluded from training. With supervised fine-tuning alone on two-box Sokoban, it solves 40% of four-box puzzles while seeing the real board only once every five moves; an action-only planner fine-tuned from the same backbone solves 2%. After RL, Agima solves more held-out Sokoban puzzles and adapts how it uses imagination: it drafts a second plan mainly when it foresees the first one failing, and branches more often at critical decisions. An agent that can picture where its actions lead can learn when and where to look ahead, in other words, to plan.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic imagination, visual planning, reasoning, multimodal agent, model-based planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_h8ZzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors provide a training framework for agentic imagination that allows vision language model-based agents to do model based planning. Within their framework, an agent receives an initial observation and plans K steps into the future, where each step consists of an action and a prediction of the next observation. The predicted observation is used as an input for the next planning step allowing optimistic linear plans as well as counterfactual reasoning over observations.

These actions are chunked when executed and there is no intermediary feedback until the next planning phase. They evaluate this work across 4 environments and test generalization to out of distribution and unseen environment instantiations, and show increased performance relative to the selected baselines under a broad range of conditions. Ablations show that the proposed framework relies heavily on the content of the predicted observations, counterfactual branching helps at critical decision points and long horizon planning helps in situations where early decisions could be detrimental. They also show that RL provides significant improvements beyond purely supervised fine tuning and that it can create an adaptive planning process. This provides a concrete foundation for visual based reasoning in VLM based agentic systems.

Critical strengths

There is a reasonable variety within the environments evaluated, testing various cases of long horizon planning. The paper is written well and easy to follow. The proposed framework is clear and utilizes appropriate SFT and RL methods. This work provides a clear and well structured foundation to approach planning through vision based reasoning in VLM based agentic systems.

Critical weaknesses

The main issue is that the literature surrounding model based planning is quite sophisticated for the environments tested, but their only baselines are their own no-imagine models and proprietary VLMs which may not be well suited for the task. Similar works are mentioned in the related works, but not directly compared.

This work also shares many similarities to Offline RL and Decision Transformers but is not adequately addressed in related work. There is, also a large overlap with belief-state planning for RL, that may provide a more standardized foundation for the planning processes discussed in the paper.

The stochastic nature of text and image generation is not addressed.

The results that are presented do not have error bars and it is unclear if any multi-seed validation was done for the results presented. This makes the results themselves potentially very brittle.

The novelty of this work in relation to prior work is not adequately addressed.

Questions for authors

Why is counterfactual reasoning only being done on alternative actions rather than alternative outcomes per action? Have you considered this as a possible cause of the failure in Pacman with its own predictions, where next observations are non-deterministic given the current action? Letting the model reason through its uncertainty over observations might provide better performance. 

Why are stronger baselines, that might have better success in these tasks, like Dreamer, not evaluated  for this work? How do more standard deep RL model-based planning baselines compare?

Can you explain some of the failure modes compared to Sol at K >= 10? It seems surprising that Maze and Sokoban have such high success rates for Sol in comparison.

How does this work compare to Transformer based world models that attempt to do a similar reasoning process without explicit image generation?

Is there any variability in the success rates, due to the stochastic nature of both text and image generation? Is there any multi-seed validation done on these environments?

6 comments
AuthorsOpenReview ↗

Response to Reviewer h8Zz

Thank you for recognizing the potential of visual reasoning for agentic planning. Agima makes imagination an adaptive reasoning process within one omni-model agent: it proposes actions, generates their visual consequences, and uses language reasoning to evaluate and compare the resulting plans. Agentic RL further teaches the model how to imagine—when to extend a rollout, reject a plan, or explore an alternative.

Our expanded experiments test this mechanism and its generalization benefits. With the same backbone, expert data, and training budget, visual imagination achieves 29%/47% on hard Sokoban/Maze2D, versus 10%/5% for latent imagination and 2%/10% for action-only planning. New comparisons with DreamerV3, Thinker, I2A, IMPALA, DT, and TT further demonstrate stronger transfer to harder, unseen configurations. Repeated inference evaluations show that the gains persist across sampling seeds. We address each concern below.

[W1, Q2] Task-specific model-based planning baselines

Paper: Appendix A.2 (Table 6; Figure 8).

Response. Thank you for identifying this gap. We have added comparisons with DreamerV3, Thinker, I2A, and IMPALA, covering both model-based planning and model-free RL approaches. The results are summarized below.

Table H1. Online baselines and Agima representation controls

MethodDataKSokoban hard (%)Maze2D hard (%)
DreamerV3, native reward10M10%11%
DreamerV3, terminal reward10M10%7%
Thinker36.17M12%—
I2A50M10%—
IMPALA48.86M10%0%
Agima40K529%47%
No Imagine / action planning40K52%10%

Importantly, we verify our training process by testing that the online baselines have learned the easy tasks before evaluating their generalization to hard levels. We report easy/hard performance from the same checkpoints:

Table H1a. Verified same-checkpoint easy/hard checks

MethodTaskCheckpoint / logged interactionsEasy (%)Hard (%)
DreamerV3 nativeSokoban10M91%0%
DreamerV3 terminalSokoban10M90%0%
DreamerV3 nativeMaze2D10M100%11%
DreamerV3 terminalMaze2D10M99%7%
ThinkerSokoban36,168,80096%2%
I2ASokoban50M92%0%
IMPALASokoban48,857,42498%0%
IMPALAMaze2D48,857,0880%0%

These results highlight Agima's stronger generalization to challenging planning tasks, alongside its potential for greater sample efficiency. On Sokoban, DreamerV3, Thinker, I2A, and IMPALA achieve 90–98% success on easy levels but only 0–2% on hard levels. Similarly, DreamerV3 achieves 99–100% on easy Maze2D but only 7–11% on hard Maze2D. These results show that existing online RL approaches can successfully learn the easier tasks but struggle to generalize to more challenging instances. In contrast, Agima achieves 29% on hard Sokoban and 47% on hard Maze2D, substantially outperforming these baselines despite receiving real observations only every five actions (K=5), compared to every action (K=1) for the online baselines.

Agima also shows promising data efficiency. The online baselines are trained for 10–50M environment transitions to achieve strong easy-task performance, yet their hard-task success remains limited. At earlier checkpoints of approximately 0.44M transitions, all baselines achieve 0% hard success and at most 8% easy success. In comparison, Agima achieves substantially higher hard-task success using only 40K offline expert-episode presentations. These results suggest that learning to plan through visual imagination can enable stronger easy-to-hard generalization with limited training data.

[W2] Offline trajectory learning and belief-state planning

Paper: Section 5; Appendix A.3 (Table 7; Figure 9).

Response.

We thank the reviewer for pointing out these connections. We have expanded our discussion and added comparisons with Decision Transformer (DT), Trajectory Transformer (TT), and Behavioral Cloning (BC).

Decision Transformer learns a return-conditioned action policy from offline trajectories, while Trajectory Transformer models trajectory sequences for planning. Agima also learns from offline expert trajectories, but explicitly trains the model to generate visual futures, evaluate their outcomes, and revise its plans based on these imagined observations. We additionally include behavioral cloning to distinguish the benefits of trajectory learning and return conditioning from those of visual imagination.

Table H2. Offline controls with same-checkpoint easy/hard results

MethodStateCheckpointSok easy/hard (%)Maze easy/hard (%)
DTStructuredMatched 40k5% / 0%10% / 0%
DTStructuredTraining-end56% / 0%41% / 0%
DTLatentMatched 40k29% / 0%56% / 0%
DTLatentTraining-end67% / 0%63% / 0%
TTStructuredMatched 40k0% / 0%0% / 0%
TTStructuredTraining-end21% / 0%41% / 3%
TTLatentMatched 40k0% / 0%0% / 0%
TTLatentTraining-end21% / 0%19% / 0%
BCStructuredMatched 40k0% / 0%0% / 0%
BCStructuredTraining-end7% / 0%1% / 3%
BCLatentMatched 40k14% / 0%35% / 0%
BCLatentTraining-end38% / 0%47% / 0%

DT and BC use K=1; TT uses K=5 with search width 1 and horizon K. Matched 40K denotes 40,000 complete expert-episode presentations. Training-end checkpoints follow the respective training budgets or stopping criteria and are not necessarily converged. “—” indicates an unfinished evaluation.

These results highlight Agima's substantially stronger easy-to-hard generalization. At the matched 40K training budget, all DT, TT, and BC baselines achieve 0% success on both hard tasks. Even with additional training, DT reaches 41–67% success on easy tasks after 176K–302K expert-episode presentations, yet still achieves 0% on both hard splits. TT and BC similarly achieve at most 3% hard success. In contrast, Agima achieves 29% on hard Sokoban and 47% on hard Maze2D with only 40K expert-episode presentations.

These results suggest that learning to imitate actions or model offline trajectories does not necessarily lead to generalization on harder planning problems. In contrast, Agima learns to explicitly imagine action consequences and reason over possible futures, enabling substantially stronger generalization with a smaller offline training budget.

Connection to belief-state planning. Agima is also related to belief-state planning, where an agent reasons about states that are not directly observed using its observation and action history. Agima similarly uses predicted visual futures as internal observations to guide planning between real environment interactions. However, unlike classical belief-state methods that maintain a probability distribution over possible states, Agima generates visual predictions without explicitly maintaining a calibrated posterior or performing Bayesian belief updates. We have clarified this distinction in the revised paper.

[W3, W4, Q5] Stochastic variability and multi-seed validation

Paper: Appendix A.5.

Response.

To evaluate the robustness of our results to stochastic inference, we conducted additional evaluations with three different inference seeds.

On Maze2D, Agima achieves 47%, 46%, and 44% success across the three seeds, compared to 10%, 10%, and 12% for No Imagine. Similarly, on Sokoban, Agima achieves 29%, 25%, and 24%, compared to 2%, 0%, and 1% for No Imagine.

The improvements are consistent across all three seeds: Agima achieves an average success rate of 45.7% vs. 10.7% on Maze2D, and 26.0% vs. 1.0% on Sokoban. These results demonstrate that the gains from visual imagination are robust to stochastic inference variability.

[W5] Novelty relative to prior work

Paper: Sections 2, 4, and 5.

Response.

We agree that using world models for planning is not new. Our novelty lies in unifying visual imagination and reasoning within a single omni model, and learning how to use imagination through agentic RL.

First, Agima unifies policy, world modeling, and outcome evaluation in a single omni model. To our knowledge, Agima is the first to train one omni model to carry out the complete agent loop: propose actions, generate their visual consequences, reason over its own predictions to evaluate possible outcomes, and revise its plans accordingly. Unlike approaches that rely on separate dynamics models and planning modules, Agima directly interleaves actions, imagined images, and language reasoning within the same model.

Second, Agima learns how to imagine through agentic RL. Rather than relying on a predefined planning procedure, Agima learns when, where, and how to imagine from task-success rewards. We observe that RL teaches the model to explore alternatives when imagined plans fail, focus branching on critical decisions, and even discover a new planning format that combines different imagination strategies without explicit demonstrations. This demonstrates that the organization of imagination itself can be learned and improved through RL.

Our contribution therefore goes beyond applying world-model planning to an omni model: we train an agent that can generate and reason over its own visual futures, and learn to adapt its imagination strategy to improve decision-making.

[Q1] Alternative outcomes for the same action

Paper: Appendix B; Table 2.

Response. “Same-action outcome sampling” means generating several candidate successor images for the same state and action, then comparing or aggregating them before acting. Under the evaluated greedy-ghost policy, Pacman’s transition is deterministic given the full state and action, so multiple samples probe decoding uncertainty rather than distinct environment outcomes. The Oracle intervention directly isolates successor fidelity: true successors raise success from 37% to 81%, identifying action-conditioned prediction accuracy as the main headroom.

[Q3] Failure modes versus GPT-5.6 Sol at long horizons

Paper: Appendix A.6 (Table 10).

Response.

A limitation of Agima at longer horizons is error accumulation in its imagined visual futures. Unlike text-based reasoning in GPT-5.6 Sol, Agima repeatedly generates visual successors and conditions subsequent planning on these predictions. At K≥10, small errors in predicted states can accumulate without real environment feedback, causing the agent to plan based on increasingly inaccurate observations.

We examine this through an Oracle successor intervention experiment reported in Appendixx A.6, where we replace generated images with ground-truth images while keeping the planning policy fixed. At K=∞, this improves success from 7% to 34% on Sokoban.

This suggests that world-model accuracy is a major bottleneck for Agima at long horizons. Improving long-horizon visual consistency is therefore an important direction for future work.

Reviewer_bXsZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In agentic and embodied applications, there could be irreversible errors or mistakes, and avoiding such mistakes requires foreseeing consequence before acting.

To this end, the authors introduce Agima, a training framework that teaches an agent to imagine before it acts. Agima proposes actions and generates images of where they lead, and then use these images as observations. Prior to executing a plan, Agima can follow a plan forward, test an alternative move or compare competing plans.

To train Agima, the authors consider both SFT (to generate or imagine the future states, for world modeling) and RL with GRPO (to optimize agent's planning).

The experimental evaluation shows the promising results obtained by Agima compared to the baselines, and RL could further the performance.

Critical strengths
  • The topic of look ahead for reasoning and planning is very interesting for agentic and embodied applications.
  • Two stage training of SFT and RL contributed to performance improvement.
  • Their findings for adaptive reasoning through Agentic RL are interesting; particularly Forward Thinking Becomes Adaptive and Branching Becomes More Exploratory and Targeted
Critical weaknesses
  • Novelty: I am not sold that the core idea behind this work is new, e.g., imagine-then-plan. This idea of looking ahead for reasoning and planning has been around for both Agentic AI and Embodied AI applications.
  • Compute: I have concerns around the efficiency of this framework. It would be appreciated to see some runtime numbers for the evaluation.
  • Performance:
    • The performance relies on imagination or the world model strongly.
    • I wish the authors would have provided more in-depth analysis of their results. For example, how could we improve the success rates (reported in Table 3) further.
Questions for authors
  • Could you explain why No Image performs better than Stale in Table 2?
  • How does counterfactual branching work? How do you identify local alternatives?
  • I am wondering why you didn't consider other domains for your experiments?
  • Similarly, no looking ahead baseline is seen. How about CoT-based baselines?
  • Did you run the baselines with their thinking mode being enabled? If not, I am wondering what would be their performance.
  • How come there is a huge gap between Own Predictions and Oracle results for Pacman but not Sokoban in Table 2?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bXsZ

[W1] Novelty of imagine-then-plan

Paper: Sections 2, 4, and 5.

We agree that imagine-then-plan is not a new concept, and we do not claim novelty in this general paradigm. Our contributions lie in how imagination is implemented and learned within a unified agent:

  • Unifying policy, world modeling, and outcome evaluation in a single omni model. To our knowledge, Agima is the first to train a single omni model to carry out the complete agent loop: propose actions, generate visual predictions of their outcomes, evaluate these imagined futures, and revise its plans based on its own generated observations. Rather than relying on separate policy, world, and reward models, Agima integrates these capabilities within a single model that can reason directly over its own imagined futures.
  • Learning how to imagine through agentic RL. Prior world-model planners typically rely on predefined planning procedures to search over predicted futures. In contrast, Agima can reason about its imagined outcomes and learn when, where, and how to imagine through agentic RL. In particular, we observe that RL teaches Agima to explore alternatives when an imagined plan fails, focus branching on critical decisions, and even discover a new planning format that combines different imagination strategies without explicit supervision. These behaviors demonstrate that imagination itself can become an adaptive, learned reasoning process, rather than a fixed procedure.

Our novelty therefore lies in training a unified visual agent that can reason over its own imagined futures and learn how to use imagination for planning, rather than proposing imagine-then-plan as a new concept.

We have revised the paper to clarify this distinction and strengthened our comparison with prior imagination-based agents and world-model planners.

[W2] Runtime and interaction cost

Paper: Appendix A.5 (Table 9).

Response.

We agree that generating imagined images can introduce additional inference cost. First, we show that this cost can be substantially reduced through step distillation without sacrificing performance.

With step distillation, we reduce the number of image-flow evaluations from 49 to 8 without losing performance. On the same ten hard Sokoban levels, this reduces inference latency from 4.99 s to 2.47 s per executed action (2.02× speedup).

We also quantify the inference overhead of the original model. Without step distillation, mean per-episode latency is 30.71 s (No Imagine) and 128.94 s (Agima) on hard Sokoban. While the original Agima does have an overhead, our step-distillation results demonstrate that imagination cost can be significantly reduced without compromising task performance, suggesting a promising direction for making imagination-based planning more computationally effective.

The motivation behind Agima is to trade additional computation for better decisions before costly or irreversible environment interactions. Internal simulation can be particularly valuable when real-world execution, resets, or failures are expensive. Furthermore, Agima supports different action-chunk lengths K, allowing practitioners to trade off imagination cost against the frequency of real-world observations.

The revised paper reports detailed latency and step-distillation measurements in Appendix A.5 (Table 9).

[W3] World-model quality and how to improve success

Paper: Section 3.1 (Table 2); Appendix A.6 (Table 10).

Response.

We expanded our successor intervention study to all four tasks in the original Table 2 to better understand the role of visual imagination and the remaining bottlenecks in Agima.

Specifically, we compare Agima's own predicted successor images against three interventions: (1) replacing them with blank frames, (2) replacing them with stale frames, and (3) replacing them with ground-truth successor images (Oracle GT). All interventions use the same trained Agima model, while No Imagine is a separately trained baseline. The results are reported below at K=5. We also updated the Table 2 in the paper.

Table B2. Agima-A7B successor intervention (K=5)

Model / conditionSokoban (%)Pacman (%)Maze2D (%)PushT (%)
No Imagine2%1%10%1%
Agima: own predictions29%37%47%18%
Agima: blank frames1%0%8%1%
Agima: stale frames0%0%8%0%
Agima: Oracle GT30%81%79%47%

First, Agima relies on the visual information in its imagined futures. Replacing predicted images with blank or stale frames causes performance to collapse across all four tasks. This confirms that the generated visual successors provide important information for planning, rather than merely serving as intermediate outputs in the reasoning process.

Second, improving world-model accuracy offers substantial headroom. Replacing Agima's predictions with ground-truth successors improves success rates from 37% to 81% on Pacman, 47% to 79% on Maze2D, and 18% to 47% on PushT. These tasks pose different prediction challenges: Pacman requires modeling ghost motion and changing game states, Maze2D requires maintaining spatial consistency, and PushT involves continuous contact dynamics. In contrast, the improvement on Sokoban is much smaller (29% to 30%). These results suggest that improving visual prediction accuracy could substantially improve Agima's performance, particularly on tasks with more challenging dynamics.

Third, accurate predictions alone are not sufficient for successful planning. Even with ground-truth successor images, Agima does not achieve perfect success on any task. This suggests that, beyond predicting accurate futures, the agent must also learn to interpret imagined outcomes, compare alternatives, and select effective actions. This motivates our use of forward thinking and task-success RL to improve how the agent reasons over its imagined futures, alongside improvements to the world model.

We further examine this effect in long-horizon imagination, where prediction errors can accumulate over multiple steps. By replacing the self predicted images with Oracles, on Sokoban, success improves from 16% to 29% at K=10 and from 7% to 34% at K=∞. These results further highlight the importance of accurate visual predictions for long-horizon planning. (See Appendix A.6)

Together, these experiments identify two complementary directions for improving Agima: better prediction of imagined futures and better reasoning over those futures. Promising approaches include more diverse action-conditioned training data, improved consistency across long imagined rollouts, and more frequent grounding in real observations to reduce prediction drift.

[Q1] Why No Imagine can outperform Stale

Paper: Section 3.1 (Table 2).

Response. The two controls answer different questions. No Imagine is separately trained to plan action sequences without intermediate images; it tests how much the visual-imagination training contributes beyond action planning from the same backbone and task data. It can reason implicitly without receiving a claimed visual successor.

Stale instead intervenes on the visual inputs of an imagination-trained Agima model, supplying an outdated frame in place of a predicted successor. It tests whether the content of those intermediate images matters. Unlike having no intermediate image, receiving an incorrect state can actively mislead subsequent decisions and create a mismatch with the expected successor input. This explains why Stale can be worse.

[Q2] How counterfactual training selects local alternatives

Paper: Appendix C.2.

Response. Appendix C.2 now documents the verified data-builder provenance. We scan expert Sokoban trajectories for critical states where the expert push preserves solvability and another legal push is certified unsolvable. At up to the three earliest such states, the trace inserts the alternative action and visual successor, rejects the harmful branch, and returns to the expert continuation.

This local construction differs from forward thinking: two candidate plans start from the same initial state with different first pushes, search supplies their continuations, and BFS labels each as solved, unfinished at the action limit, or deadlocked. Candidate order is randomized, and the shorter successful plan is preferred when both succeed. Appendix C.2 records the construction and provenance.

At inference, the learned policy chooses when to branch and which alternative to propose; the supervision-time solver does not select test-time actions.

[Q3] Domain choice and broader relevance

Paper: Appendix A.4 (Table 8); Appendix B.

Response.

To further evaluate Agima's generalization beyond the four original domains, we conducted additional experiments on Procgen Heist, a visually richer environment that requires planning directly from RGB observations.

We evaluate on 400 unseen levels disjoint from training, where Agima successfully completes 163/400 levels (40.75%), compared to 83/400 (20.75%) for No Imagine. This represents a 20 percentage point improvement, nearly doubling the success rate. For a fair comparison, both models receive real environment feedback every five actions and use the same SFT training windows and budget. These results provide additional evidence that visual imagination improves planning in more diverse, unseen environments.

Furthermore, our original four domains were selected to cover distinct planning challenges: Maze2D tests spatial navigation, Sokoban requires avoiding irreversible actions and deadlocks, Pacman involves reasoning about moving entities, and PushT requires continuous, contact-dependent control. PushT is also closely related to visuomotor manipulation tasks studied in prior robot-learning work [Chi et al., 2023].

Together, these five environments evaluate imagination-based planning across spatial navigation, irreversible decisions, dynamic interactions, and continuous control. The controlled simulators also allow us to isolate the contribution of imagination through oracle-state interventions and fixed observation intervals. While real-world deployment remains future work, these results demonstrate the potential of Agima across a range of visual planning tasks, with applications to navigation and manipulation.

[Chi et al., 2023] Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.” Robotics: Science and Systems (RSS), 2023. https://doi.org/10.15607/RSS.2023.XIX.026

Reviewer_XzafOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduce Agima, a training framework that teaches an agent to imagine before it acts in the setting of multi-modality. Three design choices are considered in the stage of SFT training, and standard RL settings after SFT. The experiments show promising results across 4 experiments, both results multi-modal reasoning. Further experiments present interesting and non-trival findings on the reasoning over alternative imagined futures and agentic behavior through RL.

Critical strengths
  1. The methodology is simple yet effective. In table 1, Agima enables a 14B model has competitive performance comparing to closed source models of Gemini 3.1 Pro and GPT-5.6 Sol. Table 1 also shows that simple SFT from imagination traces provides significant gains.
  2. More experiments demonstrates the potential usefulness of forward thinking. RL experiments demonstrates model's agentic behavior on how to utilize forward thinking, where RL changes both the format and the use of forward thinking. This is an interesting and non-trivial finding, potentially shows internalization of agentic tool usage into model's capability.
Critical weaknesses
  1. Potentially those tasks can be presented by text-state instead of images. I think a text next-state prediction should be included as a baseline as well, and potentially see how Agima performs only through text interactions. Maybe also consider benchmarks like Procgen (Heist, Chaser), where those environments can not be presented by text.
  2. Limited novelty and technical contribution. All the technical parts (SFT, RL) are common practice except the imagination.
Questions for authors
  1. Where is the objective of GRPO? Although it's obvious and common sense, I think it's a good practice to include an objective somewhere in the paper.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Xzaf

[W1] Text-state prediction and generalization

Paper: Appendix A.1 (Table 5; Figure 7).

Response. Thank you for suggesting a text-state prediction baseline.

First, we would like to emphasize that our environments already cover both discrete planning and continuous control that cannot be represented with text: in particular, PushT uses continuous (x, y) actions and contact-dependent object motion, a setting also studied in visuomotor robot learning [Chi et al., 2023].

To directly test whether visual successors can be replaced by text, we trained a structured-text successor control on Sokoban and Maze2D. It receives real RGB observations and interleaves each proposed action with a predicted JSON state before choosing the next action. The JSON records grid dimensions and object coordinates; Sokoban includes the player, boxes, walls, and targets. Successor labels are extracted from simulator states on the same expert episodes, providing privileged symbolic supervision. The backbone, task data, and training recipe follow the Agima / No Imagine controls; evaluation uses K=5. The training trajectory of the structured text is represented in Appendix A.1 Figure 7.

Table X1. Structured-text prediction: learning the training setting versus generalizing

MethodSok easy (%)Sok hard (%)Maze easy (%)Maze hard (%)
Agima90%29%100%47%
No Imagine85%2%100%10%
Structured text81%0%99%0%

The key distinction is easy-to-hard generalization. Structured text baseline reaches 81%/99% on easy Sokoban/Maze2D but 0% on both hard tasks, compared with Agima's 29%/47%.

The recorded hard-Sokoban failure shows predicted JSON retaining a 7×7, two-box description for a 10×10, four-box simulator state. (An example is shown in Appendix A.1 Figure 7).

The structured text representation is hard to generalize after SFT, whereas the image generation in Agims can directly generalize to harder configurations.

[W1] Additional Results on Procgen

Paper: Appendix A.4 (Table 8).

Response. We additionally adopt your suggestion and test Agima on Procgen Heist, where the agent collects keys, unlocks paths, and reaches a gem from RGB observations. We train on 200 fixed levels and evaluate on 400 disjoint unseen levels. A PPO expert trained for 50M interactions supplies 50,000 RGB rollout windows. Agima and No Imagine use the same backbone, windows, and 5,000-step SFT recipe with global batch size 8.

MethodData budgetReal feedback intervalUnseen success
Agima40kK=5163/400 (40.75%)
No Imagine40kK=583/400 (20.75%)
PPO expert, reference50MK=140.50%

We show that our paradigm directly transfers to tasks that’s hard to be represented by text like Procgen Heist and significantly outperforms the No-Imagine method by 20% using the same amount of training data. And using only 40k data, it achieves comparable performance with the PPO expert that’s trained on 50M data.

[W2] Technical contribution

Paper: Sections 2, 4, and 5.

Response. Agima unifies policy modeling, world modeling, and reward modeling within one omni model. The same model proposes an action, generates its visual consequence, and reasons over that consequence to accept, reject, or extend a plan. SFT trains these roles jointly through interleaved imagination traces, giving the language agent its own visual planning interface.

The second contribution is agentic RL over this interface. It teaches not only which action to take, but how to imagine: when to extend a rollout, reject a plan, or branch at a critical decision. Prior world-model agents use separate latent dynamics and fixed planners or policies; Agima instead trains one interleaved text–image omni model to generate, interpret, and adaptively use its own visual futures. We have edited the paper accordingly to emphasis our contributions.

[Q1] GRPO objective

Paper: Section 2.3; Appendix C.4.

Response. Thank you for pointing out the missing objective. We have now added the GRPO objective to Section 2.3 and its optimization details to Appendix C.4.

Thank you for your comments. I maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.