acceptodds
Under review as a conference paper at ICLR 2027

World Action Modeling with Progressive Visual Planning

Abstract

World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WAMs address this by predicting a single future frame without generating the full video, but this approach neglects how to progress toward the goal. We present ProWAM, a progressive world action model that jointly predicts actions and an ordered sequence of sparse visual sub-goals, providing explicit visual guidance to anchor action generation throughout task execution. This design scales naturally, as sub-goal prediction can be learned from large-scale action-free videos, allowing the video backbone to offload complex visual planning from the action policy. For efficient action generation, ProWAM executes a single video-backbone forward pass to cache sparse sub-goal features, eliminating iterative full-video generation and requiring only lightweight action denoising during replanning. Across extensive evaluations, ProWAM achieves superior out-of-distribution robustness. On simulation benchmarks, it sets new state-of-the-art results on LIBERO-Plus (85.8%) and randomized RoboTwin (75.7%), outperforming the strongest baseline with relative gains of up to +35.9%. On RoboCasa365, ProWAM achieves a 48.1% success rate and 18.2% on the challenging Composite-Unseen split, ranking 2nd overall. Crucially, in zero-shot real-world experiments, ProWAM achieves 70.0% success, outperforming the strongest baseline by +15.0 (from 55.0% to 70.0%, a +27.3% relative gain) in novel scenes. These results demonstrate the value of progress-indexed visual foresight for closed-loop control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Progressive Planning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_VX5XOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces ProWAM (Progressive World Action Modeling), a world-action model for robotic manipulation that augments action generation with an ordered set of visual sub-goals. Instead of relying on dense future-video rollouts or a single endpoint prediction, ProWAM parameterizes sub-goals by normalized relative task progress. At inference time, ProWAM does not perform full iterative video generation. Instead, it performs a single forward pass through the video expert. The paper reports strong results on LIBERO-Plus, randomized RoboTwin, RoboCasa365, and a small set of real-robot tasks.

Critical strengths
  1. Strong empirical performance across benchmarks. ProWAM achieves competitive results on LIBERO, RoboTwin, RoboCasa365, and real-world Franka tasks.
  2. Practical inference-efficiency improvement. The KV-cached inference strategy significantly reduces video-backbone computation, achieving a favorable trade-off between computational efficiency and performance.
Critical weaknesses
  1. Predefined temporal-progress sub-goals may not always align with meaningful task stages. ProWAM indexes sub-goals by normalized temporal progress along the remaining demonstration, rather than by explicit semantic task boundaries. While this provides a simple and consistent parameterization across trajectories of different lengths, manipulation stages such as approaching, grasping, transporting, and releasing can have substantially different durations across demonstrations. It would be helpful to explain how sensitive the method is to this potential mismatch between temporal progress and task-level structure.
  2. The use of predefined intermediate sub-goals may face challenges in stochastic or dynamic environments. ProWAM predicts sub-goals at fixed relative-progress positions, which may be effective when the task evolves along a reasonably predictable trajectory. However, when unexpected disturbances occur or the robot deviates from the nominal execution path, the originally predicted intermediate targets may become less informative. It would be useful to clarify how the method handles cases where the predicted visual progression becomes inconsistent with the actual execution.
Questions for authors
  1. How are the inference-time progress schedules selected for different benchmarks? Were kk and rir_i tuned on a validation set, and if so, what data were used for model selection?
  2. Could the authors report the wall-clock training time for both Stage 1 (video-only pretraining) and Stage 2 (joint video-action fine-tuning), including the number and type of GPUs used for each stage? In addition, please provide the minimum GPU memory requirement for inference. These details are important for assessing the practical training and deployment cost of ProWAM.
  3. Please clarify the meaning of G\mathbf G in Line 201, as this is its first occurrence in the manuscript. It appears that G\mathbf G denotes the progressive visual sub-goal latent(s), i.e., the set of sub-goal representations G1,…,Gk\\{\mathbf G_1,\ldots,\mathbf G_k\\}.
  4. Could the authors explain why ProWAM underperforms Xiaomi-Robotics-1 on RoboCasa365? In particular, could the predefined relative-progress positions used for sub-goal prediction be misaligned with meaningful task stages, thereby limiting performance on long-horizon composite tasks?
3 comments

Dear Reviewer VX5X

As the discussion deadline is approaching, we would appreciate it if you could let us know of any remaining concerns. We would be happy to address them before the deadline.

Thank you

The authors have addressed my concerns. I recommend acceptance.

Thanks for your feedback and we are glad that we address your concerns. We really appreciate your constructive comments and the discussion

Reviewer_RzM3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProWAM, a world-action-model that predicts an ordered sequence of subgoals, achiving better long-horizon planning and compatible with action-free video pretraining. By utlizing efficient model design and caching, the model is able to run low-cost inference while achieving high OOD success rate.

Critical strengths
  • The paper proposes to use sub-goal imagination for planning such that the future action prediction can be guided with the subgoal without additional higher level planners.

  • The paper provides careful evaluation in simulation environments and real-world robots.

  • By caching subgoal tokens, the model achieves efficient inference with low latency.

Critical weaknesses
  • Table 12's comparison with ProWAM-Light belongs in the main paper because it directly tests subgoal conditioning. However, LIBERO uses only one subgoal.

  • The 1.9-point gain from multiple subgoals in Figure 4(b) is not significant enough and needs uncertainty estimates to support the progressive-planning claim.

  • The OOD claims need more context, since RoboTwin's video training includes randomized scenes and the real-world setup closely follows DROID. A new evaluation scene alone does not show a substantial shift from DROID's diverse data, so the paper should explain what is unseen.

  • In the real-world experiments, the baseline checkpoints and training budgets should be explained for a fair comparison. The reported 70% versus 55% amounts to 14 versus 11 successes, so more trials and uncertainty estimates would make the comparison more convincing.

  • In Table3, the highlighted numbers are not the highest performance.

Questions for authors
  • Does ProWAM-Light use the same training recipe as ProWAM, and does it retain the subgoal-prediction loss?

  • What remains unseen after video pretraining, and do RoboTwin baselines receive the same randomized-scene videos?

  • Which real-world baseline checkpoints and fine-tuning budgets were used, and were all methods evaluated from matched initial configurations?

2 comments
Reviewer_RzM3OpenReview ↗

Post-Rebuttal Comment

The authors have carefully addressed most of my concerns. I have raised the recommendation to weak accept.

Thanks for your feedback and we are glad that we address your concerns. We really appreciate your constructive comments and the discussion

Reviewer_EBznOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose ProWAM, which is a world action model that additionally conditions actions on a few predicted visual sub-goals indexed by task progress (normalized to 0-1). The subgoal's latents are also generated by the video expert with progress-conditioned adaLN. This paper follows the Mixture of Transformer design, commonly used in WAM, and design a specific attention mask between different modality. This enables action-free video pretraining and single-pass KV-caching of the subgoal latents during inference. The authors evaluate their methods on multiple simulation benchmarks and demonstrate strong performance.

Critical strengths
  1. The paper is well-written and easy to follow: I like the general idea of progressive sub-goal prediction. This design introduces a structured understanding of the task (and progress) into the world action models, which is shown to be beneficial for policy learning in this paper's experiments. In addition, the authors also enhance the efficiency of this method by using sub-goal KV-caching.
  2. The proposed method demonstrates strong performance across three typical robotic benchmark: Libero-series, RobotTwin, and RoboCasa365. The authors also present effective ablation studies of their methods.
Critical weaknesses
  1. At line 222, it says "Trained under progress conditioning and varying sub-goal counts kk" could you provide more details on how you implement training with varying sub-goal counts kk? If this is the case, in the ablation studies Figure 4 (b), is it possible to add k=0k=0? If I understand correctly, I would expect the performance drops by a margin if no sub-goals are predicted.
  2. This leads to a following question: in Figure 4 (b), it shows that predicting more subgoals indeed can enhance the performance, but the gain is very small with only 1-2%. Could you illustrate why is that? and provide more insights on the interplay between performance and number of sub-goals and the sub-goal horizon.
Questions for authors
  1. At around line 214, it mentions that "the sub-goal slots share the same network weights and temporal position encodings as the video latents". I am confused what do you mean by "share the temporal position encodings"?
  2. In Figure 5, why the predicted images starting from obs t0 seems noticeably better than the predicted images starting from obs t1. For example, in Figure 5, the predicted images on the right side is blurry while the predicted images on the left side seems clearer.
3 comments

Dear Reviewer EBzn

As the discussion deadline is approaching, we would appreciate it if you could let us know of any remaining concerns. We would be happy to address them before the deadline.

Thank you!

Thank you for the response and providing additional experiment results. I will keep my positive evaluation of the paper.

Thanks for your feedback and we are glad that we address your concerns. We really appreciate your constructive comments and the discussion

Comments on the paper

1 comment

Dear Reviewers,

We are approaching the deadline of the discussion period (Nov. 18). Please go through authors' rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

Thank you for your time and participation.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.