acceptodds
Withdrawn from ICLR 2027

FutureWorlds:Learning Robotic World Models from Alternative Futures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robotic world models predict action-conditioned future scenes, providing a foundation for understanding action outcomes. However, turning alternative predictions into useful learning signals remains challenging: similar candidates limit informative quality comparisons, while diverging trajectories require persistent maintenance of their individual histories. We introduce , a framework that unifies candidate construction, history maintenance, and learning from relative quality. Built on a multimodal discrete autoregressive model, uses diverse beam search during reinforcement learning to construct candidate futures that balance confidence and diversity. Candidate-specific bounded memory preserves scene states and ensures that generation and policy scoring use matching histories. We further propose (Memory-Conditioned Search-Guided Policy Optimization), which converts video trajectory rewards into group-relative advantages to optimize the world model. On RT-1, BridgeV2, and RoboCasa, reduces LPIPS for 32-frame predictions by 14.78%, 20.84%, and 9.12%, respectively, relative to the strongest baseline on each dataset. Under fixed evaluation configurations, only 200 updates further improve generation quality and support continued prediction beyond the training horizon. Memory ablations, decoding sensitivity analysis, and optical-flow evaluation show that these gains extend beyond visual quality to more accurate motion prediction and more consistent object states.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic World Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Dm8XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FutureWorlds, a post-training framework for autoregressive robotic world models. Starting from a discrete autoregressive video predictor, it extends the RLVR-World-style use of outcome-based video rewards in three main ways: it maintains candidate-specific bounded histories consisting of a fixed initial scene/state anchor and a recent rolling context; it uses diverse beam search rather than independently sampled trajectories to construct alternative futures; and it applies the resulting group-relative rewards with history-matched policy scoring in the MemSPO objective. Thus, the method is primarily an integration of persistent contextual memory, diverse candidate search, and GRPO/PPO-style optimization around an existing reward-based world-model training paradigm. The method achieves strong open-loop video prediction results on RT-1, BridgeV2, and RoboCasa, but the methodological novelty and causal evidence are insufficient to justify acceptance.

Critical strengths
  • The framework is clearly described and provides a coherent integration of candidate search, branch-specific history maintenance, and trajectory-level optimization.

  • The paper reports a broad evaluation covering long-horizon prediction, motion metrics, memory ablations, decoding sensitivity, and inference resources.

Critical weaknesses
  • Limited methodological novelty. The method mainly combines existing components, including discrete autoregressive video prediction, diverse beam search, a sliding-window history, and GRPO/PPO-style optimization. The paper does not establish that this combination is necessary compared with simpler reward-based fine-tuning or candidate-selection approaches.

  • The alternative-future premise is not sufficiently validated. Candidates are rewarded according to their similarity to a single recorded future, while token-level disagreement is used as the main evidence of diversity. The paper does not show that the candidates represent distinct, physically valid futures or improve uncertainty coverage.

  • The ablations do not isolate the claimed contributions. The ordinary-beam comparison changes beam grouping and allocation in addition to the diversity penalty; the memory study is performed only at inference time; and history-consistent scoring and reward design are not directly ablated under matched training conditions.

  • The optimization provides coarse credit assignment and limited theoretical support. A single trajectory-level advantage is assigned to all (10\times80) dynamic tokens, despite their different frames and histories. Moreover, the theory only characterizes a local gradient identity and does not establish that search-selected candidates produce a better optimization direction or convergence guarantee.

  • The evaluation remains focused on oracle-guided open-loop video quality. The reward uses recorded future frames and is closely aligned with the reported PSNR/SSIM/LPIPS metrics. The experiments therefore do not establish improved planning, closed-loop control, or task success when future ground-truth observations are unavailable.

Questions for authors

see weakness above

Reviewer_jBNkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MemSPO, a GRPO-style method for post-training an iVideoGPT-based, discrete autoregressive action-conditioned video world model. Initialized from released checkpoints and fine-tuned for 50k steps per dataset, the model generates candidate futures through diverse beam search, with each candidate maintaining its own bounded context. Training uses group-normalized rewards based on negative MAE and LPIPS over ten-frame rollouts, followed by a clipped policy-gradient update that scores each candidate under its generating history. On RT-1, BridgeV2, and RoboCasa, the final models outperform twelve baselines on PSNR, SSIM, and LPIPS for 32-frame predictions across 128 held-out trajectories per dataset. After 200 updates, MemSPO also outperforms SFT, independently sampled GRPO, and an ordinary beam-search variant. Further analyses examine optical flow, memory, decoding sensitivity, inference cost, and a gradient identity at the old parameters.

Critical strengths
  1. Broad evaluation with competitive system-level performance.The evaluation spans RT-1, BridgeV2, and RoboCasa, covering both real-world and simulated robotic manipulation. Predictions are compared on 128 fixed trajectories per dataset. Under the reported configurations, the complete system achieves the best PSNR, SSIM, and LPIPS across all three datasets. This provides useful evidence of its performance across different settings, beyond a small collection of qualitative examples.
  2. Compact model with competitive performance against much larger video models. With 0.257B inference parameters and 1.89 GiB peak memory (Table 7), it outperforms the 2B–5B adapted video foundation models on all three datasets. The resource benchmark is documented carefully, including the higher latency of beam decoding (26.76 s vs 4.14 s for iVideoGPT).
Critical weaknesses
  1. Limited novelty and unclear need for RL. Most components of MemSPO are standard or inactive (e.g., clipping never applies with one update per candidate set, and the anchor adds at most 0.05 dB), and beam-based GRPO already exists (BGRPO), leaving grouped diverse beam search as the main new element. Since the reward uses the recorded future frames, supervised training on the model’s own histories toward those frames is a direct alternative, but the paper does not compare with it.
  2. Limited attribution of the headline gains to MemSPO. On BridgeV2, the SFT model already achieves an LPIPS of 0.1450 (Table 2), compared with 0.1780 for the strongest external baseline and 0.1409 for the full system (Table 1), indicating that much of the advantage predates MemSPO post-training. Furthermore, the matched comparison in Table 5 shows only 0.20% and 1.51% additional LPIPS improvements over ordinary-beam post-training on BridgeV2 and RoboCasa, respectively. Thus, the strong system-level results provide limited evidence for the incremental contribution of MemSPO’s proposed candidate construction.
Questions for authors
  1. Training data and baseline adaptation. Could the authors specify the SFT training data on each dataset (number of trajectories, split, frame rate, window length), and whether each baseline was fine-tuned on each target dataset, for how many updates, and whether it was trained to convergence? Appendix B.4 covers only three baselines on BridgeV2.
  2. See weakness
Reviewer_ZYvROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MemSPO, a post-training strategy for improving the fidelity of action conditioned world models. Specifically, the paper uses visual rewards to better align the world model generation with the ground truth generations. The paper reports results on 3 benchmarks of different robot embodiments and performs additional analyses on identifying where the improvements come from.

However, there are major issues with 1. positioning the paper with respect to the current state-of-the-art (contributions) and 2. experimental results, where the significance of this work is not clear.

Critical strengths
  • The paper introduces the beam search guiding into the GRPO branching for potentially a better use of the compute resources.

  • The paper measures the improvements on multiple axes and reports that the post-training improves across the board.

Critical weaknesses
  1. Novelty.
  • The proposed method is not novel and very similar to [RLVR] and [PersistWorld] in all suggested aspects besides the search guiding.

  • It is not clear how the candidate persistent memory is different from the shared prefix context suggested in PersistWorld. Could the authors clarify where this differs from the shared prefix context?

  • The memory anchoring is already a known detail; both specified in Ctrl-World and PersistWorld methods. To my knowledge, both papers already do the anchoring + 5/6 latest frames, which is introduced as anchored memory in this paper [Figure 7].

  • The reward formulations also seem to be inspired by recent works. There is no ablation on the choice of the rewards and their combinations.

  1. Experiments. The results are not convincing.
  • The proposed search guiding strategy brings marginal improvements compared to standard GRPO.

  • The baseline models seem to significantly underperform the numbers reported in the original papers.

  • It is unclear whether the baselines are sufficiently trained. For example, it is implied in the App. B.4 that Ctrl-World is only trained for 2,000 iterations vs 50,000 for the proposed MemSPO. And PersistWorld does only 30 updates compared to at least 200 in MemSPO. Similarly, iVideoGPT is only trained for 20k steps which is less than half the number of training steps used for MemSPO (50k). It is implied that the Wan model is only trained for 8k steps. There is no consistency among these, and, in my opinion, the main difference is largely attributed to undertraining since from Table 2, supervised finetuning (SFT) already provides most of the improvements over all the baselines.

  • PersistWorld is incorrectly run with only 25 denoising steps with CFG enabled despite the recommendation of 50 denoising steps and CFG disabled in the original paper. There are no details about how Ctrl-World is run [Table 8]. The original PersistWorld paper reports a much higher performance on a longer window (26.63 PSNR for 41 frames vs 18.77 reported here for 10 frames). Could the authors explain why there was a deviation in the inference strategy and the discrepancies in results?

  • How accurate are the metrics if each baseline runs with a different resolution? For example, Wan runs at 480p vs 256p for MemSPO.

  1. For more convincing experiments the paper should apply their method vs other post-training methods on the same or equivalent backbones to disambiguate and quantify the improvements from their proposed post-training vs supervised finetuning.

  2. The paper introduces scene and state anchors as two different kinds of anchoring information, but it is not entirely clear how the two differ?

  3. The paper implies that the world model also takes as input the language instructions, is it actually used? Wouldn’t that influence the model to better align with the task at hand since that provides a high level additional signal?

  • Could the authors try counterfactual reproduction, i.e. trying to replay actions from another set of tasks? For example, if the model has learned a mapping between language instruction and robot action, requesting the world model to replay trajectories from other kinds of tasks (while keeping the language instruction same) should perform significantly worse.

  • Is this aligned with the intent of building a simulator-like world model?

  1. Figure 2 and Figure 8 of the paper are difficult to read since they are too complex. The content could be simplified considerably.

  2. The authors measure the improvement in diversity of the sampled branches with their search guiding strategy but only discuss it briefly in App. E2. I think this can be explored a bit more to better explain. One thing authors may want to think about is how this could potentially relate to the Top-K/Bottom-K selection of candidates employed in previous methods like PersistWorld.

References: [RLVR] Wu, J., Yin, S., Feng, N., Long, M.: Rlvr-world: Training world models with reinforcement learning. arXiv preprint arXiv:2505.13934 (2025)

[PersistWorld] Learning, J. Bardhan, P. Drozdík, J. Sivic, V. Petrik, Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement, ECCV 2026

Questions for authors

Please check the critical weaknesses for the questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.