acceptodds
Withdrawn from ICLR 2027

DreamTamer: From Lucky Samples to Reliable Futures in Action Conditioned World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action-conditioned world models (ACWMs) predict future observations from visual histories and robot actions, and must faithfully capture the consequences of those actions. However, even with identical observations and actions, pretrained ACWMs can produce substantially different interaction outcomes across random seeds, making faithful prediction inconsistent across samples. To understand this variability, we conduct denoising-stage branching experiments and find that early-stage stochasticity strongly affects prediction fidelity and task-critical interaction dynamics, whereas later-stage stochasticity primarily affects visual details. This finding motivates a reinforcement learning post-training framework that concentrates optimization on denoising transitions with greater influence on prediction fidelity. Specifically, we combine timestep-aware gradient correction with variance-aware timestep sampling to direct learning toward these influential transitions. To further improve reliability across samples, we introduce a lower-bound penalty that targets low-fidelity predictions without directly penalizing favorable deviations. Experiments on Ctrl-World, DreamDojo, and Genie Envisioner demonstrate improvements in average prediction quality, the quality of low-fidelity predictions, and consistency across repeated samples. Together, these results show that our framework makes faithful action-conditioned prediction more reliable across random seeds. We will release all training and evaluation code and model checkpoints to support reproducibility and further research.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, world models, simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FaTQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The most significant contribution of this paper is that it identifies the issue of unreliable sampling in existing ACWMs. This problem can cause the model to generate visually appealing frames during rollouts while failing to faithfully simulate the real world, preventing it from serving as a solid foundation for simulation, planning, and policy evaluation.

To address this issue, the authors first conducted denoising-stage branching experiments, revealing that early stages strongly affect prediction fidelity and task-critical interaction dynamics, whereas later-stage stochasticity mainly changes visual details. Based on this observation, the authors introduce an RL post-training framework for ACWMs following the Flow-GRPO paradigm. Specifically, 1) they employ a video-similarity reward combined with a dynamics reward to evaluate the quality of rollouts. 2) They apply transition-scale correction to rectify the gradient norm imbalance in the early stages. 3) They use variance-initialized timestep sampling to allocate higher update frequencies to early stages, thereby producing more variable credit signals. 4) they introduce a downside penalty to further penalize low-fidelity outcomes while preserving favorable training signals.

Finally, the authors demonstrate the effectiveness of their proposed method and its individual components through experiments across various ACWMs.

Critical strengths

The primary strength of this paper lies in proposing a targeted post-training paradigm for ACWMs based on insights from denoising-stage branching experiments, which exploits the property that early denoising stages dictate the generated structure and interaction dynamics. Specifically, the proposed Transition-scale correction and Variance-initialized timestep sampling are both explicitly designed around this characteristic.

Critical weaknesses

1.The denoising-stage branching experiments in this paper are similar to prior findings [1] and do not offer a novel insight. Furthermore, the authors merely adapt existing RL methods for video diffusion models to the ACWM setting. For instance, techniques like Variance-initialized timestep sampling have already been explored in TempFlowGRPO and GRPO-Guard; the paper lacks a clear discussion distinguishing its approach from these existing methods.

2.Both this work and PersistWorld utilize RL post-training to enhance the rollout fidelity of ACWMs. However, the paper lacks a detailed comparison to clarify the key differences and distinctions against PersistWorld.

[1] Wang B, Vastola J J. Diffusion models generate images like painters: an analytical theory of outline first, details later[J]. arXiv preprint arXiv:2303.02490, 2023.

Questions for authors
  1. For ACWMs in embodied domain, a crucial aspect is the physical consistency of the rollouts. However, the reward design in this paper overlooks this factor and relies solely on visual-based rewards. Are there more suitable reward models specifically tailored to physical consistency for ACWMs?

  2. Why are the metrics for the "Full method" in Table 4a inconsistent with those reported for "DreamTamer (Ours)" in Table 1?

Reviewer_g6deOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses cross-seed prediction inconsistency in action-conditioned world models. Motivated by branching experiments showing greater sensitivity to early denoising transitions, it proposes DreamTamer, an RL post-training framework combining transition-scale correction, variance-based timestep sampling, and a downside penalty. Experiments on three backbones demonstrate improvements in average prediction quality, worst-of-four performance, and sampling consistency.

Critical strengths
  1. The paper addresses a practically important issue: world models should produce reliable predictions across samples for downstream planning and policy evaluation.
  2. Experiments across three distinct backbones provide encouraging evidence for the applicability of the proposed post-training approach.
  3. Reporting sampling variance and worst-of-four performance alongside average quality provides a more informative evaluation of prediction reliability.
Critical weaknesses
  1. The novelty is limited:
    • Related work with the same motivation: The paper omits Foresight Diffusion, which already studies sampling consistency in predictive diffusion models, including action-conditioned robot video prediction. This omission undermines the conclusion’s claim of being the first to formulate this problem.
    • Known behavior of diffusion models: The observation that early denoising determines coarse structure while later stages refine details is well established in the diffusion literature.
    • Established algorithms: this paper primarily adopt branch-style GRPO methods to ACWM, with some weighting techniques.
  2. Typo: The transition-scale correction paragraph largely repeats §3.2, suggesting a copy-and-paste error. And the main text does not clearly describe the implementation of it.
  3. Theoretical justification limited: The downside penalty lacks a clear connection to an explicit risk objective or a guarantee of improved lower-tail performance.
Questions for authors
  1. About Weakness 1.1, the paper should clarify what its analysis of action-conditioned dynamics adds beyond the known behavior of diffusion models.
  2. How does the problem formulation differ from Foresight Diffusion, and can you revise the novelty claims accordingly?
  3. How do variance-based sampling and the downside penalty compare with simpler alternatives, and how sensitive are the results to their hyperparameters?
Reviewer_VcZjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the inconsistency of action-conditioned world models (ACWMs): given the same visual history and robot actions, pretrained video-diffusion action-conditioned models predict substantially different interaction outcomes depending only on the random seed. The authors first show using denoising-stage branching experiments that, this variability comes mainly from early, high-noise denoising steps, which determine the interaction dynamics, while later steps mostly change visual detail. Motivated by this, they propose DreamTamer, an RL post-training framework built on Flow-GRPO with branching-based credit assignment. It adds a per-step gradient correction to boost the under-weighted early transitions, timestep sampling proportional to the base model’s per-step reward variance, and a one-sided penalty that amplifies negative credit for below-average rollouts without penalizing above-average ones, trained with a video-similarity reward plus, where action following is reliable, a motion-tracking reward. Across three backbones (Genie Envisioner and DreamDojo, Ctrl-World) and two tasks - AgiBot World and Ctrl-World, DreamTamer improves mean and worst-of-four PSNR, SSIM and LPIPS. It reduces variance across seeds by roughly 30–87% relative to the base models, and outperforms PersistWorld, a prior RL post-training method, on Ctrl-World. Ablations on Genie Envisioner indicate that each component contributes to quality and cross-seed consistency.

Critical strengths
  • The branching analysis on three backbones nicely motivates and point to where this variability mainly comes from.
  • The experimental setup is broad. On three different world models and two datasets, the method improves mean and worst-of-four quality and cuts variance across seeds substantially, by roughly 30–87%. On Ctrl-World it also improves over PersistWorld, a prior RL post-training method.
Critical weaknesses
  • Apart from PersistWorld on one backbone, every comparison is against the pretrained base models rather other RL post-training objectives. I believe RL baselines are warranted. An especially relevant baseline could be RLVR-World [1] which RL post-trains an ACWM with GRPO. This baseline is neither cited nor compared with.
  • The training reward is built from the same similarity metrics used for evaluation (PSNR, SSIM, LPIPS), leading to goodharting. Lower variance across seeds could therefore partly reflect collapsed sample diversity rather than more faithful dynamics. There is also no downstream evaluation, such as policy-evaluation accuracy or planning success, to show the predictions are more useful for the applications that motivate the work.

[1] Wu et al. (2025). RLVR-World: Training World Models with Reinforcement Learning

Questions for authors

How does standard Flow-GRPO (or Flow-GRPO-fast / TempFlow-GRPO) [1,2], trained with the same reward and compute, compare with DreamTamer on all three backbones? Or is there a reason to not compare the proposed approach with the above methods?

[1] Liu et al. (2025), Flow-GRPO: Training Flow Matching Models via Online RL [2] He et al. (2025), TempFlow-GRPO: When Timing Matters for GRPO in Flow Models

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.