acceptodds
Under review as a conference paper at ICLR 2027

DiPA-WM: Diffusion Policy Adaptation with Continual World Modeling

Abstract

World models enable sample-efficient policy learning by allowing agents to optimize action decisions through imagined rollouts. In the offline-to-online reinforcement learning setting, the data distribution often shifts as the policy improves, requiring the world model to continuously adapt to on-policy data. This creates a fundamental challenge for diffusion-based policies, whose denoising process depends on stable conditioning representations: updating the world model causes , while freezing it prevents tracking the evolving policy distribution. In this work, we propose (ffusion olicy daptation with Continual orld odeling), which addresses this tension through a factorized two-stage design. The world model is decomposed into a and a . In Stage 1, the world model is pretrained on offline data and a 1-step Gaussian policy is learned over imagination; and the context encoder is frozen after Stage 1 training. In Stage 2, a diffusion policy is initialized via behavior cloning and then trained further over imagination while the adapter is updated on freshly collected on-policy data. The frozen context encoder provides stationary conditioning features, mitigating representation drift; the world model adapter enables continual adaptation of the world model dynamics to the evolving policy distribution. To address possible non-stationarity, we further develop a nested two-loop adaptation for the world model and diffusion policy, facilitated by periodic updates of context encoder on a larger timescale. DiPA-WM is modular and general: it consistently improves multiple diffusion policy optimization methods across continuous locomotion, robotic manipulation, and autonomous driving benchmarks, outperforming frozen-world-model and model-free baselines without modifying any core training objectives.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Robotics, Control, World Models, Diffusion Policy, Online Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tbJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiPA-WM addresses representation drift when training diffusion policies within continually updated world models. It first trains a world model and Gaussian policy using offline data and early online experience, then freezes the context encoder while adapting the dynamics module and training a diffusion policy in imagination. Slower context encoder updates are also experimented with, showing that relatively low update rate can achieve comparable performance to frozen encoder.

Critical strengths
  1. The separation of representation stability and dynamics adaptation is practical and well-motivated.
  2. Encoder-update ablations cover full-rate updates, slower updates, and freezing, with consistent results supporting the design.
Critical weaknesses
  1. Limited qualitative evidence. Tables and training curves do not clearly illustrate which failures DiPA-WM avoids or how encoder-update rates affect behavior. Matched-scenario trajectories, representative policy failures, and examples of large world-model prediction errors would make the design choices more intuitive.

  2. The benefit of switching to diffusion and its sensitivity to representation drift are not fully isolated. Gaussian and diffusion policies operate in different stages, preventing a direct comparison. Starting from the same Stage1 world-model checkpoint and replay buffer, the authors could compare continued Gaussian-policy training against switching to diffusion under matched budgets, while varying encoder-update rates. This would clarify both the performance benefit and whether drift is particularly problematic for diffusion policies.

  3. The DiWA comparison requires clearer explanations. DiWA already provides a frozen-world-model baseline. However, it is unclear whether world-model configurations, warm-up, policy initialization, and subsequent budgets are matched. Please clarify these conditions.

Questions for authors
  1. Can the authors provide qualitative comparisons showing policy failures and world-model prediction gaps across methods and encoder-update rates?
  2. Under matched initialization and budgets, how does continued Gaussian-policy training compare with switching to diffusion, including sensitivity to encoder updates?
  3. Which model configurations, warm-up procedures, initializations, and training budgets are shared between DiWA and DiPA-WM experiments?
2 comments

We thank the reviewer for their insightful questions and feedback.

Q1) We note that during inference in CARLA, the Gaussian policy of stage 1 frequently applies brakes when approaching an intersection, regardless of the traffic light state or whether a vehicle is visible. The diffusion policy, which is further trained on online data, does not suffer from this symptom and can generate a smoother action chunk. For state-based tasks, we provide quantitative world model prediction gaps for MuJoCo in the appendix.

Q2) We have run additional experiments to validate the comparison between using the gaussian policy and switching to diffusion in stage 2. We continue to train the Gaussian policy for an additional 1M environment steps after stage 1, and note that performance is mostly saturated. Switching to the diffusion policy allows for further improvement beyond what is offered by a Gaussian policy. We have added these results to the main paper.

TaskReturn (Continued Gaussian)
Hopper2945.7±49.12945.7 \pm 49.1
Walker3393.4±39.83393.4 \pm 39.8
Ant3221.5±89.43221.5 \pm 89.4
Humanoid3522.3±117.93522.3 \pm 117.9

Q3) We match the configurations of DiPA-WM and DiWA where possible. We use the model architecture for both methods, and pretrain the world model equally on the same offline dataset. However, DiWA initializes its diffusion policy on the offline data, then optimizes it imagination. DiPA-WM initializes its diffusion policy on the replay buffer of the Gaussian policy. The total training budget of the diffusion policy for DiWA and DiPA-WM (stage 2) are matched, at 1 million steps.

Thank you for your rebuttal. However, I still find the answer to Q1 largely descriptive instead of being informative. By "qualitative results", I mean visualizations of policy failure and rollout trajectories. The rebuttal does not fully address that. Therefore, I'm retaining my original assessment.

Reviewer_pCRhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DiPA-WM, which uses a two-stage training strategy to avoid the pre-trained distribution to be shifted in the robot policy training. The proposed method is claimed to show general improvements to main simulated robotics tasks.

Critical strengths
  1. The main method is clearly proposed, with detailed figure supporting it.
  2. A lot of simulation-based experiments are done on various robotics tasks proving the proposed methods' effectiveness.
Critical weaknesses
  1. Sim-to-real gap. This paper is aiming to tackle the data distribution shift in robot learning. However, sim-to-real gap is one of the most significant data distribution shift, where the real robot experiments are unfortunately missing.
  2. Feature-level analysis for the proposed motivation and method is not completed. For example, experiments/visualizations showing the data distribution shift, experiments/visualizations showing the shift mitigation by the proposed method (and each stage of it), would all help the manuscript to further stand out.
Questions for authors

Please see the weaknesses part. Besides, I think it would be better to:

  1. Have std in the baseline part as well, to make the paper more rigorous.
  2. Fig. 1 and Fig. 3 can be merged to leave more rooms for experiments in the long appendix.
3 comments

We thank the reviewer for their suggestions and feedback.

W1) Our primary focus is on the distribution shift that arises from offline-to-online reinforcement learning, and we do not make any claims about the effectiveness of DiPA-WM for the distribution shift induced by the sim-to-real gap. Additionally, we are focused on CARLA experiments, and it is standard to use simulated experiments in place of real world experiments (Think2Drive, MILE). Although sim-to-real gap is an important problem in robotics, it is not the aim of our work, as our focus is on training diffusion policies continually with world models, and addressing the offline-to-online distribution shift that arises.

W2) We did not include measurements of the representation drift for all the various configurations as the representation drift is proportional to the learning rate of the context encoder, thus making most comparisons not very interesting. However, we report the representation shift for the non-frozen context encoder case during diffusion policy training in the appendix.

Q1) We have updated the tables in the paper to include standard deviations for the baselines for 3 seeded runs in each task.

Q2) We believe that figures 1 and 3 serve their own purposes, and it would make the paper less clear if they were combined. Figure 1 serves the purpose of introducing the problem DiPA-WM aims to solve, and summarizes the ideas we use to address it. Meanwhile, figure 3 complements the main text, and is used to further explain our approach in much greater detail.

I thank the authors for the rebuttals. However, I believe the issues I raised are still critical. For sim-to-real, in addition to the weaknesses above, to me, the main goal of World Models is to introduce world knowledge in real-world vision info into policies, which can have significant distribution gaps to videos provided by simulators like CARLA. I remain weak rejection.

We appreciate the reviewer’s emphasis on sim-to-real transfer as an important challenge in world modeling. However, the field encompasses many equally important research questions, and this paper addresses a distinct set of challenges. We therefore respectfully ask that the evaluation focus on the paper’s stated scope and contributions, and that concerns about sim-to-real transfer be tied specifically to its claims or findings.

Reviewer_tthQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DiPA-WM, an offline-to-online diffusion-policy learning framework with continual world-model adaptation. The key challenge is representation drift: updating the world model can change the latent context used to condition the diffusion policy and destabilize training. DiPA-WM addresses this with a two-stage design that first trains the full world model together with a Gaussian policy, then freezes the context encoder while continuing to adapt the dynamics adapter and diffusion policy using on-policy data and imagined rollouts. Experiments across MuJoCo, FrankaKitchen, RoboMimic, and CARLA with DPPO, FPO, and DSRL show consistent improvements over frozen-world-model and model-free baselines.

Critical strengths

None

Critical weaknesses

W1. Insufficient justification and specification of key design choices

While the paper motivates the overall need to decouple representation learning from continual world-model adaptation, some key architectural and training choices are insufficiently analyzed. In particular, the Stage-1 Gaussian warm-up policy is motivated by the claim that its simple parametric form better tolerates gradual representation changes. Although Table 8 supports the usefulness of the overall two-stage training procedure, it does not isolate or directly validate the choice of a Gaussian policy itself. The ablation compares the proposed two-stage procedure against single-stage variants, rather than against alternative simple warm-up policy classes, making it difficult to determine whether the Gaussian parameterization is essential.

W2. The method presentation is not sufficiently self-contained or precise

The presentation of the overall architecture makes the method difficult to follow. In Figure 3, the arrows labeled “Imagination” between the action outputs and the imagination buffer do not clearly indicate what information is being generated, stored, or retrieved, making the imagined-rollout procedure difficult to infer from the figure alone. Moreover, although the formulation explicitly decomposes the representation module into the observation encoder eϕe_\phi and context encoder cϕc_\phi, the method figure does not clearly reflect this distinction, making it difficult to map the mathematical formulation to the illustrated architecture. In addition, Algorithm 1 introduces BrealB_{\mathrm{real}} without defining it, although the rest of the paper uses BenvB_{\mathrm{env}} and DoffD_{\mathrm{off}}. These presentation inconsistencies make the overall training pipeline harder to understand.

W3. Experimental reporting lacks sufficient metric and statistical detail

The experimental section does not provide enough information to interpret or assess the reported results fully. For several benchmarks, especially MuJoCo, FrankaKitchen, and RoboMimic, the paper does not clearly define in the main text what the reported numbers represent, their ranges, or whether they correspond to episode return, success rate, or another metric. Moreover, in Tables 2–5, DiPA-WM results are reported with standard deviations, while model-free baselines are given only as point estimates, without clarifying whether they are single runs, multi-seed averages, or previously reported values. This makes it difficult to assess the statistical significance of some relatively small improvements.

Questions for authors

Q1. The paper argues that the Stage-1 Gaussian policy is more tolerant to gradual changes in the conditioning representation due to its simple parametric form. Could the authors provide a controlled empirical analysis that directly evaluates this claimed robustness to representation/input changes, ideally against alternative policy parameterizations under the same setting?

Q2. Could the authors also report the performance of the Gaussian policy alone at the end of Stage 1, before switching to the diffusion policy? Comparing this with the final Stage-2 performance would help clarify how much of the performance gain comes from the Gaussian warm-up itself versus the subsequent diffusion-policy optimization.

Q3. Could the authors provide the exact optimization objective and update rule used for the Stage-1 Gaussian policy πη\pi_\eta? The current description only refers to a “model-based RL objective” and a generic PolicyUpdate.

Q4. Please consider revising the method presentation to ensure consistent notation and terminology throughout the paper. There are several places where the same or closely related components appear to be referred to using different terms, while other symbols are introduced without definition—for example, the use of “WM dynamics,” “dynamics adapter,” and “WM adapter,” as well as the undefined BrealB_{\mathrm{real}} in Algorithm 1. A systematic cleanup of these inconsistencies would make it easier to map the mathematical formulation, figures, and algorithm to the same set of components. The main method figure should also explicitly include all modules introduced in the formulation within a single end-to-end procedure.

Q5. Could the authors provide explicit definitions and ranges for the evaluation metrics used in each benchmark, particularly MuJoCo, FrankaKitchen, and RoboMimic in the main paper? It is currently unclear from the main text whether the reported values correspond to episode return, success rate, or another metric. Providing these definitions alongside the tables would make the results easier to interpret.

Q6. Could the authors clarify how the model-free baseline results in Tables 2–5 were obtained? DiPA-WM is reported with standard deviations, whereas the DPPO/FPO/DSRL baselines are shown only as point estimates. Please specify whether these numbers are single runs, averages over multiple seeds, or values taken from prior work, and ideally report the same uncertainty statistics for all methods to enable a fairer statistical comparison.

2 comments

We thank the reviewer for their feedback and questions.

Q1 and Q2) The use of a Gaussian policy is standard in world model literature [e.g., Dreamer, STORM]. We have run additional experiments on the MuJoCo tasks to validate the comparison between using the gaussian policy and switching to diffusion in stage 2. We report the performance of the gaussian policy at the end of stage 1, together with the results for continued gaussian training. For continued training, we allow the policy to train for an additional 1M environment steps beyond stage 1. We have added these results to the main paper. As demonstrated by prior works, and by our own experimental results, a gaussian policy can be trained jointly with a world model in a stable manner.

TaskReturn (Stage 1 Gaussian)
Hopper2887.3±53.62887.3 \pm 53.6
Walker3340.2±51.13340.2 \pm 51.1
Ant3196.9±84.03196.9 \pm 84.0
Humanoid3413.8±127.33413.8 \pm 127.3
TaskReturn (Continued Gaussian)
Hopper2945.7±49.12945.7 \pm 49.1
Walker3393.4±39.83393.4 \pm 39.8
Ant3221.5±89.43221.5 \pm 89.4
Humanoid3522.3±117.93522.3 \pm 117.9

Q3) Our gaussian policy is updated using an actor-critic algorithm with a REINFORCE style estimator. The returns are computed as:

$ R^\lambda_i = \hat{r}_i + \gamma \cdot \mathbb{1}[\hat{c}_i > 0.5] ((1-\lambda),V(\hat{r}_{i+1}, \hat{x}_{i+1})+ \lambda R^\lambda_{i+1}) $

with γ=0.997\gamma = 0.997 and λ=0.95\lambda = 0.95, and R{λH}=V(h^{H},x^{H})R^\lambda_\{H\} = V(\hat{h}_\{H\}, \hat{x}_\{H\}).

and the policy is updated according to the loss:

$ \mathcal{L}_\eta = - \mathbb{E}\left[\frac{\mathrm{sg\big(R^\lambda - V(h, x)\big)}}{\max(1, S)} \log \pi_\eta (a \mid h, x) \right] - c_{\mathcal{H}} \mathbb{E}\big[\mathcal{H}(\pi_\eta)\big]$

where SS is an exponential moving average of the difference between the 9595th and the 55th percentile of the λ\lambda-returns, and the entropy coefficient is cH=3⋅10−4c_{\mathcal{H}} = 3 \cdot 10^{-4}. The value function is trained with the cross-entropy loss against the two-hot encoding of the symlog-transformed λ\lambda-returns.

We have updated the text to include this formulation.

Q4) We have updated the text and figures to consistently use the same terminology throughout the paper.

Q5) We have updated the captions of the tables to include the exact metrics being reported. We report reward for MuJoCo and CARLA navigation, and success rates for Robommic and Frankakitchen. Additionally, we restate the metrics in the main paper for extra clarity.

Q6) The reported results for the model-free baselines are point estimates. The results obtained for DiPA-WM are averaged over 3 seeded runs. We have updated the tables to report the standard deviations of the baselines averaged over 3 seeds.

I appreciate the authors' efforts and additional experiments during the rebuttal. However, citing Dreamer and STORM does not sufficiently support the claim that Gaussian policies are more robust to representation drift. Since this claim motivates the two-stage design, stronger evidence is needed to demonstrate that Gaussian policies are indeed more robust to representation drift than alternative policies. I will slightly raise my score, but my recommendation remains weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.