acceptodds
Under review as a conference paper at ICLR 2027

One Flow Transformer for Imagination and Control

Abstract

Diffusion and flow models are effective world models for visual reinforcement learning, letting agents learn behaviors in imagination. Yet existing diffusion-based online agents use them as black-box simulators: a separate policy network learns from generated rollouts, leaving the backbone’s representations unused for control. We introduce DRIFT, an efficient agent whose policy learns entirely in imagination on the same Flow Transformer backbone as its world model, trained online from scratch. DRIFT’s imagination is fast: it runs in a compact continuous latent space, using 33× fewer FLOPs per imagined transition than prior diffusion-based agents. On online RL benchmarks, DRIFT outperforms diffusion and flow baselines on Craftium and Crafter, and is competitive on Atari and with discrete latent-dynamics agents. Three modeling choices make this possible. First, we find that denoising alone yields a less effective control representation. We therefore train the backbone to predict the next latent from the policy’s compact state, a reconstruction-free objective that improves control. Second, we observe that reward predictions are less accurate on generated latents than on real ones, which can hinder policy learning. A simple and effective mitigation is to also train the world model on its own rollouts. Third, shortcut flow matching reduces imagination to one denoising step per transition, matching multi-step sampling in rollout accuracy. These results show that one generative backbone can both imagine and control in online RL.

Primary area:
reinforcement learning
Keywords:
world models, generative models, representation learning, reinforcement learning, model-based RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BCT5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRIFT, an online model-based RL framework that shares a Flow Transformer backbone between world modeling and policy learning. It incorporates next-latent prediction to improve control representations, self-generated rollout training to mitigate distribution mismatch, and shortcut flow matching for efficient one-step imagination. Experiments on Atari, Craftium, and Crafter demonstrate competitive control performance with substantially reduced imagination costs.

Critical strengths
  1. The paper addresses a meaningful problem in efficient online model-based RL. Sharing representations between generative world modeling and policy learning is a reasonable design choice.

  2. The experiments cover several benchmarks and include useful ablations. In particular, next-latent prediction and self-generated rollout training lead to substantial improvements, while one-step imagination significantly reduces computational costs.

Critical weaknesses
  1. My biggest concern is the limited methodological novelty. Most technical components are adaptations of existing approaches, including shared world-model representations (Dreamer), self-predictive representation learning, Self-Forcing, and Shortcut Flow Matching. The paper mainly combines these established techniques into a unified framework, with limited evidence of a fundamentally new methodological contribution.

  2. The paper emphasizes using one Flow Transformer for both imagination and control, but lacks a direct comparison against a separate-policy architecture using the same world model and auxiliary objectives. Without this controlled ablation, it is unclear whether representation sharing itself provides meaningful benefits beyond the additional training objectives.

  3. Self-generated rollout training introduces two objectives, (L_{\mathrm{sf}}) and (L_{\mathrm{sf-FM}}), but their individual contributions are not isolated. Similarly, the substantial gains from next-latent prediction are not compared against existing predictive or contrastive representation-learning objectives. These missing comparisons make it difficult to establish the necessity of the specific design choices.

  4. The method supervises generated latent trajectories using rewards and continuation labels from the corresponding real trajectories. However, as generated states deviate from the real trajectory, these labels may become inconsistent with the generated states. The paper provides limited analysis of this potential supervision bias and its effect on longer-horizon imagination.

Questions for authors
  1. Can the authors compare the shared-backbone architecture against an otherwise identical world model with a separate policy network under matched training and computational budgets?

  2. What are the individual contributions of (L_{\mathrm{sf}}) and (L_{\mathrm{sf-FM}})? How does next-latent prediction compare with existing predictive or contrastive representation-learning objectives?

  3. How sensitive is self-generated rollout supervision to deviations between generated and real trajectories, particularly over longer imagination horizons?

  4. Can the authors provide more direct statistical comparisons against DIAMOND and report end-to-end training efficiency under matched computational budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BCT5 (1/2)

We thank the reviewer for the careful review. We ran every requested ablation, and the shared backbone outperforms a separate policy network on all four games. We address each concern below.

W1. Novelty beyond adapting existing techniques. We understand this is the reviewer's main concern. We agree that DRIFT builds on the four techniques the reviewer lists, and we explain what each required in DRIFT's setting. What is new is that a diffusion or flow world model, trained online, also provides the policy's state.

  • Shared world-model representations (Dreamer). In DreamerV3, the world model is trained to predict future states, and such a state is naturally useful for control. A diffusion or flow model is instead trained to denoise its input. We find that, when trained online from scratch, its features alone are a poor state for the policy (Fig. 3a). Sharing becomes effective when the policy's state receives a predictive signal and the model is trained on its own generations to reduce exposure bias. Given the same predictive signal, a separate encoder still falls short of the shared backbone (W2).
  • Targeted adaptations. DRIFT's use of SPR and Self-Forcing is not a direct reuse. Each is a simple, targeted adaptation to a diffusion or flow world model. SPR matches encoders on augmented views, which do not apply to noised latent inputs, so DRIFT instead trains the readout token its policy reads to predict the next real frame's latent. Self-Forcing corrects exposure bias in video generation, but an agent also needs reward and continuation heads that work on generated latents, so DRIFT supervises them there as well. We adopt shortcut flow matching as is, to study a state-of-the-art model of this class that also makes one-step imagination cheap.
  • Significance. Prior diffusion-based agents use the world model as a black-box simulator for a separate policy. DRIFT shows that it can also be the policy's state, competitive with DIAMOND and STORM at about one-third of DIAMOND's GPU-hours (general response, End-to-end cost; reply to MEAW, W2). Improved training raises DRIFT's IQM to 0.733, lifts its median significantly above DIAMOND's and gives the most superhuman games in Tab. 1 (13 of 26) (general response, Improved training of DRIFT (added)).

Given the shift toward diffusion world models in RL, we expect the result to apply beyond DRIFT.

W2 / Q1. Shared backbone against a separate policy network. We kindly refer the reviewer to the general response (Does sharing the world model's backbone help the policy?), which reports the requested comparison with the same world model and auxiliary objectives. We also trained DIAMOND's own actor-critic in DRIFT's world model with its original settings. It exceeds the shared agent on two games, so a separate policy with its own learning algorithm can also do well on DRIFT's world model.

Final 100-episode evaluations, mean of 3 seeds. Bold marks the better of the two agents trained in DRIFT's world model.

Atari gameKungFuMasterChopperCommandCrazyClimberHero
DIAMOND's agent in DRIFT's world model21,9591,73896,25714,505
shared DRIFT (submitted)25,2951,79079,47911,189
DIAMOND, published final18,7141,37099,1685,622

W3 / Q2 (a). The two rollout losses. The two losses are best together on three of the four games, and each alone gives smaller or mixed gains. Training each loss alone, together with the submission's pair (Fig. 3b), gives all four combinations. L_sf trains the reward and continuation heads on generated latents, and L_sf-FM trains the dynamics model on sequences that contain its own generated latents.

Final evaluations, mean of 3 seeds, best per game in bold.

L_sfL_sf-FMKungFuMasterChopperCommandCrazyClimberHero
nono18,2881,05673,4059,496
yesno18,8591,26976,20510,516
noyes14,6871,37889,38810,596
yesyes (DRIFT, submitted)25,2951,79079,47911,189

We expect the pair to work best because the two losses correct different parts of the model on generated latents. L_sf corrects the reward and continuation heads, and L_sf-FM corrects the dynamics.

AuthorsOpenReview ↗

Response to Reviewer BCT5 (2/2)

W3 / Q2 (b). Next-latent prediction against a contrastive objective. We thank the reviewer for suggesting this comparison, which tests a direction the paper proposes in App. A and App. E.2.2. AC-CPC, the action-conditioned contrastive objective used by TWISTER, is a natural alternative to DRIFT's predictive objective. We replace next-latent prediction with AC-CPC.

Rows 1-2 are the online-tokenizer rows of Fig. 3a (symlog rewards, 5 seeds), and rows 3-4 are final evaluations of the submitted sign-reward agent and its AC-CPC variant (3 seeds). Best per game in bold.

objectiverolloutsKungFuMasterChopperCommandCrazyClimberHero
none (Fig. 3a)no18,17890950,660866
next-latent (Fig. 3a)no19,5161,12473,0838,582
next-latent (DRIFT, submitted)yes25,2951,79079,47911,189
AC-CPC (new)yes25,5911,68492,59512,622

AC-CPC performs as well as or better than next-latent prediction on three of the four games. We expect its gains to come from its richer predictive signal, since it looks ten steps ahead and contrasts each prediction against other states. The shared state needs a predictive objective, whichever form it takes, and a stronger one can improve DRIFT further. We will report AC-CPC in the revised ablation as a promising alternative to the simpler next-latent loss.

W4 / Q3. Validity of labels on generated states. Generated states do deviate from the real trajectory, but over DRIFT's 15-step horizon the resulting label bias is small on average. Extra wrong labels stay within one percentage point and critic bias within about 1% of the return scale at every step (first table below). Horizons beyond 15 imagined steps are outside the scope of this work, and we will list a horizon sweep as a limitation.

DRIFT uses these labels only on the first four generated steps, with weights that decay with depth (App. B.2), so most supervision falls where deviation is smallest. Steps 5 to 15 receive no labels, but the actor and critic still learn from the rewards and values predicted there, so we test all 15 steps.

We use the submitted agent's final checkpoints (4 games, 3 seeds). Starting from real states, we generate 15 steps with the recorded actions and attach the real trajectory's labels, as in training. At each step we measure the deviation from the real state, the excess rate of wrong labels on the generated state over the real one, and the critic's bias relative to the real state. A label counts as wrong when a reward head trained only on real states disagrees with it.

imagination step14815
deviation from the real state (latent MSE)0.130.190.270.38
extra wrong labels on the generated state (percentage points)0.00.80.80.5
critic bias (% of return scale)0.30.80.50.8

Deviation triples over 15 steps, yet the label and critic errors do not grow with it.

Since most steps carry no reward, we also check the steps that do. On them, the reward head is correct less often on generated states than on real ones (76% against 86% on average).

correct on rewarded steps 5-15 (%)KungFuMasterChopperCommandCrazyClimberHeroaverage
generated states9523988676
real states100471009686

The difference comes mostly from ChopperCommand, whose rewards are the rarest and are often missed even on real states. We will add these measurements to App. E.3 and list rare rewards as a limitation.

Q4. Statistics against DIAMOND and end-to-end cost.

  • A direct test finds no significant difference between DRIFT and DIAMOND, consistent with the paper's claim of competitive performance. Under the standard protocol of Agarwal et al. (2021), with both agents' per-seed scores on all 26 games and 95% stratified-bootstrap intervals from rliable, DRIFT scores slightly higher than DIAMOND in both IQM and median, but neither difference is significant (IQM +0.03 [−0.07, +0.14], median +0.15 [−0.04, +0.37]). Sec. 5.2 already states that the intervals overlap, and we will align the introduction's sentence that reports higher median and IQM.

  • On cost, DRIFT takes about 25 GPU-hours and DIAMOND 79 under the same 100k-interaction budget (general response, End-to-end cost).

Dear Reviewer,

As we are approaching the end of the rebuttal period, we kindly invite you to let us know whether our responses have addressed your concerns. We would be happy to provide further clarifications or discuss any remaining questions.

Thank you again for your time and thoughtful feedback.

Reviewer_a46vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces DRIFT, an online model-based reinforcement learning agent that uses a single Flow Transformer for both world modeling and policy learning. Unlike diffusion-based agents that train a separate policy on generated observations, DRIFT extracts the actor–critic state directly from the generative backbone. The method adds next-latent prediction to improve the control representation, training on self-generated rollouts to reduce the mismatch between real and imagined states, and shortcut flow matching to generate each imagined transition in one denoising step. The experiments cover Atari 100k, Craftium, and Crafter. Results are competitive with existing methods, but DRIFT does not consistently outperform established latent world-model agents.

Critical strengths
  • This paper explore a relevant and timely research question: whether a generative world-model backbone can also provide an effective policy representation.
  • The paper presents coherent set of design choices. There are some meaningful engineering results that give fewer denoising steps capability to DRIFT (leading to efficient 'imagination').
  • The ablations support the importance of next-latent prediction and self-generated rollout training. The paper also examines representation quality through action-conditioned probes.
Critical weaknesses
  1. The distinction from Dreamer is not sufficiently compelling. The core idea of sharing a learned world-model representation with the policy is already central to DreamerV3. DRIFT replaces Dreamer's recurrent state-space model with a continuous-latent Flow Transformer and introduces auxiliary objectives to make the resulting representation useful for control. The authors should explain what this design enables that Dreamer cannot readily achieve, beyond using a different generative model.
  • The main results do not establish a general performance advantage. Eg. On Atari 100k, DRIFT's IQM is 0.673, and 0.641 for DIAMOND. However, the confidence intervals for DRIFT and DIAMOND overlap. The results therefore support competitiveness and computational efficiency more clearly than a general improvement in model-based RL.
  • The contribution of the flow-based formulation remains unclear. The paper shows that one-step shortcut sampling reduces imagination cost. However, it does not fully establish whether continuous flow-based dynamics provide a control advantage over a suitably matched latent-dynamics model, or whether most of the gains arise from the auxiliary predictive objectives and self-generated rollout training.
  • The efficiency claim is too narrowly framed. FLOPs per imagined transition and batch-one latency are useful metrics, but they do not capture total training cost. The appendix reports about four days on a single A100 to train one Atari agent. The paper should compare end-to-end compute, environment interactions, and final performance under a common budget before presenting the method as broadly more efficient.
Questions for authors

Please see weaknesses; more specifically:

  1. What fundamentally distinguishes DRIFT from DreamerV3?
  2. Can the authors isolate the benefit of flow-based dynamics?
  3. Why and when is a flow-based world model preferable to Dreamer's latent dynamics for control?
3 comments

We thank the reviewer for the thoughtful feedback, which helped us sharpen how DRIFT differs from DreamerV3. A new comparison shows that its imagination keeps small visual details that DreamerV3 loses. We address each concern below.

W1 / Q1. What distinguishes DRIFT from DreamerV3. DRIFT's world model belongs to the model class behind high-fidelity video generation. DRIFT trains it online from scratch and reads the policy's state from it. DRIFT's imagination in continuous latent space thus keeps small visual details that discrete-latent DreamerV3 cannot readily preserve, while control stays competitive on Atari (Tab. 1). This distinction is significant and non-trivial, since a flow backbone shared as readily as DreamerV3's state is weak on its own (W3), and DRIFT makes it work with targeted adaptations that address observed failures (reply to BCT5, W1).

To test this, we compare imagined frames from DRIFT and DreamerV3 on four games. Three are the games where DIAMOND attributes part of its gains to small visual details (Asterix, Breakout and RoadRunner), and the fourth is Hero, one of the four games of our new ablations. Each model starts from the same 8 real frames, imagines the next 15 under the recorded actions, and decodes them with its own decoder. We then compute LPIPS against the real frames. Because DreamerV3's recurrent state needs a longer burn-in, we also give it 64 context frames.

Mean LPIPS over 15 imagined steps and 100 windows per game (lower is better, best per game in bold).

gameDRIFTDreamerV3, 8 framesDreamerV3, 64 frames
Asterix0.0120.1120.082
Breakout0.0270.1260.109
RoadRunner0.0230.1310.061
Hero0.0130.3720.124

DRIFT's imagined frames are closer to the real ones on all four games, even against DreamerV3 with 64 context frames. DRIFT keeps small moving objects such as the Breakout ball in place over all 15 steps, while DreamerV3 tends to blur or lose some of them. With 64 frames, DreamerV3 comes closer, and in Hero it also recovers the room layout. Fig. 4 and Figs. 10-11 already show DRIFT's imagined frames, and we will add DreamerV3's frames next to them.

W2. General performance advantage. We agree, and the paper claims competitiveness on Atari, not an advantage. We will revise the introduction's sentence that DRIFT has higher median and IQM than DIAMOND, and the abstract's results sentences (reply to MEAW, W2). The contribution is how DRIFT reaches this level, with one shared backbone at about one-third of DIAMOND's end-to-end cost. With improved training, DRIFT raises its median significantly above DIAMOND's (+0.43 [+0.08, +0.51]) and reaches the highest median and the most superhuman games in Tab. 1, although its IQM gain over DIAMOND is not yet significant (general response, Improved training of DRIFT (added)).

W3 / Q2. Isolating the benefit of flow-based dynamics. We agree that this is important and hard to isolate. A matched latent-dynamics model would change much of the agent at once, since the denoising objective also trains the policy features. Prior diffusion agents such as DIAMOND and Horizon Imagination also do not include such a comparison, and we would welcome a suggestion for a matched design.

Within DRIFT, the gains do not come from the objectives alone, since a separate encoder with the same next-latent loss still trails the shared backbone on all four games (general response, Does sharing the world model's backbone help the policy?). Fig. 3a also makes a statement about flow dynamics themselves. Shared as readily as DreamerV3's state, a flow backbone trained only to denoise gives a weak policy, so flow dynamics need a predictive objective to serve as a policy state. In return, they give high-fidelity imagination (W1) with one denoising step (Tab. 3, App. E.4.2). We do not claim a control advantage over a matched latent-dynamics model, and we will revise the abstract to state the central question explicitly, "whether a flow world model trained online can also serve as the policy's state".

W4. End-to-end cost. The cost comparison is in the general response (End-to-end cost), which also explains the four-day figure. In short, DRIFT takes about one-third of DIAMOND's GPU-hours.

Q3. When a flow model is preferable. The paper motivates DRIFT from this question (Sec. 1). Low-dimensional latent states or discrete tokens can discard relevant visual details, while flow models generate in continuous space. We therefore expect flow models to be most useful when control depends on small visual details or on several plausible high-dimensional futures, or when a video model can be reused as the world model. One-step sampling makes this practical for policy training. A control advantage from the dynamics themselves remains a hypothesis (W3).

Thanks for the detailed rebuttal.

From what I understand, DRIFT's main difference is to implement DreamerV3 with a Flow Transformer and adapt the training objectives to that architecture. In the rebuttal, the authors provide additional evidence that DRIFT produces higher-fidelity imagined observations than DreamerV3, particularly in preserving small visual details (Q1). However, while these results support the claim that flow-based imagination can better preserve visual information, they do not fully explain why this property translates into effective policy learning or improved control. I am unconvinced that better generation necessarily means better policy learning or control.

In the authors' response to Q3, the authors reiterate that low-dimensional latent states or discrete tokens can discard relevant visual details, while flow models generate in continuous space. The authors' response does not answer whether increasing the number of discrete tokens will retain the performance comparable to DRIFT (if so, the novelty of the proposed method is less clear). While the experiments show that DRIFT's approach can work well, it remains difficult to isolate the benefits of flow-based dynamics from those of the proposed training adaptations and shared representation. As a result, I remain somewhat uncertain about the underlying mechanism and how well the results support the broader motivation.

Overall, the rebuttal addresses several of my concerns and strengthens the empirical case for DRIFT. I am therefore raising my score to Weak Accept, with some reservations about the paper's explanation of why the approach works.

Thank you for the follow-up and for raising your score. The score on OpenReview may not have been updated yet. If that was missed, we would be grateful if you could update it when convenient.

Better generation and better control. We agree that better generation alone does not guarantee better control. However, faithful generation matters when the policy learns in imagination. In Breakout, every decision depends on where the ball is, so if imagination loses the ball, the policy would struggle. To test this, we tracked the objects on screen in the imagination of both DRIFT and DreamerV3. Both models start from the same 8 real frames (DreamerV3 also from 64), imagine 15 frames under the recorded actions, and are compared with the real frames, over 100 windows from each agent's play per game. We count each connected group of changing pixels (sprites, the ball, score digits) as an object.

Share of objects kept over 15 imagined steps (higher is better):

BreakoutAsterixRoadRunnerHero
DRIFT91%84%93%92%
DreamerV3, 8 frames63%35%80%47%
DreamerV3, 64 frames72%57%94%77%

We verified it manually by checking generated samples. For example, in 10 Breakout game rollouts and 10 steps into imagination, DRIFT still shows the ball in 7 of them, while DreamerV3's rollouts have mostly faded it out.

More discrete tokens, and isolating the flow dynamics. More tokens may recover more detail, and we agree this is an interesting question. However, capacity alone may not be enough, since the DreamerV3 above (official code, default configuration) already has 166M parameters, about six times DRIFT's 27.7M. A dedicated study of this question would be valuable, and we will discuss it as future work.

Thank you again for the constructive feedback. If we have missed anything, please kindly let us know.

Reviewer_MEAWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Most online diffusion world-model agents, such as DIAMOND, train a separate policy on imagined frames. DRIFT instead trains one causal Transformer online from scratch that imagines each transition in a single denoising step and also serves as the actor–critic backbone. Because denoising alone gives a weak control state, the authors add an SPR-style next-latent loss (self-predictive representations, Schwarzer et al., 2021) and train on the model's own rollouts. The agent roughly matches DIAMOND on Atari 100k, and the authors report wins over baselines they trained themselves on two procedurally generated game benchmarks.

Critical strengths
  1. The paper gets an online agent, trained from scratch, to use the features of its own flow-matching world model as the policy's state. The agent is small and imagines in one cheap step. It still reaches the level of DIAMOND and STORM on Atari, and unlike Dreamer 4 it needs no offline video pretraining.

  2. I also found the diagnostics useful. A readout from a denoising-trained backbone is a poor control state, and a one-step next-latent loss fixes most of that.

Critical weaknesses
  1. The title claim, that one backbone should serve both imagination and control, is never tested directly. I could not find an experiment that keeps the world model and imagination fixed and gives the actor–critic its own encoder over the generated latents z^\hat{z}. Fig. 3a even weakens this claim. The shared readout is poor until the SPR-style loss is added, so the useful control representation may come from that loss rather than from sharing.

  2. The performance and efficiency claims are stronger than the evidence. The gains are modest and hinge on picking the reward head per benchmark. On Atari all aggregate confidence intervals overlap with DIAMOND's. The Crafter win needs DRIFT-symlog, which predicts full-scale rewards instead of their sign. That variant is the worst method on Craftium Room, while sign-reward DRIFT trails Horizon Imagination on Crafter. So no single variant supports the abstract's claim of beating the diffusion and flow baselines on both Craftium and Crafter.

    The efficiency claim rests only on the cost per imagined step. A full run takes about four A100-days per Atari game (as mentioned by the paper), and there is no end-to-end comparison with any baseline.

Questions for authors
  1. During training, the control state sts_t is read from the same forward pass that denoises the noised latents (algorithm 1). In imagination it comes from a re-encoding pass "at the clean noise level" (algorithm 2). Which σ\sigma is that, given that σ=1\sigma = 1 is not on the training grid? Are the reward, continuation and next-latent losses computed on noised or clean passes?

  2. I recommend to add the separate-encoder control from my first weakness, with and without the same next-latent loss. It'd be better to report both return and training cost.

  3. How does DRIFT's end-to-end GPU time compare with DIAMOND, DreamerV3 and STORM on the same hardware?

  4. For the Craftium and Crafter baselines you trained, how many seeds, which hyperparameters and what tuning budget did you use?

  5. Was the reward head (sign or symlog) chosen per benchmark before seeing the results (or how to choose which reward head to use)? If one variant had to be used everywhere, which would you pick?

2 comments

We thank the reviewer for the detailed review and the concrete controls proposed. We ran the requested separate-encoder control, and it shows that sharing helps beyond the next-latent loss. We address each concern below.

W1 / Q2. A separate encoder over the generated latents. We kindly refer the reviewer to the general response (Does sharing the world model's backbone help the policy?), which reports the requested ablation with and without the same next-latent loss.

W2 / Q5. Reward-head choice and the margin over DIAMOND. We did not choose the head per benchmark from results.

  • The submission assigns the heads by reward structure, following the rule stated in the method (Sec. 4.2). The paper reports both heads on every benchmark because prior works differ, with DIAMOND training on sign rewards and DreamerV3 using symlog to keep one setting across domains. Sign rewards bound how much a reward-prediction error can affect imagined returns, while symlog keeps reward magnitude, which matters on Crafter, whose rewards span two scales (with sign rewards, making a wood pickaxe drops from 79% to 45%, Tab. 9).
  • The improved training uses symlog on every benchmark, and its Atari median is now significantly above DIAMOND's (general response, Improved training of DRIFT (added)). Already in the submission, symlog alone is competitive with DIAMOND on Atari (IQM difference −0.02 [−0.12, +0.08]) and leads the diffusion and flow baselines on three of the four Craftium tasks, with Room the exception, and on Crafter (Tabs. 1, 2 and 4). To simplify the choice of heads, we would therefore use symlog everywhere, as DreamerV3 does, and make it the headline result.
  • We will revise the abstract to report results with one fixed reward head. The following new text will replace the two results sentences (from "DRIFT's imagination is fast" to "discrete latent-dynamics agents"). "DRIFT's imagination uses 33x fewer FLOPs per imagined transition than prior diffusion-based agents, and a full run costs about one-third of DIAMOND's GPU-hours. With one reward head fixed across benchmarks, DRIFT is competitive on Atari, outperforms diffusion and flow baselines on three of the four Craftium tasks, and achieves the highest Crafter score among diffusion and flow agents. With sign rewards, it outperforms them on all four Craftium tasks."

W3 / Q3. End-to-end cost. The cost comparison is in the general response (End-to-end cost), which also explains the four-day figure. In short, DRIFT takes about one-third of DIAMOND's GPU-hours, consistent with the paper's claim against prior diffusion-based agents.

Q4. Seeds, settings and tuning of the Craftium and Crafter baselines. We evaluate DIAMOND and Horizon Imagination over five seeds using the authors' released implementations, matching the standard environment-interaction budgets without additional tuning. DRIFT also uses one architecture and hyperparameter set across all three benchmarks. Its warmup schedule and training budget were partly inspired by DIAMOND, and the number of training updates is the only benchmark-specific setting we explored. On Atari, DRIFT performs two world-model and two actor-critic updates per environment step, against four in DIAMOND. On Crafter, whose interaction budget is ten times larger, we use one update per step for faster training, with a longer 25k-step prefill. We will clarify these settings in App. C.3.

Q1. Which pass gives s_t. We thank the reviewer for catching this ambiguity. In both training and imagination, s_t comes from a separate backbone pass over the clean latent, distinct from the noisy denoising pass. The input is the encoded real latent during training and the newly generated latent during imagination. Because σ = 1 is not on the training grid, this pass is conditioned at the nearest grid point, σ = 63/64 with step size 1/64. The reward, continuation and next-latent losses are therefore all computed on this clean pass. For brevity, Algorithm 1 merged the two passes into one line, which reads as if s_t came from the denoising pass. We will show the two passes separately and give the explicit grid values in Algorithm 2. Tab. 3 already counts both passes (App. E.4.1).

Dear Reviewer,

As we are approaching the end of the rebuttal period, we kindly invite you to let us know whether our responses have addressed your concerns. We would be happy to provide further clarifications or discuss any remaining questions.

Thank you again for your time and thoughtful feedback.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers (1/2)

We thank the reviewers for their careful reviews. They find the research question timely (a46v), the shared design reasonable (BCT5), DRIFT at the level of DIAMOND and STORM with one small online model (MEAW), and one-step imagination and the predictive objective effective (all).

Below, we clarify DRIFT's contribution and address the three concerns raised by more than one reviewer, namely the benefit of sharing the backbone, end-to-end cost, and the margin over DIAMOND. (Citations refer to the submitted version, and DRIFT means the submitted agent unless marked as improved.)

What DRIFT contributes (a46v, BCT5). We agree that DreamerV3 already shares its world-model state with the policy, but its world model is not a flow model. Online agents with diffusion or flow world models (DIAMOND, Horizon Imagination) use the model only as a simulator for a separate policy. DRIFT brings DreamerV3's sharing to this state-of-the-art class of generative world models, unifying the world model and the policy's state in one network without a separate policy encoder. Below, we summarize why this is non-trivial and significant.

  • Non-trivial (a46v). Unlike DreamerV3's RSSM, a backbone trained to denoise gives a weak control state on its own (Fig. 3a).
  • Targeted adaptations (BCT5). Neither SPR nor Self-Forcing transfers directly, and DRIFT's adapted versions each fix an observed failure (Fig. 3; details in our reply to BCT5, W1).
  • Significance (a46v). DRIFT's policy reads the features of a diffusion world model with continuous latents, while the other latent agents in Tab. 1 use categorical latents or discrete tokens. Its imagination keeps visual detail that DreamerV3 loses (reply to a46v, W1), and we expect the shared design and its findings to be useful for diffusion-based agents more broadly.
  • Efficiency (MEAW, BCT5). 33x fewer FLOPs per imagined transition than DIAMOND, and about one-third of its GPU-hours end to end.

Requested experiments. Following the reviewers' suggestions, we ran new experiments on the submitted agent. Each ablation changes one component of the submitted agent and keeps everything else fixed, including training steps and updates. To fit the rebuttal window, we use three seeds on four of the paper's eight ablation games (KungFuMaster, ChopperCommand, CrazyClimber, Hero). We chose these games because they have the lowest seed variability, and they span combat, shooting, climbing and platform games. We compare final 100-episode evaluations with the submitted agent on the same three seeds, which are a subset of the five in Tab. 1.

Does sharing the world model's backbone help the policy? (MEAW, BCT5) Yes, and the gain does not come from the next-latent loss alone. We keep the submitted agent fixed and give the actor-critic its own encoder, a second copy of the world model's Transformer with the world model's optimizer settings and the same training steps. This encoder learns only from the critic loss (sep), or also from DRIFT's next-latent loss (sep+pred). The shared backbone receives no critic gradient (Sec. 4.3) and learns only from the world-model losses, which include the next-latent loss. We also test DRIFT's actor-critic on the decoded frames of the same imagination, read through a convolutional encoder (pixel policy).

Best per game in bold.

policy stateKungFuMasterChopperCommandCrazyClimberHero
shared backbone (DRIFT, submitted)25,2951,79079,47911,189
sep4,15096216,8801,776
sep+pred15,8271,26427,037994
pixel policy10,3241,62829,3189,963

Adding the next-latent loss narrows the gap on three games, but sep+pred still trails the shared backbone on all four, while adding 18.2M parameters and about 1.5x the time per epoch. Since both use the next-latent loss, the gap comes from the world-model training that shapes only the shared backbone. The pixel policy also trails on all four games, so the world model's features are a better policy state than its frames.

AuthorsOpenReview ↗

General response to all reviewers (2/2)

End-to-end cost. We agree that cost per imagined transition alone does not establish end-to-end efficiency. We timed the official implementations and default configurations on the same NVIDIA L40S, using Alien, seed 0, and 100k interactions. DRIFT's and DIAMOND's totals follow from their measured training rates, and STORM uses its default ALE v5 protocol with sticky actions.

methodGPU-hoursAtari 100k IQM
diffusion and flow
DRIFT (submitted)about 250.673
DIAMOND790.641
discrete latents
TWISTER11–
STORM4.30.636
DreamerV32.50.497

End to end, DRIFT takes about one-third of DIAMOND's GPU-hours, which is the comparison behind the paper's efficiency claim against prior diffusion-based agents. The 33x saving in Tab. 3 is per imagined transition. It shrinks to about 3x end to end, because training also updates the world model, tokenizer and policy, and DRIFT imagines about 5.6x more transitions per environment step. Transformer world models cost more than DreamerV3's recurrent one, and diffusion adds further cost, which DRIFT reduces by generating in one step. The four-day figure in App. C.4 was measured on an earlier version of our code that recomputed past context at every imagined step instead of caching it. The submitted results come from the faster version timed here, and we will replace the figure with these GPU-hours.

Improved training of DRIFT (added). We also present improved training of DRIFT as an addition, which addresses the concerns about choosing the reward head per benchmark (MEAW) and the modest margin over DIAMOND (MEAW, a46v). The architecture and the shared backbone are unchanged, and the training changes follow findings already in the paper. The main change is to next-latent prediction. Simply predicting four steps ahead did not help (App. E.2.2), so, following SPR and TWISTER, we predict up to eight steps ahead, with randomly shifted future frames as targets. We also simplify the reward-head choice by using the symlog head on every benchmark, and train the critic on replayed real states as well as imagined ones, in the spirit of the self-generated rollouts that narrow the gap between training and imagination (App. E.3). For simplicity, we also drop the warmup phase.

These changes make DRIFT clearly stronger. On Atari 100k, it reaches the highest median (0.805) and the most superhuman games (13 of 26) in Tab. 1, and its IQM rises from 0.673 to 0.733. The improvement over the submitted agent is reliable (probability of improvement 0.63 [0.56, 0.70]), and the median is now significantly above DIAMOND's (difference +0.43 [+0.08, +0.51]), although its IQM gain over DIAMOND is not significant. On Craftium, it leads all methods on three of the four tasks and more than triples the best previous return on Speleo.

Human-normalized scores over 26 games. The improved agent uses 3 seeds; the other rows are from Tab. 1.

Atari 100k#SuperhumanMeanMedianIQM
DreamerV391.0970.5600.497
STORM101.2660.5800.636
TWISTER121.6200.770–
DIAMOND111.4590.3730.641
DRIFT (submitted)121.3480.5200.673
DRIFT (improved)131.4740.8050.733

Each change contributes on the four ablation games. Removing the shift augmentation or the replay critic lowers the return on all four games, and one-step prediction lowers it on three.

Final 100-episode evaluations, mean of 3 seeds, best per game in bold.

improved DRIFTKungFuMasterChopperCommandCrazyClimberHero
full25,1883,08066,58611,471
without shift augmentation19,7402,54646,4299,532
one-step prediction16,9752,59777,6717,501
without replay critic23,9702,51957,54211,440

On Craftium, the improved agent trails DRIFT-symlog only on ChopTree, where it still outperforms the prior diffusion and flow baselines. We leave out Crafter, because 1M-step training is hard to fit into the short rebuttal window and DRIFT-symlog already leads the diffusion and flow baselines there (Tab. 4).

Mean return at 100k steps, 3 seeds; the other rows are from Tab. 2.

CraftiumSmallRoomRoomSpeleoChopTree
DIAMOND−172.6−546.1−2,047.07.3
Horizon Imag.−40.9−135.5−339.613.8
DRIFT-symlog−18.6−649.6669.831.6
DRIFT (submitted)−19.4−123.21,727.615.3
DRIFT (improved)−13.4−40.75,595[PENDING, 1 of 3 seeds: 21.1]

Planned revisions. We will upload a revised PDF before the discussion ends, with changes marked. It adds the requested experiments, the improved training with its full settings, and the changes promised in each reply.

Dear Reviewers, 

We’re about halfway through the discussion period. Please read the authors’ rebuttal and assess whether it adequately addresses the key concerns raised in your review. We encourage you to leave a follow-up comment for the authors, indicating which concerns have been resolved and whether any important issues remain or require further clarification. Please update your review and scores as appropriate in light of the rebuttal and discussion.

Thank you for your time and participation. 

Best regards, 

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.