- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DRIFT, an online model-based RL framework that shares a Flow Transformer backbone between world modeling and policy learning. It incorporates next-latent prediction to improve control representations, self-generated rollout training to mitigate distribution mismatch, and shortcut flow matching for efficient one-step imagination. Experiments on Atari, Craftium, and Crafter demonstrate competitive control performance with substantially reduced imagination costs.
Critical strengths
-
The paper addresses a meaningful problem in efficient online model-based RL. Sharing representations between generative world modeling and policy learning is a reasonable design choice.
-
The experiments cover several benchmarks and include useful ablations. In particular, next-latent prediction and self-generated rollout training lead to substantial improvements, while one-step imagination significantly reduces computational costs.
Critical weaknesses
-
My biggest concern is the limited methodological novelty. Most technical components are adaptations of existing approaches, including shared world-model representations (Dreamer), self-predictive representation learning, Self-Forcing, and Shortcut Flow Matching. The paper mainly combines these established techniques into a unified framework, with limited evidence of a fundamentally new methodological contribution.
-
The paper emphasizes using one Flow Transformer for both imagination and control, but lacks a direct comparison against a separate-policy architecture using the same world model and auxiliary objectives. Without this controlled ablation, it is unclear whether representation sharing itself provides meaningful benefits beyond the additional training objectives.
-
Self-generated rollout training introduces two objectives, (L_{\mathrm{sf}}) and (L_{\mathrm{sf-FM}}), but their individual contributions are not isolated. Similarly, the substantial gains from next-latent prediction are not compared against existing predictive or contrastive representation-learning objectives. These missing comparisons make it difficult to establish the necessity of the specific design choices.
-
The method supervises generated latent trajectories using rewards and continuation labels from the corresponding real trajectories. However, as generated states deviate from the real trajectory, these labels may become inconsistent with the generated states. The paper provides limited analysis of this potential supervision bias and its effect on longer-horizon imagination.
Questions for authors
-
Can the authors compare the shared-backbone architecture against an otherwise identical world model with a separate policy network under matched training and computational budgets?
-
What are the individual contributions of (L_{\mathrm{sf}}) and (L_{\mathrm{sf-FM}})? How does next-latent prediction compare with existing predictive or contrastive representation-learning objectives?
-
How sensitive is self-generated rollout supervision to deviations between generated and real trajectories, particularly over longer imagination horizons?
-
Can the authors provide more direct statistical comparisons against DIAMOND and report end-to-end training efficiency under matched computational budgets?
3 comments
Response to Reviewer BCT5 (2/2)
W3 / Q2 (b). Next-latent prediction against a contrastive objective. We thank the reviewer for suggesting this comparison, which tests a direction the paper proposes in App. A and App. E.2.2. AC-CPC, the action-conditioned contrastive objective used by TWISTER, is a natural alternative to DRIFT's predictive objective. We replace next-latent prediction with AC-CPC.
Rows 1-2 are the online-tokenizer rows of Fig. 3a (symlog rewards, 5 seeds), and rows 3-4 are final evaluations of the submitted sign-reward agent and its AC-CPC variant (3 seeds). Best per game in bold.
| objective | rollouts | KungFuMaster | ChopperCommand | CrazyClimber | Hero |
|---|---|---|---|---|---|
| none (Fig. 3a) | no | 18,178 | 909 | 50,660 | 866 |
| next-latent (Fig. 3a) | no | 19,516 | 1,124 | 73,083 | 8,582 |
| next-latent (DRIFT, submitted) | yes | 25,295 | 1,790 | 79,479 | 11,189 |
| AC-CPC (new) | yes | 25,591 | 1,684 | 92,595 | 12,622 |
AC-CPC performs as well as or better than next-latent prediction on three of the four games. We expect its gains to come from its richer predictive signal, since it looks ten steps ahead and contrasts each prediction against other states. The shared state needs a predictive objective, whichever form it takes, and a stronger one can improve DRIFT further. We will report AC-CPC in the revised ablation as a promising alternative to the simpler next-latent loss.
W4 / Q3. Validity of labels on generated states. Generated states do deviate from the real trajectory, but over DRIFT's 15-step horizon the resulting label bias is small on average. Extra wrong labels stay within one percentage point and critic bias within about 1% of the return scale at every step (first table below). Horizons beyond 15 imagined steps are outside the scope of this work, and we will list a horizon sweep as a limitation.
DRIFT uses these labels only on the first four generated steps, with weights that decay with depth (App. B.2), so most supervision falls where deviation is smallest. Steps 5 to 15 receive no labels, but the actor and critic still learn from the rewards and values predicted there, so we test all 15 steps.
We use the submitted agent's final checkpoints (4 games, 3 seeds). Starting from real states, we generate 15 steps with the recorded actions and attach the real trajectory's labels, as in training. At each step we measure the deviation from the real state, the excess rate of wrong labels on the generated state over the real one, and the critic's bias relative to the real state. A label counts as wrong when a reward head trained only on real states disagrees with it.
| imagination step | 1 | 4 | 8 | 15 |
|---|---|---|---|---|
| deviation from the real state (latent MSE) | 0.13 | 0.19 | 0.27 | 0.38 |
| extra wrong labels on the generated state (percentage points) | 0.0 | 0.8 | 0.8 | 0.5 |
| critic bias (% of return scale) | 0.3 | 0.8 | 0.5 | 0.8 |
Deviation triples over 15 steps, yet the label and critic errors do not grow with it.
Since most steps carry no reward, we also check the steps that do. On them, the reward head is correct less often on generated states than on real ones (76% against 86% on average).
| correct on rewarded steps 5-15 (%) | KungFuMaster | ChopperCommand | CrazyClimber | Hero | average |
|---|---|---|---|---|---|
| generated states | 95 | 23 | 98 | 86 | 76 |
| real states | 100 | 47 | 100 | 96 | 86 |
The difference comes mostly from ChopperCommand, whose rewards are the rarest and are often missed even on real states. We will add these measurements to App. E.3 and list rare rewards as a limitation.
Q4. Statistics against DIAMOND and end-to-end cost.
-
A direct test finds no significant difference between DRIFT and DIAMOND, consistent with the paper's claim of competitive performance. Under the standard protocol of Agarwal et al. (2021), with both agents' per-seed scores on all 26 games and 95% stratified-bootstrap intervals from
rliable, DRIFT scores slightly higher than DIAMOND in both IQM and median, but neither difference is significant (IQM +0.03 [−0.07, +0.14], median +0.15 [−0.04, +0.37]). Sec. 5.2 already states that the intervals overlap, and we will align the introduction's sentence that reports higher median and IQM. -
On cost, DRIFT takes about 25 GPU-hours and DIAMOND 79 under the same 100k-interaction budget (general response, End-to-end cost).
Dear Reviewer,
As we are approaching the end of the rebuttal period, we kindly invite you to let us know whether our responses have addressed your concerns. We would be happy to provide further clarifications or discuss any remaining questions.
Thank you again for your time and thoughtful feedback.
Response to Reviewer BCT5 (1/2)
We thank the reviewer for the careful review. We ran every requested ablation, and the shared backbone outperforms a separate policy network on all four games. We address each concern below.
W1. Novelty beyond adapting existing techniques. We understand this is the reviewer's main concern. We agree that DRIFT builds on the four techniques the reviewer lists, and we explain what each required in DRIFT's setting. What is new is that a diffusion or flow world model, trained online, also provides the policy's state.
Given the shift toward diffusion world models in RL, we expect the result to apply beyond DRIFT.
W2 / Q1. Shared backbone against a separate policy network. We kindly refer the reviewer to the general response (Does sharing the world model's backbone help the policy?), which reports the requested comparison with the same world model and auxiliary objectives. We also trained DIAMOND's own actor-critic in DRIFT's world model with its original settings. It exceeds the shared agent on two games, so a separate policy with its own learning algorithm can also do well on DRIFT's world model.
Final 100-episode evaluations, mean of 3 seeds. Bold marks the better of the two agents trained in DRIFT's world model.
W3 / Q2 (a). The two rollout losses. The two losses are best together on three of the four games, and each alone gives smaller or mixed gains. Training each loss alone, together with the submission's pair (Fig. 3b), gives all four combinations.
L_sftrains the reward and continuation heads on generated latents, andL_sf-FMtrains the dynamics model on sequences that contain its own generated latents.Final evaluations, mean of 3 seeds, best per game in bold.
L_sfL_sf-FMWe expect the pair to work best because the two losses correct different parts of the model on generated latents.
L_sfcorrects the reward and continuation heads, andL_sf-FMcorrects the dynamics.