acceptodds
Under review as a conference paper at ICLR 2027

ReDreamer: Geometry-Grounded Memory for Real-Time Driving World Models

12 views

Abstract

Video world models should generate evolving scenes in real time while preserving object appearance over time. Yet objects can reappear with altered appearances despite correct motion and geometry. The model reproduces where they are, but not what they looked like. Our key idea is to use the persistent object identities and geometry in scene controls as addresses for appearance memory. ReDreamer is a chunk-wise autoregressive video diffusion model that records generated appearances and retrieves them at current object locations through geometry-grounded attention. However, training with real-image references while reading generated appearances at inference introduces exposure bias in memory conditioning. We address this mismatch through rollouts that build and read their own memory, using distribution matching distillation to supervise continuations without forcing generated appearances to match the original dataset video. On our nuScenes-derived re-entry evaluation, ReDreamer improves our Re-entry Appearance Consistency (RAC) score by 10.5% over the strongest evaluated baseline. Enabling memory reads improves RAC by 12.0%. With four denoising steps per chunk, it reduces FVD by 5.7% relative to MagicDrive-V2 in our front-view evaluation and generates at 8.1 FPS.

Primary area:
generative models
Keywords:
video world models, autoregressive video generation, object-centric memory, appearance consistency, controllable video generation, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hViTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReDreamer addresses appearance inconsistency when objects re-enter the camera view in causal driving video generation. It maintains a geometry-grounded object memory bank: object features are extracted from the model's own generated outputs and bound to persistent instance identities. During generation, current object projections attend to the corresponding stored records through a region-masked attention mechanism, allowing the model to retrieve the same object's visual history as its position and visibility change. To reduce foreground occlusion in dense traffic, the paper introduces a footprint representation that replaces full 3D cuboids with projected ground footprints for layout control. Training uses causal rollouts with self-generated memory and distribution matching distillation (DMD) to address exposure bias between real-image training references and generated-memory deployment. The paper also proposes a Re-entry Appearance Consistency (RAC) metric and constructs controlled re-entry cases from nuScenes. Experiments report the highest RAC among evaluated driving generators and the lowest FVD with four denoising steps.

Critical strengths
  1. Geometry-grounded memory is a well-motivated and effective formulation for object re-entry. Binding appearance records to persistent instance identities, rather than relying solely on geometric controls or pixel-space reference frames, directly targets a failure mode that layout-conditioned generators exhibit but do not solve.
  2. The footprint representation is a simple, practical contribution to controllable driving generation. Replacing full cuboids with projected ground footprints reduces mutual occlusion among control primitives. This is a simple but effective idea.
Critical weaknesses
  1. The evaluation resolution and metric design may overstate appearance consistency. All re-entry experiments are conducted at 320×480320 \times 480. At this resolution, DINOv3 features on cropped vehicle regions are likely dominated by color and coarse silhouette; fine-grained attributes are overlooked. RAC therefore may not distinguish "same coarse appearance" from "same vehicle identity".
  2. The Wan2.1-T2V-1.3B base model advantage is not isolated. ReDreamer is built on Wan2.1-T2V-1.3B, while several baselines are built on older or even no pretrained weights. Table 3 shows ReDreamer achieving the lowest FVD, but without a backbone-matched control. Also, the contribution of the proposed memory to overall generation quality cannot be cleanly separated from the base model's generative capacity.
  3. Citation placement (minor). The mention of “Causal Forcing” (e.g., L282) should carry a citation at the first mention rather than only at the sentence end.
Questions for authors
  1. Clarification on evaluation framework. Did the authors adopt the evaluation protocol of the driving-video benchmark associated with arXiv:2507.01735 on the nuScenes validation split? Table 8 and Table 11 appear to share scene counts and temporal windows with that line of work, but I failed to see any reference to that work. If the framework was not used, a unified comparison under that protocol would strengthen the baseline evaluation.
  2. Backbone contribution. Can the authors provide a backbone-controlled experiment—for example, applying the same geometry-grounded memory to a baseline backbone (e.g., the UniMLVG or MagicDrive-V2 base), or reporting a Wan2.1-T2V-1.3B variant without memory under the identical RAC protocol—to isolate the gain attributable to memory from the gain attributable to the base model?
  3. RAC validity at low resolution. Would RAC remain discriminative between "same object" and "different object of the same color and category" at this resolution, and does the ranking of methods change at a higher generation resolution?
Reviewer_P819OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses appearance inconsistency of objects that leave and re-enter the view in layout-controlled driving world models: box controls fix where a vehicle is, but not what it looks like. ReDreamer, a chunk-wise autoregressive video diffusion model, uses the persistent instance IDs and geometry in the scene controls as addresses for an object appearance memory.

  • Write. When an object is first generated, its region is cropped from the decoded output, encoded with the frozen VAE, and stored as local features under its instance ID.
  • Read. In later chunks, a memory cross-attention lets each query patch attend only to the records of objects whose current projection covers that patch.
  • Training. After teacher-forced training and consistency distillation, a third stage applies DMD on causal rollouts that build and read their own memory. This removes the mismatch between real-image references in training and generated references at deployment, without forcing generated appearances to match the dataset video.

The authors also propose a controlled re-entry benchmark built from nuScenes and the RAC metric. ReDreamer achieves the best RAC among the evaluated driving generators and competitive video quality with four denoising steps per chunk, running at 8.1 FPS. Ablations isolate the effect of memory reads and of self-generated-memory training.

Critical strengths
  1. Clear problem definition with a well-matched design. The paper isolates a concrete failure mode: correct motion but changed appearance after re-entry. It addresses it with components whose roles are clearly motivated:

    • instance IDs select the record;
    • projected geometry restricts where it can contribute;
    • attention selects local features within that region;
    • DMD is used instead of paired reconstruction precisely so that the model preserves its own earlier appearance choices rather than being pulled toward the dataset video.

    The method and its rationale are easy to follow.

  2. Thorough and well-targeted evaluation. The paper introduces a dedicated re-entry protocol and metric (RAC) and complements them with standard video-quality metrics. Its ablations separate the contribution of memory reads (RAC 0.4506 → 0.5046) from that of self-generated-memory training (0.4569 → 0.5046, with better FID/FVD). It also reports runtime and memory-reading overhead (8.1 FPS, +13.4% pipeline time). Overall, the work is complete and convincing for the targeted problem.

Critical weaknesses
  1. Limited scope and generality. The memory relies on persistent instance IDs and 3D boxes supplied by the scene controls, and the evaluation covers only vehicles (Appendix Table 8). It is therefore unclear how the approach extends to other memory problems in world models:

    • non-rigid objects whose appearance changes with pose (e.g., pedestrians), given that each record is fixed at first observation;
    • static scene content without instance IDs (e.g., revisiting the same location);
    • settings where identities and boxes come from imperfect tracking.

    This does not undermine the results within the stated driving setting, but it limits the significance of the contribution for the broader world-model memory problem.

Questions for authors
  1. The core claim is that instance IDs and projected geometry are effective addresses for appearance memory, but Table 4 only compares memory reads on vs. off. Could you add an ablation that keeps the memory but removes the geometry-grounded addressing (e.g., memory attention without the ID/projection mask)? A comparison with InstaDrive, which Table 1 lists as a driving generator with layout control and appearance memory, would also help position the method.
  2. How robust is the memory to errors in the supplied controls, such as ID switches or box offsets that would occur with tracked rather than ground-truth identities?
Reviewer_YiDJOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ReDreamer is a real-time autoregressive video diffusion model that preserves consistent object appearance during scene re-entry by linking appearance memory to geometry-grounded spatial addresses. Using self-rollout training and distribution matching distillation to overcome exposure bias, ReDreamer achieves an 8.1 FPS generation rate and improves Re-entry Appearance Consistency

Critical strengths
  • Redreamer proposes a novel appearance memory system for autonomous driving scenarios, which stores and retrieves persistent object appearances using geometry-grounded spatial addressing to maintain consistency upon object re-entry.
  • Redreamer prevents exposure bias effectively.
  • The method is very efficient.
Critical weaknesses
  • Regarding technical novelty, the authors should clearly explain how ReDreamer differs from prior memory-augmented approaches like VMEM and WorldMem in terms of memory construction and retrieval. Specifically, is the primary distinction that ReDreamer explicitly maintains object-level appearance memory, or are there fundamental structural differences in how memory is organized?
  • The manuscript would benefit from a discussion or ablation study investigating the robustness of the memory retrieval mechanism under significant camera viewpoint variations.
  • Only a minimal number of video results are provided, limiting the qualitative evaluation of the proposed approach.
Questions for authors
  • Why is "an identity having a consistent appearance", or "Re-entry Appearance Consistency" a critical problem in autonomous driving? This should be highlighted in the introduction.
  • Does the memory gets updated after object re-entry? Or does the memory kept same once it's observed intially?
  • In L266, how does one know which pixel is the surrounding context and which is the object?
  • Regarding the training strategy to prevent exposure bias, its not clear what exactly the ReDreamer is different against Causal Forcing series.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.