- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a controllable driving-video diffusion model that addresses two challenges simultaneously: precise scene and object control and long-term consistency across autoregressive video chunks. The core idea is to represent every dynamic and static scene instance as a 3D block, render these blocks into the camera view as a dense conditioning signal, and use instance-level KV caching to provide explicit object-level memory across chunks. The data-curation pipeline is largely automatic: it combines segmentation and geometry estimation to lift image observations into a canonical 3D blocks-world representation. The video generator itself builds on an off-the-shelf pretrained backbone, Cosmos-Predict2.5, which is kept frozen, together with a trainable ControlNet-style branch for blocks-world conditioning.
Critical strengths
-
For the video-generation component, a major strength is the paper’s elaborate data-curation pipeline for constructing a dense, persistent, 3D-grounded scene representation from raw videos, rather than largely relying on the structured annotations already provided by autonomous-driving datasets.
-
The second strength of the work is its instance-addressable long-term memory. To improve consistency across autoregressive video chunks, the authors maintain a bank of internal DiT attention features indexed by explicit object identity and geometry. This allows the model to retrieve and spatially realign object-specific memories when an instance reappears, rather than relying only on frame-level context or raw image patches or dense KV-caching (which could be expensive).
Critical weaknesses
I have several concerns regarding the interpretation of the reported results, the practical requirements for applying MemOIR to new scenes, the evaluation of long-term video quality, memory performance and the presentation.
1. Differences in model inputs raise a fairness concern.
The authors compare the performance of models that receive different inputs, which raises concerns about the fairness of the comparisons in Table 1 and Figure 4. For example, in the temporal quality comparison, MemOIR receives a real initial frame, while some of the other models (such as MagicDrive) receives no image. The authors acknowledge this difference in the appendix, but stating it clearly in Section 4.2 would help readers interpret the results.
On the other hand, these differences make it difficult to determine how much of MemOIR’s performance advantage comes from its memory, the initial image, the blocks-world representation, or the underlying video model. The memory ablations provide useful evidence of the benefit of memory within MemOIR, but they do not address the differences in inputs across methods.
2. The input requirements for generating new scenes need clarification.
MemOIR relies on object geometry and consistent object identities across frames. It would be helpful to explain how users can obtain this information for new scenes when no source video or existing layout data is available.
Related work provides useful examples of preparing generation inputs from more limited information. Ctrl-V [1] predicts bounding-box sequences from an initial RGB image and box constraints, while DriveDreamer-2 [2] generates agent trajectories and road layouts from text. These methods use different scene representations, but they address the practical step of creating the inputs needed for video generation.
For MemOIR, I would appreciate a clear description of the minimum inputs needed without a source video and how users can create a compatible blocks-world scene. An estimate of the preparation effort would also help readers assess how easily the method can be applied to new scenes.
3. The decline in quality during longer generation needs further discussion.
Table A4 suggests that MemOIR’s video quality declines more rapidly than that of the other methods as generation continues. The authors note that the real initial frame helps early generation and that quality drifts as generation moves farther from that frame.
Given the role of KV memory in preserving information across video chunks, I would expect it to help limit this degradation. However, its effect on the rate of decline is unclear from Table A4. Further analysis or discussion of what causes the decline would be helpful for the readers.
[1] Luo, G. Y., Luo, Z. H., Gosselin, A., Jolicoeur-Martineau, A., & Pal, C. (2025). Ctrl-V: Higher fidelity autonomous vehicle video generation with bounding-box controlled object motion. Transactions on Machine Learning Research. https://openreview.net/forum?id=n6To2wAOKL
[2] Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., & Wang, X. (2025). DriveDreamer-2: LLM-enhanced world models for diverse driving video generation. Proceedings of the AAAI Conference on Artificial Intelligence, 39. https://ojs.aaai.org/index.php/AAAI/article/view/33130
Questions for authors
Thank you to the authors for their hard work on this paper. I would appreciate it if the authors could address the concerns raised in Weaknesses 1-3. I have also noted several minor comments below on the experiments and writing, which I hope will help improve the paper. I would be willing to reconsider my score if the authors adequately address all of them.
-
Minor concern: Instance consistency could be evaluated beyond vehicles. The vehicle ReID evaluation provides useful evidence of car consistency. It would also be valuable to evaluate buildings and other static landmarks. In Figure 8 and the minute-long demos, some buildings appear to change in appearance or structure after U-turns, raising questions about how well the memory preserves these objects. A car’s broad appearance can often be described by a few attributes, such as color, size, and brand. Preserving a building’s identity may require remembering a more complex combination of architectural details, including its overall shape, material, and window layout. Buildings could therefore provide a more demanding test of whether the memory retains detailed visual and structural information beyond a small set of object attributes.
-
Minor concern: Some presentation details could be improved. It would be helpful to use consistent terminology. For example, in Table 1 and Section 4, using either “video quality” or “visual quality” when referring to the same metric category. In Figure 2, showing the reference, target, and generated frames from the same scene would also make the model diagram easier to follow and clarify the relationship between its inputs and output.
-
Minor concern: Ego-motion error reporting. Reporting percentage errors and variances for ego-motion controllability metrics would help clarify typical performance and whether a few large errors dominate the reported averages.
-
Could the authors consider using the same real initial frame as a motion-conditioning frame for DreamForge, if supported by the model, to enable a fairer comparison with MemOIR?
-
The Reproducibility Statement section appears to still contain the original template instructions. Could the authors please replace them with an actual statement or remove the placeholder?
10 comments
3. Our full MemOIR gives the best quality while keeping the control. Full MemOIR achieves the best visual quality on all four FID/FVD columns and improves on the frame-only backbone (FID 37.2 → 23.6; FVD 780.4 → 548.9). The blocks-world condition therefore contributes to quality as well as control: by specifying the static scene environments and the ego motion, it consistently keeps the rollout in a similar environment to the initially observed scene. Its controllability stays at the level of the variant without an initial frame (vehicle tracking 73.0% vs. 73.4%) and far above every other frame-conditioned model (ego error 8.4% vs. 40.6–59.1%). The small remaining gap to this variant (ego error 4.4% vs. 8.4%; pedestrian mIoU 0.464 vs. 0.421) is expected, and we attribute it to the model having to satisfy two conditions at once. Without a frame, the model is free to synthesize content that fits the blocks exactly. With a real frame, it must also preserve the true geometry of the observed scene, which the coarse boxes and footprints only approximate; wherever the two disagree, the output follows the real scene, and so deviates slightly from the specified layout and trajectory. Qualitatively, the baselines reproduce the given frame but, within about a second, replace the scene with generic, nuScenes-like content, whereas MemOIR stays faithful to the observed street throughout (rebuttal page)).
The remaining component, memory, is isolated in the memory ablation (Table 6 of the revised paper) and evaluated further in A4.
W2. What are the minimum inputs for generating a new scene?
A2. The model requires only a rendered blocks-world condition. HD maps, footprints and LiDAR are optional sources for building it**, and the blocks world can be built from a single initial image with off-the-shelf models alone.** Thank you for this practical question. Like Ctrl-V and DriveDreamer-2, which prepare generation inputs from an initial image or from text, our construction pipeline addresses the practical step of creating the inputs needed for generation.
- What the model requires. The view-space rendered blocks-world condition, a text prompt and, optionally, the real first frame. The model never reads HD maps, LiDAR, footprints or annotation files, so any procedure that outputs a 3D box world can drive it.
- The blocks world can be built from a single initial image, using off-the-shelf models only. Our construction pipeline is purely algorithmic: it takes a single initial image and its camera intrinsics, and combines off-the-shelf models with deterministic geometry. It runs as a series of algorithms; we train no model for it, and it needs no HD map, 3D annotations, footprints or LiDAR.
- Camera and ground: monocular metric depth (Depth Anything 3) lifts the image into 3D, and the camera height and pitch are estimated by fitting the ground plane.
- Dynamic instances: 3D detections (WildDet3D) are associated with instance masks (SAM 3), giving one 3D box with a persistent instance identity per vehicle and pedestrian.
- Static structure: semantic segmentation (Mask2Former, Mapillary Vistas classes [2]) lifted through the depth gives building footprints and heights fitted to the visible facades and rooflines, vegetation volumes, road boundaries and lane lines. Traffic lights, signs and poles come from a detector (D-FINE [3]).
- Motion: each instance block follows a trajectory along a lane-direction field estimated from the lane lines, or one specified by the user.
- What our training and evaluation used. In the paper, we built both the training and the validation blocks worlds with ground-truth annotations for the road layer and the dynamic instances (HD map with lane lines, and 3D boxes), the same inputs the baselines require. Everything else is built with off-the-shelf models such as Depth Anything 3 and Mask2Former; public building footprints and LiDAR are used only where available, each with a video-based fallback.
The result is a manipulable code world: every element is an editable primitive with an instance identity. It renders directly into our blocks-world condition, from which MemOIR generates the full video. The whole pipeline runs automatically in minutes per scene. Users only need an image and, optionally, the desired instance motions. Examples, from a single input image through the constructed world to the generated video, are on our rebuttal page).
We additionally find that an LLM agent can further improve and extend the world. Beyond the automatic pipeline, we find that an LLM agent can refine the constructed world in a closed loop, following the refinement approach of SpatialClaw [4]. The agent renders the world, compares it with the image, corrects building heights and footprints, adds missing structures (far facades, the end of the road, overpasses) and removes spurious ones. It also helps users expand and edit the scene, for example by populating plausible per-instance block behaviours across the whole scene from a single initial image. Examples are on our rebuttal page, and we describe this refinement in Appendix B.8 of the revised paper.
W3. What causes the quality decline over long generation?
A3. We identify three causes of the relative decline: in-domain recall by the baselines, a similarity bias in window-wise FID/FVD, and a mismatch in chunk length between training and inference. Thank you for this careful observation.
1. On nuScenes, the competitive baselines benefit heavily from recall. As detailed in A1, MagicDrive-V2 and DreamForge are trained only on nuScenes, whose training and validation splits are recorded in the same city areas, and their BEV maps and ego pose effectively identify the location. They can therefore recall scenery seen during training, which keeps their later windows close to the ground truth in Table A6 in the revised paper. MemOIR, conditioned only on a view-space blocks-world condition, must generate this scenery instead. We therefore evaluate on zero-shot Lyft L5 (80 scenes), which removes this advantage. Every method also receives the same real initial frame and generates the same number of frames, giving the fairest possible ground for comparison. Under these conditions, as shown in Table R2, MemOIR outperforms both baselines by a wide margin in every window, from the first (FID 24.9 vs. 89.2 and 99.6) to the last (FID 49.3 vs. 139.3 and 100.7; FVD 638.9 vs. 885.0 and 878.4). The qualitative comparisons in A1 show the same gap (rebuttal page).
Table R2. Per-window FID/FVD on Lyft L5 (zero-shot, 80 scenes), same-scene reference.
| Method | FID 0–4 s | 4–8 s | 8–12 s | 12–15.3 s | FVD 0–4 s | 4–8 s | 8–12 s | 12–15.3 s |
|---|---|---|---|---|---|---|---|---|
| MagicDrive-V2 | 99.6 | 105.4 | 97.1 | 100.7 | 941.7 | 985.4 | 935.3 | 878.4 |
| DreamForge | 89.2 | 111.2 | 129.8 | 139.3 | 789.4 | 887.9 | 913.4 | 885.0 |
| MemOIR (ours) | 24.9 | 33.9 | 41.9 | 49.3 | 326.3 | 487.8 | 585.7 | 638.9 |
2. With few samples, window-wise FID/FVD partly measure similarity to the specific scenes. FID and FVD compare distributions, but with only frames of 80 videos, the reference set is essentially the evaluated scenes themselves. Scored against the same scenes’ real frames, these metrics reward reproducing those exact scenes. Because MemOIR follows the real initial frame closely, its first window scores exceptionally well. Its score then rises as it generates content the initial frame never showed, even when that content is realistic. Therefore, to measure the realism of the generated videos independently of scene similarity, we additionally score each window against a cross-scene reference: an equal number of real frames drawn from other scenes in the test set, rather than from each video’s own ground truth (Table R3).
Table R3. Per-window FID/FVD on Lyft L5 (zero-shot, 80 scenes), cross-scene reference. (Quality-focused measurement)
| Method | FID 0–4 s | 4–8 s | 8–12 s | 12–15.3 s | FVD 0–4 s | 4–8 s | 8–12 s | 12–15.3 s |
|---|---|---|---|---|---|---|---|---|
| MagicDrive-V2 | 87.0 | 91.6 | 87.8 | 91.7 | 876.9 | 877.7 | 956.8 | 891.4 |
| DreamForge | 80.6 | 102.0 | 125.8 | 137.5 | 758.2 | 867.4 | 913.2 | 963.3 |
| MemOIR (ours) | 44.9 | 44.1 | 48.0 | 55.7 | 552.2 | 595.0 | 641.0 | 703.3 |
Against the cross-scene reference, MemOIR’s FID stays flat over the first 8 s. Its gap to the same-scene score shrinks from 20.0 to about 6 by 8–12 s, so most of the early rise reflects divergence from the specific scene, not lower realism. The baselines, by contrast, score better against other scenes than against their own (e.g., DreamForge 80.6 vs. 89.2 at 0–4 s), which shows that they depart from the given scene almost immediately. Their curves therefore start from a much worse level, leaving little room to degrade further: MagicDrive-V2’s FID stays around 87–92 from the first window onward, which reflects early failure rather than stability. Under both references, MemOIR achieves the best FID and FVD in every window, so its advantage holds for both fidelity to the observed scene and overall realism.
W1. Fairness of comparisons across different model inputs.
A1. We have added an input-matched, zero-shot comparison that also separates the contribution of each component. Thank you for raising this important point. In the original Table 1, we ran MagicDrive-V2 and DreamForge under their default protocol without an initial frame, which the revised paper now states clearly (Sec. 4.1, Table 1 and the Fig. 4 caption). To remove this difference, we re-ran the evaluation with matched inputs, giving both baselines the same real initial frame as MemOIR.
We also moved the evaluation to a zero-shot setting on Lyft Level 5 [1] dataset, which no method (ours or the baselines) saw in training, as we find that nuScenes favours the competitive baselines - MagicDrive-V2 and DreamForge are trained on nuScenes alone, and its training and validation splits are recorded in identical city areas, differing only in the driving scenarios. Because their BEV road maps and ego pose effectively identify the location, these models can largely recall the scenery seen there during training, keeping their outputs close to the ground-truth frames. MemOIR receives only a view-space blocks-world condition of coarse geometric primitives, with no global map or location input, so it must generate the scenery rather than recall it. On nuScenes, frame-level similarity to the ground truth therefore understates the realism of its videos. Using zero-shot Lyft removes this advantage and compares pure generation capability rather than recall. Given the rebuttal timeline, we use 80 of Lyft’s 180 scenes sampled randomly. Each method generates the same 15.4 s window (185 frames at 12 Hz), scored against the real camera keyframes.
Table R1 reports this comparison. To separate the contribution of each component, it also includes two controls: the Cosmos-Predict2.5 backbone given the same initial frame but no blocks-world condition, and MemOIR with the blocks-world condition only, without any initial frame. The results are given in the rebuttal page.
Table R1. Input-matched comparison and component analysis on Lyft L5 (zero-shot, 80 scenes). [Result videos]
1. Controllability comes from the blocks world. With the blocks-world condition alone and no initial frame, MemOIR already tracks 73.4% of vehicles ([email protected] 79.6) with an ego error of 4.4% of the path. The same backbone without the blocks-world condition reaches only 20.8% and 42.5%, even though it receives the real initial frame, and the baselines reach 27.6–32.1% and 40.6–59.1%. Neither the backbone nor the initial frame provides control; the blocks-world condition does.
2. The initial frame provides similarity to the ground truth. The frame fixes the appearance of the specific scene, which is what FID/FVD against the same scenes reward: with the frame alone, the uncontrolled backbone already reaches an FID of 37.2. Without it, MemOIR must invent the appearance of the street solely from coarse blocks. Even so, MemOIR with the blocks-world condition only receives no image at all, yet its FID is comparable to that of the baselines, which do receive the real frame (92.4 vs. 87.0 and 97.7; 95.5 vs. 89.5 and 111.5 against other scenes), with a somewhat higher FVD (1057.8 vs. 845.0–928.9).