acceptodds
Under review as a conference paper at ICLR 2027

Driving Video Generation from View-Space Blocks World with Object-Level Memory

Abstract

Video generative models promise driving simulators that can render arbitrary traffic scenarios, but this requires both precise control over the scene and consistency across long rollouts. Existing driving generators condition only on traffic participants and road layout, leaving the static world, and with it the ego motion, underspecified, while frame-level memories grow quickly and cannot tell which cached tokens belong to which instance. We present MemOIR, a driving video diffusion model conditioned on a view-space blocks-world representation that renders every dynamic and static instance as a 3D block encoding its class, identity, and occlusion, curated automatically from raw driving videos. This representation densely specifies both agent motion and the ego trajectory, and enables object-level KV caching: each instance's tokens are cached from its most complete observations and re-embedded where its block projects when it reappears, attending to far fewer tokens than frame-level caching. MemOIR achieves the most accurate instance placement and ego-motion control among driving generators with competitive visual quality, faithfully places vehicles performing unseen maneuvers such as spin-outs, and substantially improves the identity consistency of returning vehicles.

Primary area:
generative models
Keywords:
Video generation, controllable generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tkmbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a controllable driving-video diffusion model that addresses two challenges simultaneously: precise scene and object control and long-term consistency across autoregressive video chunks. The core idea is to represent every dynamic and static scene instance as a 3D block, render these blocks into the camera view as a dense conditioning signal, and use instance-level KV caching to provide explicit object-level memory across chunks. The data-curation pipeline is largely automatic: it combines segmentation and geometry estimation to lift image observations into a canonical 3D blocks-world representation. The video generator itself builds on an off-the-shelf pretrained backbone, Cosmos-Predict2.5, which is kept frozen, together with a trainable ControlNet-style branch for blocks-world conditioning.

Critical strengths
  1. For the video-generation component, a major strength is the paper’s elaborate data-curation pipeline for constructing a dense, persistent, 3D-grounded scene representation from raw videos, rather than largely relying on the structured annotations already provided by autonomous-driving datasets.

  2. The second strength of the work is its instance-addressable long-term memory. To improve consistency across autoregressive video chunks, the authors maintain a bank of internal DiT attention features indexed by explicit object identity and geometry. This allows the model to retrieve and spatially realign object-specific memories when an instance reappears, rather than relying only on frame-level context or raw image patches or dense KV-caching (which could be expensive).

Critical weaknesses

I have several concerns regarding the interpretation of the reported results, the practical requirements for applying MemOIR to new scenes, the evaluation of long-term video quality, memory performance and the presentation.

1. Differences in model inputs raise a fairness concern.

The authors compare the performance of models that receive different inputs, which raises concerns about the fairness of the comparisons in Table 1 and Figure 4. For example, in the temporal quality comparison, MemOIR receives a real initial frame, while some of the other models (such as MagicDrive) receives no image. The authors acknowledge this difference in the appendix, but stating it clearly in Section 4.2 would help readers interpret the results.

On the other hand, these differences make it difficult to determine how much of MemOIR’s performance advantage comes from its memory, the initial image, the blocks-world representation, or the underlying video model. The memory ablations provide useful evidence of the benefit of memory within MemOIR, but they do not address the differences in inputs across methods.

2. The input requirements for generating new scenes need clarification.

MemOIR relies on object geometry and consistent object identities across frames. It would be helpful to explain how users can obtain this information for new scenes when no source video or existing layout data is available.

Related work provides useful examples of preparing generation inputs from more limited information. Ctrl-V [1] predicts bounding-box sequences from an initial RGB image and box constraints, while DriveDreamer-2 [2] generates agent trajectories and road layouts from text. These methods use different scene representations, but they address the practical step of creating the inputs needed for video generation.

For MemOIR, I would appreciate a clear description of the minimum inputs needed without a source video and how users can create a compatible blocks-world scene. An estimate of the preparation effort would also help readers assess how easily the method can be applied to new scenes.

3. The decline in quality during longer generation needs further discussion.

Table A4 suggests that MemOIR’s video quality declines more rapidly than that of the other methods as generation continues. The authors note that the real initial frame helps early generation and that quality drifts as generation moves farther from that frame.

Given the role of KV memory in preserving information across video chunks, I would expect it to help limit this degradation. However, its effect on the rate of decline is unclear from Table A4. Further analysis or discussion of what causes the decline would be helpful for the readers.

[1] Luo, G. Y., Luo, Z. H., Gosselin, A., Jolicoeur-Martineau, A., & Pal, C. (2025). Ctrl-V: Higher fidelity autonomous vehicle video generation with bounding-box controlled object motion. Transactions on Machine Learning Research. https://openreview.net/forum?id=n6To2wAOKL

[2] Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., & Wang, X. (2025). DriveDreamer-2: LLM-enhanced world models for diverse driving video generation. Proceedings of the AAAI Conference on Artificial Intelligence, 39. https://ojs.aaai.org/index.php/AAAI/article/view/33130

Questions for authors

Thank you to the authors for their hard work on this paper. I would appreciate it if the authors could address the concerns raised in Weaknesses 1-3. I have also noted several minor comments below on the experiments and writing, which I hope will help improve the paper. I would be willing to reconsider my score if the authors adequately address all of them.

  1. Minor concern: Instance consistency could be evaluated beyond vehicles. The vehicle ReID evaluation provides useful evidence of car consistency. It would also be valuable to evaluate buildings and other static landmarks. In Figure 8 and the minute-long demos, some buildings appear to change in appearance or structure after U-turns, raising questions about how well the memory preserves these objects. A car’s broad appearance can often be described by a few attributes, such as color, size, and brand. Preserving a building’s identity may require remembering a more complex combination of architectural details, including its overall shape, material, and window layout. Buildings could therefore provide a more demanding test of whether the memory retains detailed visual and structural information beyond a small set of object attributes.

  2. Minor concern: Some presentation details could be improved. It would be helpful to use consistent terminology. For example, in Table 1 and Section 4, using either “video quality” or “visual quality” when referring to the same metric category. In Figure 2, showing the reference, target, and generated frames from the same scene would also make the model diagram easier to follow and clarify the relationship between its inputs and output.

  3. Minor concern: Ego-motion error reporting. Reporting percentage errors and variances for ego-motion controllability metrics would help clarify typical performance and whether a few large errors dominate the reported averages.

  4. Could the authors consider using the same real initial frame as a motion-conditioning frame for DreamForge, if supported by the model, to enable a fairer comparison with MemOIR?

  5. The Reproducibility Statement section appears to still contain the original template instructions. Could the authors please replace them with an actual statement or remove the placeholder?

10 comments

W1. Fairness of comparisons across different model inputs.

A1. We have added an input-matched, zero-shot comparison that also separates the contribution of each component. Thank you for raising this important point. In the original Table 1, we ran MagicDrive-V2 and DreamForge under their default protocol without an initial frame, which the revised paper now states clearly (Sec. 4.1, Table 1 and the Fig. 4 caption). To remove this difference, we re-ran the evaluation with matched inputs, giving both baselines the same real initial frame as MemOIR.

We also moved the evaluation to a zero-shot setting on Lyft Level 5 [1] dataset, which no method (ours or the baselines) saw in training, as we find that nuScenes favours the competitive baselines - MagicDrive-V2 and DreamForge are trained on nuScenes alone, and its training and validation splits are recorded in identical city areas, differing only in the driving scenarios. Because their BEV road maps and ego pose effectively identify the location, these models can largely recall the scenery seen there during training, keeping their outputs close to the ground-truth frames. MemOIR receives only a view-space blocks-world condition of coarse geometric primitives, with no global map or location input, so it must generate the scenery rather than recall it. On nuScenes, frame-level similarity to the ground truth therefore understates the realism of its videos. Using zero-shot Lyft removes this advantage and compares pure generation capability rather than recall. Given the rebuttal timeline, we use 80 of Lyft’s 180 scenes sampled randomly. Each method generates the same 15.4 s window (185 frames at 12 Hz), scored against the real camera keyframes.

Table R1 reports this comparison. To separate the contribution of each component, it also includes two controls: the Cosmos-Predict2.5 backbone given the same initial frame but no blocks-world condition, and MemOIR with the blocks-world condition only, without any initial frame. The results are given in the rebuttal page.

Table R1. Input-matched comparison and component analysis on Lyft L5 (zero-shot, 80 scenes). [Result videos]

MethodInitial frameFID↓FVD ↓Veh. Track (%) ↑ (% of real)Veh. [email protected] ↑Veh. mIoU ↑Ped. Track (%) ↑ (% of real)Ped. mIoU ↑Ego err % path ↓Final heading ° ↓
MagicDrive-V2✓87.0928.932.1 (36.0)40.80.5073.2 (6.1)0.32959.113.70
DreamForge✓97.7845.027.6 (30.9)36.40.4813.2 (6.1)0.38740.62.11
Baseline (Cosmos) + Initial Frame✓37.2780.420.8 (23.3)28.90.4911.1 (2.0)0.34642.51.06
MemOIR w/o Initial Frame (blocks-world condition only)✗92.41057.873.4 (82.3)79.60.64021.1 (40.0)0.4644.40.34
MemOIR w/ Initial Frame✓23.6548.973.0 (81.8)77.80.63120.0 (38.0)0.4218.40.55

1. Controllability comes from the blocks world. With the blocks-world condition alone and no initial frame, MemOIR already tracks 73.4% of vehicles ([email protected] 79.6) with an ego error of 4.4% of the path. The same backbone without the blocks-world condition reaches only 20.8% and 42.5%, even though it receives the real initial frame, and the baselines reach 27.6–32.1% and 40.6–59.1%. Neither the backbone nor the initial frame provides control; the blocks-world condition does.

2. The initial frame provides similarity to the ground truth. The frame fixes the appearance of the specific scene, which is what FID/FVD against the same scenes reward: with the frame alone, the uncontrolled backbone already reaches an FID of 37.2. Without it, MemOIR must invent the appearance of the street solely from coarse blocks. Even so, MemOIR with the blocks-world condition only receives no image at all, yet its FID is comparable to that of the baselines, which do receive the real frame (92.4 vs. 87.0 and 97.7; 95.5 vs. 89.5 and 111.5 against other scenes), with a somewhat higher FVD (1057.8 vs. 845.0–928.9).

3. Our full MemOIR gives the best quality while keeping the control. Full MemOIR achieves the best visual quality on all four FID/FVD columns and improves on the frame-only backbone (FID 37.2 → 23.6; FVD 780.4 → 548.9). The blocks-world condition therefore contributes to quality as well as control: by specifying the static scene environments and the ego motion, it consistently keeps the rollout in a similar environment to the initially observed scene. Its controllability stays at the level of the variant without an initial frame (vehicle tracking 73.0% vs. 73.4%) and far above every other frame-conditioned model (ego error 8.4% vs. 40.6–59.1%). The small remaining gap to this variant (ego error 4.4% vs. 8.4%; pedestrian mIoU 0.464 vs. 0.421) is expected, and we attribute it to the model having to satisfy two conditions at once. Without a frame, the model is free to synthesize content that fits the blocks exactly. With a real frame, it must also preserve the true geometry of the observed scene, which the coarse boxes and footprints only approximate; wherever the two disagree, the output follows the real scene, and so deviates slightly from the specified layout and trajectory. Qualitatively, the baselines reproduce the given frame but, within about a second, replace the scene with generic, nuScenes-like content, whereas MemOIR stays faithful to the observed street throughout (rebuttal page)).

The remaining component, memory, is isolated in the memory ablation (Table 6 of the revised paper) and evaluated further in A4.

 

W2. What are the minimum inputs for generating a new scene?

A2. The model requires only a rendered blocks-world condition. HD maps, footprints and LiDAR are optional sources for building it**, and the blocks world can be built from a single initial image with off-the-shelf models alone.** Thank you for this practical question. Like Ctrl-V and DriveDreamer-2, which prepare generation inputs from an initial image or from text, our construction pipeline addresses the practical step of creating the inputs needed for generation.

  1. What the model requires. The view-space rendered blocks-world condition, a text prompt and, optionally, the real first frame. The model never reads HD maps, LiDAR, footprints or annotation files, so any procedure that outputs a 3D box world can drive it.
  2. The blocks world can be built from a single initial image, using off-the-shelf models only. Our construction pipeline is purely algorithmic: it takes a single initial image and its camera intrinsics, and combines off-the-shelf models with deterministic geometry. It runs as a series of algorithms; we train no model for it, and it needs no HD map, 3D annotations, footprints or LiDAR.
    • Camera and ground: monocular metric depth (Depth Anything 3) lifts the image into 3D, and the camera height and pitch are estimated by fitting the ground plane.
    • Dynamic instances: 3D detections (WildDet3D) are associated with instance masks (SAM 3), giving one 3D box with a persistent instance identity per vehicle and pedestrian.
    • Static structure: semantic segmentation (Mask2Former, Mapillary Vistas classes [2]) lifted through the depth gives building footprints and heights fitted to the visible facades and rooflines, vegetation volumes, road boundaries and lane lines. Traffic lights, signs and poles come from a detector (D-FINE [3]).
    • Motion: each instance block follows a trajectory along a lane-direction field estimated from the lane lines, or one specified by the user.
  3. What our training and evaluation used. In the paper, we built both the training and the validation blocks worlds with ground-truth annotations for the road layer and the dynamic instances (HD map with lane lines, and 3D boxes), the same inputs the baselines require. Everything else is built with off-the-shelf models such as Depth Anything 3 and Mask2Former; public building footprints and LiDAR are used only where available, each with a video-based fallback.

The result is a manipulable code world: every element is an editable primitive with an instance identity. It renders directly into our blocks-world condition, from which MemOIR generates the full video. The whole pipeline runs automatically in minutes per scene. Users only need an image and, optionally, the desired instance motions. Examples, from a single input image through the constructed world to the generated video, are on our rebuttal page).

We additionally find that an LLM agent can further improve and extend the world. Beyond the automatic pipeline, we find that an LLM agent can refine the constructed world in a closed loop, following the refinement approach of SpatialClaw [4]. The agent renders the world, compares it with the image, corrects building heights and footprints, adds missing structures (far facades, the end of the road, overpasses) and removes spurious ones. It also helps users expand and edit the scene, for example by populating plausible per-instance block behaviours across the whole scene from a single initial image. Examples are on our rebuttal page, and we describe this refinement in Appendix B.8 of the revised paper.

 

W3. What causes the quality decline over long generation?

A3. We identify three causes of the relative decline: in-domain recall by the baselines, a similarity bias in window-wise FID/FVD, and a mismatch in chunk length between training and inference. Thank you for this careful observation.

1. On nuScenes, the competitive baselines benefit heavily from recall. As detailed in A1, MagicDrive-V2 and DreamForge are trained only on nuScenes, whose training and validation splits are recorded in the same city areas, and their BEV maps and ego pose effectively identify the location. They can therefore recall scenery seen during training, which keeps their later windows close to the ground truth in Table A6 in the revised paper. MemOIR, conditioned only on a view-space blocks-world condition, must generate this scenery instead. We therefore evaluate on zero-shot Lyft L5 (80 scenes), which removes this advantage. Every method also receives the same real initial frame and generates the same number of frames, giving the fairest possible ground for comparison. Under these conditions, as shown in Table R2, MemOIR outperforms both baselines by a wide margin in every window, from the first (FID 24.9 vs. 89.2 and 99.6) to the last (FID 49.3 vs. 139.3 and 100.7; FVD 638.9 vs. 885.0 and 878.4). The qualitative comparisons in A1 show the same gap (rebuttal page).

Table R2. Per-window FID/FVD on Lyft L5 (zero-shot, 80 scenes), same-scene reference.

MethodFID 0–4 s4–8 s8–12 s12–15.3 sFVD 0–4 s4–8 s8–12 s12–15.3 s
MagicDrive-V299.6105.497.1100.7941.7985.4935.3878.4
DreamForge89.2111.2129.8139.3789.4887.9913.4885.0
MemOIR (ours)24.933.941.949.3326.3487.8585.7638.9

 

2. With few samples, window-wise FID/FVD partly measure similarity to the specific scenes. FID and FVD compare distributions, but with only frames of 80 videos, the reference set is essentially the evaluated scenes themselves. Scored against the same scenes’ real frames, these metrics reward reproducing those exact scenes. Because MemOIR follows the real initial frame closely, its first window scores exceptionally well. Its score then rises as it generates content the initial frame never showed, even when that content is realistic. Therefore, to measure the realism of the generated videos independently of scene similarity, we additionally score each window against a cross-scene reference: an equal number of real frames drawn from other scenes in the test set, rather than from each video’s own ground truth (Table R3).

Table R3. Per-window FID/FVD on Lyft L5 (zero-shot, 80 scenes), cross-scene reference. (Quality-focused measurement)

MethodFID 0–4 s4–8 s8–12 s12–15.3 sFVD 0–4 s4–8 s8–12 s12–15.3 s
MagicDrive-V287.091.687.891.7876.9877.7956.8891.4
DreamForge80.6102.0125.8137.5758.2867.4913.2963.3
MemOIR (ours)44.944.148.055.7552.2595.0641.0703.3

Against the cross-scene reference, MemOIR’s FID stays flat over the first 8 s. Its gap to the same-scene score shrinks from 20.0 to about 6 by 8–12 s, so most of the early rise reflects divergence from the specific scene, not lower realism. The baselines, by contrast, score better against other scenes than against their own (e.g., DreamForge 80.6 vs. 89.2 at 0–4 s), which shows that they depart from the given scene almost immediately. Their curves therefore start from a much worse level, leaving little room to degrade further: MagicDrive-V2’s FID stays around 87–92 from the first window onward, which reflects early failure rather than stability. Under both references, MemOIR achieves the best FID and FVD in every window, so its advantage holds for both fidelity to the observed scene and overall realism.

Reviewer_yJohOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemOIR addresses controllability and identity consistency in driving video generation. It represents dynamic and static objects as view-space blocks that encode their class, identity, geometry, and occlusion. These rendered blocks condition a pretrained video diffusion model through a control branch. To preserve object appearance across generation chunks, the method stores instance-specific KV features, aligns their spatial positional embeddings with the objects’ current projections, and limits memory access according to instance ownership.

Critical strengths
  1. The scene representation supports both control and memory in a coherent way. Instance identities establish which historical features to retrieve, while object geometry determines where to place them. This is a meaningful design for preserving object appearance when objects return at different image locations.

  2. The memory ablation supports the main technical contribution. ReID Top-1 improves from 26.1% without memory to 27.3% with dense caching, 38.1% with instance-level caching, and 45.6% after adding warped RoPE. This suggests that explicit object association and spatial alignment are more useful than simply retaining historical tokens. The reduced memory footprint further strengthens the practical value.

Critical weaknesses
  1. The actual requirements of the scene-construction pipeline are not sufficiently clear in the main text. Appendix B.4 describes the use of HD maps, existing 3D object annotations, building footprints, and, where available, LiDAR or external height data. These requirements qualify the claim of constructing the representation automatically from raw videos and affect its applicability to less structured data. Differences in backbones, training data, and conditioning settings also make it difficult to attribute the controllability gains specifically to the proposed representation. The stationary-object ablation partially addresses this issue, but uses only 1K training iterations and a selected evaluation subset.

  2. The evidence for long-horizon consistency remains limited. The quantitative re-appearance benchmark includes 30 vehicles in 29 scenes, and the 55-second results are mainly qualitative. Table A4 also reports increasing FVD over time, from 221.9 in the first window to 375.2 in the last, while MagicDrive-V2 remains around 340–351. Thus, the results establish improved identity preservation for returning objects more clearly than sustained visual quality and world consistency over long rollouts.

Questions for authors
  1. Could you specify the annotation and external-data requirements for each dataset and object category? In particular, which results use only video-derived geometry, and how much does performance change without maps, building footprints, or LiDAR?

  2. Were the dense-KV and instance-KV variants in Table 5 trained under the same adaptation protocol and budget, including backbone updates, key dropout, and returning-instance loss weighting?

  3. Could you report uncertainty estimates for the re-appearance benchmark and provide a breakdown by absence duration and viewpoint change? Quantitative results across multiple longer rollouts would also help assess the scope of the long-horizon claims.

9 comments

W1-1. Scene-construction requirements.

A1-1. The model requires only a rendered blocks-world condition. HD maps, footprints and LiDAR are optional sources for building it, and the blocks world can be built from a single initial image with off-the-shelf models alone. Thank you for this important question.

  1. What the model requires. The view-space rendered blocks-world condition, a text prompt and, optionally, the real first frame. The model never reads HD maps, LiDAR, footprints or annotation files, so any procedure that outputs a 3D box world can drive it.
  2. The blocks world can be built from a single initial image, using off-the-shelf models only. Our construction pipeline is purely algorithmic: it takes a single initial image and its camera intrinsics, and combines off-the-shelf models with deterministic geometry. It runs as a series of algorithms; we train no model for it, and it needs no HD map, 3D annotations, footprints or LiDAR.
    • Camera and ground: monocular metric depth (Depth Anything 3) lifts the image into 3D, and the camera height and pitch are estimated by fitting the ground plane.
    • Dynamic instances: 3D detections (WildDet3D) are associated with instance masks (SAM 3), giving one 3D box with a persistent instance identity per vehicle and pedestrian.
    • Static structure: semantic segmentation (Mask2Former) lifted through the depth gives building footprints and heights fitted to the visible facades and rooflines, vegetation volumes, road boundaries and lane lines. Traffic lights, signs and poles come from a detector (D-FINE [1]).
    • Motion: each instance block follows a trajectory along a lane-direction field estimated from the lane lines, or one specified by the user.
  3. What our training and evaluation used. In the paper, we built both the training and the validation blocks worlds with ground-truth annotations for the road layer and the dynamic instances (HD map with lane lines, and 3D boxes), the same inputs the baselines require. Everything else is built with off-the-shelf models such as Depth Anything 3 and Mask2Former; public building footprints and LiDAR are used only where available, each with a video-based fallback.

The result is a manipulable code world: every element is an editable primitive with an instance identity. It renders directly into our blocks-world condition, from which MemOIR generates the full video. The whole pipeline runs automatically in minutes per scene. Users only need an image and, optionally, the desired instance motions. Examples, from a single input image through the constructed world to the generated video, are on our rebuttal page.

We additionally find that an LLM agent can further improve, extend, and manipulate the world. Beyond the automatic pipeline, we find that an LLM agent can refine the constructed world in a closed loop, following the refinement approach of SpatialClaw [2]. The agent renders the world, compares it with the image, corrects building heights and footprints, adds missing structures (far facades, the end of the road, overpasses) and removes spurious ones. It also helps users expand and edit the scene, for example by populating plausible per-instance block behaviours across the whole scene from a single initial image. Examples are on our rebuttal page, and we describe this refinement in Appendix B.8 of the revised paper.

W1-2. Can the controllability gains be attributed to the blocks-world representation?

A1-2. Yes: a controlled comparison shows that the controllability comes from the blocks world, not from the backbone or the initial frame. We evaluate on zero-shot Lyft L5 [3] (80 randomly sampled scenes, unseen by every method), where all baselines receive the same real initial frame as MemOIR. We add two controls: the Cosmos-Predict2.5 backbone with the initial frame but no blocks-world condition, and MemOIR with the blocks-world condition only, without any initial frame. Table R1 reports the full results. The results are given in the rebuttal page.

Table R1. Input-matched comparison and component analysis on Lyft L5 (zero-shot, 80 scenes). [Result videos]

MethodInitial frameFID↓FVD ↓Veh. Track (%) ↑ (% of real)Veh. [email protected] ↑Veh. mIoU ↑Ped. Track (%) ↑ (% of real)Ped. mIoU ↑Ego err % path ↓Final heading ° ↓
MagicDrive-V2✓87.0928.932.1 (36.0)40.80.5073.2 (6.1)0.32959.113.70
DreamForge✓97.7845.027.6 (30.9)36.40.4813.2 (6.1)0.38740.62.11
Baseline (Cosmos) + Initial Frame✓37.2780.420.8 (23.3)28.90.4911.1 (2.0)0.34642.51.06
MemOIR w/o Initial Frame (blocks-world condition only)✗92.41057.873.4 (82.3)79.60.64021.1 (40.0)0.4644.40.34
MemOIR w/ Initial Frame✓23.6548.973.0 (81.8)77.80.63120.0 (38.0)0.4218.40.55
  • Controllability comes from the blocks world. With the blocks-world condition alone and no initial frame, MemOIR tracks 73.4% of vehicles with an ego error of 4.4% of the path. The same backbone without the blocks-world condition reaches only 20.8% and 42.5%, even with the real frame, and MagicDrive-V2 and DreamForge reach 27.6–32.1% and 40.6–59.1%.
  • The initial frame provides similarity to the ground truth. It fixes the appearance of the scene: the frame-only backbone already reaches an FID of 37.2. MemOIR with the blocks-world condition only receives no image at all, yet its FID (92.4) is comparable to that of the baselines, which do receive the frame (87.0 and 97.7).
  • Our full MemOIR gives the best quality while keeping the control. Full MemOIR achieves the best FID and FVD (23.6 and 548.9, against 37.2 and 780.4 for the frame-only backbone) while retaining the controllability of the blocks-world condition (vehicle tracking 73.0%, ego error 8.4%). The small gap to the variant without an initial frame (ego error 4.4%) is expected: without a frame, the model is free to synthesize content that fits the blocks exactly, whereas with a real frame it must also preserve the true geometry of the observed scene, which the coarse blocks only approximate.

The two controls share the same backbone, so the gain is explained neither by the backbone nor by the initial frame. Both MemOIR variants are trained to convergence and evaluated on 80 randomly sampled scenes, so this comparison is subject to neither the short training schedule nor the selected subset of the stationary-object ablation.

W2-1. Is the evidence for long-horizon instance consistency limited to a small set of returning vehicles?

A2-1. Our instance-level memory generalizes across object classes and video backbones. We ran two additional experiments: (i) memory for buildings and vehicles with MemOIR, and (ii) the same memory ported to a different video backbone, Wan2.1-VACE [4], for pedestrians (WalkingTours [5]) and human faces (CelebV-HQ [6]).

1. Our memory preserves the fine details of revisited buildings and vehicles. With the Cosmos-Predict2.5 backbone, we evaluate memory on vehicles and on buildings, whose identity depends on fine facade detail over large image areas. We design a forward–reverse revisit protocol on building-heavy scenes from the Lyft L5 and AV2 datasets. The ego vehicle drives forward in the first chunk, and in the second chunk the ego and all instance trajectories are replayed in reverse. Every building and vehicle in the second chunk has therefore already been observed in the first. For each revisited instance, we compare its crop in the reverse chunk with its crop at the same pose in the forward chunk (Table R2):

  • Crops: the visible area of each building’s footprint prism; the projected 3D box for vehicles.
  • Structural, perceptual and pixel agreement: SSIM, LPIPS and PSNR.

Table R2. Forward–reverse revisit self-consistency evaluation [Result videos].

ClassMemorySSIM ↑LPIPS ↓PSNR ↑
Buildings✗0.4650.39217.05
✓0.5500.34019.54
Vehicles✗0.5050.42915.63
✓0.6000.36917.81

Without memory, revisited instances drift after the chunk boundary: buildings lose or rearrange facade details such as windows, and vehicles change appearance. With memory, both retain their appearance consistency across chunk boundaries, even after leaving and re-entering the view in a new generating chunk.

2. Our instance-level memory mechanism is generally applicable across backbones and domains, including pedestrians and faces. We implement our instance-level memory into a general-purpose video backbone (Wan2.1-VACE) and fine-tune it on in-domain data. Videos are generated in consecutive chunks, each conditioned on the last frame of the previous chunk. A model could simply copy identity from that frame, so we place the chunk boundary where the identity is not visible or heavily occluded. The identity can then only be recovered from what the model saw earlier.

Pedestrians (WalkingTours). We select 26 clips in which a target person leaves the frame and later re-enters, and cut at a frame where that person is out of view. We crop the returning person and measure identity with a person re-identification (ReID) model: cosine similarity to the person’s first appearance and to the real person at the same frame, plus ReID top-1 accuracy. LPIPS and PSNR measure agreement with the real frame.

Table R3. Pedestrian identity after re-entering the frame across the chunk boundary (WalkingTours, 26 clips) [Result videos].

ModelReID cos vs. first appearance ↑ReID cos vs. real ↑ReID top-1 ↑LPIPS ↓PSNR ↑
Wan2.1-VACE (no memory)0.5460.5840.3850.49011.35
+ instance-level KV memory (ours)0.6030.6640.5380.43413.06

With memory, the returning person is matched to the correct identity far more often (ReID top-1 38.5% → 53.8%). They are also closer both to their first appearance (0.546 → 0.603) and to the real person (0.584 → 0.664), with better perceptual and pixel agreement (LPIPS 0.490 → 0.434; PSNR +1.7 dB).

Human faces (CelebV-HQ). We select 29 clips in which the person turns away and back, and cut mid-turn, when only their profile or the back of the head is visible. We measure face-identity similarity between the frontal face that reappears in the next chunk and the face before the turn. To this end, we use ArcFace [7] and CurricularFace [8], face-recognition models that map an aligned face crop to an identity embedding, and we report the cosine similarity between the two embeddings, averaged over all clips.

Table R4. Face identity after a turn-away across the chunk boundary (CelebV-HQ) [Result videos].

ModelFaceSim-Arc ↑FaceSim-Cur ↑
Wan2.1-VACE (no memory)0.2560.262
+ instance-level KV memory (ours)0.3470.356

With memory, face-identity similarity rises from 0.256 to 0.347 (ArcFace) and from 0.262 to 0.356 (CurricularFace). In both domains, the backbone without memory regenerates a plausible but different person or face; with our memory, it restores the original identity.

Reviewer_tTz6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of controllability and long-term consistency in driving video generation. The problem is relevant. The authors propose to use a view-space blocks-world representation to control the motion of dynamic and static objects, as well as the ego vehicle. They also introduce an instance-level KV caching mechanism to preserve object appearance when objects disappear and reappear during long video generation.

Critical strengths

The paper addresses a very relevant problem. Maintaining memory and consistency over long sequences remains an important challenge for driving world models.

The combination of structured scene conditioning and object-level memory is interesting. The experimental results are encouraging, particularly for object controllability and identity consistency over long sequences.

Critical weaknesses

I have identified 4 potential weaknesses to discuss with their corresponding questions in the next section:

  1. Limited related work. Several closely related papers are missing, including ConsisDrive (ICLR 2026), InstaDrive (2026), MAD (CVPR 2026), GEM (CVPR 2025), and StreetCrafter (CVPR 2025).ConsisDrive and InstaDrive are particularly relevant to instance-level consistency and geometric alignment. MAD and GEM investigate structured representations for controlling ego-motion, object dynamics, and scene composition, while StreetCrafter demonstrates the benefits of dense geometric conditioning for camera control. The authors should better position their contributions relative to these works, particularly regarding the novelty of the blocks-world representation and instance-level memory.

  2. Dependence on accurate geometry and instance identities.

The method relies on 3D bounding boxes, camera poses, and consistent instance identities. However, these quantities can be noisy. How robust is the proposed method to errors in these inputs?

  1. Limited evaluation of instance-level memory.

The long-term consistency experiments focus primarily on vehicles. How does it do on pedestrians, whose appearance and pose can change significantly over time?

  1. Computational efficiency. Can we have more details on the computational cost of the pipeline, including geometry estimation, blocks-world construction, memory management, and video generation.
Questions for authors
  1. How sensitive is MemOIR to errors in 3D bounding boxes, camera poses, and instance identities?
  2. What happens when an object receives a different instance ID after reappearing? Does the method have a mechanism to recover from identity switches, or does it rely entirely on consistent identities provided by the conditioning representation?
  3. Does instance-level memory also improve the appearance and identity consistency of pedestrians after occlusion? Could the authors provide qualitative results on pedestrians?
  4. What is the end-to-end computational cost, including blocks-world construction, geometry estimation, memory writing, and video generation? How does it compare with existing driving world models?
6 comments

W1. Limited related work.

A1. Related work. Thank you for pointing us to these works. Below, we clarify MemOIR’s novelty in comparison to the related works you have stated, and have revised the Related Work section of the paper (Sec. 2) accordingly:

  • Instance-level consistency (ConsisDrive, InstaDrive). MemOIR is differentiated from these works in that it preserves instance identity across chunk boundaries through a persistent, per-instance key–value memory, while both works make instance identity consistent only within a single generation window. Both rely on track-ID-guided attention: ConsisDrive lets tokens of the same instance attend to one another across frames, and InstaDrive links a reappearing instance to its last visible frame. These mechanisms work only while the earlier frames remain in context. In long autoregressive rollouts, they do not: once an instance leaves the view and a chunk boundary passes, its identity could not be recovered. MemOIR instead caches each instance’s keys and values and retrieves them by identity when it reappears, preserving its appearance over long-horizon generation.
  • Structured control (MAD, GEM). MemOIR’s blocks world is differentiated from these works in that it represents the entire scene as simple, editable 3D primitives that people or LLM agents can build and manipulate directly. MAD controls generation through a 2D pose video of vehicle and pedestrian skeletons and lane lines. GEM controls it through sparse DINOv2 feature tokens, ego waypoints and human poses. Both cover only moving agents and road layout, and neither is easy to author by hand. Our blocks-world condition instead encodes every scene element, including buildings, vegetation and traffic signals as well as dynamic instances, as instance-coloured boxes and footprints in 3D. Building or editing a scene therefore amounts to placing, moving or removing blocks, which a user can do in a bird’s-eye-view editor and an LLM agent can do in code (see A2 in our response to Reviewer tkmb).
  • Dense geometric conditioning (StreetCrafter). MemOIR is differentiated from StreetCrafter in that its geometric condition requires no GT sensor data, so it can craft and generate new scenes very easily. StreetCrafter conditions on renderings of the scene’s real LiDAR points, which gives precise camera control for novel-view synthesis but requires LiDAR scans and object tracks of an existing recording. Our blocks-world condition is likewise geometric, but is built from compact boxes and footprints. These can be estimated from a single image (see A2 in our response to Reviewer tkmb) or authored from scratch, and each carries the instance identity that our memory retrieves.

Our paper now makes these distinctions explicit; the core novelty of MemOIR is the combination of an easily craftable, manipulable instance-level blocks world with instance-keyed KV memory. Thank you for your valuable comment.

 

W2. Robustness in case of errors in blocks world condition?

A2-1. MemOIR remains robust and visually coherent under substantial noise in its block conditions. As a generative model, MemOIR does not need pixel-exact conditions, and we demonstrate that it renders a coherent scene that follows the overall layout and tolerates considerable noise in individual boxes. To verify this, we take nuScenes validation scenes and perturb one clearly visible target vehicle per scene in the blocks-world condition in two ways:

  • (i) location jitter, an independent Gaussian offset of the box in every frame (σ = 0.1 / 0.2 / 0.3 m per axis), and
  • (ii) box dropout, where the box is removed in a random 10% / 20% / 30% of frames, causing flicker.

As Table R1 shows (Table A12 in the appendix), controllability and visual quality of the generated videos stay close to the clean setting despite increasing noise levels. The model generates a smooth, temporally coherent vehicle that follows the underlying trajectory instead of the frame-to-frame noise, and it bridges the frames where the box is missing. Qualitative results are available on our rebuttal page.

Table R1. Robustness to noisy conditioning on nuScenes (one target vehicle per scene perturbed). Target-vehicle controllability, measured against the clean box in visible frames.

PerturbationNoise levelBox recall (%) ↑Track coverage (%) ↑mIoU ↑Center err. (% of box diag.) ↓
Real video (reference)N/A99.899.70.7623.66
No noiseN/A96.096.40.7225.77
Location jitterσ = 0.1 m96.697.00.7136.15
σ = 0.2 m94.995.40.6976.61
σ = 0.3 m94.394.70.6817.43
Box dropout (flicker)p = 10%95.495.30.7066.56
p = 20%96.496.30.6966.95
p = 30%94.193.80.6887.46

Regarding identity-switch errors, we distinguish two cases:

A2-2. Within a chunk, MemOIR is robust to ID switches. When a vehicle briefly leaves the frame and returns with a different ID inside the same chunk, MemOIR preserves its identity. Full spatio-temporal attention within the chunk keeps the vehicle’s appearance in context, and an abrupt change of appearance would be out of distribution for the model (Appendix D of the revised paper). We show several such cases on the rebuttal page.

Across chunks, instance ID is explicitly fixed in the blocks world itself, which the user controls directly. If a vehicle fully leaves the view, a chunk boundary passes, and it then returns under a different ID, MemOIR has no built-in mechanism to recover: the memory is retrieved by ID, so the vehicle is treated as a new instance. We note two points, however.

  1. In simulation, identity is controlled, not estimated. MemOIR is designed as a controllable world model: scenes are composed or edited directly in the blocks world, so identities are specified by the user and are consistent by construction. ID switches arise only when reconstructing a given real video, where IDs come from an upstream tracker.
  2. Switches can be fixed upstream, on the blocks world itself. Since every instance is an explicit 3D block in world space, a returning instance can be re-associated with a stored one before generation, using class, box size, and trajectory continuity. This is a lightweight step on the conditioning, independent of the generator.

We have added this discussion as a limitation in Appendix E of the revised paper. Thank you for the valuable review.

W3. Limited evaluation of instance-level memory.

A3. Our instance-level memory generalizes across object classes and video backbones. We ran two additional experiments: (i) memory for buildings and vehicles in the driving domain with MemOIR, and (ii) the same memory ported to a different video backbone, Wan2.1-VACE [1], for pedestrians (WalkingTours [2]) and human faces (CelebV-HQ [3]). Videos are on our rebuttal page.

1. Our memory preserves the fine details of revisited buildings and vehicles. With the Cosmos-Predict2.5 backbone, we evaluate memory on vehicles and on buildings, whose identity depends on fine facade detail over large image areas. We design a forward–reverse revisit protocol on building-heavy scenes from the Lyft L5 [4] and AV2 datasets. The ego vehicle drives forward in the first chunk, and in the second chunk the ego and all instance trajectories are replayed in reverse. Every building and vehicle in the second chunk has therefore already been observed in the first, at the same pose. For each revisited instance, we compare its crop in the reverse chunk with its crop at the same pose in the forward chunk (Table R2):

  • Crops: the visible area of each building’s footprint prism; the projected 3D box for vehicles.
  • Structural, perceptual and pixel agreement: SSIM, LPIPS and PSNR.

Table R2. Forward–reverse revisit self-consistency evaluation [Result videos].

ClassMemorySSIM ↑LPIPS ↓PSNR ↑
Buildings✗0.4650.39217.05
✓0.5500.34019.54
Vehicles✗0.5050.42915.63
✓0.6000.36917.81

Without memory, revisited instances drift after the chunk boundary: buildings lose or rearrange facade details such as windows, and vehicles change appearance. With memory, both retain their appearance consistency across chunk boundaries, even after leaving and re-entering the view in a new generating chunk.

 

2. Our instance-level memory mechanism is generally applicable across backbones and domains, including pedestrians and faces. We implement our instance-level memory into a general-purpose video backbone (Wan2.1-VACE) and fine-tune it on in-domain data. Videos are generated in consecutive chunks, each conditioned on the last frame of the previous chunk. A model could simply copy identity from that frame, so we place the chunk boundary where the identity is not visible or heavily occluded. The identity can then only be recovered from what the model saw earlier.

Pedestrians (WalkingTours). We select 26 clips in which a target person leaves the frame and later re-enters, and cut at a frame where that person is out of view. We crop the returning person and measure identity with a person re-identification (ReID) model: cosine similarity to the person’s first appearance and to the real person at the same frame, plus ReID top-1 accuracy. LPIPS and PSNR measure agreement with the real frame.

Table R3. Pedestrian identity after re-entering the frame across the chunk boundary (WalkingTours, 26 clips) [Result videos].

ModelReID cos vs. first appearance ↑ReID cos vs. real ↑ReID top-1 ↑LPIPS ↓PSNR ↑
Wan2.1-VACE (no memory)0.5460.5840.3850.49011.35
+ instance-level KV memory (ours)0.6030.6640.5380.43413.06

With memory, the returning person is matched to the correct identity far more often (ReID top-1 38.5% → 53.8%). They are also closer both to their first appearance (0.546 → 0.603) and to the real person (0.584 → 0.664), with better perceptual and pixel agreement (LPIPS 0.490 → 0.434; PSNR +1.7 dB).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their helpful suggestions and constructive reviews. We are encouraged by their positive assessment of our work as addressing a very relevant problem (tTz6), with our core contributions—the view-space blocks-world representation and instance-level KV memory—recognized as an interesting combination (tTz6) that supports both control and memory in a coherent way (yJoh). The reviewers acknowledged our instance-addressable memory, which retrieves and realigns object-specific features instead of dense KV caching (yJoh, tkmb), our elaborate data-curation pipeline for a dense, 3D-grounded scene representation (tkmb), and our encouraging results on controllability and identity consistency (tTz6), supported by our memory ablation (yJoh). We are particularly encouraged by Reviewer yJoh’s assessment of our design as “a meaningful design for preserving object appearance when objects return at different image locations”.

We have prepared a rebuttal webpage (https://memoir-wm-anon.github.io/rebuttal/) with videos for every new experiment below. As controllability and long-horizon consistency are best judged qualitatively from the generated videos, we kindly encourage the reviewers and the AC to view it. These results are also included in our revised manuscript. Key points are summarized below.

 

Key concerns and our responses:

  1. Generality of instance-level memory (Reviewers tTz6, yJoh, tkmb) All reviewers asked whether instance consistency extends beyond vehicles. We evaluate our memory on new classes and a second backbone, proving the generalizability and applicability of our method. These results strongly demonstrate that our instance-level KV memory is not tied to the driving domain or to our backbone, but is a general mechanism for instance consistency in long-video generation.
    • Buildings and vehicles: a forward–reverse revisit protocol, with gains on every metric (Webpage Sec. 2.1 / revised paper Table A13)
    • Pedestrians: ported to Wan2.1-VACE, our memory raises ReID top-1 on WalkingTours from 38.5% to 53.8% (Webpage Sec. 2.2 / revised paper Table 4 and Fig.7)
    • Human faces: ArcFace identity similarity on CelebV-HQ rises as well, from 0.256 to 0.347 (Webpage Sec. 2.3 / revised paper Table 4)
  2. Fairness of comparisons and attribution of gains (Reviewers tkmb, yJoh) Reviewers noted that the baselines received different inputs, making our gains hard to attribute. We provide a controlled comparison:
    • Input-matched comparison: MagicDrive-V2 and DreamForge receive the same real initial frame as MemOIR, on zero-shot Lyft L5 dataset, unseen by every method (Webpage Sec. 1.1 / revised paper Sec. 4.1, Table A14)
    • Component analysis: the Cosmos-Predict2.5 backbone with the initial frame but no blocks-world condition, and MemOIR with the blocks-world condition only (Table R1 in response to Reviewers yJoh and tkmb)
    • MemOIR achieves the best quality and control, and its controllability comes from the blocks-world condition, not from the backbone or the initial frame.
  3. Long-horizon quality and consistency (Reviewers yJoh, tkmb)
    • Our method demonstrates superior quality-over-time: per-window FID/FVD on zero-shot Lyft L5 dataset, where MemOIR is best in every window (Webpage Sec. 1.2 / revised paper Table A7)
    • Analysis of the decline in visual quality: we find a train–inference mismatch in chunk length; fine-tuning on longer clips flattens the decline (revised paper Table A7)
    • Re-appearance benchmark: paired bootstrap confidence intervals, and breakdowns by absence duration and viewpoint change (revised paper Appendix D.3, Tables A15–A17) MemOIR’s advantage is significant against every baseline, and its scores do not decay as the absence grows.
  4. Scene-construction requirements (Reviewers yJoh, tkmb)
    • We demonstrate that our method can construct a viable blocks world purely from an initial image or video using off-the-shelf models (revised paper Sec. 3.1, Appendix B.6, Table A3)
    • A purely algorithmic pipeline builds the blocks world from a single initial image with off-the-shelf models, without annotations, maps or LiDAR (Webpage Sec. 4.1 / Appendix B.7)
    • We also demonstrate LLM agent can further refine and extend the constructed world in a closed loop (Webpage Sec. 4.2 / Appendix B.8)

 

Additional key updates include:

  • Robustness to noisy boxes and identity switches (Webpage Sec. 3 / Table A12, Appendix E, Reviewer tTz6)
  • Per-stage computational cost (Appendix B.4, Table A2, Reviewer tTz6)
  • Matched training protocol for all memory variants (Sec. 4.5, Reviewer yJoh)
  • Ego-motion errors as percentages of path length, with per-scene distributions (Table 1, Table A11, Reviewer tkmb)
  • Unified terminology, revised Figure 1 (Reviewer tkmb)
  • Positioning against ConsisDrive, InstaDrive, MAD, GEM and StreetCrafter in the revised Related Work (Sec. 2, Reviewer tTz6)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.