acceptodds
Under review as a conference paper at ICLR 2027

WAM-Sim: World Anchored Memory Meets Online RL for Consistent and Realistic Driving Simulation

Abstract

Autoregressive driving world models systematically suffer from structural drift, identity drift, and quality decay over long rollout horizons due to transient conditioning signals that lack persistent state. We present WAM-Sim, a driving world model anchored by a persistent hierarchical memory and optimized via online flow-matching reinforcement learning. WAM-Sim structures persistent context across three temporal and semantic scales: static structural tokens encoding road topology, read-only scene tokens supplying geolocated background appearance, and write-once instance tokens preserving individual traffic participant identities across occlusions and chunk boundaries. To restore visual sharpness and physical plausibility degraded by multi-scale conditioning, we introduce an online post-training framework that optimizes self-generated rollouts via flow-matching RL guided by hierarchy-aligned visual rewards and a vision-language model physical judge. The invariant memory manifold bounds policy optimization to suppress semantic collapse and reward hacking, while online RL systematically enforces physical commonsense and fine-grained visual fidelity. On the nuScenes benchmark, WAM-Sim achieves a superior generation fidelity (FID=7.07, FVD=42.01) and strong downstream perception performance (19.67 mAP, 68.08 road mIoU). On minute-scale rollouts of self-collected data, WAM-Sim substantially mitigates structural and identity drift, preserving high scene similarity and subject identity coherence across extended horizons.

Primary area:
generative models
Keywords:
driving world model, video generation, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ApscOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to deal with the systematic degradation of autoregressive driving world models over minute-scale rollouts in terms of structural drift, identity drift, and quality decay. WAM-Sim anchors generation with a three-tier persistent world memory and restores visual sharpness and physical plausibility via Drive DiffusionNFT. On nuScenes it achieves strong fidelity and downstream perception, and on minute-scale self-collected rollouts it substantially mitigates structural/identity drift, with human-preference wins over strong baselines.

Critical strengths

The paper decomposes long-horizon failure into structural drift, identity drift, and quality decay, and argues that the root cause is the lack of persistent, addressable state rather than mere context-window length.

Critical weaknesses
  • As nuScenes lacks geolocated reference databases, WAM-Sim runs without scene memory there. The headline FID=7.07 is therefore produced by a reduced variant that abandons a central component, so the paper’s central contribution is validated almost entirely on a self-collected, non-standard dataset rather than on the public benchmark it leads with. This also happens with the minute-scale rollouts evaluation.
  • The paper presents this synergy as a key insight but provides no formal characterization or controlled measurement of how the memory constrains the policy search space or prevents reward hacking. Support is limited to qualitative figures and ablation trends.
  • It is unclear whether the fidelity/consistency gains justify the overhead relative to cheaper trajectory-conditioned or single-chunk baselines, especially for scalable closed-loop simulation.
  • Scene memory requires a geolocated reference database, limiting applicability and fairness of comparison.
Questions for authors
  • How well does WAM-Sim perform on nuScenes when its scene memory is enabled, and is there a public long-horizon benchmark where all memory tiers are active?
  • What exactly does the invariant memory manifold bound, and how is reward hacking actually prevented? A quantitative characterization e.g., measuring policy-space drift, reward-distribution shift, or hallucination rate over training should be provided.
  • How does the method’s cost scale, and is the fidelity gain worth it for practical closed-loop simulation?
  • How sensitive are the reported gains to the VLM physical judge and the scene-memory reference quality?
7 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We sincerely appreciate Reviewer Apsc for the insightful, thorough, and constructive review. Your sharp observations—from full-memory validation on public benchmarks and quantitative bounds on RL policy drift to computational scaling and reference sensitivity—directly pinpoint what is needed to make our claims airtight and have substantially strengthened our paper. Below, we address each point concisely.

1. How well does WAM-Sim perform on nuScenes when its scene memory is enabled, and is there a public long-horizon benchmark where all memory tiers are active? (W1, Q1)

To validate our contribution on a public benchmark, we conduct additional experiments on nuScenes with all memory tiers enabled. We evaluate the full validation sequences (~20 seconds, ~200 frames) using multi-chunk autoregressive rollouts, substantially longer than the clips (8-16 frames) used by many baselines in Table 1. To our knowledge, nuScenes is currently the only public driving benchmark combining HD maps, 3D tracking boxes, and a geographic reference database (nuScenes-Geography from Spatial Retrieval), as Waymo and Argoverse lack publicly available paired geographic references. Thus, multi-chunk autoregressive rollouts on nuScenes sequences with all memory tiers enabled represent the most feasible long-horizon evaluation with currently available public datasets.

Enabling memory reduces FID from 7.07 to 5.75 and FVD from 42.01 to 32.40. It also improves mAP and NDS by 2.63 and 1.06 points, respectively, and road and vehicle mIoU by 2.24 and 0.89 points. For long-horizon generation on nuScenes, the full-memory variant improves Subject Coherence from 0.623 to 0.666 and Scene Similarity from 0.854 to 0.891. These additional experiments provide evidence on a widely used public driving dataset for the benefits of the world memory across generation fidelity, downstream perception, and long-horizon scene and identity consistency.

MethodFID ↓FVD ↓mAP ↑NDS ↑mIoU_road ↑mIoU_vehicle ↑Subject Coherence ↑Scene Similarity ↑
w/o memory7.0742.0119.6729.9568.0831.530.6230.854
w/ memory5.7532.4022.3031.0170.3232.420.6660.891

2. What exactly does the invariant memory manifold bound, and how is reward hacking actually prevented? A quantitative characterization e.g., measuring policy-space drift, reward-distribution shift, or hallucination rate over training should be provided. (W2, Q2)

We sincerely thank the reviewer for raising this point. To provide a controlled measurement beyond qualitative evidence, we report the complete memory on/off × RL on/off comparison under the same setting as in the original manuscript. We also quantify policy-space drift, measured as the DINOv2 Distance between rollouts generated before and after RL. During RL, we update only the self-attention LoRA parameters of the DiT and keep the memory cross-attention and all other modules frozen. Generation thus stays anchored to the manifold consistent with the road structure, geographic background, and instance appearance provided by memory. We empirically observe that memory helps limit policy drift to some extent: the model moves noticeably less from its pre-RL behavior when memory is present (DINOv2 Distance of 0.087 vs. 0.134). Under the same training settings, RL yields larger gains on held-out metrics with memory than without (+2.26 vs. +1.33 on Subject Fidelity, +5.05 vs. +3.21 on Physical Plausibility), which suggests that the improvements reflect genuine quality gains rather than reward hacking. RL also preserves the persistence gains from memory, even improving them slightly. Overall, these results reflect an interaction benefit in which each component reinforces rather than offsets the other. We will revise the manuscript so that the claims about this interaction benefit are stated more accurately and grounded in these results.

  Persistence  Object & Physics  Policy-space Drift
MethodSub. Coh. ↑Scene Sim. ↑Sub. Fid. ↑Phys. Plaus. ↑DINOv2 Dist. to Pre-RL ↓
Base0.6460.86342.1651.45-
RL only0.6520.86143.4954.660.134
Memory only0.6780.90541.7450.91-
Memory + RL0.6810.91044.0055.960.087
AuthorsOpenReview ↗

Official Comment by Authors (2)

3. It is unclear whether fidelity/consistency gains justify the overhead relative to cheaper trajectory-conditioned or single-chunk baselines, especially for scalable closed-loop simulation. How does the method’s cost scale? (W3, Q3)

MethodResolutionParamsThroughput ↑VRAM ↓FVD ↓Scene Similarity ↑Subject Coherence ↑
trajectory-conditioned1280×7042B8.00 FPS45.71286.540.6970.322
Ours1280×7043.1B6.94 FPS47.45137.920.8280.547

At the same resolution, compared with the trajectory-conditioned baseline, WAM-Sim incurs a 13.3% reduction in throughput and a 3.8% increase in peak memory usage, while achieving 51.9% lower FVD over minute-scale rollouts and substantially improving instance consistency and scene similarity. This trade-off demonstrates the value of persistent memory for long-horizon simulation.

Our memory design scales as O(1) with rollout duration: scene memory applies cross-attention with a single reference frame and a fixed number of tokens per segment, while instance memory maintains a bounded cache of up to 32 active instances. Consequently, per-chunk inference latency and peak VRAM remain constant as rollout duration increases (e.g., from 10s to 60s and beyond), supporting scalable long-horizon closed-loop simulation.

4. How sensitive are the reported gains to the VLM physical judge and the scene-memory reference quality? (Q4)

Sensitivity to VLM physical judge. To test the sensitivity of the reported gains to the VLM judge, we re-evaluated Physical Plausibility with an additional paraphrased prompt set (Prompt Set 2) and a stronger judge (Qwen3.8-27B). Although absolute scores vary across settings, our final model outperforms the strongest baseline (HorizonDrive) in every case, and RL post-training consistently improves over our model without RL by 2.72 to 5.86 points.

Prompt Set 2—Q1: Look carefully at every pedestrian who is walking or running in this video, from the first frame to the last: is their limb motion always anatomically correct and fully natural, with no implausible joint movement and no abnormal limb paths at any moment? Answer with yes or no. | Q2: Whenever pedestrians are moving, are their arms and legs always easy to tell apart, with no limbs fusing into the torso, fusing with one another, vanishing, or becoming visually tangled at any point? Answer with yes or no.

  Physical Plausibility↑  
Qwen3.5-9BQwen3.5-9BQwen3.8-27B   Qwen3.8-27B
(Prompt Set 1)(Prompt Set 2)(Prompt Set 1)   (Prompt Set 2)
HorizonDrive51.1964.2440.77   40.70
Ours (w/o RL)50.9163.7439.36   38.99
Ours (w/ RL)55.9666.4644.97   44.85

Sensitivity to reference quality. We assess sensitivity to scene-reference quality by adding noise to the reference images. Clean references reduce FVD from 81.64 to 76.68 and improve Scene Similarity from 0.863 to 0.911. With noise-corrupted references, FVD reaches 78.16, and Scene Similarity reaches 0.903, corresponding to an FVD degradation of 1.48 and a Scene Similarity decrease of only 0.008 relative to clean references. These results show that the benefits of scene memory largely persist under tested noise corruption.

Scene MemoryNoise CorruptionFVD ↓Scene Similarity ↑
81.640.863
✓\checkmark76.680.911
✓\checkmark✓\checkmark78.160.903
AuthorsOpenReview ↗

Official Comment by Authors (3)

5. Scene memory requires a geolocated reference database, limiting applicability and fairness of comparison. (W4)

Applicability: The geographic database is constructed offline from geotagged training images, enabling reuse of existing driving logs. Scene-reference conditioning is optional at the chunk level: for chunks with missing references, generation proceeds using the remaining conditioning inputs. We evaluate robustness to incomplete reference availability by independently dropping scene references during inference. At dropout probabilities of 0.1 and 0.3, Scene Similarity reaches 0.908 and 0.901, respectively, compared with 0.910 using all references. These results demonstrate graceful degradation under partial reference availability and support the applicability of scene memory with incomplete geographic coverage.

Drop Prob. pScene Similarity ↑FVD ↓
0.00.91074.20
0.10.90877.03
0.30.90178.60

Fairness:We evaluate UniMLVG, OpenDWM, and HorizonDrive using their original publicly released checkpoints and architectures. Their inputs exclude the retrieved reference images. To assess the contribution of our memory design under matched reference access, we additionally implement a baseline on OmniDreams, providing it with exactly the same scene references and instance appearance information as our method.

With the same reference information, our method further reduces FVD from 76.84 to 73.77 and improves Subject Coherence from 0.658 to 0.678 and Scene Similarity from 0.888 to 0.905. This matched-reference fair comparison supports the additional contribution of persistent memory management beyond the gains obtained by supplying appearance references.

FVD ↓Subject Coherence ↑Scene Similarity ↑
OmniDreams81.640.6460.863
OmniDreams + Ref76.840.6580.888
Ours (w/o RL)73.770.6780.905
Reviewer_8pHXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WAM-Sim addresses long-horizon appearance and identity drift in driving video generation. It combines persistent structural, scene, and instance memory with DiffusionNFT post-training to improve consistency and visual quality. The three memory tiers address road geometry, background appearance, and instance identity, respectively, forming a coherent design for mitigating drift in long videos. Automated evaluations, long-horizon ablations, and blinded human assessments provide encouraging evidence. The human study also reports statistical tests at the scenario level.

Critical strengths

Structural memory aligns HD maps with the ego trajectory, providing persistent road geometry guidance. Instance memory stores appearance features by track ID and restricts retrieval through instance masks, directly targeting identity changes across video chunks.

Reporting both FID and FVD on nuScenes helps assess the quality of image and video distributions. The inclusion of 3D detection mAP and NDS provides additional evidence that generated videos preserve information relevant to perception.

The human study covers 60 scenarios and 15 assessors, with randomized presentation order and spatial placement to reduce presentation bias.

Critical weaknesses

The temporal scope and sources of appearance references available at test time remain unclear. Section 3.2 describes selecting instance references from multiple temporal segments, while Table 11 allows up to four reference frames and specifies one-time registration at first appearance. The paper does not explain how these reference frames are obtained when an instance is first registered.

The relationship between the scene-memory reference database and the test targets, and whether all methods have equal access to reference information, also need clarification. The current experiments do not fully separate the benefits of persistence from those of additional appearance references.

nuScenes uses multi-view evaluation, whereas the self-collected dataset contains front-view videos. Further details on the multi-view implementation and training adaptation are needed. The paper explicitly states that scene memory is disabled on nuScenes. These gaps limit attribution and make the relevant experiments harder to reproduce.

The paper argues that persistent memory bounds the policy optimization manifold and suppresses reward hacking, but the current evidence does not establish this mechanism. Fixed memory can influence generation through conditioning; its invariance alone does not establish a hard constraint on generated outputs or the feasible policy set.

The ablations and qualitative comparisons in Tables 3 and 6 and Figure 8 do not isolate the interaction between memory and RL. Tables 3 and 6 examine memory components and rewards, respectively, without reporting the complete memory on/off × RL on/off factorial comparison. This makes the interaction benefit difficult to quantify.

The rewards exhibit trade-offs: the scene reward decreases from 62.68 in the scene-only setting to 61.48 in the full setting, while the object reward drops slightly from 0.273 to 0.271. It is therefore inaccurate to describe all optimization signals as consistently improving as rewards are added.

The problem formulation initially uses (P_c) for the global pose, while Equation (1) and subsequent text use (G_c) for the full pose trajectory. Their indexing and mapping need clarification. The number of tokens per scene reference is initially denoted by (L), routing indices use (P), and the instance-membership matrix reuses (L) for the number of video tokens, making distinct token dimensions easy to confuse.

Questions for authors

Are memory construction and access to reference information sufficiently controlled? Please specify the sources, temporal scope, and availability of reference frames from multiple temporal segments at first registration, and explain the separation between the scene-memory database and the test targets/evaluation videos.

Does the current evidence support attributing constraints on RL optimization to memory? Please provide the complete memory on/off × RL on/off factorial comparison with matched training budgets.

Can the notation and dimensions be made consistent? Please clarify the indexing and mapping between global poses and full pose trajectories, and distinguish the token dimensions used for scene references, routing, and the instance-membership matrix.

7 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We really thank Reviewer 8pHX for the thorough, constructive review and encouraging assessment of our work. Below, we address all technical questions, provide the requested controlled experiments (matched-reference baseline, nuScenes with scene memory, and the complete 2×2 Memory × RL factorial), and clarify the notation and implementation details.

1.Section 3.2 describes selecting instance references from multiple temporal segments, while Table 11 allows up to four reference frames and specifies one-time registration at first appearance. The paper does not explain how these reference frames are obtained when an instance is first registered. (W1)

For instances visible in the conditioning frame, that frame is used directly for registration. For instances appearing later, reference frames are selected from their first-appearance chunk after it has been fully generated. We divide the chunk into up to four temporal segments and select the frame with the largest bounding box from each segment. Instance crops from these frames are then encoded and written together in a single registration step.

2. The relationship between the scene-memory reference database and the test targets. (W2)

All recordings used for evaluation are excluded from the training set. We construct the geographic reference database DgeoD_{geo} offline from the training set. Geographic reference images come from independently collected recording sessions covering the evaluated locations. The reference database and evaluation videos are separated at the recording level, excluding the target sequences and temporally adjacent segments of the same recordings from the reference database.

3. Whether all methods have equal access to reference information, also need clarification. The current experiments do not fully separate the benefits of persistence from those of additional appearance references. (W2)

We evaluate UniMLVG, OpenDWM, and HorizonDrive using their original publicly released checkpoints and architectures. Their inputs exclude the retrieved reference images. To assess the contribution of our memory design under matched reference access, we additionally implement a baseline on OmniDreams, providing it with exactly the same scene references and instance appearance information as our method.

The results distinguish two sources of improvement. After adding reference information, OmniDreams decreases FVD from 81.64 to 76.84, improves Subject Coherence from 0.646 to 0.658, and Scene Similarity from 0.863 to 0.888, demonstrating the benefit of additional appearance references.

With the same reference information, our method further reduces FVD from 76.84 to 73.77 and improves Subject Coherence from 0.658 to 0.678 and Scene Similarity from 0.888 to 0.905. This matched-reference comparison supports the additional contribution of persistent memory management beyond the gains obtained by supplying appearance references.

MethodFVD ↓Subject Coherence ↑Scene Similarity ↑
OmniDreams81.640.6460.863
OmniDreams + Ref76.840.6580.888
Ours (w/o RL)73.770.6780.905

4. Further details on the multi-view implementation and training adaptation are needed. (W3)

Multi-view implementation: We extend our pretrained single-view model to jointly generate six camera views from the nuScenes dataset. We introduce a Cross-View Attention module between the text cross-attention and MLP layer of each block. Considering the spatial arrangement of surround-view cameras, each view attends only to its two adjacent views, explicitly modeling spatial relationships between neighboring cameras. Additionally, we introduce learnable View Embeddings, which are projected into nine sets of shift, scale, and gate parameters and added to the original AdaLN modulation signals, enabling view-aware feature modulation. Self-attention is performed within each view to capture temporal dependencies, while Cross-View Attention facilitates information exchange between adjacent views to promote cross-view consistency.

Training details: The model is trained for 40,000 optimization steps with a global batch size of 4, a learning rate of 1e-4, and a context parallelism (CP) size of 8. During training, the pretrained backbone is frozen, and only the Cross-View Attention modules, View Embeddings, and AdaLN layers are optimized. The output projection of the Cross-View Attention module is zero-initialized to ensure that the newly introduced branch does not perturb the pretrained representations at the beginning of training.

AuthorsOpenReview ↗

Official Comment by Authors (2)

5. The paper explicitly states that scene memory is disabled on nuScenes. These gaps limit attribution and make the relevant experiments harder to reproduce. (W3)

As original nuScenes dataset lacks geolocated reference databases, we use the open-source nuScenes-Geography dataset from Spatial Retrieval as a geolocated reference database and have added an evaluation with memory enabled on nuScenes. As shown in the table, enabling scene memory reduces FID from 7.07 to 5.75 and FVD from 42.01 to 32.40. It also improves mAP and NDS by 2.63 and 1.06 points, respectively, and road and vehicle mIoU by 2.24 and 0.89 points. These consistent improvements provide public-benchmark evidence for the benefits of memory in both generation fidelity and downstream perception.

MethodFID ↓FVD ↓mAP ↑NDS ↑mIoU_road ↑mIoU_vehicle ↑
w/o memory7.0742.0119.6729.9568.0831.53
w/ memory5.7532.4022.3031.0170.3232.42

6. The paper argues that persistent memory bounds the policy optimization manifold and suppresses reward hacking, but the current evidence does not establish this mechanism. Tables 3 and 6 examine memory components and rewards, respectively, without reporting the complete memory on/off × RL on/off factorial comparison. (W4, W5, Q2)

We really appreciate your insightful suggestion. We report the complete memory on/off × RL on/off comparison, with all variants trained under the budget reported in the original manuscript. Policy-space drift is measured as the DINOv2 Distance between rollouts generated before and after RL. During RL, we update only the self-attention LoRA parameters of the DiT and keep the memory cross-attention and all other modules frozen. Generation thus stays anchored to the manifold consistent with the road structure, geographic background, and instance appearance provided by memory. In the factorial comparison, we empirically observe that memory helps limit policy drift to some extent: under identical RL settings, the model moves noticeably less from its pre-RL behavior when memory is present (DINOv2 Distance of 0.087 vs. 0.134). At the same time, with memory, RL achieves larger gains on held-out metrics (+2.26 vs. +1.33 on Subject Fidelity, +5.05 vs. +3.21 on Physical Plausibility), indicating that the optimization leads to real quality gains instead of reward hacking. RL also preserves the persistence gains from memory, even improving them slightly. Overall, these results reflect an interaction benefit in which each component reinforces rather than offsets the other. We will revise the manuscript so that the claims about this interaction benefit are stated more accurately and grounded in these results.

  Persistence  Object & Physics  Policy-space Drift
MethodSub. Coh. ↑Scene Sim. ↑Sub. Fid. ↑Phys. Plaus. ↑DINOv2 Dist. to Pre-RL ↓
Base0.6460.86342.1651.45-
RL only0.6520.86143.4954.660.134
Memory only0.6780.90541.7450.91-
Memory + RL0.6810.91044.0055.960.087

7. It is therefore inaccurate to describe all optimization signals as consistently improving as rewards are added. (W6)

We agree and thank the reviewer for pointing this out. Adding further rewards indeed introduces trade-offs among the optimization signals. We have rephrased the sentence accordingly.

AuthorsOpenReview ↗

Official Comment by Authors (3)

8. Are memory construction and access to reference information sufficiently controlled? Please specify the sources, temporal scope, and availability of reference frames from multiple temporal segments at first registration, and explain the separation between the scene-memory database and the test targets/evaluation videos. (Q1)

Yes, both are explicitly controlled. For the reference frames used at first registration:

Source. For instances visible in the conditioning frame, the reference is cropped from that user-provided frame. For instances that appear later, the reference frames are the model’s own generated frames. No GT appearance or external instance reference is used in either case.

Temporal scope. For later-appearing instances, selection is restricted to the chunk in which the instance first appears. We divide this chunk into up to four temporal segments and take the frame with the largest bounding box of the instance from each.

Availability. Registration takes place only after the first-appearance chunk has been fully generated, so all selected reference frames already exist when the entry is written. Instances in the conditioning frame are registered before generation begins.

Separation between the scene-memory database and the evaluation videos. All recordings used for evaluation are excluded from the training set. We construct the geographic reference database D_geo offline from the training set. Geographic reference images come from independently collected recording sessions covering the evaluated locations. The reference database and evaluation videos are separated at the recording level, excluding the target sequences and temporally adjacent segments of the same recordings from the reference database. The scene-preservation results reported in the paper already follow this independent-session reference protocol.

9. Can the notation and dimensions be made consistent? Please clarify the indexing and mapping between global poses and full pose trajectories, and distinguish the token dimensions used for scene references, routing, and the instance-membership matrix. (W7, Q3)

Thank the reviewer for pointing out the inconsistency of notation and dimensions. We have standardized the notation in the updated submission. We acknowledge the notation inconsistency: G_c and P_c denote the same global ego-pose trajectory. We use L for the number of tokens per scene reference, N_b for the number of temporal segments, L_v for the number of video tokens, and N for the number of instances. Scene references have dimensions (N_b * L) × D, the scene-routing mask has dimensions L_v × (N_b * L), and the instance-membership matrix has dimensions L_v × N.

Reviewer_ycxrOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WAM-Sim, an autoregressive driving video generation framework aimed at reducing structural drift, identity drift, and quality degradation over long rollouts. Building on a pretrained driving world model, it introduces a hierarchical memory comprising map-based structural conditioning, read-only geolocated scene references, and a write-once instance appearance bank accessed through instance-masked attention. It further applies DriveDiffusionNFT, an online flow-matching reinforcement learning stage that combines rewards for image quality, object recognizability, and physical plausibility. The intended contribution is the integration of persistent scene and instance conditioning with reward-guided post-training to improve both consistency and realism. Validation includes nuScenes and a self-collected driving dataset, measuring generation fidelity, downstream perception performance, scene and identity preservation, and physical plausibility. Component and reward ablations, rollouts extending to 60 seconds, and human preference studies provide evidence of improvements over the evaluated baselines.

Critical strengths
  1. The persistent memory design addresses a consequential limitation of long-horizon driving generation. Separating road structure, geographic appearance, and instance identity gives each information source an appropriate persistence and retrieval mechanism. The 60-second evaluation provides concrete support: enabling scene and instance memory improves Subject Coherence from 0.322 to 0.547 and reduces FVD from 286.54 to 137.92. These results directly support the paper’s central contribution of mitigating accumulated drift over extended rollouts.

  2. The improvements are supported by complementary evaluations beyond the optimized rewards. The reward ablation reports gains in independently evaluated Subject Fidelity (41.74 to 44.00) and pedestrian-motion Physical Plausibility (50.91 to 55.96). Additionally, a blinded study involving 60 scenarios and 15 expert assessors reports an 86.7% scenario-level preference rate for the full system over the OmniDreams baseline. Together, these evaluations provide useful evidence that the framework’s improvements translate into perceptually meaningful benefits.

Critical weaknesses
  1. The inference-time memory construction and reference-data protocols are insufficiently specified. Section 3.2 describes instance memory as being written once upon an agent’s initial appearance, while constructing its representation from multiple crops selected across temporal segments. It is unclear which observations are available when this write occurs and whether inference-time crops originate from generated history, externally supplied references, or ground-truth sequences. Similarly, Appendix D.4 describes geographic indexing without adequately specifying how scene references are sourced and separated from evaluation recordings. These details materially determine the difficulty of the generation task and the interpretation of the reported scene and identity preservation gains. Without an explicit causal memory-construction procedure and documented reference-data separation, it is difficult to assess whether the evaluation matches the claimed persistent-memory setting and whether the comparisons provide comparable conditioning information.
  2. The incremental methodological contribution is insufficiently established. DriveDiffusionNFT largely follows the existing DiffusionNFT objective, and ReinDriveGen already applies DiffusionNFT to driving video generation with a driving-specific reward mechanism. Geographic reference conditioning and instance-masked attention also have close precedents in Spatial Retrieval and ConsisDrive. The distinctive contribution therefore appears concentrated in the write-once appearance bank, hierarchical memory routing, and their integration with reward-guided post-training. However, the experiments do not sufficiently isolate these specific advances. The memory ablations lack comparisons against simpler conditioning strategies using the same reference information, while the claimed memory–RL synergy lacks a clearly controlled comparison of the base model, memory alone, RL alone, and their combination. This leaves insufficient evidence that the proposed integration offers a substantial advance over existing components.
Questions for authors
  1. What are the specific methodological advances over the closest prior approaches? Please distinguish the components inherited from DiffusionNFT, Spatial Retrieval, and ConsisDrive from the proposed additions. In particular, given that ReinDriveGen already applies DiffusionNFT to driving video generation, what changes does DriveDiffusionNFT introduce to the optimization procedure beyond its conditioning inputs and reward formulation? Please also explain the distinctive role of write-once persistence and hierarchical routing relative to existing reference-conditioning mechanisms.

  2. Can controlled experiments isolate the proposed memory design and its interaction with RL? Please provide a comparison of base model / memory only / RL only / memory + RL, using the same backbone, data, and sampling protocol, with training budgets explicitly reported. Please clarify which variants in Table 3 received RL post-training. To isolate the memory-specific contribution, please additionally include a simpler conditioning baseline with access to the same scene references and instance appearance information. Reporting long-horizon identity consistency, scene similarity, and independent quality metrics would help establish whether the proposed design provides benefits beyond adding reference information and post-training separately.

  3. How is instance memory constructed causally during inference? Please provide a step-by-step description or pseudocode specifying the observation window used to select the appearance crops, their source, and the ordering of generation, memory registration, and subsequent retrieval. How does selecting up to four crops across temporal segments reconcile with writing the bank once at an instance’s initial appearance? What happens when insufficient valid observations are available? Please distinguish training-time and inference-time construction. If the reported evaluation uses future ground-truth appearance crops, please also evaluate a causal variant using only observations available at registration.

  4. How are geographic references separated from the evaluation recordings? Please describe the construction of the reference database and the split among training recordings, evaluation sequences, and reference imagery. Can retrieved references originate from the target sequence or temporally adjacent portions of the same recording, or are they obtained from independent recording sessions? If recording-level separation is not already enforced, please repeat the key scene-preservation evaluation using independently captured references, while retaining geographic coverage. This would help assess whether the gains persist under the intended geographic-retrieval setting.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We really appreciate Reviewer ycxr for the insightful feedback and for recognizing the effectiveness of our hierarchical memory design, 60-second drift mitigation, and independent/human evaluations. Below, we address all weaknesses and questions.

1. It is unclear which observations are available when this write occurs and whether inference-time crops originate from generated history, externally supplied references, or ground-truth sequences. (W1)

Two types of observations are available at write time. For instances visible in the conditioning first frame, the observations are crops from that frame. For instances appearing later, the observations are crops from generated frames within their first-appearance chunk.

The inference-time crops come exclusively from the conditioning frame and model-generated frames, not from GT videos or external instance references.

2. What are the specific methodological advances over the closest prior approaches? Please distinguish the components inherited from DiffusionNFT, Spatial Retrieval, and ConsisDrive from the proposed additions. In particular, given that ReinDriveGen already applies DiffusionNFT to driving video generation, what changes does DriveDiffusionNFT introduce to the optimization procedure beyond its conditioning inputs and reward formulation? (Q1, W2)

Like ReinDriveGen, we use the DiffusionNFT objective unchanged. The distinction between ReinDriveGen and our RL method is mainly in reward design. ReinDriveGen uses a single reward that targets the visual quality of vehicles. Ours is a combination of three rewards designed for driving scenes, which extends the target from the visual quality of vehicles alone to overall scene quality, the visual quality of vehicles and pedestrians, and the physical plausibility of motion.

Similar to Spatial Retrieval, our method leverages location information to retrieve geographically relevant reference images. The key distinction lies in memory design: while Spatial Retrieval focuses solely on background references, our method introduces a hierarchical memory structure to maintain consistency in both the background and foreground objects.

Following ConsisDrive, we adopt instance-masked attention. The core difference between our method and ConsisDrive lies in the content stored in instance memory. ConsisDrive represents instances through category, bounding-box size, and track ID, without directly encoding their specific visual appearance. Our instance representation stores appearance tokens extracted from instance image crops and retrieves them by track ID in subsequent chunks.

3. Please also explain the distinctive role of write-once persistence and hierarchical routing relative to existing reference-conditioning mechanisms. (Q1)

Write-once persistence. ConsisDrive provides instance conditions (category, box size, and track ID) within each chunk and carries no state across chunks. We instead extract appearance tokens from each instance’s image crops, write them once at registration, and reuse them in all subsequent chunks.

Hierarchical routing. Compared with existing global reference conditioning, where all video tokens can access all reference features, our routing restricts access according to spatial correspondence and instance identity. Scene references guide their corresponding scene regions, while instance appearance tokens are accessible only to video tokens associated with the same track ID.

4. The memory ablations lack comparisons against simpler conditioning strategies using the same reference information. Please additionally include a simpler conditioning baseline with access to the same scene references and instance appearance information. (W2)

We have added a simpler conditioning baseline for memory ablation. It uses exactly the same scene and instance reference tokens as our method, concatenating them for global injection through standard self-attention. This baseline reduces FVD from 81.64 to 76.84 and improves Subject Coherence from 0.646 to 0.658 and Scene Similarity from 0.863 to 0.888, demonstrating the benefit of additional reference information. Our method further reduces FVD to 73.77 and improves Subject Coherence to 0.678 and Scene Similarity to 0.905. This comparison directly evaluates our hierarchical memory design against a simpler conditioning strategy under matched reference access and supports the additional benefit of hierarchical memory routing.

MethodFVD ↓Subject Coherence ↑Scene Similarity ↑
w/o memory81.640.6460.863
Reference concat76.840.6580.888
Ours (w/o RL)73.770.6780.905
AuthorsOpenReview ↗

Official Comment by Authors (2)

5. Can controlled experiments isolate the proposed memory design and its interaction with RL? Please provide a comparison of base model / memory only / RL only / memory + RL, using the same backbone, data, and sampling protocol, with training budgets explicitly reported. Please clarify which variants in Table 3 received RL post-training. (Q2, W2)

We sincerely thank the reviewer for this suggestion and report the comparison of base model / memory only / RL only / memory + RL below. All four models share the same backbone, data, and sampling protocol. As reported in the paper, "Base" is the checkpoint trained for 130k steps with HD-map structural conditioning, and "Memory only" continues from it with a further 15k and 5k steps for scene memory and instance memory, respectively. The two RL variants are trained under an identical budget (300 steps, group size 16, LoRA rank 256), each starting from its corresponding pre-RL checkpoint. Added to Base individually, each component improves what it is designed for: memory raises Subject Coherence from 0.646 to 0.678 and Scene Similarity from 0.863 to 0.905, while RL raises Subject Fidelity from 42.16 to 43.49 and Physical Plausibility from 51.45 to 54.66. When RL is applied on top of memory, its gains are larger (+2.26 vs. +1.33 on Subject Fidelity, +5.05 vs. +3.21 on Physical Plausibility), and they are obtained with a smaller policy drift (0.087 vs. 0.134). RL also does not compromise persistence but slightly improves it. Together, these results reflect an interaction benefit in which each component reinforces rather than offsets the other.

In Table 3, only the last row (the full model) received RL post-training. The "Memory only" results without RL are included in the table below.

  Persistence  Object & Physics  Policy-space Drift
MethodSub. Coh. ↑Scene Sim. ↑Sub. Fid. ↑Phys. Plaus. ↑DINOv2 Dist. to Pre-RL ↓
Base0.6460.86342.1651.45-
RL only0.6520.86143.4954.660.134
Memory only0.6780.90541.7450.91-
Memory + RL0.6810.91044.0055.960.087

6. How is instance memory constructed causally during inference? Please provide a step-by-step description or pseudocode specifying the observation window used to select the appearance crops, their source, and the ordering of generation, memory registration, and subsequent retrieval. (Q3)

Instance memory is constructed causally by using only the conditioning frame and completed generated chunks, with newly registered entries available to subsequent chunks.

Observation window: For a new instance, appearance crops are selected only from its fully generated first-appearance chunk. Instances already visible in the conditioning frame use that frame directly.

Source of the crops: They come from the conditioning frame or frames already generated by the model, without future GT appearance or external instance references. GT bounding boxes and track IDs provide only spatial locations and identity associations.

Ordering: Instances visible in the conditioning frame are registered first. Each subsequent chunk follows: read existing memory → generate the current chunk → register new instances. A new instance has no dedicated appearance-memory entry while its first-appearance chunk is being generated. Its entry is written only after that chunk is complete and is read starting from subsequent chunks.

AuthorsOpenReview ↗

Official Comment by Authors (3)

7. How does selecting up to four crops across temporal segments reconcile with writing the bank once at an instance’s initial appearance? What happens when insufficient valid observations are available? Please distinguish training-time and inference-time construction. If the reported evaluation uses future ground-truth appearance crops, please also evaluate a causal variant using only observations available at registration. (Q3)

Multi-crop selection and write-once registration. These are compatible: multiple crops are used to construct the content written in a single registration event. Specifically, we divide the candidate observations into four temporal segments and select the largest-box observation from each segment. These four observations are encoded and written together in a single registration step.

Insufficient observations. If fewer than four valid crops are available, we fill the empty slots with zeros. If there is no observation, we skip registration.

Training versus inference. During training, we use GT crops from individual chunks to teach the model to read appearance tokens. Training does not simulate cross-chunk registration and persistence. During inference, instances visible in the conditioning frame are registered from that frame before generation begins; later instances are registered using completed generated chunks.

Evaluation. The reported evaluation does not use future GT appearance crops.

8. How are geographic references separated from the evaluation recordings? Please describe the construction of the reference database and the split among training recordings, evaluation sequences, and reference imagery. (Q4)

We construct the geographic reference database D_geo offline from the training set and use SQLite to organize reference images and their latitude and longitude coordinates. Scene references are sourced from the retrieval results of the reference database constructed from training datasets. At inference time, we query the database using the coordinates associated with each temporal segment and retrieve the nearest image within the local neighborhood. An empty neighborhood yields an empty retrieval result.

All recordings used for evaluation are excluded from the training set. Geographic reference images come from independently collected recording sessions covering the evaluated locations. The reference database and evaluation videos are separated at the recording level, excluding the target sequences and temporally adjacent segments of the same recordings from the reference database. The scene-preservation results reported in the paper already follow this independent captured references protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.