- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to deal with the systematic degradation of autoregressive driving world models over minute-scale rollouts in terms of structural drift, identity drift, and quality decay. WAM-Sim anchors generation with a three-tier persistent world memory and restores visual sharpness and physical plausibility via Drive DiffusionNFT. On nuScenes it achieves strong fidelity and downstream perception, and on minute-scale self-collected rollouts it substantially mitigates structural/identity drift, with human-preference wins over strong baselines.
Critical strengths
The paper decomposes long-horizon failure into structural drift, identity drift, and quality decay, and argues that the root cause is the lack of persistent, addressable state rather than mere context-window length.
Critical weaknesses
- As nuScenes lacks geolocated reference databases, WAM-Sim runs without scene memory there. The headline FID=7.07 is therefore produced by a reduced variant that abandons a central component, so the paper’s central contribution is validated almost entirely on a self-collected, non-standard dataset rather than on the public benchmark it leads with. This also happens with the minute-scale rollouts evaluation.
- The paper presents this synergy as a key insight but provides no formal characterization or controlled measurement of how the memory constrains the policy search space or prevents reward hacking. Support is limited to qualitative figures and ablation trends.
- It is unclear whether the fidelity/consistency gains justify the overhead relative to cheaper trajectory-conditioned or single-chunk baselines, especially for scalable closed-loop simulation.
- Scene memory requires a geolocated reference database, limiting applicability and fairness of comparison.
Questions for authors
- How well does WAM-Sim perform on nuScenes when its scene memory is enabled, and is there a public long-horizon benchmark where all memory tiers are active?
- What exactly does the invariant memory manifold bound, and how is reward hacking actually prevented? A quantitative characterization e.g., measuring policy-space drift, reward-distribution shift, or hallucination rate over training should be provided.
- How does the method’s cost scale, and is the fidelity gain worth it for practical closed-loop simulation?
- How sensitive are the reported gains to the VLM physical judge and the scene-memory reference quality?
7 comments
Official Comment by Authors (2)
3. It is unclear whether fidelity/consistency gains justify the overhead relative to cheaper trajectory-conditioned or single-chunk baselines, especially for scalable closed-loop simulation. How does the method’s cost scale? (W3, Q3)
| Method | Resolution | Params | Throughput ↑ | VRAM ↓ | FVD ↓ | Scene Similarity ↑ | Subject Coherence ↑ |
|---|---|---|---|---|---|---|---|
| trajectory-conditioned | 1280×704 | 2B | 8.00 FPS | 45.71 | 286.54 | 0.697 | 0.322 |
| Ours | 1280×704 | 3.1B | 6.94 FPS | 47.45 | 137.92 | 0.828 | 0.547 |
At the same resolution, compared with the trajectory-conditioned baseline, WAM-Sim incurs a 13.3% reduction in throughput and a 3.8% increase in peak memory usage, while achieving 51.9% lower FVD over minute-scale rollouts and substantially improving instance consistency and scene similarity. This trade-off demonstrates the value of persistent memory for long-horizon simulation.
Our memory design scales as O(1) with rollout duration: scene memory applies cross-attention with a single reference frame and a fixed number of tokens per segment, while instance memory maintains a bounded cache of up to 32 active instances. Consequently, per-chunk inference latency and peak VRAM remain constant as rollout duration increases (e.g., from 10s to 60s and beyond), supporting scalable long-horizon closed-loop simulation.
4. How sensitive are the reported gains to the VLM physical judge and the scene-memory reference quality? (Q4)
Sensitivity to VLM physical judge. To test the sensitivity of the reported gains to the VLM judge, we re-evaluated Physical Plausibility with an additional paraphrased prompt set (Prompt Set 2) and a stronger judge (Qwen3.8-27B). Although absolute scores vary across settings, our final model outperforms the strongest baseline (HorizonDrive) in every case, and RL post-training consistently improves over our model without RL by 2.72 to 5.86 points.
Prompt Set 2—Q1: Look carefully at every pedestrian who is walking or running in this video, from the first frame to the last: is their limb motion always anatomically correct and fully natural, with no implausible joint movement and no abnormal limb paths at any moment? Answer with yes or no. | Q2: Whenever pedestrians are moving, are their arms and legs always easy to tell apart, with no limbs fusing into the torso, fusing with one another, vanishing, or becoming visually tangled at any point? Answer with yes or no.
| Physical Plausibility↑ | ||||
|---|---|---|---|---|
| Qwen3.5-9B | Qwen3.5-9B | Qwen3.8-27B | Qwen3.8-27B | |
| (Prompt Set 1) | (Prompt Set 2) | (Prompt Set 1) | (Prompt Set 2) | |
| HorizonDrive | 51.19 | 64.24 | 40.77 | 40.70 |
| Ours (w/o RL) | 50.91 | 63.74 | 39.36 | 38.99 |
| Ours (w/ RL) | 55.96 | 66.46 | 44.97 | 44.85 |
Sensitivity to reference quality. We assess sensitivity to scene-reference quality by adding noise to the reference images. Clean references reduce FVD from 81.64 to 76.68 and improve Scene Similarity from 0.863 to 0.911. With noise-corrupted references, FVD reaches 78.16, and Scene Similarity reaches 0.903, corresponding to an FVD degradation of 1.48 and a Scene Similarity decrease of only 0.008 relative to clean references. These results show that the benefits of scene memory largely persist under tested noise corruption.
| Scene Memory | Noise Corruption | FVD ↓ | Scene Similarity ↑ |
|---|---|---|---|
| 81.64 | 0.863 | ||
| 76.68 | 0.911 | ||
| 78.16 | 0.903 |
Official Comment by Authors (3)
5. Scene memory requires a geolocated reference database, limiting applicability and fairness of comparison. (W4)
Applicability: The geographic database is constructed offline from geotagged training images, enabling reuse of existing driving logs. Scene-reference conditioning is optional at the chunk level: for chunks with missing references, generation proceeds using the remaining conditioning inputs. We evaluate robustness to incomplete reference availability by independently dropping scene references during inference. At dropout probabilities of 0.1 and 0.3, Scene Similarity reaches 0.908 and 0.901, respectively, compared with 0.910 using all references. These results demonstrate graceful degradation under partial reference availability and support the applicability of scene memory with incomplete geographic coverage.
| Drop Prob. p | Scene Similarity ↑ | FVD ↓ |
|---|---|---|
| 0.0 | 0.910 | 74.20 |
| 0.1 | 0.908 | 77.03 |
| 0.3 | 0.901 | 78.60 |
Fairness:We evaluate UniMLVG, OpenDWM, and HorizonDrive using their original publicly released checkpoints and architectures. Their inputs exclude the retrieved reference images. To assess the contribution of our memory design under matched reference access, we additionally implement a baseline on OmniDreams, providing it with exactly the same scene references and instance appearance information as our method.
With the same reference information, our method further reduces FVD from 76.84 to 73.77 and improves Subject Coherence from 0.658 to 0.678 and Scene Similarity from 0.888 to 0.905. This matched-reference fair comparison supports the additional contribution of persistent memory management beyond the gains obtained by supplying appearance references.
| FVD ↓ | Subject Coherence ↑ | Scene Similarity ↑ | |
|---|---|---|---|
| OmniDreams | 81.64 | 0.646 | 0.863 |
| OmniDreams + Ref | 76.84 | 0.658 | 0.888 |
| Ours (w/o RL) | 73.77 | 0.678 | 0.905 |
Official Comment by Authors (1)
We sincerely appreciate Reviewer Apsc for the insightful, thorough, and constructive review. Your sharp observations—from full-memory validation on public benchmarks and quantitative bounds on RL policy drift to computational scaling and reference sensitivity—directly pinpoint what is needed to make our claims airtight and have substantially strengthened our paper. Below, we address each point concisely.
1. How well does WAM-Sim perform on nuScenes when its scene memory is enabled, and is there a public long-horizon benchmark where all memory tiers are active? (W1, Q1)
To validate our contribution on a public benchmark, we conduct additional experiments on nuScenes with all memory tiers enabled. We evaluate the full validation sequences (~20 seconds, ~200 frames) using multi-chunk autoregressive rollouts, substantially longer than the clips (8-16 frames) used by many baselines in Table 1. To our knowledge, nuScenes is currently the only public driving benchmark combining HD maps, 3D tracking boxes, and a geographic reference database (nuScenes-Geography from Spatial Retrieval), as Waymo and Argoverse lack publicly available paired geographic references. Thus, multi-chunk autoregressive rollouts on nuScenes sequences with all memory tiers enabled represent the most feasible long-horizon evaluation with currently available public datasets.
Enabling memory reduces FID from 7.07 to 5.75 and FVD from 42.01 to 32.40. It also improves mAP and NDS by 2.63 and 1.06 points, respectively, and road and vehicle mIoU by 2.24 and 0.89 points. For long-horizon generation on nuScenes, the full-memory variant improves Subject Coherence from 0.623 to 0.666 and Scene Similarity from 0.854 to 0.891. These additional experiments provide evidence on a widely used public driving dataset for the benefits of the world memory across generation fidelity, downstream perception, and long-horizon scene and identity consistency.
2. What exactly does the invariant memory manifold bound, and how is reward hacking actually prevented? A quantitative characterization e.g., measuring policy-space drift, reward-distribution shift, or hallucination rate over training should be provided. (W2, Q2)
We sincerely thank the reviewer for raising this point. To provide a controlled measurement beyond qualitative evidence, we report the complete memory on/off × RL on/off comparison under the same setting as in the original manuscript. We also quantify policy-space drift, measured as the DINOv2 Distance between rollouts generated before and after RL. During RL, we update only the self-attention LoRA parameters of the DiT and keep the memory cross-attention and all other modules frozen. Generation thus stays anchored to the manifold consistent with the road structure, geographic background, and instance appearance provided by memory. We empirically observe that memory helps limit policy drift to some extent: the model moves noticeably less from its pre-RL behavior when memory is present (DINOv2 Distance of 0.087 vs. 0.134). Under the same training settings, RL yields larger gains on held-out metrics with memory than without (+2.26 vs. +1.33 on Subject Fidelity, +5.05 vs. +3.21 on Physical Plausibility), which suggests that the improvements reflect genuine quality gains rather than reward hacking. RL also preserves the persistence gains from memory, even improving them slightly. Overall, these results reflect an interaction benefit in which each component reinforces rather than offsets the other. We will revise the manuscript so that the claims about this interaction benefit are stated more accurately and grounded in these results.