- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents a two-stage driving-scene generator that first predicts 12 Hz semantic occupancy from BEV layouts and then renders multi-view videos using occupancy-derived semantic/coordinate buffers and camera embeddings. The integrated system is ambitious and reports strong in-domain video and reconstruction results.
Critical strengths
S1. Strong reported in-domain results. Table 2 reports FVD values of 35.98, 43.19, and 57.57 at 8, 16, and 49 frames. The reported 38.26 mIoU and 27.73 mAP exceed GenieDrive by 7.29 and 8.56 points under the authors’ re-evaluation protocol.
S2. Potentially valuable high-frequency occupancy resource. The nuCraftv2 pipeline combines refined static reconstruction, dynamic-instance assets, HD maps, and 12 Hz poses over a large 3D region.
Critical weaknesses
W1: Limited methodological novelty and insufficient isolation of the core contributions. The proposed framework appears to combine several existing techniques, including occupancy-conditioned generation, autoregressive temporal modeling, geometry-aware conditioning, and pretrained video diffusion with ControlNet-style adaptation. While the integration is potentially useful, the paper does not sufficiently establish which components constitute a fundamentally new methodological contribution beyond prior occupancy-centric driving scene generation approaches.
W2. Insufficient quantitative evaluation of zero-shot generalization and flexible-camera generation. Table 6 reports results only on nuCraftv2-val, while Table 2 adopts a reference-view-conditioned evaluation protocol because the baselines do not support the claimed reference-free setting, making it difficult to assess the advantages of the proposed approach under a consistent and directly comparable protocol. The authors should evaluate AnyScene on additional benchmarks, such as nuPlan and WOMD, under a strictly train-free setting, and compare it against relevant state-of-the-art methods using consistent evaluation protocols and quantitative metrics.
Questions for authors
Q1. What rollout positions contribute to Table 6, and how do occupancy scores change with autoregressive horizon?
Q2. Will nuCraftv2, preprocessing code, models, prompts, and evaluation scripts be released?
2 comments
Response #2 to Reviewer nQ4j
W2.3: Quantitative Evaluation of Flexible-camera Generation.
Direct quantitative evaluation of flexible-camera generation remains challenging due to the lack of suitable baselines, ground-truth views, and established evaluation tools. Qualitatively, Fig. 4(b) and Figs. 18–19 show consistent scene geometry and appearance when expanding 6 cameras to 12, 18, and 24 views. Quantitatively, Table 7 validates this capability through sparse-view 3D reconstruction: adding 6 or 18 generated views to the 6 physical cameras consistently improves completeness, recall, and BEV coverage under both Depth Anything 3 and SurroundNEXO, while accuracy stays within 0.03 m of the real-camera baseline. For example, with Depth Anything 3 and 18 added views, the completeness error decreases from 1.10 to 0.68 m, and both [email protected] and BEV coverage increase from 0.56 to 0.68. These results provide quantitative evidence of the practical utility of flexible-camera generation.
Q1: Rollout Evaluation and Autoregressive Horizon.
- Evaluation Protocol of Occupancy Generation. The occupancy generation evaluation covers all 33.3K frames from 150 nuCraftv2 validation scenes at 12 Hz, covering rollout positions from 0 to approximately 20 seconds (18.5 seconds on average). Each scene is generated through a continuous autoregressive rollout, where the first frame is sampled without history and subsequent frames are conditioned on the three most recently generated frames, without any ground-truth re-anchoring. All frames contribute equally to the reported metrics.
- Autoregressive Horizon Analysis. We further evaluate occupancy quality at different rollout positions, with representative results shown below.
| Metric | 0–1 s | 4–5 s | 9–10 s | 14–15 s | 16–17 s | All |
|---|---|---|---|---|---|---|
| 3D mIoU | 18.8 | 18.9 | 19.4 | 19.3 | 18.7 | 19.01 |
| 3D IoU | 15.1 | 15.6 | 16.1 | 16.0 | 15.2 | 15.58 |
| BEV-vs-Layout mIoU | 52.4 | 51.9 | 52.6 | 52.6 | 52.5 | 52.45 |
All 150 scenes cover at least 17 seconds. Over this interval, both 3D mIoU and BEV-to-layout mIoU remain close to their overall values, showing no measurable degradation with increasing autoregressive horizon.
- Long-horizon Zero-shot Evaluation. To further investigate long-term stability, we conduct additional 60-second (720-frame) autoregressive rollouts on 10 nuPlan logs under a strictly zero-shot setting, without fine-tuning or ground-truth re-anchoring. Since nuPlan lacks occupancy ground truth, we evaluate BEV-to-layout consistency.
| Metric | 0–10 s | 20–30 s | 40–50 s | 50–60 s |
|---|---|---|---|---|
| BEV-vs-Layout mIoU (static) | 43.1 | 44.1 | 44.4 | 44.2 |
| BEV-vs-Layout mIoU (dynamic) | 54.4 | 51.4 | 50.1 | 45.2 |
| BEV-vs-Layout IoU | 70.4 | 71.7 | 72.0 | 72.3 |
Static map consistency and IoU remain stable throughout the 60-second rollout, while dynamic categories decline gradually, from 54.4 to 50.1 mIoU over the first 50 seconds and to 45.2 in the final 10 seconds. Overall, these results demonstrate stable autoregressive occupancy generation over standard scene durations and promising long-horizon layout controllability, while revealing gradual degradation in dynamic object generation over extended horizons.
Q2: Open-source Release.
Yes. Upon acceptance, we will release the nuCraftv2 dataset (data and per-instance assets), the preprocessing code, the models, the prompts, and the evaluation scripts.
Response #1 to Reviewer nQ4j
Dear Reviewer,
We sincerely thank you for acknowledging our strong in-domain video generation and perception results, as well as the potential value of nuCraftv2 as a high-frequency occupancy resource. We address your concerns through further clarification of our contributions and novelty and additional evaluations of zero-shot generalization, flexible-camera generation, and autoregressive stability, as detailed below.
As summarized in the Overall Response, AnyScene treats scalable driving-data generation as a unified task, requiring faithful layout control, high visual quality, temporal and cross-view consistency, and flexible camera configurations together. This task determines the BEV-to-occupancy-to-video route, as BEV layouts provide a portable control interface across data sources and occupancy provides explicit, camera-independent geometry. Existing approaches following this route remain limited in layout adherence, temporal resolution, and dependence on reference frames and fixed camera rigs. AnyScene addresses these limitations through a co-design of the entire pipeline, with each stage verified quantitatively:
Our contribution therefore lies in this task-driven co-design, which satisfies these requirements together and makes camera configurations flexible queries over a shared, layout-controlled scene, rather than in any individual module. Accordingly, we do not claim the occupancy VAE or the video generation architecture as standalone methodological innovations. The resulting layout-faithful, high-frequency, and reference-free generation enables view expansion, counterfactual scenario generation, data generation across diverse locations, and improved scene reconstruction.
We conduct additional zero-shot evaluations on nuPlan, WOMD, and OpenStreetMap (OSM) without any fine-tuning, using UniScene and X-Scene as baselines under a consistent evaluation protocol. Since WOMD and OSM do not provide surround-view camera images, video-level metrics cannot be computed consistently across these sources. We therefore evaluate BEV-to-layout consistency, which requires no ground truth and applies uniformly to all layout sources. AnyScene consistently outperforms UniScene and X-Scene on all three sources. These results provide quantitative evidence that AnyScene maintains strong layout controllability across unseen data sources.
Existing baselines do not support reference-free view expansion with flexible camera configurations, so the reference-view-conditioned protocol is the only setting in which all methods can be compared consistently. Under this common protocol, AnyScene still achieves the best FVD, mIoU, and mAP in Table 2. Reference-free generation and flexible camera configurations, which the baselines do not support, are demonstrated in Fig. 4 and evaluated separately in W2.3.