acceptodds
Under review as a conference paper at ICLR 2027

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

Abstract

Recorded driving datasets cannot exhaustively cover the various road layouts, traffic flows, and rare safety-critical scenarios needed for end-to-end autonomous driving. Scaling synthetic data beyond recorded scenes therefore requires a generator that can faithfully realize user-specified layouts, maintain temporally coherent geometry, and render scenes from flexible viewpoints without matching real-world footage. Existing methods struggle to meet these requirements together: end-to-end video generators offer limited geometric control, while occupancy-guided approaches may deviate from BEV layouts and remain constrained by low-frequency signals, reference frames, or fixed camera rigs. We propose AnyScene, a unified occupancy-centric framework that autoregressively generates semantic occupancy sequences from diverse BEV layouts and uses the resulting geometry to synthesize temporally consistent multi-view driving videos. AnyScene follows user-designed layouts and generalizes zero-shot to layouts from unseen datasets and global maps, while supporting reference-free video synthesis and flexible camera configurations. To support the framework, we build nuCraftv2, a 12 Hz dataset based on nuScenes with synchronized BEV layouts and high-quality dense semantic occupancy. Experiments demonstrate zero-shot occupancy generation across these layout sources and state-of-the-art video generation, together with applications to counterfactual observation generation and sparse-view 3D reconstruction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Controllable Generation, World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nQ4JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a two-stage driving-scene generator that first predicts 12 Hz semantic occupancy from BEV layouts and then renders multi-view videos using occupancy-derived semantic/coordinate buffers and camera embeddings. The integrated system is ambitious and reports strong in-domain video and reconstruction results.

Critical strengths

S1. Strong reported in-domain results. Table 2 reports FVD values of 35.98, 43.19, and 57.57 at 8, 16, and 49 frames. The reported 38.26 mIoU and 27.73 mAP exceed GenieDrive by 7.29 and 8.56 points under the authors’ re-evaluation protocol.

S2. Potentially valuable high-frequency occupancy resource. The nuCraftv2 pipeline combines refined static reconstruction, dynamic-instance assets, HD maps, and 12 Hz poses over a large 3D region.

Critical weaknesses

W1: Limited methodological novelty and insufficient isolation of the core contributions. The proposed framework appears to combine several existing techniques, including occupancy-conditioned generation, autoregressive temporal modeling, geometry-aware conditioning, and pretrained video diffusion with ControlNet-style adaptation. While the integration is potentially useful, the paper does not sufficiently establish which components constitute a fundamentally new methodological contribution beyond prior occupancy-centric driving scene generation approaches.

W2. Insufficient quantitative evaluation of zero-shot generalization and flexible-camera generation. Table 6 reports results only on nuCraftv2-val, while Table 2 adopts a reference-view-conditioned evaluation protocol because the baselines do not support the claimed reference-free setting, making it difficult to assess the advantages of the proposed approach under a consistent and directly comparable protocol. The authors should evaluate AnyScene on additional benchmarks, such as nuPlan and WOMD, under a strictly train-free setting, and compare it against relevant state-of-the-art methods using consistent evaluation protocols and quantitative metrics.

Questions for authors

Q1. What rollout positions contribute to Table 6, and how do occupancy scores change with autoregressive horizon?

Q2. Will nuCraftv2, preprocessing code, models, prompts, and evaluation scripts be released?

2 comments
AuthorsOpenReview ↗

Response #1 to Reviewer nQ4j

Dear Reviewer,

We sincerely thank you for acknowledging our strong in-domain video generation and perception results, as well as the potential value of nuCraftv2 as a high-frequency occupancy resource. We address your concerns through further clarification of our contributions and novelty and additional evaluations of zero-shot generalization, flexible-camera generation, and autoregressive stability, as detailed below.

W1: Clarification of Contributions and Novelty.

As summarized in the Overall Response, AnyScene treats scalable driving-data generation as a unified task, requiring faithful layout control, high visual quality, temporal and cross-view consistency, and flexible camera configurations together. This task determines the BEV-to-occupancy-to-video route, as BEV layouts provide a portable control interface across data sources and occupancy provides explicit, camera-independent geometry. Existing approaches following this route remain limited in layout adherence, temporal resolution, and dependence on reference frames and fixed camera rigs. AnyScene addresses these limitations through a co-design of the entire pipeline, with each stage verified quantitatively:

  • Occupancy Data and Generation. nuCraftv2 provides 12 Hz dense occupancy supervision, enabling occupancy generation at the video frame rate. Under the same training budget, the occupancy generator achieves 52.45/68.97 BEV-to-layout mIoU/IoU, compared with 36.09/47.35 for UniScene and 12.62/6.81 for X-Scene, outperforms both on zero-shot layouts (W2.1), and shows no measurable degradation over 17-second autoregressive rollouts (Q1).
  • Video Generation. Grounding video generation in occupancy geometry reduces FVD from 112.20 to 43.19 (Table 6), and AnyScene achieves the best FVD, mIoU, and mAP under the common protocol (Table 2). Because scene and camera configuration are decoupled, reference-free, flexible-camera views are generated from the same occupancy, and their utility is validated quantitatively through sparse-view 3D reconstruction (Table 7 and W2.3).

Our contribution therefore lies in this task-driven co-design, which satisfies these requirements together and makes camera configurations flexible queries over a shared, layout-controlled scene, rather than in any individual module. Accordingly, we do not claim the occupancy VAE or the video generation architecture as standalone methodological innovations. The resulting layout-faithful, high-frequency, and reference-free generation enables view expansion, counterfactual scenario generation, data generation across diverse locations, and improved scene reconstruction.

W2.1: Quantitative Zero-shot Generalization on nuPlan, WOMD, and OSM.

We conduct additional zero-shot evaluations on nuPlan, WOMD, and OpenStreetMap (OSM) without any fine-tuning, using UniScene and X-Scene as baselines under a consistent evaluation protocol. Since WOMD and OSM do not provide surround-view camera images, video-level metrics cannot be computed consistently across these sources. We therefore evaluate BEV-to-layout consistency, which requires no ground truth and applies uniformly to all layout sources. AnyScene consistently outperforms UniScene and X-Scene on all three sources. These results provide quantitative evidence that AnyScene maintains strong layout controllability across unseen data sources.

MethodnuPlan mIoU / IoU ↑WOMD mIoU / IoU ↑OSM mIoU / IoU ↑
AnyScene (Ours)50.5 / 73.052.2 / 60.650.7 / 60.9
UniScene32.3 / 46.229.7 / 34.233.5 / 48.5
X-Scene14.1 / 6.418.1 / 7.06.5 / 4.5

W2.2: Evaluation Protocol of Table 2.

Existing baselines do not support reference-free view expansion with flexible camera configurations, so the reference-view-conditioned protocol is the only setting in which all methods can be compared consistently. Under this common protocol, AnyScene still achieves the best FVD, mIoU, and mAP in Table 2. Reference-free generation and flexible camera configurations, which the baselines do not support, are demonstrated in Fig. 4 and evaluated separately in W2.3.

AuthorsOpenReview ↗

Response #2 to Reviewer nQ4j

W2.3: Quantitative Evaluation of Flexible-camera Generation.

Direct quantitative evaluation of flexible-camera generation remains challenging due to the lack of suitable baselines, ground-truth views, and established evaluation tools. Qualitatively, Fig. 4(b) and Figs. 18–19 show consistent scene geometry and appearance when expanding 6 cameras to 12, 18, and 24 views. Quantitatively, Table 7 validates this capability through sparse-view 3D reconstruction: adding 6 or 18 generated views to the 6 physical cameras consistently improves completeness, recall, and BEV coverage under both Depth Anything 3 and SurroundNEXO, while accuracy stays within 0.03 m of the real-camera baseline. For example, with Depth Anything 3 and 18 added views, the completeness error decreases from 1.10 to 0.68 m, and both [email protected] and BEV coverage increase from 0.56 to 0.68. These results provide quantitative evidence of the practical utility of flexible-camera generation.

Q1: Rollout Evaluation and Autoregressive Horizon.

  • Evaluation Protocol of Occupancy Generation. The occupancy generation evaluation covers all 33.3K frames from 150 nuCraftv2 validation scenes at 12 Hz, covering rollout positions from 0 to approximately 20 seconds (18.5 seconds on average). Each scene is generated through a continuous autoregressive rollout, where the first frame is sampled without history and subsequent frames are conditioned on the three most recently generated frames, without any ground-truth re-anchoring. All frames contribute equally to the reported metrics.
  • Autoregressive Horizon Analysis. We further evaluate occupancy quality at different rollout positions, with representative results shown below.
Metric0–1 s4–5 s9–10 s14–15 s16–17 sAll
3D mIoU18.818.919.419.318.719.01
3D IoU15.115.616.116.015.215.58
BEV-vs-Layout mIoU52.451.952.652.652.552.45

All 150 scenes cover at least 17 seconds. Over this interval, both 3D mIoU and BEV-to-layout mIoU remain close to their overall values, showing no measurable degradation with increasing autoregressive horizon.

  • Long-horizon Zero-shot Evaluation. To further investigate long-term stability, we conduct additional 60-second (720-frame) autoregressive rollouts on 10 nuPlan logs under a strictly zero-shot setting, without fine-tuning or ground-truth re-anchoring. Since nuPlan lacks occupancy ground truth, we evaluate BEV-to-layout consistency.
Metric0–10 s20–30 s40–50 s50–60 s
BEV-vs-Layout mIoU (static)43.144.144.444.2
BEV-vs-Layout mIoU (dynamic)54.451.450.145.2
BEV-vs-Layout IoU70.471.772.072.3

Static map consistency and IoU remain stable throughout the 60-second rollout, while dynamic categories decline gradually, from 54.4 to 50.1 mIoU over the first 50 seconds and to 45.2 in the final 10 seconds. Overall, these results demonstrate stable autoregressive occupancy generation over standard scene durations and promising long-horizon layout controllability, while revealing gradual degradation in dynamic object generation over extended horizons.

Q2: Open-source Release.

Yes. Upon acceptance, we will release the nuCraftv2 dataset (data and per-instance assets), the preprocessing code, the models, the prompts, and the evaluation scripts.

Reviewer_83crOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose AnyScene, a unified occupancy-centric framework that autoregressively generates semantic occupancy sequences from BEV and user-design layouts. To train it, authors build nuCraftv2, a 12 Hz dataset based on nuScenes with synchronized BEV layouts and high-quality dense semantic occupancy. Resulting generated occupancy helps to synthesize temporally consistent multi-view driving videos. Experiments show that the occupancy generator generalizes zero-shot across user-designed layouts, unseen datasets, and global maps, while the video generator achieves state-of-the-art performance.

Critical strengths
  1. Authors build a high-frequency dataset with synchronized BEV layouts and high-quality dense semantic occupancy, which is important for the community.
  2. Authors provide broad range of evaluations and applications. Compared to other methods, AnyScene achieves state-of-the-art performance.
Critical weaknesses
  1. Main part of the models is semantic occupancy generation from BEV-layout, but authors only measure occupancy VAE reconstruction (Table 1), and quality of generation from BEV is not compared to baselines.
  2. The technical novelty is not clear. Previous methods also generate occupancy from BEV-layouts and condition video generation on it, and each part is mostly a combination of existing components.
Questions for authors
  1. Could you compare to baselines on occupancy generation quality?
  2. Evaluation on WOMD is only qualitative, is it possible to measure it quantitatively?
1 comment
AuthorsOpenReview ↗

Response #1 to Reviewer 83cr

Dear Reviewer,

We sincerely appreciate your recognition of the value of nuCraftv2 as a high-frequency occupancy dataset, as well as the broad range of evaluations and applications demonstrating AnyScene's strong performance. We address your concerns through additional occupancy generation comparisons, clarification of our contributions and novelty, and quantitative zero-shot evaluations, as detailed below.

W1 & Q1: Comparison with Occupancy Generation Baselines.

We conduct additional comparisons with UniScene and X-Scene under the same training budget (120 GPU-hours for the VAE and 300 GPU-hours for occupancy generation). Both baselines are retrained on nuCraftv2 from their official implementations, using the same data splits, ground-truth occupancy, and BEV layout conditions as AnyScene. The results demonstrate that AnyScene achieves higher voxel-level occupancy accuracy and stronger BEV layout consistency than both baselines.

MethodObserved 3D Volume (↑) mIoU/IoU3D Volume (↑) mIoU/IoUBEV Top-down (↑) mIoU/IoUBEV vs. Layout (↑) mIoU/IoU
AnyScene (ours)24.14/24.6419.01/15.5835.04/60.5352.45/68.97
UniScene19.26/23.6215.26/14.5527.86/57.3736.09/47.35
X-Scene3.84/5.363.86/3.77.96/33.0612.62/6.81

These demonstrate stronger geometric fidelity and, more importantly, substantially better adherence to the specified BEV layout, validating the effectiveness of our occupancy generation pipeline beyond VAE reconstruction.

W2: Clarification of Contributions and Novelty.

As summarized in the Overall Response, AnyScene treats scalable driving-data generation as a unified task: faithful layout control, high visual quality, temporal and cross-view consistency, and flexible camera configurations must be achieved together. This task determines the BEV-to-occupancy-to-video route, as BEV layouts provide a portable control interface across data sources and occupancy provides explicit, camera-independent geometry. Existing methods following this route still face the following limitations, which AnyScene addresses through a co-design of the entire pipeline, from data construction to occupancy and video generation:

  • Layout Adherence. Existing BEV-to-occupancy generation may deviate from the specified layouts. Under the same training budget, AnyScene achieves 52.45/68.97 BEV-to-layout mIoU/IoU on nuCraftv2, compared with 36.09/47.35 for UniScene and 12.62/6.81 for X-Scene (W1 & Q1), and maintains this advantage on zero-shot layouts (Q2).
  • Temporal Resolution. Existing pipelines rely on low-frequency (2 Hz) or coarse conditioning, which limits frame-level control. nuCraftv2 provides 12 Hz dense occupancy supervision, so AnyScene generates occupancy at the frame rate required by video generation.
  • Reference Frames and Camera Rigs. Existing occupancy-guided video generation, such as UniScene, InfiniCube, and GenieDrive, depends on reference frames and fixed camera rigs. AnyScene decouples scene specification from camera configuration: the same generated occupancy is rendered into semantic and coordinate buffers for any camera count and pose, so multi-view videos are generated without reference frames and 6 cameras can be expanded to up to 24 views without retraining. It also achieves the best FVD, mIoU, and mAP in Table 2.

Our contribution therefore lies in this task-driven co-design, which satisfies these requirements together and makes camera configurations flexible queries over a shared, layout-controlled scene, rather than in any individual module. The resulting layout-faithful, high-frequency, and reference-free generation enables the applications demonstrated in the paper, including view expansion, counterfactual scenario generation, data generation across diverse locations, and improved scene reconstruction.

Q2: Quantitative Zero-shot Evaluation on WOMD.

We conduct additional quantitative evaluations on WOMD under a strictly zero-shot setting, without any fine-tuning. AnyScene substantially outperforms UniScene and X-Scene.

MethodnuPlan mIoU / IoU ↑WOMD mIoU / IoU ↑OSM mIoU / IoU ↑
AnyScene (Ours)50.5 / 73.052.2 / 60.650.7 / 60.9
UniScene32.3 / 46.229.7 / 34.233.5 / 48.5
X-Scene14.1 / 6.418.1 / 7.06.5 / 4.5
Reviewer_dEqCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AnyScene is a two-stage, occupancy-centric driving scene generator. A BEV-conditioned latent DiT first generates 12 Hz semantic occupancy from BEV layout sequences. Semantic and coordinate buffers rendered from the occupancy then condition a control branch on a frozen Wan2.1-14B video model, which supports reference-free generation and autoregressive view expansion to arbitrary camera rigs. The authors also build nuCraftv2, a 12 Hz dense occupancy dataset derived from nuScenes. On nuScenes, the video generator reports the best FVD and BEVFormer mIoU/mAP among the compared methods, and the extra generated views densify sparse-view reconstruction.

Critical strengths
  1. Conditioning the video model on geometry rendered from occupancy makes the camera rig something chosen at inference time. One model can generate surround views without a reference frame, expand from 6 to 24 cameras, and re-render a laterally shifted ego path. Fixed-rig generators cannot do this without retraining.
  2. The occupancy generator is well designed, and the ablations support the design. Joint attention over BEV and occupancy tokens clearly beats additive conditioning (19.01 vs. 13.59 mIoU), and teacher forcing is essential for stable autoregressive rollout. The rendered semantic and coordinate buffers also matter a lot for video quality (Table 4: FVD drops from 112.2 to 43.2).
  3. nuCraftv2 (12 Hz dense occupancy with per-instance assets) could be a useful resource for occupancy generation and world modeling, if it is released.
Critical weaknesses
  1. Controllability is not compared with prior work. Table 6 reports only AnyScene, although Appendix D.1 says the metrics follow X-Scene and UniScene. Neither method, nor any other BEV-to-occupancy method, is included in the table. The BEV-vs-layout score of 52.45 mIoU is also hard to interpret without the same score for ground-truth occupancy. The 3D scores are low (19.01 mIoU, 15.58 IoU). The paper explains this by sparse ground truth, but nuCraftv2 is presented as dense, and Fig. 11 cannot separate plausible completion from hallucination. Reporting mIoU only over observed voxels would help.
  2. Zero-shot generalization is claimed but not measured. The abstract and contributions state that experiments demonstrate zero-shot generation on nuPlan, WOMD, OpenStreetMap and user layouts. The evidence is only Figs. 1, 15 and 16. Appendix C.1 does not explain how these sources are mapped to the 15-channel layout, and it never lists the five static classes. For OSM, which has no agents and only partial lane-level geometry, the conversion matters a lot. Layout-consistency scores on each source would make this claim checkable.
  3. The comparison in Table 2 needs clarification. The text says all methods were re-evaluated, but several baseline rows match the values reported in GenieDrive. Two entries differ from that source: the MagicDrive-V2 clip length and the GenieDrive mAP. The methods also differ in clip length, frame rate, resolution, backbone size (1.3B vs. 14B) and conditioning, so the FVD gaps are not directly comparable. In addition, the mIoU/mAP listed in the 49-frame row are identical to the 16-frame results in Table 4.
Questions for authors
  1. Could you add occupancy-generation baselines (e.g., UniScene, X-Scene) and the BEV-vs-layout score of ground-truth occupancy?
  2. How are nuPlan, WOMD and OSM converted to the 15-channel layout, and where do agents come from for OSM? Can you report layout consistency on each source?
  3. What are the inference time and memory costs, especially for 24-view expansion?
3 comments
AuthorsOpenReview ↗

Response #1 to Reviewer dEqC

Dear Reviewer,

We sincerely thank you for the thoughtful assessment and for recognizing the flexibility of our camera generation framework, the effectiveness of our occupancy generator and supporting ablations, and the potential value of nuCraftv2 as a high-frequency occupancy resource. We address your concerns regarding controllability comparisons, zero-shot generalization, and evaluation details as follows.

W1.1 & Q1: Comparison with Occupancy Generation Baselines.

MethodObserved 3D Volume mIoU/IoU3D Volume (↑) mIoU/IoUBEV Top-down (↑) mIoU/IoUBEV vs. Layout (↑) mIoU/IoU
AnyScene (ours)24.14/24.6419.01/15.5835.04/60.5352.45/68.97
UniScene19.26/23.6215.26/14.5527.86/57.3736.09/47.35
X-Scene3.84/5.363.86/3.77.96/33.0612.62/6.81
GT (val full)---47.73/48.86

We have conducted additional comparisons with UniScene and X-Scene and evaluated the BEV-to-layout consistency of ground-truth (GT) occupancy, along with 3D occupancy metrics computed over observed voxels, as shown in the table above. We compare all methods under the same training budget (120 GPU-hours for the VAE and 300 GPU-hours for occupancy generation). Both baselines are retrained on nuCraftv2 from their official implementations, using the same data splits, ground-truth occupancy, and BEV layout conditions as AnyScene. AnyScene consistently outperforms UniScene and X-Scene on 3D occupancy accuracy and BEV layout consistency. These results demonstrate stronger geometric fidelity and, more importantly, substantially improved adherence to the specified BEV layout.

W1.2 & Q1: Layout Consistency of Ground-truth Occupancy.

The last row of the table above reports the BEV-to-layout consistency of GT occupancy. On the full validation set, AnyScene achieves a higher BEV-to-layout accuracy than GT occupancy. This is because GT occupancy is reconstructed from aggregated LiDAR scans and remains incomplete due to temporal boundaries, occlusions, and sparse long-range observations, as illustrated in Fig.11. In contrast, BEV layouts are derived from HD maps and 3D bounding boxes, providing more complete structural information. Consequently, GT occupancy may contain unobserved regions where the layout indicates valid road surfaces or lane markings. Conditioned on these layouts, AnyScene learns to generate plausible occupancy in such regions, resulting in higher layout consistency.

W1.3: Low 3D Scores and Evaluation over Observed Voxels.

A relatively low 3D mIoU does not necessarily indicate poor geometric quality, for two reasons. First, GT occupancy is inevitably incomplete (W1.2), so layout-consistent occupancy generated in regions missing from GT is counted as errors. Second, voxel-level metrics are sensitive to elevation: small height offsets on otherwise well-shaped roads shift them into different voxels, especially on the varying-elevation roads common in nuScenes and nuCraftv2. As suggested, we additionally evaluate 3D occupancy over camera-observed voxels, as reported in the first column of the table above. Restricting the evaluation to these voxels raises the scores of all methods, and AnyScene remains the best with 24.14/24.64 mIoU/IoU, compared with 19.26/23.62 for UniScene and 3.84/5.36 for X-Scene.

W2.1 & Q2: Quantitative Zero-shot Generalization.

We conduct additional zero-shot evaluations on nuPlan, WOMD, and OpenStreetMap (OSM) data without fine-tuning. The BEV-to-layout consistency results are summarized below.

MethodnuPlan mIoU / IoU ↑WOMD mIoU / IoU ↑OSM mIoU / IoU ↑
AnyScene (Ours)50.5 / 73.052.2 / 60.650.7 / 60.9
UniScene32.3 / 46.229.7 / 34.233.5 / 48.5
X-Scene14.1 / 6.418.1 / 7.06.5 / 4.5

AnyScene consistently outperforms both baselines across all three unseen data sources. These results demonstrate that AnyScene maintains strong layout controllability across different map formats and agent annotations without additional training, providing quantitative evidence of its zero-shot generalization capability.

W2.2 & Q2: Layout Conversion for nuPlan, WOMD, and OSM.

We convert all data sources into a unified 15-channel BEV layout, consisting of five static map classes (drivable area, pedestrian crossing, walkway, road divider, and lane divider) and ten dynamic agent classes following the nuScenes definitions. For nuPlan and WOMD, HD maps and agent annotations are transformed into the ego coordinate system and rasterized into the corresponding semantic channels. For OSM, map elements are converted based on their tags, with road widths estimated from lane counts. Since OSM lacks dynamic agent annotations, vehicle footprints are generated using a simulator with traffic simulation capabilities to create traffic configurations.

AuthorsOpenReview ↗

Response #2 to Reviewer dEqC

W3.1: Sources of Baseline Results in Table 2.

OmniNWM and the 16-frame and 49-frame GenieDrive results were re-evaluated using the same evaluation toolkit and code as AnyScene, while the 8-frame GenieDrive results and the results for the remaining baselines were adopted from the GenieDrive paper. The MagicDrive-V2 results reported in its original paper correspond to the 16-frame evaluation setting, consistent with our Table 2.

W3.2: Fairness of FVD Comparison.

We acknowledge that the compared methods differ in clip length, frame rate, resolution, backbone size, and conditioning, as is common in prior work. We therefore compare FVD under the same number of evaluation frames, following the evaluation protocol of GenieDrive. Under matched frame counts, AnyScene achieves the best FVD at 8, 16, and 49 frames, including against the re-evaluated GenieDrive (43.19/57.57 vs. 63.65/92.52 at 16/49 frames) and OmniNWM (43.19 vs. 60.49 at 16 frames). Regarding backbone size, the gains do not stem from our larger 14B backbone alone: in Table 6, the same backbone without geometric buffers yields an FVD of 112.20 at 16 frames, worse than GenieDrive (63.65) and MagicDrive-V2 (94.84), whereas adding the semantic and coordinate buffers reduces it to 43.19.

W3.3: Identical mIoU/mAP Results.

Following the W-CODA 2024 Track 2 evaluation protocol, mIoU and mAP are computed on three fixed frames (indices 0, 6, and 12), corresponding to the first three nuScenes keyframes. Since the 16-frame and 49-frame settings share these evaluation frames, their identical mIoU/mAP scores are expected rather than a reporting error.

Q3: Inference Time and Memory Costs of AnyScene.

We report inference costs on NVIDIA H200 GPUs for 49-frame clips at 480×832 resolution. Video generation uses 4-step DMD sampling without classifier-free guidance (CFG).

StageViewsTime (1 GPU)Peak GPU Memory
Occupancy generation—~11 min7.7 GiB
Video generation1~0.25 min~48 GiB
Video generation3~1.4 min~57 GiB
Video generation6~4.2 min~57 GiB
Video generation24~29 min~57 GiB

For 24-view generation, parallel execution on 8 GPUs reduces the latency to ~8 min. To control memory usage, we generate at most three views jointly and expand additional views through independent outpainting calls. Consequently, peak GPU memory remains approximately constant, while inference time scales with the number of generated views. This setting allows AnyScene to run on a single GPU regardless of the number of generated views. Further engineering optimizations remain possible, offering substantial opportunities to reduce inference latency and computational costs toward practical deployment.

Reviewer_dEqCOpenReview ↗

Reply to authors

I thank the authors for the detailed responses, which address most of my concerns. I will raise my score to a positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification of AnyScene’s Contributions and Summary of Additional Evidence

We would like to clarify that AnyScene’s central contribution is a complete driving-data generation framework and the capabilities it enables.

1. Addressing scalable driving-data generation as a unified task.

Our objective is to jointly achieve faithful layout control, high visual quality, temporal and cross-view consistency, and flexible camera configurations within a scalable data-generation pipeline. AnyScene connects user-specified BEV layout sequences to explicit 3D geometry and multi-view videos, supporting scene construction beyond recorded environments and synthesis without matching real reference images. The emphasis is on satisfying these requirements together throughout the pipeline, rather than optimizing an isolated intermediate module.

2. Novelty in consistent generation beyond fixed camera rigs.

The novelty we emphasize lies in the task-driven co-design that makes camera configurations flexible queries over a shared, layout-controlled scene. Scene specification and camera configuration are decoupled, allowing the same generated geometry to support consistent observations under different camera counts and poses without retraining. This provides a generative route to extending observation coverage beyond the recorded rig and alleviating sparse-view constraints in downstream reconstruction.

We do not claim the specific occupancy VAE design or video-generation architecture as standalone methodological innovations. These components implement and support the underlying design principle; our novelty claim concerns the co-design that realizes controllable scene generation and consistent, flexible-view synthesis together.

3. Additional experiments supporting the system-level contribution.

Our rebuttal adds direct comparisons with UniScene and X-Scene on occupancy-generation quality and layout controllability, going beyond VAE reconstruction. We also provide quantitative zero-shot layout-consistency evaluations on nuPlan, WOMD, and OpenStreetMap without fine-tuning. Finally, we clarify the continuous autoregressive evaluation on nuCraftv2 and analyze 60-second rollouts on nuPlan, characterizing both sustained layout control and degradation in dynamic categories at longer horizons.

These additions strengthen the empirical basis for AnyScene’s capabilities while clarifying their evaluation conditions and limitations. The corresponding results and protocol clarifications are provided in our point-by-point responses and will be incorporated into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.