- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work focuses on multi-view driving video generation. The authors propose a method named V-REG that exploits a pretrained foundation model (DINOv3) to accelerate the optimization of a multi-view video diffusion transformer for driving scene video generation. To facilitate feature alignment between the pretrained foundation model and the diffusion transformer, the author propose frame-view aligned representation entanglement and alignment. Experiments on the nuScenes dataset show the effectiveness of the proposed method.
Critical strengths
- The proposed idea is technically sound and easy to follow.
Critical weaknesses
- The contribution of the proposed method is not significant. The idea of accelerating the optimization of diffusion transformers with supervision from foundation models has been widely explored in the literature. REPA is one of the pioneering works that proposes representation alignment to accelerate diffusion transformer training. Subsequent works, such as iREPA, VideoREPA, and GeometryForcing, have introduced various strategies for representation alignments. The proposed frame-view alignment is a simple variant of existing techniques rather than a technically new method.
[1] REPRESENTATION ALIGNMENT FOR GENERATION: TRAINING DIFFUSION TRANSFORMERS IS EASIER THAN YOU THINK. Yu et al. ICLR 2025.
[2] WHAT MATTERS FOR REPRESENTATION ALIGNMENT: GLOBAL INFORMATION OR SPATIAL STRUCTURE?. Singh et al. ICLR 2025.
[3] VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models. Zhang et al. NeurIPS 2025.
[4] GEOMETRY FORCING: MARRYING VIDEO DIFFUSION AND 3D REPRESENTATION FOR CONSISTENT WORLD MODELING. Wu et al. NeurIPS 2025.
- Many state-of-the-art driving video generation methods are overlooked. For example, [5,6,7] achieve significantly better performance than the proposed method on the nuScenes dataset, but these methods are not compared or discussed in this work.
[5] MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction. Ni et al. CVPR 2025.
[6] UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving. Chen et al. ICCV 2025.
[7] MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer. Wu et al. CVPRF 2026.
-
The statement about the existing multi-modal DiT paradigm is overly narrow. There are different paradigms to train multi-modal multi-view video diffusion transformers. For example, [7] proposes a unified multi-modal multi-view video transformer for driving video generation without auxiliary control DiT or auxiliary alignment.
-
The claim “training-efficient” is not well supported by the experiments. Throughout the paper, the authors use the 30K step optimization as the evident to demonstrate the training efficiency. However, the proposed requires an additional foundation model, DINOv3, to process the inputs. Although this additional foundation model is frozen, this will still significantly increase the training cost. Thus, training step is not a good metric to measure training efficiency. Instead, a common practice is training the compared methods on the same machine to measure the GPU hours required for model training.
-
Experiments need improvement. In this work, the authors mainly report results on video, depth, and semantic generation. However, in the area of driving video generation, additional experiments are usually required, such as scene controllability experiments (measured by mAP and mIoU) and data augmentation experiments (use synthetic data to train a perception model, measured by mAP and NDS), to evaluate the quality of the generated video data.
-
Writing needs improvement, especially the Related Work section. Many state-of-the-art methods are not discussed in this section.
Questions for authors
- What is technically new in the proposed method? What is the key contribution of this work to the community?
- Please address the critical weaknesses.
4 comments
We welcome any further discussion or feedback from the reviewer Lfxs.
Dear Reviewer Lfxs:
Hoping this message finds you well. Your comments and the provided insights are very valuable to our work. We will attach these analysis and additional experiments to our final version. As the rebuttal-discussion period is nearing its end, could you please review our response to see if it addresses your concerns? Your timely feedback will be extremely valuable to us. Could you read and let us know if there are more questions? We would be very grateful! Your decision is of utmost importance to us, and we earnestly hope that you will consider the contribution of this research to the field of controllable multi-view driving video generation. Thank you very much !
Best,
The Authors
Thanks for the response. I appreciate the authors' effort. I encourage the authors to carefully revise the manuscript to address the critical weaknesses pointed out by the reviewers.
-
Novelty: My concern is that the proposed alignment method is primarily an extension of existing techniques from general tasks to autonomous driving tasks (with multiple cameras).
-
Performance: State-of-the-art methods have achieved significantly better performance on the nuScenes dataset, and the authors' response does not address this concern. For example, MaskGWM reports an FVD of 65.4 (see Table 1 of the MaskGWM paper), UniMLVG reports an FVD of 36.1 (see Table 1 of the UniMLVG paper), MoVieDrive reports an FVD of 46.8 (see Table 1 of the MoVieDrive paper), while the proposed method only achieves an FVD of 98.75.
-
Scene controllability: I encourage the authors to report scene controllability results on nuScenes, which can be directly compared with state-of-the-art methods.
-
Data augmentation: The improvement achieved by adding the synthetic data generated by the proposed method is marginal (only +0.13). This raises concerns about the practical value of the proposed approach.
Response for additional concerns
We thank the reviewer for the prompt follow-up. We appreciate the clear guidance on the remaining concerns. We address each point below, focusing on methodological novelty, comparisons on nuScenes, scene controllability, and the practical value of synthetic data augmentation.
Novelty. Thank you for clarifying this concern. We respectfully believe that V-REG’s contribution goes beyond applying an existing alignment loss to an autonomous driving dataset. V-REG couples depth- and semantic-based conditioning with supervision of intermediate DiT features, using the same timestep and camera view for both. This correspondence matters in synchronized driving video: cameras at the same instant observe different objects, road regions, and occlusions, so an alignment target from another view would not describe the content being generated.
To the best of our knowledge, V-REG is the first to couple view-specific control signals and view-specific representation alignment in this way for synchronized multi-view driving video generation. The resulting design achieves strong generation quality while substantially reducing training time, as discussed in our response to W3 for Reviewer fary. We would welcome a reference to any earlier work using this same coupling and would clarify the distinction in the manuscript accordingly.
Performance
Thanks. We fully understand the concern that strong performance on a large-scale benchmark such as nuScenes is essential for assessing the effectiveness of V-REG. The original 98.75 FVD was measured at a 30K-step checkpoint before convergence. After continuing training on 6 H100 GPUs until convergence (50K steps), V-REG achieves 7.57 FID and 43.38 FVD. We have updated this result in Tab 1&2&3.
The following table summarizes the principal differences between the compared methods:
| Configuration | MaskGWM-mview | UniMLVG | MoVieDrive | V-REG (ours) |
|---|---|---|---|---|
| Multiple training datasets | ✓ | ✓ | × | × |
| Multiple training stages | ✓ | ✓ | × | × |
| Training GPUs | 32 × A800 | N/R | N/R | 6 × H100 |
| Generation conditions | R/A, | R/T/C/B/M, | R/T/C/M/O, | R/C/Traj. |
| Multiple output modalities | × | × | ✓ | ✓ |
| FID ↓ | 8.90 | 5.80 | — | 7.57 |
| FVD ↓ | 65.40 | 36.10 | 46.80 | 43.38 |
R: reference frames; A: action controls; T: text; C: camera parameters; B: 3D boxes; M: HD maps, road maps, or object-box maps; O: 3D occupancy; Traj.: ego trajectories; N/R: not reported.
MaskGWM uses OpenDV-2K pretraining and three training stages, while UniMLVG uses multiple datasets (OpenDV-YouTube,
nuScenes, Waymo, Argoverse 2) and stages; its ablation shows that additional datasets and final-stage training substantially reduce FVD. The methods also differ in generation conditions and evaluation protocols. Consequently, their published FVD values provide useful context but are not a fair comparison under matched data, conditioning, compute, and evaluation settings. We detail these differences in Appendix C (Tab.5). Crucially, V-REG's converged 43.38 FVD directly addresses the reviewer's concern about the earlier 30K-step result.
Scene controllability. We thank the reviewer for this constructive suggestion. We have evaluated scene controllability on the nuScenes validation set using a pretrained BEVFormer. Following established practice, we report mAP for 3D object detection and mIoU for bird’s-eye-view (BEV) segmentation.
| Method | mAP ↑ | mIoU ↑ |
|---|---|---|
| MagicDrive | 9.70 | 15.60 |
| MagicDrive-V2 | 11.50 | 17.40 |
| MoVieDrive | 22.70 | 35.80 |
| V-REG (ours) | 22.74 | 35.91 |
V-REG achieves 22.74 mAP and 35.91 mIoU, exceeding MoVieDrive’s reported 22.70 mAP and 35.80 mIoU on both metrics. These results provide the requested nuScenes controllability comparison. We have attached the results in Appendix D(Tab. 6). |
Data augmentation.
We thank the reviewer for raising the question of practical value. We respectfully disagree that a +0.13%t gain, by itself, establishes that the benefit of synthetic data is marginal. The gain must be interpreted in the context of the downstream model’s training schedule and the amount of synthetic data added. The +0.13% gain on PandaSet comes from a preliminary experiment using only 1,000 generated samples, rather than an optimized augmentation study. Its size should be interpreted in light of the downstream training schedule: MagicDrive (Appendix F, Tab.6) reports gains as small as +0.25 mAP and +0.03 NDS under its longer 2× schedule. We have clarified the scope of our result in Appendix E(Tab. 7).
We hope that these clarifications will further help the reviewer to solve the remaining concerns. To facilitate a direct assessment of the generated results, we also provide an anonymous project page (https://anonymous.4open.science/w/VREG-4C74/).
We thank the reviewer for the thoughtful feedback. We address the remaining concerns bellow.
Good question! V-REG adapts representation alignment to controllable multi-view driving video generation. It uses the same timestep–camera correspondence to inject depth and semantic conditions and to supervise intermediate DiT features. This ties the training target for each view to the geometry and semantics used to condition it.
On PandaSet, we compare multi-view adaptations of the cited methods using the same backbone, initialization, input conditions, training protocol, and KV-cache setting.
Relative to VideoREPA, the strongest comparator on FID and FVD, V-REG reduces FVD by 21.2% and recorded training time by 68.0%. With the auxiliary branches fixed and caching disabled, enabling alignment also improves FID from 37.17 to 29.57 and FVD from 140.71 to 129.79 (Table 4, ID 4 → ID 6).
Thanks. MaskGWM’s public release provides partial inference code, and its repository issue discussion explains that the original OpenSora training pipeline is unavailable. MoVieDrive’s code was unavailable to us. We therefore focus our additional controlled comparison on UniMLVG, using PandaSet due to GPU memory constraints.
Under the matched PandaSet evaluation, V-REG achieves lower FID (27.56 vs. 30.46) and lower FVD (89.79 vs. 113.84) with 1.6 hours of recorded training time compared with 12.0 hours for UniMLVG.
Thanks. We believe there exists a misunderstanding about the multimodal DiT paradigm. Our Fig. 1 illustrates multimodal conditioning at the architecture level. MoVieDrive [7] falls within this broader category and illustrates block-level conditioning inside a unified DiT, adding 3D occupancy embeddings to denoising features (Fig. 3 and Eq. 4 in [7]). These two paradigms share the same sprits.
Thanks. The 30K steps in Appendix Fig. 9 specify the optimization budget. We supplement them with reported wall-clock time and GPU-hours.
These reuslts indicate that our approach can achive better results with lower resources.
Thanks. We conduct two additional experiments on PandaSet.
Scene controllability. We evaluate how well the generated videos follow the requested scene layout using detection mAP and segmentation mIoU.
V-REG improves mAP by 6.67 points and mIoU by 11.70 points over the baseline, indicating closer agreement between the generated videos and the scene controls.
Data augmentation. We train a perception model using real data alone or real data augmented with 1,000 V-REG-generated samples. Both models are evaluated on the same real PandaSet validation set. We report detection mAP under the PandaSet protocol.
Adding synthetic samples yields a modest gain of 0.13 mAP points on real validation data. We will include both experiments in the revision and describe their evaluation protocols.
Thanks. As detailed in W1, V-REG couples view-specific depth and semantic conditioning with view-specific representation alignment while preserving the main denoising path. This gives practitioners a way to integrate pretrained visual priors into synchronized multi-camera generation, with measured gains in training efficiency and generation quality on PandaSet.