acceptodds
Under review as a conference paper at ICLR 2027

DrivENVS: Generative Novel View Synthesis from Monocular Driving Videos

13 views

Abstract

Generative novel view synthesis (NVS) methods for dynamic scenes typically reconstruct a full 4D representation of the scene and then generate refined video outputs using a video diffusion model (VDM), conditioned on the rendered views of the 4D structure. However, conventional VDMs require extensive training, exhibit significant memory inefficiency, and remain constrained to generating short video sequences. We present DrivENVS, an efficient alternative to VDM-based enhancement that employs a few-step image-diffusion model to sequentially refine coarse rendered views into a realistic and temporally coherent video. A key architectural novelty is an efficient conditioning mechanism that involves leveraging several reference frames selected from the source video using a reconstruction-aware strategy. Incorporating relative camera pose and capture time-step information for reference frames also provides useful spatio-temporal context. Across nuScenes, Waymo, and Argoverse 2, DrivENVS outperforms baseline methods on video quality metrics in challenging NVS settings, while also providing memory-efficient inference and enabling the generation of extended video sequences.

Primary area:
generative models
Keywords:
novel view synthesis, video generation, few-step diffusion models, 4D view synthesis for driving scenes

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s5TROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DrivENVS, a framework for novel-view synthesis from calibrated monocular driving videos. It reconstructs a motion-aware point cloud using estimated depth and object tracks, then renders coarse views along a target camera trajectory. A four-step image diffusion enhancer refines these renderings autoregressively using previously generated frames and geometrically selected source references. The highest-ranked reference is processed through joint attention, while additional references are incorporated through cross-attention with relative camera and time conditioning. Experiments on nuScenes, Waymo, and Argoverse 2 show improvements over image-based baselines in video quality and several reconstruction metrics, with lower reported inference memory than NeoVerse.

Critical strengths
  • The proposed method is simple and efficient, the experiments demonstrate its effectiveness compared to baseline methods.
  • The paper is well-written and easy to understand.
Critical weaknesses
  • The paper emphasizes its advantages over video-based enhancement, including in the abstract, but the comparison with video models is not sufficiently controlled. The comparison uses a 2B image model and a 14B NeoVerse model with different resolutions and training settings, while Table 11 shows lower memory usage but also lower throughput. I would recommend additional comparisons with relevant video-based methods such as ReCamDriving and ReCamMaster, including smaller models, reporting generation quality, inference time, and peak GPU memory under comparable settings.
  • The method relies on estimated depth, object tracking, and a fixed motion threshold for both coarse rendering and reference selection, but its robustness to errors in these components is not sufficiently evaluated. I would recommend that the authors provide controlled ablations and moving-object evaluations to clarify how reconstruction errors affect object appearance, position, and motion in the generated videos.
Questions for authors
  • Could the authors provide a comparison with a smaller video-based NVS model at matched resolution and sequence length, reporting quality, throughput, and peak GPU memory?
  • How sensitive is the method to depth errors, tracking errors, and the motion threshold, particularly for moving vehicles and pedestrians?
4 comments
AuthorsOpenReview ↗

Response to Reviewer s5TR (1-2/3)

C1. Request to compare with a smaller video model at matched resolution and sequence length.

Please see our response to a similar question to Reviewer QiWQ C2 and C4 and the new results we have reported in Tables A and B above. We re-did the baseline comparison based on your feedback and added two new small-video models (StreetCrafter, ReCamMaster -- as the reviewer suggested) and compared to NeoVerse at matched resolution and sequence length.

ReCamDriving. We tested ReCamDriving using GaussianStorm for feed-forward driving reconstruction. We found that the model can only generate videos at eight predefined lateral camera shifts: 1, 2, 3, and 4 m to left and right. The model does not support vertical camera shift or camera rotation. As the released models/code does not generalize to arbitrary camera shifts, we excluded it in the comparison.

C2. Sensitivity to depth errors, motion threshold, etc.

Please see our response to Reviewer QiWQ's C5 question about depth estimation sensitivity, etc. We have reported the new results in Tables C and D above.

AuthorsOpenReview ↗

Response to Reviewer s5TR (3/3)

C3. Robustness to reference selection needs more evaluation

The result of ablating the reference selection is reported in Tables 8–9 in the main paper. We also ablated the number of reference frames. To summarize, increasing the reference count from one to eight reduces FVD from 173.20 to 150.52. Our pose-aware strategy achieves FVD = 150.52, which outperforms a simpler selection strategy, which has FVD = 170.64.

AuthorsOpenReview ↗

Follow-up response to Reviewer s5TR

Dear Reviewer s5TR,

Thank you again for your helpful suggestions.

We have added comparisons with smaller video models at matched resolution and sequence length (Table 5), together with motion-threshold and alternative depth/segmentation/tracking evaluations (Tables 7 and 15).

Before the discussion period ends, we would appreciate hearing your assessment of these additions and whether any specific concerns remain. We would be happy to discuss them further.

The Authors

Hello, we are approaching the discussion deadline. Can you please provide your thoughts and comments?

Reviewer_kco7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DrivENVS, an image diffusion-based method for novel-view synthesis from monocular driving videos. By conditioning the image diffusion model on 4D geometry and selected reference frames, the method autoregressively generates temporally consistent novel-view videos. Experiments show improvements over Difix3D and DiffusionHarmonizer across different viewpoint changes and datasets.

Critical strengths
  • The paper is well structured, and the method is well motivated and easy to understand. The extensive experiments demonstrate improvements over other image diffusion baselines on autonomous driving scenes.
  • Incorporating 4D geometry and reference-frame conditioning into diffusion models is a useful approach to improving geometric and temporal consistency in novel-view synthesis, as demonstrated by prior work. Incorporating them into an image diffusion model to enable autoregressive video generation is particularly interesting.
Critical weaknesses
  • The comparison with video diffusion uses pretrained NeoVerse on nuScenes. This is not fully fair, since NeoVerse’s pretraining data do not include nuScenes. NeoVerse should therefore be finetuned on nuScenes, as was done for Difix3D and DiffusionHarmonizer. Alternatively, the comparison could be conducted on Waymo, which is included in NeoVerse’s pretraining data. That said, a stronger ablation to isolate the advantage of image diffusion over video diffusion would be to finetune a Cosmos-2B video diffusion model with the same geometric multi-reference conditioning.
  • The supplementary videos show some recurring failure modes. For example, in several lateral-shift examples, half of the car ahead is missing, or trees appear inconsistent across frames. Since the paper does not include a limitations section, could the authors explain the possible causes of these failures?
Questions for authors

Please see the Critical Weaknesses section above. Additionally:

  • Since reference frames are selected based on co-visibility, do they substantially overlap with the previous frames used for temporal conditioning?
  • The paper claims that the model can generate arbitrarily long sequences. Does it suffer from the drift commonly seen in autoregressive generation? If so, what is the practical limit on generation length, and how does it compare with autoregressive video diffusion models?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kco7 (1-2/4)

C1. NeoVerse training mismatch and comparing with smaller video-model

Thank you for the feedback. Following your suggestion, we re-evaluated NeoVerse on Waymo. Please see our respones to Reviewer QiWQ C2 and C4 with Table A and B above.

We re-evaluated NeoVerse on Waymo. Unfortunately we could not finetune NeoVerse due to prohibitive compute requirements. We have added two small video diffusion baselines (StreetCrafter, ReCamMaster) as requested.

C2. Missing car regions, unstable trees, and discussing limitations in the paper.

Missing car regions. This happens due to incomplete 4D reconstructions. When the moving car is not fully visible in the source video frame, the complete geometry is absent in the 4D point cloud, producing holes in the coarse target views rendered from the shifted camera trajectory (see Appendix A.3). The enhancer can complete small holes, but large missing regions can be problematic for our method. We feel that a more sophisticated 4D reconstruction technique could be used to address this limitation but we feel it is beyond the scope of this work.

Unstable trees, poles, and fine structures. These regions can be sensitive to depth errors, changing visibility, and variation between generated frames. Our second training stage addresses temporal consistency through temporal-context conditioning and a flow-based consistency objective. Our ablation study in the paper provides quantitative evidence of an overall temporal benefit (in paper Table 6, rows 1 and 7). These results support improved overall temporal quality, while residual flicker in trees, poles, and other fine structures remains a limitation.

For qualitative comparisons, please see Figs. 6–17 in the paper and our supplementary video from 2:53 onward, which present the original views, 4D reconstruction renderings, and DrivENVS outputs side-by-side. These comparisons show the reconstruction artifacts present in the coarse rendered views and the corresponding refined frames. We acknowledge these limitations and we will discuss them in the revised final paper.

AuthorsOpenReview ↗

Response to Reviewer kco7 (3-4/4)

C3. Since reference frames are selected based on co-visibility, do they substantially overlap with the previous frames used for temporal conditioning?

At inference time, the amount of spatio-temporal overlap between the previous output frames and the "past" reference frames depends on the amount of camera shift, or the pose difference between the source and target camera trajectories. Scene depth also plays a role. With viewpoint change and fairly low scene depth, large parallax will be present between the reference frame and the past frame even when there is overlap.

During stage 2 of our training procedure, history is teacher-forced using GT clean frames (Appendix A.1), so the same source frame appears in both reference-conditioning and temporal conditioning. This is because we train on monocular videos and rely on double reprojection.

C4. Does it suffer from the drift commonly seen in autoregressive generation? If so, what is the practical limit on generation length, and how does it compare with autoregressive video diffusion models?

DrivENVS first performs a full 4D scene reconstruction from the source video, then renders coarse frames, and refines those frames frame-by-frame to generate the output video. Thus, if the 4D reconstruction is drift-free and accurate, the refiner would not introduce any additional drift when the target trajectory stays fairly close to the reconstructed scene. Arbitrarily long videos can be generated by our method, if a scalable 4D reconstruction technique is available, and the target cameras do not deviate too much from the source cameras. We currently use DepthAnything-3 which handles moderate length sequences, but more scalable techniques are needed for longer source videos.

Autoregressive video diffusion models (AR-VDMs) that condition on a single start frame, or a few input images, can generate an unlimited number of frames but rely on the generative model to create plausible scene content not observed in the input images. To reduce drift and produce 4D consistency, such models must be trained on vast datasets. In contrast, our method aims to strongly condition on the 4D reconstructon derived from the source video, and thus needs relatively less training data.

However, AR-VDMs conditioned on 4D reconstructions of long source videos, are conceptually similar to our method DriveENVS. They also generate frames sequentially using previous frames, but potentially in larger batches of frames in each inference step. For DrivENVS, that batch size is essentially one frame. Thus, it is possible that in certain cases the longer spatio-temporal memory/context learned by AR-VDMs may produce more complete results compared to DrivENVS. But, the larger model is typically very resource intensive and data intensive to train, with two stages of training involving self-forcing, etc.

Thank you for the additional comparisons with video diffusion models and the clarifications, including the visual limitation caused by imperfect 4D reconstruction. These responses address most of my concerns. I will maintain my original weak-accept rating.

Thank you for reviewing the additional comparisons and clarifications, and for your continued positive assessment.

Your feedback helped us strengthen the evaluation and clarify the limitations arising from imperfect 4D reconstruction. If there is any remaining point you would like us to clarify, we would be happy to address it.

Best Regards,

The Authors

Reviewer_QiWQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an autoregressive image-diffusion framework for generative novel view synthesis from monocular driving videos. It first reconstructs a motion-aware 4D point cloud to render coarse target views, then refines them using a few-step rectified-flow enhancer. The enhancer introduces geometry-aware multi-reference selection, efficient cross-attention, and relative camera-time conditioning to improve appearance transfer and temporal consistency. Experiments on nuScenes, Waymo, and Argoverse 2 demonstrate improved video-level quality over image-diffusion baselines, with lower inference memory than a video-diffusion baseline.

Critical strengths
  1. The paper addresses the high computational cost and limited sequence length of video-diffusion-based novel view synthesis, which are realistic limitations of driving-scene simulation.

  2. The evaluation covers three driving datasets and several camera transformations, including lateral shifts, stereo-view synthesis, and side-view synthesis. The ablation studies also investigate the individual contributions of the proposed components.

Critical weaknesses
  1. The framework builds substantially on existing image-diffusion enhancement and autoregressive generation methods, particularly Difix3D and Harmonizer. The main additions are geometry-aware multi-reference conditioning, relative camera-time embeddings, and few-step rectified-flow refinement. Although these components are reasonable, the paper does not sufficiently establish that their combination constitutes a major methodological advance beyond integrating existing techniques. The authors should more clearly distinguish the proposed contributions from prior reference-conditioning, camera-conditioning, and autoregressive diffusion approaches.

  2. The main comparisons focus on Difix3D and Harmonizer, with NeoVerse evaluated separately. The paper would benefit from comparisons with a broader range of recent camera-controlled video-generation and dynamic novel-view-synthesis methods. In particular, the authors should clarify which methods can support arbitrary target trajectories and compare them under equivalent input information, output resolution, and evaluation protocols. This would help establish the actual performance and efficiency advantages of DrivENVS.

Questions for authors
  1. Given the reported throughput of 0.717 FPS, how does the method compare with NeoVerse in total runtime and computational cost when generating sequences of different lengths? What practical benefits does autoregressive generation offer other than reduced peak memory?

  2. How sensitive is the framework to errors in depth estimation, object tracking, and static/dynamic classification? Would the proposed conditioning mechanism remain effective when the reconstructed point cloud contains substantial artifacts?

  3. Does the model generalize to camera trajectories with larger translations or rotations than those evaluated, and to scenes with limited overlap between the source and target views?

10 comments
AuthorsOpenReview ↗

Response to Reviewer QiWQ (1/6)

C1. Distinguish the proposed contributions from prior reference-conditioning, camera-conditioning, and autoregressive diffusion approaches.

While prior works propose reference-conditioning, camera-conditioning etc. for generative NVS, almost all of them have an underlying video diffusion model (VDM). These VDMs can only generate a fixed, pre-determined number of video frames (eg. 81). To create longer videos, the VDMs can output multiple batches independently, but 4D consistency is absent across different batches. Thus, existing VDM-based methods cannot efficiently generate 4D reconstruction grounded videos of unlimited length. In contrast, existing autoregressive methods can generate very long videos but are not designed for challenging 4D driving NVS tasks. Sensor2Sensor [Wang et al. 2026] does address that task, but uses a different strategy (very data intensive), where no 4D reconstruction is used, nor is there any geometric grounding like ours.

We demonstrate that a compact image diffusion model can be used for frame-by-frame video generation to generate arbitrarily long videos, without sacrificing the benefits of geometric-conditioning and 4D reconstruction-grounding that is crucial for generative NVS. Even though our method combines ideas from prior works, we believe that the context in which we have combined them is crucial to judging the novelty.

We are inspired by Difix3D and Harmonizer, but Difix3D neither generates coherent videos nor handles dynamic scenes. Harmonizer neither focuses on NVS nor incorporates reference or camera-conditioning. We go beyond Difix and Harmonizer as follows.

  • Difix has single reference frame-conditioning but their approach is too memory intensive for multiple frames. In contrast, we present a novel and efficient approach for multiple reference frame conditioning, that combines self and cross attention. Furthermore, our reference frame selection method (derived from camera pose, geometry) is also novel.
  • We adopt Harmonizer's temporal-context training idea for smooth video generation, but that alone is insufficient for NVS with camera shifts. Relative camera-time conditioning, originally proposed for VDMs, is something we bring into image diffusion, thereby improving NVS performance while maintaining efficiency.

In our main paper, Tables 6 and 9 validates our design choice.

ConfigurationFVDsee main paper
Full DrivENVS150.52Table 6, row 7
Without the joint top-1 reference path189.46Table 6, row 3
Without relative camera and time conditioning167.49Table 6, row 4
Without pose-aware reference frame selection170.64Table 9

We believe that our specific combination of ideas (autoregressive generation, reference-conditioning, reference selection, relative camera and time-conditioning, etc.) is non-trivial and is crucial for achieving accurate and efficient image diffusion-based long video generation without sacrificing 4D-geometric grounding and conditioning.

AuthorsOpenReview ↗

Response to Reviewer QiWQ (2/6)

C2. How does the method compare with NeoVerse in total runtime and computational cost when generating sequences of different lengths?

We re-evaluated NeoVerse and DrivENVS (ours) on Waymo (2 m. shift) at the same resolution (560 ×\times 336), and report a fair comparison below. NeoVerse could be run only in two fixed settings that generate 25 and 81 frames respectively (it cannot output more than 81 frames). In contrast, our method can generate any sequence length, given a sufficient number of coarse rendered target frames.

Table A: DrivENVS vs NeoVerse on different sequence length (Table 14 in the revised paper)

MethodResolutionSequence LengthTotal Runtime (s)Peak GPU Memory (GB)
NeoVerse560 ×\times 336254.355.9
DrivENVS560 ×\times 336256.88.1
NeoVerse560 ×\times 3368112.288.2
DrivENVS560 ×\times 3368122.18.1
NeoVerse560 ×\times 336200N/Anot possible
DrivENVS560 ×\times 33620054.68.1
DrivENVS1024 ×\times 576200279.519.3

Given a 4D reconstruction, DrivENVS refines successive target frames using fixed-size reference and fixed number of past generated frames. Increasing the output length thus does not increase the number of frames jointly processed, allowing continuous framewise generation at fixed memory usage.

Furthermore, DrivENVS can generate hi-res 1024×576 output using only slightly higher memory (19.4 GB) for any sequence length, whereas NeoVerse cannot generate frames at that resolution.

AuthorsOpenReview ↗

Response to Reviewer QiWQ (3-4/6)

C3. What practical benefits does autoregressive generation offer other than reduced peak memory?

It enables generating the video frame-by-frame, albeit using a small and fixed number of past output frames for temporal conditioning. The memory footprint is fixed, does not grow with sequence length. Thus, the output video can be arbitrarily long, based on how many coarse rendered frames need to be refined.

C4. Consistent comparison with VDM baselines

We re-evaluated DrivENVS and VDM baselines more uniformly as requested. The output resolution is 560 ×\times 336. All methods use the same held-out Waymo scenes, 81 frame sequences, and target camera trajectories (2 m left, right, and up shifts). Camera coordinates and scale are converted consistently for each implementation. FID and FVD use the same real-reference image pool, feature extraction, preprocessing, etc.

We added two VDM baselines (StreetCrafter, ReCamMaster) with smaller backbones (the released ReCamMaster version uses Wan2.1-T2V-1.3B). Our method uses an image diffusion backbone (IDM) unlike the VDM baselines.

Table B: Comparison with VDM baselines. (Table 5 in the revised paper)

ModelBackboneTypeFIDFVDFPS (s)Peak GPU memory (GB)
NeoVerseWan-T2V-14BVDM27.54249.466.7588.2
StreetCrafterSVD-Vista-1.5BVDM28.03274.2614.2119.6
ReCamMasterWan-T2V-1.3BVDM231.021332.0926.3416.8
OursCosmos-T2I-2BIDM12.49179.453.668.1

Our method also outperforms ReCamMaster and StreetCrafter.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all three reviewers for their thoughtful and constructive feedback and for recognizing DrivENVS’s practical relevance, simplicity, and effectiveness. We have expanded the experimental evaluation and clarified the technical contributions, trade-offs, and limitations. The revised manuscript adds five experimental tables (Tables 5, 7, 14, 15 and 16) covering additional video diffusion baselines, runtime and memory across sequence lengths, sensitivity to reconstruction components, and larger camera translations. The new results and accompanying discussions are highlighted in blue. We summarize the main additions and clarifications below:

  • Expanded video-diffusion baseline comparisons. We evaluate NeoVerse, StreetCrafter, and ReCamMaster on held-out Waymo scenes at matched resolution and sequence length, reporting quality, throughput, and peak GPU memory (Table 5). We additionally report runtime and memory across different generation lengths (Table 14).
  • Reconstruction sensitivity studies. We evaluate the motion-classification threshold and substitute alternative depth-estimation and segmentation/tracking models (Tables 7 and 15). These experiments measure robustness of the preprocessing steps.
  • Larger viewpoint changes. We extend the Waymo evaluation to 3 m and 4 m translations using the same frozen checkpoint without translation-specific fine-tuning, and report the quality drop at larger offsets (Table 16).
  • Methodological clarifications and limitations. We explain the differences from Difix3D and Harmonizer, emphasizing geometric reference selection, asymmetric multi-reference fusion, and relative camera-time conditioning. We also discuss limitation and artifacts caused by incomplete reconstructions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.