acceptodds
Under review as a conference paper at ICLR 2027

FreeView: Multi-View Driving Video Generation with Spatiotemporal Memory

Abstract

Robust autonomous driving requires diverse multi-view observations of common traffic scenarios and long-tail events. Dashcam and online monocular driving videos provide a rich source of these scenarios. Their limited view coverage makes them difficult to use directly in autonomous driving systems that require synchronized multi-camera observations. Expanding these videos into coherent multi-view sequences requires preserving the reference scene across viewpoints and integrating observations from different times. We propose FreeView, a multi-view driving video generation framework with spatiotemporal memory. Our reference-as-view design incorporates the reference video as an additional conditioning view, using shared view-wise self-attention for memory encoding and cross-view attention for memory interaction. This design leverages the strong representation and interaction capabilities of the pretrained generation model. We further retrieve relevant reference observations across time and use relative temporal and spatial encodings to guide attention matching. Experiments demonstrate the effectiveness of our method in preserving visual fidelity and consistency with the reference scene. We also demonstrate generation under flexible camera configurations and applications to real-world dashcam videos.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QYgzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FreeView generates synchronized multi-view driving videos from a monocular reference video and camera parameters. It builds on an internal pretrained multi-view video diffusion model and introduces a reference-as-view design: clean reference tokens are processed using shared per-view self-attention, while target tokens access reference features through cross-view attention. A geometry-based retrieval mechanism selects historical reference observations, supplemented by relative temporal and spatial encodings. The model also incorporates camera-layout conditioning. Experiments on nuScenes and an internal driving dataset demonstrate improvements over several reconstruction and novel-view generation pipelines.

Critical strengths
  1. Expanding monocular driving videos into consistent surround-view observations is practically useful. Jointly generating target views and reusing pretrained attention for reference conditioning is a reasonable design that avoids introducing a separate reference encoder.
Critical weaknesses
  1. The comparison does not sufficiently establish an advantage over existing joint multi-view generators. Much of the evaluation concerns reconstruction or geometry-assisted novel-view pipelines, including Street Gaussians, GEN3C, NeoVerse, and Vista4D. AnyView and OpenLongTail are relevant references, but generate target views separately or through a cascade. The joint-generation baseline, Cosmos-Drive-Dreams, uses a default layout without target-pose control and is explicitly included for reference only.

    Recent work provides closer precedents. Sensor2Sensor(The author mentioned it in the paper and think it is a corcurrent work, but it is CVPR2026) converts monocular observations into multi-camera outputs; CVD-STORM provides a joint multi-view driving-video architecture; and Cosmos-Transfer2.5 has camera-controlled multi-view model which explicitly generates multiple target videos from a source video and target camera trajectories, albeit in robotics. These do not all constitute interchangeable driving benchmarks, but they make a controlled comparison with an existing joint-generation architecture necessary. A particularly important baseline is the same backbone trained to keep the complete reference view clean and denoise the missing views, without FreeView’s specialized memory design. I suggest the authors should select more multi-view video generation model for comparsion. Most of them have involved similar tasks during training.

  2. The internal backbone obscures the source of the improvements and limits reproducibility. The method inherits its main generation architecture from an unnamed pretrained model. Its pretraining data, training recipe, and original conditioning capabilities are insufficiently documented. It is also unclear whether competing methods receive comparable driving-domain fine-tuning. Consequently, the reported gains cannot be confidently attributed to the proposed memory modules rather than the backbone or training data. The exact temporal attention structure and the construction of the adjacency graph for new camera layouts also require clarification.

  3. Ablations use only four evaluation scenes at reduced resolution, with 1,000–2,000 training steps and no reported variability. In Table 4, adding the complete retrieval/encoding design raises PSNR from 12.69 to 12.94, but reduces SSIM from 0.340 to 0.337 and only slightly improves LPIPS from 0.594 to 0.592. Retrieval alone worsens PSNR and SSIM. In Table 7, shared reference self-attention improves PSNR by only 0.02 over independent reference self-attention. These results provide limited evidence for robust component-level gains, and the ablations do not directly measure the claimed improvements in cross-view consistency.

  4. Flexible-view generation is insufficiently validated. Quantitative experiments primarily evaluate standard dataset camera rigs. Figure 1 offers qualitative cross-layout examples, but does not establish reliable generation at arbitrary unseen camera positions or orientations. Camera-pose conditioning alone is insufficient evidence of continuous viewpoint generalization. Evaluation at held-out intermediate angles—for example, between front-right and back-right—would better distinguish geometric controllability from adaptation to familiar camera layouts.

  5. No video result is provided.

Questions for authors

Please refer to the weakness section and particularly answer the following questions:

  1. What is the internal pretrained model, what data was it pretrained on, and which capabilities already existed before FreeView? Were the evaluated baselines fine-tuned on the same driving data?

  2. Can you compare against standard masked reference-view conditioning on the same backbone, using identical training data and compute? How much improvement remains after controlling for joint target generation and backbone strength?

  3. Can the model accurately generate intermediate camera orientations and unseen translations or intrinsics? How is the adjacency graph defined for these layouts, and can pose adherence be measured quantitatively?

1 comment
AuthorsOpenReview ↗

Response to Reviewer QYgz

Thank you for your detailed feedback. We address your comments below.

W1, W3 & Q2 · Same-backbone reference conditioning and expanded ablations

Thank you for this suggestion. Our no-retrieval variant provides a same-backbone joint-generation baseline: it retains reference-as-view conditioning and jointly generates the target views without retrieved memory.

We expanded the ablation and evaluation sets from 20 to 40 clips. For a fair comparison, both variants use the same backbone and evaluation settings, with the same reduced batch size in these additional ablations in 480p. Adding our retrieval method improves overall performance on both sets.

20 clips

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
AnyView12.140.4260.61971.64617.85
Ours (multi-view, w/o retrieval)12.670.43240.608153.60403.55
Ours (full)12.980.43730.599153.42397.32

40 clips

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
AnyView12.280.4410.62258.97361.27
Ours (multi-view, w/o retrieval)13.010.44340.606038.87265.48
Ours (full)13.530.45520.594739.57237.61

W2 & Q1 · Backbone pretraining and task-specific training of baselines

We use an internally trained model for its multi-view generation and camera-pose control capabilities. It was initialized from Cosmos-Predict2.5-2B and trained on our internal seven-camera driving data, without public datasets at this stage. All evaluation scenes and clips were excluded from both this stage and subsequent FreeView training.

The baselines also received task-specific training. They use different backbones and datasets. For example, AnyView [4] uses Cosmos-Predict2-2B-Video2World and trains on datasets including Waymo and Lyft, while OpenLongTail fine-tunes Wan2.1-VACE-1.3B on datasets including PandaSet and nuScenes.

W3 · Reference self-attention weight sharing in Table 7

Table 7 examines whether reference self-attention requires separate weights. Both variants use memory, and their similar performance supports weight sharing. The effectiveness of our memory design is demonstrated by the same-backbone comparisons above.

W4 & Q3 · Unseen-layout generation and joint camera-pose evaluation

We extended the comparison with AnyView [4] to the unseen Waymo target layout using the same evaluation setup. FreeView achieves higher matching success rates on both reference-to-generated (RG) and generated-to-generated (GG) pairs, improving overall GMSR@10 by 13.39 percentage points.

MethodRG ↑GG ↑Overall ↑
AnyView47.93%49.94%49.27%
Ours73.78%57.10%62.66%

FreeView also achieves lower mean rotation and translation errors with both pose estimators, Pi3X [2] and VGGT-Omega [3]. Following Vista4D [1], we use reconstructed poses to evaluate camera control. We jointly input the real front-view video and the generated videos, then compare their estimated relative poses with the requested poses, without aligning each view separately.

EstimatorMethodMean rotation error (°) ↓Mean translation error (m) ↓
Pi3XAnyView26.8212.55
Pi3XOurs19.612.74
VGGT-OmegaAnyView26.388.48
VGGT-OmegaOurs23.244.27

W2 & Q3 · Temporal attention and camera adjacency

Temporal attention. In each DiT block, the reference and target views use the same self-attention weights. Within each view, tokens attend across space and time.

Camera adjacency. Following MagicDrive [5] and MagicDrive-V2 [6], we connect neighboring target views according to their left-to-right arrangement around the vehicle. We use the same rule for new camera layouts.

W5 · Video results

We have uploaded video results in the supplementary material.

[1] Vista4D: Video Reshooting with 4D Point Clouds, CVPR 2026.

[2] π³: Permutation-Equivariant Visual Geometry Learning, ICLR 2026.

[3] VGGT-Ω, CVPR 2026.

[4] AnyView: Synthesizing Any Novel View in Dynamic Scenes, ECCV 2026.

[5] MagicDrive: Street View Generation with Diverse 3D Geometry Control, ICLR 2024.

[6] MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control, ICCV 2025.

Reviewer_Y2viOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FreeView, a framework for generating synchronized multi-view driving videos from a monocular reference video. The method introduces a reference-as-view design that incorporates the reference video as an additional conditioning view, reusing pretrained view-wise self-attention and cross-view attention for memory encoding and interaction. It further retrieves historical reference frames based on geometric coverage and incorporates relative temporal and spatial encodings to improve cross-time information utilization. Experiments on nuScenes and an internal driving dataset demonstrate improvements in visual quality and multi-view consistency over existing methods.

Critical strengths
  1. The paper addresses a meaningful and practical problem. Generating synchronized multi-view driving videos from monocular inputs could help leverage abundant real-world dashcam videos for autonomous driving applications.

  2. The proposed framework is simple and well-motivated. The reference-as-view design effectively reuses pretrained attention modules for memory encoding and interaction without requiring a separate reference encoder. The overall framework is straightforward and integrates naturally into existing multi-view video generation architectures.

  3. The experimental results are promising. The method achieves competitive results on nuScenes and an internal driving dataset, with particularly noticeable improvements in perceptual metrics such as FID and FVD. The experiments also include comparisons with several recent methods and ablations of the proposed components.

Critical weaknesses
  1. The paper lacks multi-frame comparisons, videos results and visualizations of retrieved reference frames, which are important for demonstrating temporal consistency and the effectiveness of the proposed memory mechanism.

  2. The camera layout selection in Figure 1(e) is questionable. The five-camera generated layout includes the front view, which corresponds to the input reference view, while omitting the back view. The back view is particularly challenging due to its large viewpoint difference from the reference and is highly relevant to the motivation for cross-time memory retrieval. It is unclear why this specific layout was selected and whether the front view was generated or directly reused from the input. The authors should clarify the layout selection and provide more comprehensive results on challenging unseen camera configurations.

  3. Although the paper aims to convert real-world monocular driving videos into multi-view videos, most quantitative experiments are conducted on nuScenes and an internal driving dataset. Figure 1(a) shows only one internet dashcam example. More evaluations on diverse internet videos, including challenging environments and camera configurations, are needed to demonstrate practical generalization.

  4. The robustness to inaccurate camera poses is not evaluated. The proposed memory retrieval and relative spatial encoding rely heavily on camera intrinsics and extrinsics. While Figure 1(a) uses MapAnything to estimate camera parameters for internet videos, the paper does not investigate how pose estimation errors affect retrieval accuracy or generation quality.

  5. Table 8 shows that increasing the number of retrieved reference frames from one to two reduces PSNR from 12.84 to 12.62 and worsens LPIPS from 0.589 to 0.595, despite a slight improvement in SSIM. Moreover, retrieval alone provides limited improvements in Table 4. These results raise questions about how effectively the proposed memory mechanism utilizes additional historical observations and whether it can benefit from longer temporal contexts.

Questions for authors

Please refer to the weaknesses listed above. I would be willing to raise my score if the authors can adequately address these concerns and provide convincing clarifications or additional experimental evidence.

1 comment
AuthorsOpenReview ↗

## Response to Reviewer Y2vi

Thank you for recognizing the motivation of FreeView and its integration of reference conditioning into multi-view generation.

W1 & W3: Additional visualizations and video results

We have added more visualizations to the appendix of the revised paper and uploaded video results in the supplementary material. Figure 5 in Appendix A shows how a retrieved reference helps recover crosswalk markings that are no longer visible in the current front view, with a comparison against the model without retrieval.

W2: Camera layout and the input front view

Figure 1(e) uses Waymo's five-camera layout to test a target layout not used in our training. This layout has no directly backward-facing camera. The front view is copied from the input, so the example contains one input view and four generated views. We did not train on Waymo data. Figure 1(d) uses an unseen Waymo input video to generate a six-camera layout, including a back view. Thus, (e) tests an unseen output layout, while (d) tests an unseen input dataset.

W4: Robustness to estimated camera poses

Generation quality remains close when calibrated poses are replaced with MapAnything estimates. GT denotes dataset-calibrated poses, and Estimated denotes MapAnything poses. We evaluate on the 20 nuScenes clips and an expanded set of 40 clips, generating 41-frame videos at 480p.

20 clips. Replacing calibrated poses with estimated poses decreases our full model's PSNR by only 0.041 dB, leaves SSIM unchanged, and increases LPIPS by 0.0026.

MethodPosePSNR ↑SSIM ↑LPIPS ↓
AnyViewGT11.460.43310.6355
AnyViewEstimated11.450.43130.6377
Vista4DGT11.290.34490.6648
Vista4DEstimated11.150.33200.6820
Ours w/o retrievalGT12.670.43240.6081
Ours w/o retrievalEstimated12.670.43580.6096
OursGT12.980.43730.5991
OursEstimated12.940.43730.6017

After expanding the evaluation to 40 clips, PSNR decreases by only 0.014 dB, SSIM increases by 0.0003, and LPIPS increases by 0.0015. These results show that our method is robust to the pose estimation errors in this evaluation.

MethodPosePSNR ↑SSIM ↑LPIPS ↓
AnyViewGT11.940.44480.6304
AnyViewEstimated11.920.44330.6321
Vista4DGT11.260.34840.6707
Vista4DEstimated11.130.33970.6846
Ours w/o retrievalGT13.010.44340.6060
Ours w/o retrievalEstimated13.010.44780.6073
OursGT13.530.45520.5947
OursEstimated13.520.45550.5962

W5: Retrieving two frames and longer temporal contexts

Retrieving two frames usually adds little geometric coverage over one. We compare one-frame and two-frame coverage on 40 clips. Coverage measures how much of the current target view is geometrically covered by the retrieved reference frames. Overlapping regions are counted only once, and added coverage is the extra coverage provided by the second frame.

A. One versus two reference frames. With 41-frame clips, the second frame adds only 0.0182 percentage points on average, with no added coverage in 94.5% of cases.

One-frame coverage (%)Two-frame coverage (%)Added coverage (pp)No added coverage (%)
18.741918.76010.018294.5

B. Longer temporal contexts. Keeping the targets fixed, we extend the temporal context from 41 to 101 frames. One-frame coverage increases from 30.77% to 76.78%. Even at 101 frames, the second frame adds only 0.39 percentage points on average, with no added coverage in 78.5% of cases.

Temporal context (frames)One-frame coverage (%)Two-frame coverage (%)Added coverage (pp)No added coverage (%)
4130.771930.78830.016594.0
6152.278252.41270.134588.0
8167.946868.24390.297182.0
10176.782677.16930.386778.5

These results show that retrieving from longer temporal contexts increases coverage, while a second reference often provides redundant coverage.

Reviewer_44jdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FreeView studies how to expand a monocular driving video into synchronized videos under a specified multi-camera configuration. It incorporates the reference video as an additional view in a pretrained multi-view video generator. Reference and target views share view-wise self-attention, while cross-view attention allows target views to access clean reference features. The method also retrieves reference frames from earlier timestamps based on additional field-of-view coverage and uses relative temporal and spatial embeddings to guide attention. Camera configurations and motion are represented through static and dynamic Plücker embeddings.

Experiments on nuScenes and an internal driving dataset show improvements over the evaluated baselines in perceptual quality and reference fidelity. Additional evaluations measure estimated camera motion and cross-view feature matching. Qualitative examples demonstrate generation with different camera layouts and transfer to dashcam videos.

Critical strengths
  1. The problem is practically relevant, and the architecture provides a coherent solution. Generating target views jointly avoids the error accumulation associated with cascaded view generation. Treating the reference video as an additional conditioning view also allows the model to reuse pretrained attention modules without introducing a separate reference encoder.

  2. The main comparisons show consistent improvements in perceptual quality across the two datasets and both evaluated sequence lengths on nuScenes. For 41-frame nuScenes generation, FreeView achieves an FVD of 372.63, compared with 557.79 for Vista4D, and an LPIPS of 0.596, compared with 0.619 for AnyView. The camera-motion and cross-view matching evaluations are useful additions to the standard image and video quality metrics.

Critical weaknesses
  1. The evidence for the proposed memory components is not yet convincing. In Table 4, adding retrieval and both relative encodings increases PSNR from 12.69 to 12.94, but decreases SSIM from 0.340 to 0.337 and only slightly improves LPIPS from 0.594 to 0.592. Retrieval alone reduces both PSNR and SSIM. These experiments cover only four scenes at reduced resolution, with no uncertainty estimates or direct evaluation of cross-time content preservation. Table 7 also shows little difference between shared and independent reference self-attention. The results support parameter sharing as a reasonable implementation choice, but do not clearly establish a substantial benefit from the proposed spatiotemporal memory design.

  2. It is difficult to determine how much of the improvement comes from the proposed method rather than the internal pretrained backbone and its training data. The backbone’s pretraining setup is insufficiently documented, and comparable training conditions for the baselines are not established. The main benchmarks use only 20 reference clips per dataset, which also makes the reliability of FID/FVD comparisons difficult to assess without uncertainty estimates. Finally, unseen-layout and dashcam generalization are demonstrated qualitatively, and the independently aligned camera trajectories do not verify whether the generated views jointly preserve the requested camera rig. These gaps limit the strength of the empirical conclusions.

Questions for authors
  1. What data and camera configurations were used to pretrain the internal backbone? Please clarify whether there is any overlap with the evaluation data and whether the baselines received comparable task-specific training. How does the same backbone with reference-as-view conditioning, but without retrieval and relative encodings, perform on the main benchmarks?

  2. Are the improvements from retrieval and relative encodings consistent across a larger evaluation set at the main resolution? Results on cross-view consistency and regions visible only in earlier reference frames, together with variability across scenes or runs, would help clarify their contribution.

  3. Can the authors quantify performance on unseen camera layouts and report cross-view matching separately for reference-to-generated and generated-to-generated pairs? Can the requested relative camera poses also be evaluated jointly, without independent per-view Sim(3) alignment?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 44jd

Thank you for recognizing the practical value of our task. We address your comments below.

W2 & Q1 · Backbone pretraining and task-specific training of baselines

We use an internally trained backbone for its multi-view generation and camera-pose control capabilities. It was initialized from Cosmos-Predict2.5-2B and trained on our internal driving data, without public datasets at this stage. All evaluation scenes and clips were excluded from both this stage and subsequent FreeView training.

The baselines also received task-specific training, using different backbones and datasets. For example, AnyView uses Cosmos-Predict2-2B-Video2World and trains on datasets including Waymo and Lyft, while OpenLongTail fine-tunes Wan2.1-VACE-1.3B on datasets including PandaSet and nuScenes.

W1, W2 & Q1, Q2 · Same-backbone comparisons, memory effectiveness, and evaluation scale

Thank you for this suggestion. We expanded the ablation and evaluation sets from 20 to 40 clips. For a fair comparison, both variants use the same backbone and evaluation settings, with the same reduced batch size in these additional ablations in 480p. Adding our retrieval method improves overall performance on both sets.

20 clips

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
AnyView12.140.4260.61971.64617.85
Ours (multi-view, w/o retrieval)12.670.43240.608153.60403.55
Ours (full)12.980.43730.599153.42397.32

40 clips

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
AnyView12.280.4410.62258.97361.27
Ours (multi-view, w/o retrieval)13.010.44340.606038.87265.48
Ours (full)13.530.45520.594739.57237.61

Appendix A, Figure 5 further illustrates how a retrieved reference helps recover crosswalk markings no longer visible in the current front view.

W1 · Reference self-attention weight sharing in Table 7

Table 7 examines whether reference self-attention requires separate weights. Both variants use memory, and their similar performance supports weight sharing. The effectiveness of our memory design is demonstrated by the same-backbone comparisons above.

W2 & Q3 · Cross-view matching and joint camera-pose evaluation

Thank you for your interest in Table 5. Using the same evaluation setup with the target layout changed to the unseen Waymo layout, FreeView outperforms AnyView [4] on both reference-to-generated (RG) and generated-to-generated (GG) matching, improving overall GMSR@10 by 13.39 percentage points.

MethodRG ↑GG ↑Overall ↑
AnyView47.93%49.94%49.27%
Ours73.78%57.10%62.66%

FreeView also achieves lower mean rotation and translation errors with both pose estimators, Pi3X [2] and VGGT-Omega [3]. Following Vista4D [1], we use reconstructed poses to evaluate camera control. We jointly input the real front-view video and the generated videos, then compare their estimated relative poses with the requested poses, without aligning each view separately.

EstimatorMethodMean rotation error (°) ↓Mean translation error (m) ↓
Pi3XAnyView26.8212.55
Pi3XOurs19.612.74
VGGT-OmegaAnyView26.388.48
VGGT-OmegaOurs23.244.27

[1] Vista4D: Video Reshooting with 4D Point Clouds, CVPR 2026.

[2] π³: Permutation-Equivariant Visual Geometry Learning, ICLR 2026.

[3] VGGT-Ω, CVPR 2026.

[4] AnyView: Synthesizing Any Novel View in Dynamic Scenes, ECCV 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Area Chair and Reviewers,

Thank you for the careful reviews. All three reviewers recognized the practical value of generating surround-view videos from a monocular driving video and highlighted our reference-as-view design, which reuses pretrained attention without a separate reference encoder. We summarize the main findings from the rebuttal below.

1. Retrieval improves generation with the same backbone (44jd, QYgz). Both variants retain reference-as-view conditioning and jointly generate the target views under matched training and evaluation settings. On the expanded 40-clip evaluation, the full model increased PSNR from 13.01 to 13.53 dB compared with the no-retrieval variant. We also clarified the backbone’s training data and confirmed that all evaluation scenes were excluded from both backbone adaptation and subsequent FreeView training.

2. FreeView better preserves the geometry of an unseen camera layout (44jd, QYgz). On the held-out Waymo target layout, FreeView outperformed AnyView on both reference-to-generated and generated-to-generated matching, improving overall matching success by 13.39 percentage points. When the real front view and generated views were evaluated jointly, both Pi3X and VGGT-Omega also found lower mean rotation and translation errors for FreeView.

3. Generation quality remains stable with estimated poses (Y2vi). On the expanded 40-clip evaluation, replacing calibrated camera poses with MapAnything estimates reduced FreeView’s PSNR by only 0.014 dB.

4. A longer retrieval window improves coverage, while a second frame adds little (Y2vi). With the targets fixed, expanding the window from 41 to 101 frames increased the share of the target view covered by a single retrieved frame from 30.77% to 76.78%. Retrieving a second frame added little coverage even with the longer window.

Figure 5 in Appendix A illustrates what retrieval contributes: an earlier front-view frame helps recover crosswalk markings that are no longer visible in the current front view but appear in the target rear view. We also added video results to the supplementary material.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.