acceptodds
Withdrawn from ICLR 2027

OccluCast: Casting Hidden Backgrounds from Multi-View Driving Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Removing foreground vehicles from driving videos is a fundamental step toward editable, empty-street reconstruction, yet it remains challenging due to severe occlusions, multi-view consistency requirements, and the need to recover geometrically plausible background content. We present OccluCast, a multi-view temporal diffusion framework for vehicle-free driving-scene inpainting. Given synchronized three-view videos and foreground vehicle masks, OccluCast augments a pretrained video diffusion backbone with a lightweight ViewSync branch that injects same-timestamp multi-view attention and temporal attention to propagate background evidence across cameras and time. An empty-street classifier-free guidance strategy further suppresses foreground-vehicle hallucinations by steering the denoising trajectory away from a jointly learned foreground-reconstruction direction. To lift the inpainted videos into an explicit 3D representation, we perform mesh-guided Gaussian distillation, where a LiDAR-derived background mesh filters the supervision loss so that inpainted content is applied only to genuinely occluded regions while real observations supervise visible surfaces. Experiments on the Waymo benchmark demonstrate that OccluCast produces visually realistic, temporally stable, and geometrically consistent empty-street reconstructions, providing a reliable tool for autonomous driving simulation.

Primary area:
generative models
Keywords:
Generative Models for Vision, Autonomous driving simulation, Gaussian splatting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1TGkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents OccluCast, a method for removing foreground vehicles from synchronized three-view driving videos, generating empty-street videos, and distilling them into renderable 3D Gaussian scenes. The authors construct 3,200 RGB clips and 3,200 mask clips from Waymo by pairing foreground masks from other scenes with empty regions in the current scene. The model is based on Wan2.1 VACE-14B and inserts a ViewSync branch every five Wan-DiT blocks, with same-timestamp attention across adjacent cameras and temporal attention within each view. It also uses two learnable text conditions for empty-street classifier-free guidance. For the 3D stage, background geometry is aggregated from LiDAR and processed with VDBFusion to build a background mesh, which is then used to filter supervision from generated videos. Table 1 compares OccluCast with StreetUnveiler, DriveEditor, and WanVACE, while Table 2 ablates ViewSync, empty-street CFG, and mesh-guided Gaussian distillation. Figures 4–5 and the supplementary video provide qualitative comparisons.

Critical strengths

The problem is practically relevant. The paper combines vehicle removal, multi-view and temporal consistency, and empty-street 3D scene construction in one pipeline, with a clear connection to scene editing for autonomous-driving simulation.

The method components are connected to the task constraints. ViewSync exchanges information across adjacent views and time, while mesh-guided distillation uses LiDAR background geometry to filter generated supervision. Table 2 includes ablations of the three main designs.

Under the reported evaluation protocol, OccluCast is best on all six generation and reconstruction metrics in Table 1. Table 2 shows that removing ViewSync or empty-street CFG degrades both generation and reconstruction, while removing mesh-guided distillation mainly affects reconstruction. Figures 4–5 and the supplementary video also provide qualitative comparisons.

Critical weaknesses

The evaluation does not isolate the main completion claims. Section 4.1 compares vehicle-free outputs with original videos containing vehicles, without specifying the FID/FVD evaluation regions. These scores can support overall visual quality but do not establish the quality of the filled regions. For the 3D results, Tables 1 and 2 report the same perceptual metrics and overlap-based MV Cons.; Figure 4 is qualitative. This leaves the geometric consistency of the completed background insufficiently tested.

The technical contribution appears mainly to be a task-specific combination of cross-view/temporal attention, foreground/background CFG, and LiDAR-filtered Gaussian supervision. The paper does not clearly distinguish these mechanisms from the closest video-editing and reconstruction methods. The scene-level train/test split, test-set size, and baseline adaptation details are also missing.

Questions for authors

Which reference videos and image regions are used for FID/FVD? Is there evidence specifically evaluating completion quality inside the removal masks?

What evidence beyond Figure 4 and the image/video metrics supports geometric consistency in the completed 3D regions?

What mechanism distinguishes ViewSync and the learned foreground/background guidance from the closest existing approaches, beyond adapting them to this pipeline?

Please specify the scene-level split and test-set size. How were the video baselines converted to 3D, and which reconstruction inputs and optimization settings were shared?

At inference, do u_bg and u_fg in Eqs. 15–17 use the same foreground mask? If not, how are the two spatial conditions constructed?

Reviewer_s1SaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes OccluCast for removing vehicles from multi-view driving videos and reconstructing empty streets with 3D Gaussian Splatting. It builds on Wan2.1 VACE and introduces three components. ViewSync shares background information across cameras and time. Empty-street CFG suppresses vehicle hallucinations. Mesh-guided Gaussian distillation uses LiDAR geometry to reduce conflicts between generated content and real observations. Experiments on Waymo report better video generation and reconstruction results than the compared methods.

Critical strengths

S1. ViewSync combines cross-view and temporal attention to share visible background information. This helps complete regions hidden by vehicles, rather than processing each camera independently.

S2. Mesh-guided distillation preserves supervision from real observations and limits generated supervision to occluded regions. This addresses conflicts between generated backgrounds and recorded scene content during reconstruction.

Critical weaknesses

W1. The methodological novelty is limited and not clearly distinguished from prior work.

  1. Sec. 3.3 spends much space describing VACE’s frozen backbone, condition branch, and residual injection, but does not clearly explain which operations ViewSync changes in the original Context Adapter.
  2. Combining cross-view and temporal attention is used in Panacea and MagicDriveDiT. The paper should clarify what ViewSync adds beyond adapting this established design to VACE.
  3. Empty-street CFG learns foreground/background conditions in standard guidance, while mesh-guided distillation filters generated supervision to preserve observed surfaces. Both are useful task-specific changes, but their methodological advances appear limited.

W2. The experiments do not sufficiently support key claims.

  1. The paper criticizes prior methods for relying on segmentation and heuristic masks, yet its own pipeline uses Mask2Former and projected 3D-box annotations. Without tests under mask or annotation errors, it is unclear how the method addresses this limitation.
  2. ViewSync contains 3B trainable parameters, but generation runtime, peak memory, and added cost over VACE are not reported. Thus, the claims of being “lightweight, scalable, and general” lack evidence.

W3. Presentation issues make the method harder to understand.

  1. Fig. 2 has very small labels and mixes training with inference, without explicitly showing the reconstruction inputs from LiDAR and original images. Fig. 3 spends much space on repeated blocks but does not clearly show branch-to-backbone injection.
  2. Sec. 3.3 and the implementation details state that only ViewSync and camera embeddings are trained, whereas Sec. 3.4 and Fig. 2 introduce two learnable text embeddings. The trainable parameters should be stated consistently.
Questions for authors

Please refer to weaknesses.

Reviewer_p7wwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies vehicle removal and background completion in driving scenes. OccluCast adds a trainable ViewSync residual branch to a pretrained Wan-VACE backbone to share information across synchronized camera views and time. It creates training examples by transferring foreground masks from other scenes onto visible background regions and uses empty-street guidance to discourage vehicle hallucinations. The completed videos are then distilled into an editable 3D Gaussian representation. A LiDAR-derived background mesh determines where generated content is used for supervision.

Critical strengths
  1. The pipeline connects multi-view vehicle removal with an explicit, editable 3D scene representation. This is a practical contribution to driving-scene simulation.
  2. ViewSync is a well-motivated way to adapt a pretrained video model to synchronized cameras. It combines cross-view attention at the same timestamp with temporal attention, and both the reported comparisons and the overall ViewSync ablation show meaningful improvements.
Critical weaknesses
  1. The evaluation does not directly show how well the completed regions are reconstructed or whether they are geometrically consistent across views. FID and FVD are computed over full frames or videos, so unedited content may substantially influence the results. The reference videos also still contain the vehicles that the method removes. For MV Cons., the paper describes the overall protocol but leaves out how regions from adjacent cameras are aligned and how LPIPS distances are normalized. This makes the metric difficult to reproduce and interpret as evidence of geometric consistency. The evaluation would be more convincing if it measured the completed regions separately and included a geometry-aware cross-view metric.

  2. The 3D reconstruction stage appears only loosely coupled to the proposed video generator. It largely follows an existing Gaussian reconstruction framework, with the LiDAR-mesh supervision mask as the main addition. The reported gain from this mask is small. A comparison with reconstruction supervised only by generated videos is needed to assess whether the proposed integration is necessary. The lack of a relevant 3D completion baseline, such as GS-RoadPatching, also limits this assessment.

Questions for authors
  1. What proportion of the pixels used to compute MV Cons. fall within the inpainting masks? Could you evaluate the completed regions separately, for example by synthetically masking background that is visible elsewhere in the sequence? Where reliable depth is available, could you also report a geometry-aware reprojection or correspondence measure?

  2. Could you explain why SceneCrafter and GS-RoadPatching are not included, or provide comparisons under a comparable setup? Please also ablate cross-view and temporal attention separately and compare reconstruction supervised only by generated videos.

  3. How do you ensure that a foreground mask transferred from another scene creates a plausible training occlusion at its new location? How sensitive are the two learned guidance directions to the initial text prompts, particularly the potentially ambiguous prompt "An empty realistic autonomous driving scene"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.