acceptodds
Under review as a conference paper at ICLR 2027

Beyond Observed Rays: Query-Guided Completion of Layered Gaussian Driving Worlds

11 views

Abstract

Driving-scene reconstruction with 3D Gaussian Splatting (3DGS) turns sensor logs into 3D environments, making recorded driving scenes renderable and editable for closed-loop simulation. However, reconstructing scenes solely from recorded camera observations leaves weakly observed or unseen object surfaces and occluded background regions poorly reconstructed. During closed-loop simulation, ego-vehicle motion may shift the camera away from the recorded trajectory, exposing previously unobserved regions and associated reconstruction defects, which can compromise simulation reliability. We introduce FullSplat, a task-query completion framework that uses expected exposure within a bounded camera–action distribution to identify low-support regions that later queries may expose and complete them in a reusable 3D scene. A structured 3D prior generates object-canonical Gaussian residuals that follow rigid object motion, while masked rectified flow generates world-fixed background residuals; cross-layer conflict resolution and observation-preserving gates integrate both into one persistent scene for later rendering and editing without further generation. Experiments on Waymo and nuScenes cover reconstruction and novel-view synthesis, with rigid-object edits and 4–6 m camera-offset stress tests beyond the 3 m range on Waymo. Results show better captured-view fidelity, novel-view quality, and temporal consistency, with improvements under object translation, rotation, and removal and in views beyond the completion range.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Driving Scene Completion, Novel View Synthesis, Scene Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CuYFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FullSplat, a method for completing driving scenes using 3D Gaussian Splatting (3DGS). The approach identifies object surfaces likely to be revealed in the future, as well as currently occluded background regions. It then leverages structured 3D priors to generate Gaussian residuals within the local coordinate systems of objects, while employing a masked rectified flow model to complete the background in the world coordinate system. Experiments on the Waymo and nuScenes datasets demonstrate improvements in novel-view reconstruction, and experiments involving rigid-body manipulation on the Waymo dataset also show enhanced performance.

Critical strengths
  1. The problem is clearly defined and represents an important issue within the field.
  2. The idea of \textit{query-guided target selection} is novel and intuitive.
  3. The experimental metrics are competitive, and the ablation studies are thorough.
Critical weaknesses
  1. In query-guided target selection, reconstruction priority is linked to projected area. This may creates an inherent bias: some key NPCs with smaller projected areas(pedestrians, traffic sign) are at a disadvantage compared to objects with larger projected areas(vehicles).
  2. The contribution of this paper is the \textit{completion of unseen geometry}. However, all metrics used for novel-view synthesis and editing are either distribution-based or reference-free. In other words, the evaluation focuses on an indirect metric rather than the actual effectiveness of the geometric completion itself.
  3. The computational overhead of the method was not quantified.
Questions for authors
  1. How are the projected area components unified across different objects?
  2. What’s the accuracy of completed geometry
  3. Will FullSplat add notable computational cost, or reduce it?
Reviewer_iKnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Most classical 3d reconstruction methods for driving scenes produce good novel views when cameras are close to input views, but their quality degrades significantly under lateral shifts and they cannot render unobserved regions of scene and vehicles, which is important for simulation. This paper introduces Fullsplat, a method for complete scene reconstruction, generating additional gaussians for objects and background using corresponding generative models. Experiments on benchmark datasets show its effectiveness in reconstruction, novel view synthesis and rigid edits.

Critical strengths
  1. Complete driving scene reconstruction is an important task and authors provide reasonable design to solve it.
  2. Fullsplat achieves state-of-the-art results across multiple evaluation scenarios on Waymo and nuScenes datasets, including reconstruction, various lane shifts and rigid-objects edits.
Critical weaknesses
  1. Paper is very difficult to follow, key components of the method are very vague or missing, which makes results difficult to reproduce. For example, shared encoder, which is supposed to encode images, gaussians and queries, is not described. Its output is a set of tokens, but for object generation authors utilize Trellis model, which works on structural latent space, not clear how they correspond to each other. Additionally, Trellis consists of two sequential and uses text or image conditioning, authors don't write anything about that and how they adapt it to their model.
  2. Evaluation is incomplete.
  • Important baselines are missing. MM-Trellis (https://github.com/HongliXiao/MM-TRELLIS) is Trellis-based model for car completion, which, combined with OmniRe, can produce scene reconstruction with complete cars and serve as a baseline. Novelty compared to this baseline is limited. Additionally, recent methods explore feed-forward approaches for driving scene reconstruction (e.g. WorldSplat, StreetForward), this methods are not included in the paper.
  • Missing evaluation protocol description for Waymo and Nuscenes. OmniRe surprisingly underperforms StreetGS in evaluation. Additionally, as Fullsplat is based on OmniRe, +4-5 PSNR gain on reconstruction scenario is not clear where is coming from.
  • Evaluation is almost only quantitative, paper has only visual comparison to OmniRe on lane shift and car movement and removal, all other baselines and other evaluation scenarios are missing in qualitative evaluation.
Questions for authors
  1. What is shared encoder architecture? How tokens are split into world and object tokens? How Trellis is adapted to the pipeline (what is F_obj design)?
  2. How good is method for car completion? Please provide qualitative results and comparison with MM-Trellis.
  3. Please specify, how many scenes and which scenes were used for Waymo and Nuscenes evaluation, which cameras were selected, how many frames in reconstruction and was there train-test split in reconstruction frames list.
  4. Could you explain the reason why Fullsplat is 4-5 PSNR better in reconstruction compared to OmniRe? It might be, that improvement comes from number of gaussians, could you reconstruct scenes with higher budget and compare?
Reviewer_MTwyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies the issue that existing driving-scene reconstructions often fail to represent occluded or unobserved regions, which can lead to artifacts when the scene is viewed from novel viewpoints or subjected to object transformations. It argues that scene completion should be guided by potential future usage rather than only existing observations, and proposes a unified framework to improve scene completeness and consistency. This work aims to make reconstructed driving scenes more reliable and reusable for simulation and editing.

Critical strengths

The paper addresses the problem of completing occluded regions in driving-scene reconstruction. It proposes a geometry-consistent approach to scene completion, with a coherent overall framework. The task is meaningful, and the proposed methods are generally well aligned with the task requirements.

Critical weaknesses

The proposed completion targets and coverage criteria rely on manually predefined distributions of camera viewpoints and object actions, including query ranges, sampling strategies, and weighting schemes. These choices may not adequately reflect diverse real-world scenarios and could bias the regions prioritized for completion. However, the paper provides limited analysis of the method's sensitivity to these design choices, particularly the number of sampled queries and query-distribution weights. Consequently, the robustness and generalizability of the proposed framework under different query distributions remain insufficiently validated.

In Figure 2(b), some background textures in the completed scene appear inconsistent with those in the input, particularly the stripe patterns on the barrier. These discrepancies raise concerns about the method's ability to preserve the appearance of already observed regions during scene completion. The paper lacks detailed visual comparisons or quantitative analyses of appearance consistency, making it difficult to assess whether the completion process introduces unintended modifications to existing scene content.

Questions for authors

Appendix F.3 states that rejected completions fall back to the base reconstruction and remain in the evaluation. Could the authors clarify what percentage of evaluated scenes on each dataset required this fallback?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.