acceptodds
Withdrawn from ICLR 2027

GaussianDiffuser: Diffusion-based 3D Gaussian Pre-training for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised pre-training has become a fundamental component for advancing 3D scene understanding in autonomous driving. While existing approaches commonly rely on rendering-based reconstruction to enforce geometric consistency, their objectives remain largely confined to camera-observable surface signals, which can make the learned representations overly dependent on directly visible rendering cues. In this paper, we propose GaussianDiffuser, an efficient feedforward framework that integrates diffusion-based denoising into 3D Gaussian Splatting (3DGS) pre-training. Diffusion-based denoising provides a complementary pre-training objective for 3D Gaussian scene representations by regularizing the model to recover clean signals from corrupted latents, encouraging the representation to exploit multi-view context and spatial consistency rather than locally clean rendering cues. To realize this objective, GaussianDiffuser introduces a diffusion-aware Gaussian block for a denoising objective during pre-training, learning enriched spatial representations with minimal architectural modifications. Experimental results on the nuScenes dataset demonstrate that GaussianDiffuser improves depth reconstruction and occupancy prediction, reducing Abs Rel by 17% and improving Occ3D mIoU by 2.4 over previous state-of-the-art methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous driving, Pre-training, Self-supervised learning, Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fFGYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GaussianDiffuser is a self-supervised pre-training framework for vision-centric 3D driving perception. It corrupts multi-view image latents using a flow-matching-style interpolation, lifts the corrupted features into continuous 3D Gaussians, refines them with timestep-conditioned AdaLN-Zero blocks, and rasterizes RGB, LiDAR depth, and feature predictions toward clean targets. During downstream fine-tuning, the noise scheduler and projection head are removed, t=0t=0 is used, and probabilistic voxelization connects the representation to occupancy and detection heads. The method is evaluated on nuScenes/Occ3D, SurroundOcc, 3D detection, and AV2-to-nuScenes transfer; the controlled comparisons against a rendering-only GaussianFormer-2 baseline show improvements from 0.159 to 0.151 Abs Rel and from 35.58 to 36.56 Occ3D mIoU.

Critical strengths
  1. The paper includes meaningful matched controls. Against the same GaussianFormer-2 pipeline, GaussianDiffuser improves depth, occupancy, corruption-domain transfer, and ImageNet-initialized performance.

  2. The transfer evaluation is broad. The AV2-to-nuScenes zero-shot depth result is particularly promising, reducing Abs Rel from 0.626 for NeRP3D to 0.273.

Critical weaknesses
  1. The headline gains overstate the controlled evidence. The abstract emphasizes a 17% Abs Rel reduction and +2.4 Occ3D mIoU using cross-paper comparisons, whereas the matched gains are approximately 5% Abs Rel and +0.98 mIoU. SurroundOcc also uses a different input resolution, and the detection comparison uses different downstream detectors across methods (Abstract, p. 1; Tables 3 and 5, pp. 8 and 13).

  2. Key quantitative and implementation facts are internally inconsistent. If Table 11 is correct, GaussianDiffuser adds 112.84 GFLOPs, or 11.7%, rather than 0.39 GFLOPs; the claimed 49.2% reduction relative to NeRP3D would also be incorrect. The 37.9/36.8 mIoU discrepancy affects another headline result, while the clean-versus-corrupted feature paths remain under-specified (Section 4.4, p. 8; Figure 2 and Equation 3, pp. 4–5; Table 11 and Appendix C, p. 18).

Questions for authors
  1. Please specify the complete information flow during pre-training: do the depth network, Gaussian initialization, Equation 3's FimgF_{\rm img}, Gaussian colors/features, and projection head receive xσ′x'_\sigma or x0x_0? Is x0x_0 stop-gradient, and is the ConvNeXt backbone updated (Figure 2 and Sections 3.1–3.3, pp. 3–6)?

  2. What is the correct 17-class Occ3D result, 37.9 or 36.8 mIoU? Please explain the source of the discrepancy and confirm which value underlies the +3.8 and +2.4 comparisons (Table 2 and Section 4.4, p. 8).

  3. Which computation figure is correct, 1079.05 or 966.60 GFLOPs? Please define the measurement boundary consistently; if 1079.05 is correct, the claims of +0.39 GFLOPs and negligible overhead should be revised (Table 11 and Appendix C, p. 18).

Reviewer_uGZ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GaussianDiffuser, a diffusion-based 3D Gaussian pre-training framework designed for 3D scene understanding in autonomous driving. The framework incorporates an additional denoising pre-training objective: noise is injected into 2D image latents, and clean visual signals are subsequently recovered following Gaussian lifting and refinement. The authors evaluated the proposed method on the nuScenes and Argoverse 2 datasets, and the experimental results demonstrate that GaussianDiffuser achieves competitive performance across multiple tasks.

Critical strengths
  1. The research question has practical significance=
  2. The ablation studies are thorough.
  3. Solid engineering and changes are easy to follow
Critical weaknesses
  1. The primary contribution of this paper lies in integrating diffusion-style noise corruption, a flow-matching noise schedule, AdaLN-Zero conditioning, and a feature-denoising objective into a 3D Gaussian pre-training framework. While the design concept is sound, the paper lacks a sufficient analysis of the motivations and underlying mechanisms for the inclusion of each of these techniques.
  2. According to Figure 2, clean image latents appear to be used not only for constructing noisy inputs but also for contributing to the subsequent Gaussian refinement process via a separate pathway. Furthermore, while the deformable cross-attention in Eq. 3 utilizes image features (FimgF_{\mathrm{img}}), the paper does not explicitly state whether these features undergo the same corruption process during the pre-training phase. If the deformable cross-attention retains direct access to unperturbed features, the model could potentially leverage these clean features to obtain local visual information, rather than relying solely on the spatial context provided by the Gaussian representations for reconstruction.
  3. Although the title mentions \textit{autonomous driving}, this is reflected only in the dataset used. The proposed method itself does not demonstrate a sufficient connection to autonomous driving tasks; in other words, the approach could equally be applied to reconstruction tasks in other domains.
Questions for authors
  1. Is it possible for the denoising process to form a shortcut via clean image features?
  2. Could the authors add a baseline that omits noise corruption while retaining the same feature reconstruction loss and projection head? In other words, why is diffusion-based corruption superior to standard feature reconstruction?
Reviewer_sa25OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new feedforward Gaussian prediction framework named GaussianDiffuser, which integrates diffusion-based denoising into 3D Gaussian representation learning. Compared with rendering-only pre-training, GaussianDiffuser introduces denoising as an additional pre-training objective to learn better 3D Gaussian scene representations. And the results show that this framework achieves SOTA performance in depth estimation and 3D occupancy prediction.

Critical strengths

The authors introduce denoising as an additional pre-training objective to learn better 3D Gaussian representations, which I consider a novel idea. Experimental results show that GaussianDiffuser performs well across multiple tasks. In addition, Table 4 shows its robustness in zero-shot depth estimation.

Critical weaknesses

I have questions about the difference between GaussianDiffuser and data augmentation in latent space. Based on the paper, GaussianDiffuser adds noise to 2D latent features during pre-training and requires the model to recover clean features. This seems similar to using random noise augmentation to improve representation learning. Could the authors compare their method with random noise augmentation under the same training settings to clarify the additional benefits of GaussianDiffuser. Furthermore, the timestep is fixed at t=0 during inference. This makes me more inclined to interpret the method as a noise augmentation strategy used during pre-training. Could the authors explain how their method differs from noise augmentation or denoising pre-training? If different levels of noise are introduced during inference, with the corresponding timesteps, how would the final predictions be affected?

Could the authors mark the best and second-best results in Tables 2, 3, and 5, as well as other relevant tables, to make the comparisons easier to follow?

Questions for authors

Please refer to the Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.