acceptodds
Under review as a conference paper at ICLR 2027

Gaussian for Gaussian: Unifying Feature Representation and Rendering in Feed-Forward Driving Scene Reconstruction

Abstract

Vehicle-mounted surround-view cameras have limited overlap, making it difficult to exploit cross-view information while preserving local detail and reconstructing complete novel views. Existing methods address this challenge through cross-view matching or dense spatial grids, but face limited correspondence support or a resolution–memory trade-off. We propose Gaussian for Gaussian, a feed-forward framework that uses cross-camera features as spatially sparse supplements to local features before constructing a separate renderable scene. Using estimated metric depth, we lift multi-scale image features into Feature Gaussians in a shared 3D space. We splat these Gaussians into the source views and fuse the projected features with local image features, retaining pixel-aligned detail alongside geometrically aligned support. To turn the fused features into high-quality novel views, we further predict independent Render Gaussians with layered geometry and learnable local offsets, providing geometric flexibility beyond the initial depth surfaces. Under known ego-motion, our complete system achieves 28.803 dB PSNR on nuScenes, surpassing the DrivingForward baseline by 7.136 dB, and transfers to PandaSet without fine-tuning with a 4.611 dB advantage. Qualitative and regional analyses show improved reconstruction in newly revealed regions and multi-view overlap regions, while feature aggregation yields larger gains where cross-camera support is stronger.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Feed-Forward 3D Reconstruction, Novel View Synthesis, Autonomous Driving, Multi-View Feature Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_y6pyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Feature Gaussians for sharing features across surround-view cameras, followed by separate Render Gaussians for novel-view synthesis. The method is evaluated mainly on nuScenes, with an additional zero-shot test on PandaSet.

Critical strengths
  • The overall design is simple and easy to follow.
  • The method improves clearly over DrivingForward.
  • The ablations provide some evidence for the proposed components.
Critical weaknesses
  1. Limited to static 3D reconstruction. The method does not explicitly model time or object motion. It is unclear what advantage this static formulation offers over recent feed-forward 4D methods, such as STORM, DrivingRecon, UniSplat, and DGGT.

  2. Unclear comparison protocol. The original settings of Omni-Scene, pixelSplat, MVSplat, and other baselines differ substantially in training data, inputs, resolution, and depth priors. The paper should clarify whether these methods were retrained under a unified setting and whether the comparisons are fair.

  3. Missing recent baselines. Several relevant 4D driving-scene reconstruction methods are neither discussed nor compared, including STORM (Yang et al., ICLR 2025), DrivingRecon (Lu et al., NeurIPS 2025), UniSplat (Shi et al., ICLR 2026), and DGGT (Chen et al., CVPR 2026). This makes it difficult to assess the method against the current state of the art.

  4. Limited evidence for cross-domain generalization. PandaSet still uses a six-camera surround-view setup similar to nuScenes. Evaluation on datasets with different camera numbers and layouts, such as Waymo or Argoverse 2, would better support the zero-shot claim.

  5. Insufficient qualitative comparison. The main nuScenes visualization only compares against DrivingForward. Comparisons with stronger recent methods, especially on dynamic objects, overlap regions, and failure cases, are needed.

Questions for authors

Refer to the weaknesses for details.

Reviewer_TRhoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a feed-forward network for surround view driving scene reconstruction. The paper addresses the limited camera overlap problem and introduces two Gaussian representations, including Feature Gaussians for multi-view feature aggregation and Render Gaussians for novel view synthesis. The experiments show their improvement over existing methods.

Critical strengths
  • The design of Feature Gaussians and Render Gaussians is interesting, which combines multi-view geometric information while maintaining flexibility for NVS.
  • The improvement over previous methods is large on both the nuScenes and PandaSet datasets.
Critical weaknesses
  • The method design has limited novelty. This combination of feature splatting and geometry-guided feature aggregation has been proposed by other works like FreeSplat [1], GSRecon [2], C3G[3]. Therefore, the paper should also compare with these methods and clearly clarify its technical contribution.
  • The comparisons with existing methods are not fairly controlled. While the improvement over DrivingForward is large, the proposed method uses UniDepthV2 and two Gaussian layers which DrivingForward lacks. More comparable baselines with matched depth priors, Gaussian capacities, and training budgets are necessary to establish the effectiveness of the proposed architecture.
  • Insufficient isolation of cross-camera feature aggregation. The key contribution is geometry-guided cross-camera feature fusion. However, the local-only ablation in Table 2 removes all projected features, including those from the receiving camera itself. Therefore, the substantial improvement cannot be directly attributed to cross-camera information sharing.
  • Limited geometric and dynamic reconstruction evaluation. The method relies heavily on estimated metric depth and accurate camera poses, but the evaluation focuses primarily on image rendering metrics. There is insufficient quantitative analysis of reconstructed geometric accuracy, cross-view consistency, and robustness to depth or pose estimation errors. Moreover, the method does not explicitly model dynamic object motion between T0 and T1, limiting its applicability to dynamic driving scene reconstruction.
  • The analysis of efficiency is insufficient. The comparison between Feature Gaussians and dense voxels is not fully controlled, as the Gaussian representation aggregates features at five scales while the voxel baseline produces only one aggregated feature scale. Furthermore, Table 4 shows nearly identical rendering quality, relatively modest allocated-memory savings, and slightly higher latency for Feature Gaussians. These results do not convincingly establish the superiority of Gaussian-based intermediate representations in either reconstruction quality or computational efficiency.

[1] Wang, Yunsong, et al. "Freesplat: Generalizable 3d gaussian splatting towards free view synthesis of indoor scenes." Advances in Neural Information Processing Systems 37 (2024): 107326-107349.

[2] Yang, Hang, et al. "GSRecon: Efficient Generalizable Gaussian Splatting for Surface Reconstruction from Sparse Views." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[3] An, Honggyu, et al. "C3g: Learning compact 3d representations with 2k gaussians." arXiv preprint arXiv:2512.04021 (2025).

Questions for authors

See weaknesses

Reviewer_8wQcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary

The paper proposes Gaussian for Gaussian, a feed-forward framework for novel-view synthesis from low-overlap surround-view images. It uses estimated metric depth to lift multi-scale image features into Feature Gaussians in a shared 3D space. It splats these Gaussians back into each source view and fuses the projected features with local image features as a sparse supplement. It then predicts separate Render Gaussians with 2 layered depth and learnable local offsets to render more complete novel views.

Contributions

  1. Two separate Gaussian sets: one for intermediate features, one for the final renderable scene.
  2. Geometry-guided, multi-scale, sparse feature fusion that avoids correspondence search and dense grids.
  3. Results on nuScenes, zero-shot transfer to PandaSet, and region-level analysis.
  4. Minor: Layered depth plus local offsets for flexible geometry. This idea is adopted from Flash3D-style work.

Validation

  1. Main results. On nuScenes, the method beats DrivingForward by 7.1 dB PSNR. On PandaSet, without fine-tuning, it still leads by 4.6 dB. Visual results show fewer holes in newly revealed regions and more consistent appearance across cameras.
  2. Every component helps. Feature aggregation matters most (−3.6 dB without it), followed by the second depth layer (−1.8 dB). Rendering through Gaussians beats direct decoding, and local offsets add a limited gain.
  3. Regional analysis. The biggest gain is in newly revealed regions (+12.2 dB). In overlap regions, DrivingForward gets worse while this method gets better. Paper also shows the gain w/ and w/o cross-camera support, but this evidence is only weakly correlational.
  4. Efficiency. Feature Gaussians match a dense voxel baseline in quality and latency, with slightly lower memory. This is tested only at low resolution.
Critical strengths
  1. Large performance gains. The method outperforms DrivingForward by 7.1 dB PSNR on nuScenes and by 4.6 dB on zero-shot PandaSet transfer. The ablation study shows the contribution of each component, and the region-level analysis shows where the gains come from.
  2. Clear design. Separating Feature Gaussians from Render Gaussians is a reasonably novel idea. Reusing 3DGS splatting to project Feature Gaussians into feature maps is a simple and effective way to share information across cameras.
Critical weaknesses
  1. Unclear source of the gains. Despite the large improvement, the paper does not show that the proposed Gaussian representation is the main cause. The main comparison with DrivingForward also changes the depth prior (frozen UniDepthV2) and the number of Gaussians per pixel. The only representation comparison is against the authors’ own voxel baseline at low resolution (128×384), and the two perform about the same (28.334 vs. 28.330 dB).
  2. The memory advantage over dense voxels is small (peak allocated memory 4830 → 4501 MB).
Questions for authors
  1. Can you retrain DrivingForward with frozen UniDepthV2 depth, or retrain your model with DrivingForward’s depth? The current comparison changes the depth prior and the method at the same time.
  2. Your model uses two Gaussians per pixel, while DrivingForward uses one. How does your model perform with G = 1 at 352×640, compared with DrivingForward at the same resolution? This separates the effect of the representation from the effect of extra capacity.
  3. What is the time gap between T0 and T1? How does performance change with larger gaps (e.g., 2–5 frames), and in regions containing moving objects?
Reviewer_FKbDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a feed-forward framework for driving-scene reconstruction using separate Gaussians for feature aggregation and rendering. However, the geometric behavior learned through image supervision, the contribution of cross-camera aggregation, and the fairness of baseline comparisons require further validation. If the authors can address these concerns with additional experiments and analysis, I would be willing to reconsider my rating.

Critical strengths

The paper addresses an important challenge in low-overlap surround-view reconstruction: exploiting cross-camera information while preserving local detail and recovering newly revealed regions. Using separate Feature Gaussians for 3D feature aggregation and Render Gaussians for scene rendering provides a coherent framework. Retaining local features is well motivated, while layered depths and learnable offsets provide additional geometric flexibility.

The experimental results are promising. The complete system achieves 28.803 dB PSNR on nuScenes, outperforming DrivingForward by 7.136 dB, and improves PSNR by 4.611 dB in zero-shot transfer to PandaSet. The retrained ablations and regional analyses also provide evidence for the effectiveness of feature aggregation, layered geometry, and local offsets, helping assess the contributions of the proposed components.

Critical weaknesses

Although Appendix C.4 reports GPU latency and memory measurements, Ours and DrivingForward use different timing scopes, TF32 settings, and profiling sample counts, limiting direct comparison. The total training time and GPU-hours are also not reported. Given the depth estimation, multi-scale feature projection, and two Gaussian representations, a more complete resource analysis is needed to assess practical efficiency.

The connection between the training objective and the claimed geometric behavior is insufficiently explained. The method uses image reconstruction losses without additional depth supervision or offset regularization. It remains unclear whether the second Gaussian layer represents meaningful hidden surfaces and whether unconstrained offsets produce stable, geometrically consistent structures.

The contribution of cross-camera information is not independently established. The aggregation ablation removes both same-camera and other-camera projections, making it difficult to distinguish gains from reorganizing local features from those provided by additional camera observations.

The main performance comparison involves several confounding factors. Ours and DrivingForward differ in depth systems, Gaussian layer counts, and training configurations, making it difficult to attribute the overall improvement specifically to the proposed architectural design.

Questions for authors

Can the authors compare training and inference costs under the same hardware, numerical precision, and timing scope? Please report GPU models and counts, total training time, GPU-hours, inference latency, and peak memory, specifying whether depth estimation and feature projection are included.

Can the authors visualize the contribution of the second Gaussian layer and report distributions of depth increments, opacity, and position offsets? These diagnostics would help establish whether image supervision alone learns meaningful layered geometry.

Can the authors add a same-camera-only projection baseline to independently quantify the improvement provided by other-camera observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.