acceptodds
Withdrawn from ICLR 2027

Beyond the Visible Surface: Geometry-Guided Joint Optimization of Multi-Layer Gaussians for Single-View Scene Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward Gaussian reconstruction predicts a renderable 3D scene from a single image, yet reconstructing content beyond visible surfaces remains challenging. Limited hypotheses along each source ray restrict the representation of newly exposed regions, while depth-based initialization alone provides insufficient geometric guidance for feature learning. We propose GeoJMG, a geometry-guided framework for jointly learning multi-layer Gaussian representations for single-view scene reconstruction. Ray-Stratified Gaussian Modeling (RSGM) constructs multiple depth-ordered surface hypotheses along each source ray and predicts their Gaussian parameters through independent decoder branches over shared multi-scale features. All layers are trained jointly under a common reconstruction objective without predefined visibility roles or layer-specific supervision. To guide this learning process, we derive a compact geometric descriptor from depth, camera-space coordinates, depth gradients, surface normals, and reliability. A lightweight Geometry-Conditioned Residual Adapter (GCRA) integrates these cues into multiple encoder scales through zero-initialized projections, preserving the base features at initialization. The resulting model reconstructs a layered Gaussian scene in a single forward pass without per-scene optimization. Experiments on RealEstate10K, NYUv2, and KITTI demonstrate improved novel-view synthesis quality and effective generalization across indoor and outdoor scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Single-view scene reconstruction, Novel-view synthesis, 3D Gaussian splatting, Layered Gaussian representations, Geometric conditioning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oC5nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents GeoJMG, a feed-forward framework for single-view scene reconstruction. GeoJMG extends Flash3D’s layered Gaussian representation and injects depth-derived geometric features through a multi-scale residual adapter to improve single-view scene reconstruction. Experiments on RealEstate10K, NYUv2, and KITTI report improvements in novel-view synthesis.

Critical strengths

None

Critical weaknesses
  1. Unclear presentation. The paper does not clearly distinguish inherited components from new contributions, making the methodological progression difficult to follow. Table 2 lists “Du et al.” without identifying the method, and several figures and tables require clearer labels and explanations.
  2. Limited novelty and insufficient validation of the central claim. The framework closely follows Flash3D’s depth anchors, independent decoders, and joint rendering-based training. The main additions appear to be increased layer capacity and a geometric residual adapter. Although regional evaluations show modest rendering improvements, they do not establish that the method recovers more accurate hidden-surface geometry. Both the methodological novelty and the claimed advance beyond visible surfaces therefore remain insufficiently substantiated.
Questions for authors
  1. Could the authors clearly distinguish inherited components from new contributions and clarify ambiguous presentation details, including the “Du et al.” entry in Table 2 and the labels and explanations accompanying figures and tables?
  2. What is the methodological novelty of RSGM and joint optimization beyond Flash3D, and what evidence demonstrates improved hidden-surface geometry rather than only improved rendered appearance?
Reviewer_aBNYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents GeoJMG, a feed-forward framework for single-view 3D scene reconstruction using multi-layer 3D Gaussian Splatting. Building directly upon the pixel-aligned, multi-layer Gaussian formulation of Flash3D (which uses a frozen UniDepth model to anchor K=2K=2 Gaussian layers along each camera ray), GeoJMG aims to improve reconstruction quality in disoccluded and newly revealed regions through two main modifications: Ray-Stratified Gaussian Modeling (RSGM):

Expands the number of depth-ordered surface hypotheses per pixel from K=2K=2 to K=4K=4 (predicting cumulative positive log-depth increments ΔDk\Delta D_k and predicting per-layer Gaussian parameters via KK independent decoder heads). All layers are optimized jointly via standard end-to-end rendering loss without explicit visibility masks or layer-specific supervision.

Geometry-Conditioned Residual Adapter (GCRA): Analytically derives an 11-channel geometric descriptor directly from the frozen UniDepth map DsD_s and camera intrinsics KK—comprising normalized linear depth, log-depth, 3D camera coordinates, finite-difference depth gradients, cross-product surface normals, and a gradient-based smoothness/reliability score. This descriptor is processed by two 3×33\times 3 convolutional layers, bilinearly resized, and injected into the 5 scales of the ResNet-50 encoder pyramid via zero-initialized 1×11\times 1 convolutions.

Critical strengths

The experimental execution is unusually thorough and honest. Rather than simply showing that K=4K=4 layers beat K=2K=2 layers, the authors conduct parameter-matched and FLOP-matched ablations (Table 5: widening the K=2K=2 decoder to ~70.7M parameters vs. a shared K=4K=4 decoder vs. K=4K=4 independent decoders) to prove that the gain comes from ray-stratified multi-surface capacity rather than raw parameter count. Furthermore, the visibility-partitioned evaluation with 10,000-sample scene-bootstrap confidence intervals (Appendix C, Tables 9–10) and computational overhead profiling (Appendix B, Table 8) are exemplary practices that substantiate every empirical claim.

Critical weaknesses

Architecturally, GeoJMG is a relatively straightforward extension of Flash3D (Szymanowicz et al., 2025). Flash3D already introduced the core paradigm used here: feeding UniDepth into a ResNet-50 encoder, predicting cumulative positive depth offsets (ΔDk\Delta D_k) along source rays, and using independent decoders to predict multi-layer pixel-aligned Gaussians optimized purely via joint rendering loss (with K=2K=2). Consequently, "Ray-Stratified Gaussian Modeling (RSGM)" is essentially scaling Flash3D's layer hyperparameter from K=2K=2 to K=4K=4. While GCRA adds a well-engineered multi-scale feature adapter using standard analytical derivatives of the depth map (XYZXYZ, ∇D\nabla D, normals) and zero-init 1×11\times 1 convolutions (ControlNet-style), the overall algorithmic contribution feels more like a well-executed engineering refinement of Flash3D than a fundamentally new paradigm.

On KITTI zero-shot transfer (Table 3), GeoJMG achieves the exact same PSNR as Flash3D (21.96 dB) and only marginal changes in SSIM (+0.001) and LPIPS (-0.004), despite having 34.6% more parameters (70.8M vs. 52.6M), 53% more FLOPs (39.0G vs. 25.5G), and twice as many Gaussians to rasterize (4HW4HW vs. 2HW2HW).

In the layer-removal analysis (Section 4.4, Table 7, Figure 4), layers L2L_2 and L3L_3 are always grouped and removed together (L2-3L_{2\text{-}3}), whereas L1L_1 and L4L_4 are removed individually. Removing two layers simultaneously naturally causes a much larger PSNR drop (3.25–5.27 dB) than removing a single layer (~0.86–1.39 dB), making the comparison asymmetric and leaving it unclear how L2L_2 and L3L_3 differentiate from one another individually.

Questions for authors

1.In Table 7 and Figures 4–5, why were L2L_2 and L3L_3 merged into a single group (L2-3L_{2\text{-}3}) during the removal sensitivity analysis rather than evaluated individually (L1,L2,L3,L4L_1, L_2, L_3, L_4)? What are the individual PSNR drops and regional contrasts when removing only L2L_2 versus only L3L_3, and how do their learned depth increments (ΔD2\Delta D_2 vs. ΔD3\Delta D_3) and opacities differ statistically?

2.In Table 3, GeoJMG shows strong zero-shot gains on NYUv2 (+0.37 dB over Flash3D) but no PSNR gain on KITTI (21.96 dB for both). Is this because KITTI's forward-facing driving motion exhibits different disocclusion patterns, or because GCRA's fixed depth normalization constants (d0=20 m,τd=τx=5d_0 = 20\text{ m}, \tau_d = \tau_x = 5) are tuned for indoor/RealEstate10K scales and saturate on outdoor driving depths?

3.In Table 1, Niagara slightly outperforms Flash3D across all three standard RealEstate10K settings. However, in Table 2 (Interpolation/Extrapolation), Niagara drops by over 2.1 dB below Flash3D (21.73 vs. 23.87 dB). Could you clarify why Niagara degrades so sharply under the 256×256256\times 256 interpolation/extrapolation protocol?

Reviewer_YWE8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GeoJMG is a feed-forward, per-scene-optimization-free method for single-view scene reconstruction. It assigns K depth-ordered surface hypotheses per ray (RSGM) with independently predicted Gaussian parameters, and injects depth-derived geometric cues (depth, camera-space coordinates, gradients, normals, reliability) into the encoder through a zero-initialized residual adapter (GCRA). All layers are trained jointly under a single reconstruction objective. On RealEstate10K it outperforms all compared single-view SOTA across 5-frame/10-frame/random-offset, interpolation, and extrapolation, and transfers to NYUv2/KITTI with gains mainly in SSIM/LPIPS.

Critical strengths
  1. The manuscript is well-written and easy to follow.
  2. The method design is reasonable and well-motivated.
Critical weaknesses
  1. The method is relatively simple and several design choices are confusing. (A) Given the estimated monocular depth, one could just anchor Gaussians at the point map and render directly; the added value of the multi-layer-hypothesis network and the independent per-layer decoders over such a simple depth-anchored Gaussian map is not clearly justified. (B) The only geometric information source is the single-view depth, and the network exhibits no explicit generative capacity: it is unclear how ray-stratified sampling alone can recover content behind occluded or newly exposed regions beyond re-rendering the depth-derived surface.
  2. A representative SOTA is missing from both the experiments and the related work: LVSM [1]. LVSM is also a feed-forward method in this field.
  3. Reconstruction quality is claimed but not directly evaluated. The paper reports only rendered-image metrics (PSNR/SSIM/LPIPS) and asserts that reconstruction quality is good. Direct reconstruction/geometry evaluation, e.g., depth accuracy of the reconstructed Gaussians against ground truth, is requested to substantiate this claim, especially given the emphasis on recovering content beyond visible surfaces.
Questions for authors
  1. Can LVSM [1] be added to the comparisons and related work?
  2. Can direct geometry evaluation (e.g., depth accuracy of the reconstructed Gaussians vs. GT) be provided?
  3. What is the actual gain of RSGM/GCRA over a simple depth-anchored Gaussian baseline?
  4. Given only single-view depth and ray-stratified sampling, how is content in occluded/newly exposed regions recovered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.