acceptodds
Under review as a conference paper at ICLR 2027

StereoGaussians: Feed-Forward 3D Gaussian Splatting from Stereo Images

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) enables reconstruction without per- scene optimisation, but practical stereo-camera applications require nearby-view extrapolation beyond the input views. Stereo depth anchors visible surfaces, yet rendering newly exposed regions also requires learned appearance and additional scene capacity. We introduce StereoGaussians, which predicts a metric 3DGS representation from a single calibrated stereo pair. It reuses intermediate repre- sentations from frozen pretrained stereo networks to predict Gaussian attributes, while calibrated disparity anchors the geometry. A second Gaussian layer and an expanded image canvas provide capacity for disoccluded and outside-field-of- view content. For training, we construct SceneSplat-Stereo from quality-filtered 3DGS teachers, pairing stereo inputs with nearby target views across 803 training scenes. Experiments on unseen real and photorealistic stereo benchmarks demon- strate improvements over strong view-synthesis baselines, while ablation studies support our main design choices.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Stereo Matching, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Luj1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper estimates 3DGS from stereo image pairs.

To train the model, the authors curate a stereo dataset from SceneSplat's ScanNet subsets through calibrated 3DGS rendering of nearby stereo views. It also samples more target views for further supervision.

The authors then learn a network to regress Gaussians from multi-scale features, RGB, and stereo depth in a residual fashion. It combines the FoundationStereo backbone and the DPT-style decoder inspired by SHARP. Several amendments are applied to extend FoV and background fidelity.

The zero-shot evaluation on StereoNVS and ReplicaGS is comparable to baselines.

Critical strengths
  • The paper defines an approach to curate stereo datasets from mutliview / 3DGS datasets.
  • The paper enhances FoV for better outpainting.
Critical weaknesses
  • The approach is a bit bland. It replaces SHARP's monocular depth backbone with a stereo depth backbone and did minor surgery.
  • The dataset's scope is narrow. Although holding multiple frames, the diversity is limited. The ScanNetGS and ScanNetPPGS subsets focus on small rooms, mostly offices, conference rooms, and hotel rooms. The curation lacks large scale indoor scenes like cathedral and ballrooms in TanksAndTemples, and lacks outdoor scenes in general.
  • Similarly, the "zero shot" evaluations are on synthetic data with the similar modality: indoor, small scale, and less vibrant.

If the paper extends / enhances SHARP with stereo input, it should highlight:

  • exploit its power of more accurate metric depth at larger viewing angles (to the extend where we clearly see distortions of SHARP)
  • showcase it's power in accurate detail, high-fidelity close-ups, and faraway contents in the wild
  • preserve / enhance the ability to model view-dependent effects given the two layer representation AND enhanced spherical harmonics representation
Questions for authors
  • Could you try to show case results on high resolution Middlebury 2014 subset at larger viewpoints?
  • Similarly, could you try to evaluate on ETH3D/KITTI stereo benchmark as applications to real-world outdoor scenes?
  • Could you please prepare video supplement in SHARP's demo fashion to show case it's ability to inpainting at a reasonable resolution?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Luj1

Thank you for the feedback on the method’s contribution and the scope of the evaluation. We address the concern in turn.

W1 — Transferring stereo representations to Gaussian prediction

StereoGaussians adopts SHARP’s residual attribute composer. To test whether switching to stereo depth alone could account for our gains, we replaced SHARP’s front-layer monocular depth with calibrated stereo depth at inference. We used the stereo depth for both unprojection and attribute decoding and kept SHARP’s released weights fixed.

MethodStereoNVS PSNR ↑ / SSIM ↑ / LPIPS ↓ReplicaGS PSNR ↑ / SSIM ↑ / LPIPS ↓
SHARP, released18.85 / 0.614 / 0.26023.39 / 0.788 / 0.210
SHARP + stereo depth20.59 / 0.718 / 0.22623.84 / 0.802 / 0.214
StereoGaussians without descriptor (Table 4A)20.75 / 0.742 / 0.18728.28 / 0.936 / 0.105
StereoGaussians21.26 / 0.755 / 0.16530.27 / 0.945 / 0.082

Stereo depth improves SHARP’s PSNR on both benchmarks. With the same depth source, StereoGaussians achieves 0.67 dB higher PSNR on StereoNVS and 6.43 dB higher PSNR on ReplicaGS, with higher SSIM and lower LPIPS on both.

StereoGaussians follows SHARP, DepthSplat, MVSplat and Flash3D in using estimated depth to place Gaussians in 3D. However, our research question in this work is whether intermediate features learned for stereo matching can also guide Gaussian attribute prediction. Our decoder reuses these features from a frozen stereo network. To supervise its Gaussian predictions, we built SceneSplat-Stereo, which pairs calibrated stereo inputs with novel-view targets. Fine-tuning DepthSplat on these data improves its zero-shot PSNR on both benchmarks (Sec. 5.3, Table 3, and our response to g86s, W2/Q3). Sec. 5.4 (Table 4A) isolates the contribution of the stereo descriptor. With RGB and stereo depth held fixed, removing the stereo descriptor lowers PSNR by 0.51 dB on StereoNVS and 1.98 dB on ReplicaGS.

W2 / W3 — Training diversity and zero-shot generalisation

We agree that SceneSplat-Stereo primarily covers small indoor rooms from ScanNet and ScanNet++ (Sec. 3 and Appendix B). Appendix C clarifies that the held-out ReplicaGS scenes share our training renderer and camera-sampling protocol. StereoNVS evaluates zero-shot transfer to real photographs captured with a stereo rig in eight unseen scenes. Appendix J extends the evaluation to outdoor scenes.

Q1 / Q2 — Evaluation on ETH3D and KITTI

ETH3D. We rectified 17 stereo pairs from six outdoor scenes and used 52 held-out photographs as novel-view targets. We computed PSNR over pixels with valid laser depth inside the reference field of view. StereoGaussians achieved 21.90 dB, compared with 15.32 dB for SHARP, with higher PSNR in all six scenes (Appendix J). Figure 14 shows our renderings with held-out photographs from courtyard and electro.

KITTI. Appendix J (Fig. 15) shows novel views of two KITTI street scenes rendered after moving the camera 2 m forward.

Larger camera displacements on StereoNVS. To address the request for larger viewpoint changes using real target images, we compared methods across camera displacements. Within each scene, we ranked target views by their distance from the input camera and split them into three groups (near, mid and far). Appendix H.1 (Table 14) reports mean PSNR (dB) for each group.

MethodNearMidFar
StereoGaussians22.4921.0520.21
Flash3D20.5619.5818.91
LVSM20.1319.8819.37
SHARP (mono)19.6318.6718.22
DepthSplat (RE10K)18.7017.9017.32

StereoGaussians achieves higher mean PSNR than all four baselines in each displacement group. In the farthest group, it leads SHARP by 1.99 dB and the strongest baseline, LVSM, by 0.84 dB. The lead over LVSM narrows with displacement, and our model leads it in five of the eight scenes in the farthest group.

We hope these additions address the reviewer's concerns, and would be glad to clarify anything further.

Thank you for the additional experiments and detailed explanations.

  • For Figure 14 and 15, would you please add SHARP rendering for visual comparison? Would you please clarify the resolution at inference time?
  • As noted in my previous comment, I wonder if it would be feasible for you to evaluate on Middlebury and/or prepare video supplements with spiral motion to showcase the 3DGS rendering comparison between SHARP and StereoGaussians?

Thank you!

the strongest baseline, LVSM

I believe LagerNVS has replaced LVSM as the new SOTA. However, it was my fault for bringing this up late, so I would not ask for new experiments. It would not affect my final grading. This is only for your information.

My final grading depends on the two requests I posted above. Thank you.

Thank you for the follow-up and for pointing us to LagerNVS. We have added SHARP visual comparisons for Figures 14 and 15, prepared video comparisons on Middlebury, StereoNVS and ReplicaGS, and evaluated LagerNVS on both benchmarks. The new qualitative figures are collected in a separate PDF (StereoGaussians_Qualitative_Results_for_Luj1.pdf), accompanied by video comparisons in the supplementary files for this follow-up. Given the limited time, these additions have yet to be integrated into the manuscript and its existing supplement; we will incorporate them in a future revision.

SHARP comparisons and inference resolution.

Pages 3–4 of the additional PDF show SHARP renderings alongside StereoGaussians for the ETH3D and KITTI examples corresponding to Figures 14 and 15. The inference input resolution is 256 × 256 for ETH3D and 256 × 896 for KITTI. Both methods produce plausible outdoor renderings. In the ETH3D examples, StereoGaussians preserves scene structure that aligns more closely with the target images, particularly around the doorway in the Electro scene. This is consistent with the geometric grounding provided by calibrated, metric-scale stereo depth.


Video comparisons

We provide comparisons with SHARP on Middlebury, StereoNVS and ReplicaGS, including results at high inference resolutions. The videos show that StereoGaussians can produce visually detailed novel-view renderings at high resolution, preserving scene structure and appearance as the camera moves.

We observe some artifacts near foreground boundaries, which may arise from stereo-depth errors at discontinuities that propagate through our depth-anchored Gaussian representation. Boundary-aware depth refinement or constrained Gaussian displacements are potential remedies whose effectiveness we leave for future investigation. Improving the underlying stereo estimates at depth discontinuities is beyond the scope of this work.


Additional LagerNVS evaluation.

We evaluated the released RE10K-trained LagerNVS checkpoint under our Table 2 zero-shot protocol, consistent with our use of released RE10K checkpoints for the other applicable baselines. The first table reports overall results, and the second compares per-scene PSNR with our default 16-pixel-padding model.

MethodStereoNVS PSNR ↑ / SSIM ↑ / LPIPS ↓ReplicaGS PSNR ↑ / SSIM ↑ / LPIPS ↓
LagerNVS17.48 / 0.5280 / 0.294625.28 / 0.8211 / 0.1554
StereoGaussians21.26 / 0.7549 / 0.165330.27 / 0.9447 / 0.0818

StereoNVS averages over all 224 target views. ReplicaGS averages over 8,000 target views, with 1,000 per scene.

BenchmarkSceneLagerNVS PSNR ↑StereoGaussians PSNR ↑
StereoNVSscene321.1921.91
StereoNVSscene1118.5424.77
StereoNVSscene1315.2714.23
StereoNVSscene2515.9920.89
StereoNVSscene2619.7723.84
StereoNVSscene3014.2215.85
StereoNVSscene3820.1725.85
StereoNVSscene4014.9623.98
ReplicaGSoffice027.3432.31
ReplicaGSoffice128.7034.04
ReplicaGSoffice224.0230.00
ReplicaGSoffice325.1430.59
ReplicaGSoffice424.4529.17
ReplicaGSroom024.9230.31
ReplicaGSroom123.9427.86
ReplicaGSroom223.7727.83

StereoGaussians achieves higher overall PSNR and SSIM and lower LPIPS on both benchmarks. Its overall PSNR gains are approximately 3.78 dB on StereoNVS and 4.99 dB on ReplicaGS, with higher per-scene PSNR in seven of eight StereoNVS scenes and all eight ReplicaGS scenes. Pages 1–2 of the additional PDF show LagerNVS renderings alongside the input and target images.

We hope these additional comparisons and clarifications address your remaining concerns and provide further evidence for your assessment of our work.

Reviewer_AYshOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StereoGaussians predicts a renderable, metric-scale 3D Gaussian scene from one calibrated stereo pair. A frozen stereo network supplies disparity and intermediate features; a trainable decoder and attribute predictor combine these with RGB-D inputs. The front Gaussian layer is anchored to stereo depth, while a deeper second layer and an expanded image canvas provide additional rendering support. The authors construct SceneSplat-Stereo from quality-filtered real-scene 3DGS teachers, using 803 training scenes and approximately 324K stereo pairs. Experiments evaluate zero-shot transfer to captured StereoNVS images and teacher-rendered ReplicaGS targets, with comparisons to released models and controlled architectural/data ablations.

Critical strengths
  1. In Table 2 (p. 6), the default model leads the evaluated methods on all three metrics on both benchmarks. Relative to LVSM, PSNR improves by 1.465 dB on StereoNVS and 4.494 dB on ReplicaGS. StereoNVS LPIPS is 0.1653, versus 0.2024 for DepthSplat and 0.2460 for LVSM. The captured-image results provide important evidence beyond matching teacher-rendered targets, and the qualitative comparisons show plausible improvements in boundaries and scene coverage.

  2. Tables 3-4 compare training sources and vary input conditioning, the frozen stereo encoder, the second layer, composition, and padding under a stated common schedule. Removing the descriptor while retaining RGB and stereo geometry reduces PSNR by 0.509/1.982 dB on StereoNVS/ReplicaGS and worsens LPIPS on both. The separately trained Fast-FoundationStereo variant also performs well. The scene-disjoint teacher-data construction is detailed enough to understand the supervision, and the authors report cases where an ablated model improves a metric rather than claiming uniform superiority of every choice.

Critical weaknesses
  1. Table 2 and Sec. 5.1 (p. 6, lines 307-313) retain each baseline's original pretraining, while the proposed model benefits from a new, task-aligned stereo-to-target dataset. Table 3 also changes source scale, camera/target distributions, appearance, and—in IRS—the recovered camera poses. These are useful practical comparisons, but do not isolate the proposed architecture from the training-data advantage or identify which property of the new data is decisive. The existing controlled descriptor ablation partly addresses this and is important to the positive assessment. Requested improvement: keep the headline claim explicitly scoped to the evaluated zero-shot systems, provide complete checkpoint/preprocessing details, and report scene-level results or uncertainty. Training a strong baseline on the same data would strengthen attribution, but I do not make that additional training a condition for acceptance of the narrower system/data contribution.

  2. Sec. 5.5 and Table 4 (pp. 8-9) show substantial benefits from expanded support, but aggregate image scores do not identify how much improvement occurs in mutually visible, disoccluded, or outside-FoV regions. Padding also changes the Gaussian count: at 256-square resolution, two layers produce 131,072 Gaussians without padding and 165,888 with the default padding, an increase of about 26.6%. The back layer's PSNR gain is 0.180 dB on captured StereoNVS versus 3.826 dB on ReplicaGS, making this distinction particularly relevant to real-image claims. The authors already recognize region-specific evaluation as a limitation. Requested improvement: state clearly which conclusions are whole-image rendering results and, preferably, report visibility-region metrics for the existing default/no-back-layer/no-padding outputs. A count-matched coverage control would further clarify the mechanism. These limits temper the interpretation rather than negate the demonstrated rendering gains.

Questions for authors
  1. Can the authors provide per-scene Table 2 results and the main descriptor/back-layer/padding differences, or paired scene-level uncertainty? Each benchmark has eight scenes, so thousands of target images should not be treated as independent evidence of scene generalization. These results would help assess whether the gains are broad or concentrated.

  2. How do the model and key ablations perform separately on mutually visible, disoccluded, and outside-reference-FoV pixels? Does the second layer improve disoccluded-region quality on captured images despite its small aggregate StereoNVS PSNR gain? This would most directly support the paper's geometric interpretation.

  3. Please identify the exact baseline checkpoints and image/intrinsic preprocessing, including the FS-naive-GS settings and mask implementation for PSNR/SSIM/LPIPS. What are end-to-end prediction time, subsequent rendering time, and peak memory for the default and higher-resolution variants on stated hardware? A code/data release plan with scene manifests and camera-generation settings would also improve reuse. Feed-forward prediction alone does not quantify these costs.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AYsh - Part 1

Thank you for the constructive suggestions. We address the concerns in turn.

W1 — Scope of the headline claim

We agree that Table 2 compares complete zero-shot systems. The final sentence of the Abstract and the third contribution in Sec. 1 now limit our claim to the zero-shot systems evaluated on StereoNVS and ReplicaGS.

W1 — Reproducibility

Appendix A describes the training setup, and Appendix B details scene selection, camera sampling and dataset generation. Our response to Q3 below gives the baseline and evaluation settings and our code release plan.

W1 / Q1 — Are the gains consistent across scenes?

Each benchmark has eight scenes. We computed the PSNR difference within each scene and gave all eight scenes equal weight. Each cell reports the mean paired difference in dB, a descriptive 95% bootstrap interval over scenes, and the number of scenes in which our default model has higher PSNR.

Ours (16 px) vsStereoNVSReplicaGS
w/o descriptor+0.51 [+0.37, +0.64] (8/8)+1.98 [+1.66, +2.30] (8/8)
w/o back layer+0.18 [+0.08, +0.29] (7/8)+3.83 [+2.81, +4.83] (8/8)
w/o padding (same-base retrain)+3.11 [+1.69, +4.76] (8/8)+0.80 [+0.69, +0.92] (8/8)
SHARP-mono+2.55 [+0.93, +4.42] (7/8)+6.88 [+6.07, +7.82] (8/8)
LVSM+1.62 [−0.27, +3.52] (5/8)+4.49 [+3.93, +5.01] (8/8)

On ReplicaGS, the default model has higher PSNR in all eight scenes for every comparison shown. On StereoNVS, the descriptor and padding improve PSNR in 8/8 scenes and the back layer in 7/8. Against LVSM, our model has higher PSNR in 5/8 StereoNVS scenes, and the interval for the mean difference includes zero. Appendix H shows the distribution of paired gains (Fig. 13), the paired statistics (Table 11) and the per-scene scores on each benchmark (Tables 12–13).

W1 — Strong baseline on the same training data

Fine-tuning the released DepthSplat checkpoint on SceneSplat-Stereo improves zero-shot PSNR by 3.2 dB on StereoNVS and 12.6 dB on ReplicaGS. Sec. 5.3 and Table 3 compare the released and fine-tuned checkpoints directly. Appendix D describes the fine-tuning setup, also summarised in our response to g86s (W2/Q3).

W2 / Q2 — Whole-image and visibility-region results on ReplicaGS

Appendix I.1 (Table 15) reports the corresponding region-wise gains on ReplicaGS. On ReplicaGS, the teacher 3DGS provides rendered depth for both the reference and target cameras, allowing us to estimate visibility. We back-project each valid target pixel and project it into the reference crop. Pixels projecting outside the crop are outside-FoV. Inside the crop, we label a pixel disoccluded if its projected depth lies behind the rendered reference surface by more than 3% of the reference depth plus 2 cm. The remaining pixels are mutually visible.

Full StereoGaussians vs.Mutually visible ΔPSNRDisoccluded ΔPSNROutside-FoV ΔPSNR
w/o back layer+1.11 (8/8)+2.67 [+1.74, +3.54] (8/8)+4.05 [+2.92, +5.20] (8/8)
w/o padding−0.03 (1/8)+0.26 (7/8)+1.51 [+1.31, +1.70] (8/8)
w/o descriptor+0.57 (8/8)+2.93 (8/8)+2.01 (8/8)

On ReplicaGS, all three components improve PSNR outside the input FoV (+1.51 to +4.05 dB). The back layer also improves PSNR in disoccluded regions in all eight scenes, with a mean gain of 2.67 dB. Padding extends the canvas and improves PSNR primarily outside the input FoV (+1.51 dB), with little change in mutually visible regions (−0.03 dB). The stereo descriptor improves PSNR in all three regions across all eight scenes, with its largest mean gain in disocclusions (+2.93 dB).

W2 — Does padding help through coverage or Gaussian count?

While preparing the additional padding control, we found that the “without padding” row in Table 4(D) used an earlier training configuration. We retrained the unpadded model with the full model’s configuration and corrected Table 4(D) and the accompanying text in Sec. 5.5. The main correction is on ReplicaGS, where unpadded PSNR rises from 21.534 to 29.462 dB. With this matched baseline, 16 px padding improves PSNR by 3.223 dB on StereoNVS and 0.804 dB on ReplicaGS.

To separate coverage from Gaussian count, we trained a control with the same 165,888 Gaussians as the padded model, placing them on a denser grid over the original FoV. Appendix I.2 (Table 16) reports all three configurations under the same training recipe.

ModelGaussiansStereoNVS PSNR (dB)ReplicaGS PSNR (dB)
No padding131,07218.0429.46
Denser grid, same FoV165,88817.9929.47
16 px padding, expanded FoV165,88821.2630.27

Increasing density within the original FoV leaves PSNR nearly unchanged on both benchmarks. At the same Gaussian count, extending coverage raises mean PSNR from 17.99 to 21.26 dB on StereoNVS and from 29.47 to 30.27 dB on ReplicaGS.

[continued in the next comment]

AuthorsOpenReview ↗

Response to Reviewer AYsh - Part 2

Q3 — Baseline checkpoints and evaluation protocol

The baseline checkpoints and evaluation protocol is included in Appendix D.

Q3 — Prediction time, rendering time and peak memory

Appendix E.1 (Table 10) reports these measurements. We measured the time from a stereo pair to a Gaussian scene, then the time to render one additional view from that scene. On one NVIDIA H200 at batch size 1, we report median CUDA-event timings over 50 runs after 10 warm-ups. Inference used FP32, with FP16 autocast in the stereo backbone (32 GRU iterations).

ModelPrediction (ms)Rendering per view (ms)Peak allocated / reserved GPU memory (GiB)
256², pad 16 (default)1921.61.9 / 2.1
256², pad 641991.62.2 / 2.4
512²2752.03.3 / 4.7
1024²7343.98.6 / 14.3

Q3 — Code and data release plan

Upon acceptance, we will release the code and configurations for data generation and preprocessing, training, inference and evaluation, including the baseline adapters. The release will include SceneSplat-Stereo train/validation/test scene manifests, camera-generation settings and scripts, and teacher-filtering scripts so researchers can regenerate the dataset from SceneSplat-7K and reproduce the training and evaluation setup.

We hope these additions address the reviewer's concerns, and would be glad to clarify anything further.

Reviewer_g86sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a feed-forward 3D Gaussian Splatting prediction model from a calibrated stereo pair. The main idea is to reuse frozen stereo features from a pretrained foundation stereo model, use calibrated metric disparity as a geometric anchor, and predict Gaussian attributes on top of them. This paper also proposes SceneSplat-Stereo, a dataset generated from quality-filtered 3DGS teachers to provide calibrated stereo pairs together with nearby target views. Experiments on real and rendered benchmarks demonstrate state-of-the-art quantitative performance against existing baselines.

Critical strengths
  • The proposed framework that reuses frozen foundation stereo features paired with a lightweight prediction head trained in only 30K steps is computationally practical and effective for capturing metric geometry.
  • The proposed training dataset fills a meaningful training data gap by providing calibrated stereo pairs together with nearby target views rendered from quality-filtered 3DGS teachers, enabling consistent supervision for nearby-view extrapolation.
Critical weaknesses
  • The novelty of the proposed architecture is limited, as the core mechanisms rely on combining existing work. The two-layer Gaussian representation and the residual attribute composition pipeline are directly adapted from SHARP, while the depth-offset back layer and out-of-FoV border padding were already introduced in Flash3D.
  • Dataset and architectural contributions are not separated in Table 2. Comparing the full pipeline (StereoGaussians trained on SceneSplat-Stereo) against off-the-shelf pretrained baselines in Table 2 is reasonable as an end-to-end system check, but the paper claims the SceneSplat-Stereo dataset (Contribution 1) and the StereoGaussians model (Contribution 2) as separate contributions. Table 3 varies the training dataset on StereoGaussians and Table 4 ablates internal variants, however, the StereoGaussians architecture is never compared against any external baseline trained on the same dataset, e.g., LVSM or DepthSplat trained on SceneSplat-Stereo. Furthermore, ReplicaGS is generated using the same gsplat renderer and the same zero-rotation spherical target offsets used to create SceneSplat-Stereo, according to the dataset generation details, so ReplicaGS closely matches StereoGaussians' training distribution while remaining out-of-distribution for the off-the-shelf baselines. Training existing baselines on SceneSplat-Stereo is needed both for an apples-to-apples model comparison and to show that SceneSplat-Stereo benefits other methods too.
  • 3DGS baseline selection in Table 2 is not comprehensive, missing Flash3D. None of the 3DGS baselines evaluated in Table 2 combine metric depth with out-of-FoV extrapolation capacity. MVSplat and DepthSplat lock Gaussians to unpadded input rays and suffer severe black-border penalties during extrapolation, while SHARP relies only on monocular scale guesses without stereo depth. Meanwhile, Flash3D already combines a frozen metric depth input, two-layer depth-offset Gaussians, and out-of-FoV border padding, but it is omitted from Table 2.
Questions for authors
  • Could you clarify the architectural differences between the proposed depth-offset back layer and out-of-FoV border padding vs Flash3D, and add a proper citation to Flash3D in Section 4.4?
  • Could you evaluate Flash3D in Table 2 on StereoNVS and ReplicaGS, both using the released pretrained weights and when replacing the monocular depth input with FoundationStereo depth?
  • To separate the model contribution of StereoGaussians from the dataset contribution of SceneSplat-Stereo, and to show that SceneSplat-Stereo also improves existing methods, Could you train or fine-tune baselines such as LVSM and DepthSplat on SceneSplat-Stereo under the same setup, both to separate the model contribution from the dataset contribution and to verify that SceneSplat-Stereo also improves existing methods?
1 comment
AuthorsOpenReview ↗

Response to Reviewer g86s

We thank the reviewer for the detailed comments. We address the concerns in turn.

W1 / Q1 — Novelty relative to SHARP and Flash3D, and the Flash3D citation

In the revised manuscript, we discuss Flash3D in Sec. 4.4 and include its results in Table 2. Flash3D, SHARP and StereoGaussians share a layered, pixel-wise Gaussian representation.

Design choiceFlash3DSHARPStereoGaussians
InputSingle imageSingle imageStereo image
Geometry sourceFrozen monocular metric depth + learned front-layer offsetLearned monocular metric depthStereo depth
Second layerLearned absolute depth offset from the front layerSecond learned depth layerAnchored at local background depth
Out-of-FoV supportZero-pads the input image before the encoderN/APads the decoded canvas
Features for attributesTrained ResNet-50 on RGB and depthSHARP's trained encoderFrozen stereo network's intermediate features, decoded by a DPT with RGB-D injected at every scale
Attribute parameterisationDirect predictionResidual attribute composerSHARP's residual attribute composer
Training dataRealEstate10KLarge-scale simulated and real dataSceneSplat-Stereo

Our design supports view extrapolation in four ways. Calibrated stereo depth places the front Gaussians at metric depth. Intermediate stereo features guide the prediction of Gaussian attributes. We adopt SHARP’s residual attribute composer because it reduces Gaussian artifacts at object boundaries. For regions newly exposed by camera motion, we share the two-layer design with Flash3D and SHARP, but anchor our back layer at the local background depth. We also extend coverage beyond the input view by padding the decoded canvas, while Flash3D pads the input image.

Our main research question is whether intermediate stereo features improve novel-view synthesis when calibrated stereo depth already fixes the front-layer geometry. Table 4 (block A, “Without descriptor”) shows that removing the DPT descriptor while retaining RGB and stereo depth lowers PSNR by 0.51 dB on StereoNVS and 1.98 dB on ReplicaGS.

W3 / Q2 — Flash3D in Table 2

We evaluated the released Flash3D checkpoint zero-shot and added the results to Table 2. We also evaluated the same checkpoint with its monocular depth replaced at test time by metric depth from FoundationStereo, using Flash3D’s external-depth input and keeping all weights fixed. Appendix D (Table 8) reports the three-way comparison below, with StereoGaussians evaluated at Flash3D’s 32-pixel border width.

MethodStereoNVS PSNR ↑ / SSIM ↑ / LPIPS ↓ReplicaGS PSNR ↑ / SSIM ↑ / LPIPS ↓
Flash3D19.69 / 0.654 / 0.27724.79 / 0.819 / 0.194
Flash3D + FoundationStereo depth21.82 / 0.726 / 0.24827.97 / 0.916 / 0.152
StereoGaussians22.40 / 0.760 / 0.16430.19 / 0.945 / 0.082

Replacing monocular depth with stereo depth improves Flash3D on all three metrics. With the same depth source and border width, StereoGaussians still achieves higher SSIM and lower LPIPS on both benchmarks. Its mean PSNR advantage is modest on StereoNVS (+0.58 dB) and larger on ReplicaGS (+2.22 dB). Thus, replacing depth alone in the released Flash3D model recovers part of the gap to StereoGaussians.

W2 / Q3 — Separating the dataset from the architecture; same-data baselines

We fine-tuned DepthSplat on SceneSplat-Stereo, as suggested by the reviewer. Like StereoGaussians, it first predicts depth and uses it to place 3D Gaussians. Sec. 5.3 (Table 3) compares the released and fine-tuned checkpoints directly, and Appendix D describes the training setup. Starting from the released RealEstate10K (RE10K) checkpoint, we used the same data split, resolution, training budget and evaluation protocol as StereoGaussians.

Model and trainingStereoNVS PSNR / SSIM / LPIPSReplicaGS PSNR / SSIM / LPIPS
DepthSplat, released RE10K checkpoint17.98 / 0.750 / 0.20216.39 / 0.616 / 0.327
DepthSplat, RE10K → SceneSplat-Stereo21.15 / 0.731 / 0.20629.00 / 0.921 / 0.091
StereoGaussians, SceneSplat-Stereo21.26 / 0.755 / 0.16530.27 / 0.945 / 0.082

Fine-tuning the released checkpoint on SceneSplat-Stereo improves DepthSplat’s zero-shot PSNR by 3.2 dB on StereoNVS and 12.6 dB on ReplicaGS.

We agree with the reviewer that ReplicaGS shares our training renderer and target-camera sampling protocol, even though its scenes are unseen. As detailed in Appendix C, StereoNVS tests zero-shot transfer to photographs captured with a stereo rig in eight unseen scenes. On StereoNVS, StereoGaussians achieves 21.26 dB without target-domain fine-tuning, close to the 21.15 dB of fine-tuned DepthSplat, with higher SSIM (0.755 vs. 0.731) and lower LPIPS (0.165 vs. 0.206).

We hope these additions address the reviewer's concerns, and would be glad to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for highlighting the value of SceneSplat-Stereo (g86s, Luj1), the lightweight head trained on frozen stereo features (g86s), the results on captured images and controlled ablations (AYsh), and the expanded field of view (Luj1). Below, we summarise our findings on the reviewers’ main concerns. Our individual responses provide detailed comparisons and address the reviewers’ remaining questions and concerns.

A frozen stereo network provides both metric depth and features that improve Gaussian attribute prediction. Replacing monocular depth with calibrated stereo depth improves PSNR for both SHARP (response to Luj1, W1) and Flash3D (Appendix D, Table 8). Nevertheless, StereoGaussians still achieves higher PSNR and SSIM and lower LPIPS than these stereo-depth variants on both benchmarks. Our decoder also uses intermediate features from the stereo network. With RGB and stereo depth held fixed, removing these features lowers PSNR by 0.51 dB on StereoNVS and 1.98 dB on ReplicaGS (Table 4A).

SceneSplat-Stereo can also improve the zero-shot performance of other models. Fine-tuning the released DepthSplat checkpoint on our dataset raises its PSNR by 3.2 dB on StereoNVS and 12.6 dB on ReplicaGS (Sec. 5.3, Table 3). While StereoGaussians still outperforms fine-tuned DepthSplat on both benchmarks across all three metrics (PSNR, SSIM and LPIPS). Its mean PSNR is 1.27 dB higher on ReplicaGS and 0.11 dB higher on StereoNVS. On StereoNVS, it also achieves 3.2% higher SSIM and 19.6% lower LPIPS.

Padding helps by extending coverage, and the back layer improves newly visible regions. Adding Gaussians within the original field of view barely changes PSNR. Using the same number to extend the canvas increases PSNR on both benchmarks. On ReplicaGS, the back layer improves PSNR by 2.67 dB in disoccluded regions and 4.05 dB outside the input view (Appendix I, Tables 15–16).

The indoor-trained model transfers to real photographs, including outdoor scenes. ReplicaGS shares our training renderer and camera protocol. On captured StereoNVS images, StereoGaussians leads the four evaluated baselines in mean PSNR at near, middle and far camera displacements (Appendix H.1, Table 14). On six outdoor ETH3D scenes, the indoor-trained model achieves 21.90 dB versus 15.32 dB for SHARP on laser-valid pixels within the reference view (Appendix J).

We limit our headline claim to the evaluated zero-shot systems. Appendix D documents baseline checkpoints and preprocessing, Appendix H gives per-scene results, and Appendix E.1 reports prediction time, rendering time and peak memory. Upon acceptance, we will release the data generation, training and evaluation code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.