- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper estimates 3DGS from stereo image pairs.
To train the model, the authors curate a stereo dataset from SceneSplat's ScanNet subsets through calibrated 3DGS rendering of nearby stereo views. It also samples more target views for further supervision.
The authors then learn a network to regress Gaussians from multi-scale features, RGB, and stereo depth in a residual fashion. It combines the FoundationStereo backbone and the DPT-style decoder inspired by SHARP. Several amendments are applied to extend FoV and background fidelity.
The zero-shot evaluation on StereoNVS and ReplicaGS is comparable to baselines.
Critical strengths
- The paper defines an approach to curate stereo datasets from mutliview / 3DGS datasets.
- The paper enhances FoV for better outpainting.
Critical weaknesses
- The approach is a bit bland. It replaces SHARP's monocular depth backbone with a stereo depth backbone and did minor surgery.
- The dataset's scope is narrow. Although holding multiple frames, the diversity is limited. The ScanNetGS and ScanNetPPGS subsets focus on small rooms, mostly offices, conference rooms, and hotel rooms. The curation lacks large scale indoor scenes like cathedral and ballrooms in TanksAndTemples, and lacks outdoor scenes in general.
- Similarly, the "zero shot" evaluations are on synthetic data with the similar modality: indoor, small scale, and less vibrant.
If the paper extends / enhances SHARP with stereo input, it should highlight:
- exploit its power of more accurate metric depth at larger viewing angles (to the extend where we clearly see distortions of SHARP)
- showcase it's power in accurate detail, high-fidelity close-ups, and faraway contents in the wild
- preserve / enhance the ability to model view-dependent effects given the two layer representation AND enhanced spherical harmonics representation
Questions for authors
- Could you try to show case results on high resolution Middlebury 2014 subset at larger viewpoints?
- Similarly, could you try to evaluate on ETH3D/KITTI stereo benchmark as applications to real-world outdoor scenes?
- Could you please prepare video supplement in SHARP's demo fashion to show case it's ability to inpainting at a reasonable resolution?
4 comments
Thank you for the additional experiments and detailed explanations.
- For Figure 14 and 15, would you please add SHARP rendering for visual comparison? Would you please clarify the resolution at inference time?
- As noted in my previous comment, I wonder if it would be feasible for you to evaluate on Middlebury and/or prepare video supplements with spiral motion to showcase the 3DGS rendering comparison between SHARP and StereoGaussians?
Thank you!
the strongest baseline, LVSM
I believe LagerNVS has replaced LVSM as the new SOTA. However, it was my fault for bringing this up late, so I would not ask for new experiments. It would not affect my final grading. This is only for your information.
My final grading depends on the two requests I posted above. Thank you.
Thank you for the follow-up and for pointing us to LagerNVS. We have added SHARP visual comparisons for Figures 14 and 15, prepared video comparisons on Middlebury, StereoNVS and ReplicaGS, and evaluated LagerNVS on both benchmarks. The new qualitative figures are collected in a separate PDF (StereoGaussians_Qualitative_Results_for_Luj1.pdf), accompanied by video comparisons in the supplementary files for this follow-up. Given the limited time, these additions have yet to be integrated into the manuscript and its existing supplement; we will incorporate them in a future revision.
SHARP comparisons and inference resolution.
Pages 3–4 of the additional PDF show SHARP renderings alongside StereoGaussians for the ETH3D and KITTI examples corresponding to Figures 14 and 15. The inference input resolution is 256 × 256 for ETH3D and 256 × 896 for KITTI. Both methods produce plausible outdoor renderings. In the ETH3D examples, StereoGaussians preserves scene structure that aligns more closely with the target images, particularly around the doorway in the Electro scene. This is consistent with the geometric grounding provided by calibrated, metric-scale stereo depth.
Video comparisons
We provide comparisons with SHARP on Middlebury, StereoNVS and ReplicaGS, including results at high inference resolutions. The videos show that StereoGaussians can produce visually detailed novel-view renderings at high resolution, preserving scene structure and appearance as the camera moves.
We observe some artifacts near foreground boundaries, which may arise from stereo-depth errors at discontinuities that propagate through our depth-anchored Gaussian representation. Boundary-aware depth refinement or constrained Gaussian displacements are potential remedies whose effectiveness we leave for future investigation. Improving the underlying stereo estimates at depth discontinuities is beyond the scope of this work.
Additional LagerNVS evaluation.
We evaluated the released RE10K-trained LagerNVS checkpoint under our Table 2 zero-shot protocol, consistent with our use of released RE10K checkpoints for the other applicable baselines. The first table reports overall results, and the second compares per-scene PSNR with our default 16-pixel-padding model.
| Method | StereoNVS PSNR ↑ / SSIM ↑ / LPIPS ↓ | ReplicaGS PSNR ↑ / SSIM ↑ / LPIPS ↓ |
|---|---|---|
| LagerNVS | 17.48 / 0.5280 / 0.2946 | 25.28 / 0.8211 / 0.1554 |
| StereoGaussians | 21.26 / 0.7549 / 0.1653 | 30.27 / 0.9447 / 0.0818 |
StereoNVS averages over all 224 target views. ReplicaGS averages over 8,000 target views, with 1,000 per scene.
| Benchmark | Scene | LagerNVS PSNR ↑ | StereoGaussians PSNR ↑ |
|---|---|---|---|
| StereoNVS | scene3 | 21.19 | 21.91 |
| StereoNVS | scene11 | 18.54 | 24.77 |
| StereoNVS | scene13 | 15.27 | 14.23 |
| StereoNVS | scene25 | 15.99 | 20.89 |
| StereoNVS | scene26 | 19.77 | 23.84 |
| StereoNVS | scene30 | 14.22 | 15.85 |
| StereoNVS | scene38 | 20.17 | 25.85 |
| StereoNVS | scene40 | 14.96 | 23.98 |
| ReplicaGS | office0 | 27.34 | 32.31 |
| ReplicaGS | office1 | 28.70 | 34.04 |
| ReplicaGS | office2 | 24.02 | 30.00 |
| ReplicaGS | office3 | 25.14 | 30.59 |
| ReplicaGS | office4 | 24.45 | 29.17 |
| ReplicaGS | room0 | 24.92 | 30.31 |
| ReplicaGS | room1 | 23.94 | 27.86 |
| ReplicaGS | room2 | 23.77 | 27.83 |
StereoGaussians achieves higher overall PSNR and SSIM and lower LPIPS on both benchmarks. Its overall PSNR gains are approximately 3.78 dB on StereoNVS and 4.99 dB on ReplicaGS, with higher per-scene PSNR in seven of eight StereoNVS scenes and all eight ReplicaGS scenes. Pages 1–2 of the additional PDF show LagerNVS renderings alongside the input and target images.
We hope these additional comparisons and clarifications address your remaining concerns and provide further evidence for your assessment of our work.
Response to Reviewer Luj1
Thank you for the feedback on the method’s contribution and the scope of the evaluation. We address the concern in turn.
W1 — Transferring stereo representations to Gaussian prediction
StereoGaussians adopts SHARP’s residual attribute composer. To test whether switching to stereo depth alone could account for our gains, we replaced SHARP’s front-layer monocular depth with calibrated stereo depth at inference. We used the stereo depth for both unprojection and attribute decoding and kept SHARP’s released weights fixed.
Stereo depth improves SHARP’s PSNR on both benchmarks. With the same depth source, StereoGaussians achieves 0.67 dB higher PSNR on StereoNVS and 6.43 dB higher PSNR on ReplicaGS, with higher SSIM and lower LPIPS on both.
StereoGaussians follows SHARP, DepthSplat, MVSplat and Flash3D in using estimated depth to place Gaussians in 3D. However, our research question in this work is whether intermediate features learned for stereo matching can also guide Gaussian attribute prediction. Our decoder reuses these features from a frozen stereo network. To supervise its Gaussian predictions, we built SceneSplat-Stereo, which pairs calibrated stereo inputs with novel-view targets. Fine-tuning DepthSplat on these data improves its zero-shot PSNR on both benchmarks (Sec. 5.3, Table 3, and our response to g86s, W2/Q3). Sec. 5.4 (Table 4A) isolates the contribution of the stereo descriptor. With RGB and stereo depth held fixed, removing the stereo descriptor lowers PSNR by 0.51 dB on StereoNVS and 1.98 dB on ReplicaGS.
W2 / W3 — Training diversity and zero-shot generalisation
We agree that SceneSplat-Stereo primarily covers small indoor rooms from ScanNet and ScanNet++ (Sec. 3 and Appendix B). Appendix C clarifies that the held-out ReplicaGS scenes share our training renderer and camera-sampling protocol. StereoNVS evaluates zero-shot transfer to real photographs captured with a stereo rig in eight unseen scenes. Appendix J extends the evaluation to outdoor scenes.
Q1 / Q2 — Evaluation on ETH3D and KITTI
ETH3D. We rectified 17 stereo pairs from six outdoor scenes and used 52 held-out photographs as novel-view targets. We computed PSNR over pixels with valid laser depth inside the reference field of view. StereoGaussians achieved 21.90 dB, compared with 15.32 dB for SHARP, with higher PSNR in all six scenes (Appendix J). Figure 14 shows our renderings with held-out photographs from courtyard and electro.
KITTI. Appendix J (Fig. 15) shows novel views of two KITTI street scenes rendered after moving the camera 2 m forward.
Larger camera displacements on StereoNVS. To address the request for larger viewpoint changes using real target images, we compared methods across camera displacements. Within each scene, we ranked target views by their distance from the input camera and split them into three groups (near, mid and far). Appendix H.1 (Table 14) reports mean PSNR (dB) for each group.
StereoGaussians achieves higher mean PSNR than all four baselines in each displacement group. In the farthest group, it leads SHARP by 1.99 dB and the strongest baseline, LVSM, by 0.84 dB. The lead over LVSM narrows with displacement, and our model leads it in five of the eight scenes in the farthest group.
We hope these additions address the reviewer's concerns, and would be glad to clarify anything further.