- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper mainly studies how to reconstruct indoor scenes with highly reflective objects using Gaussian splatting. The paper proposes a method that combines learned uncertainty, visibility filtering, and SAM segmentation to generate masks for reflective metallic objects. These masks supervise the metallic parameters in a subsequent rendering stage, which uses parallax-corrected cubemap sampling to model near-field reflections. The authors also introduce the IndoorRef12 dataset and report improvements on both synthetic and real scenes.
Critical strengths
-
The paper designs a detailed pipeline for metallic object localization and mask generation. The combination of uncertainty estimation, visibility filtering, spatial candidate selection, and SAM segmentation is carefully considered. The experiments show that the generated masks provide useful supervision and improve reconstruction quality in the evaluated scenes.
-
The paper evaluates not only overall image quality but also reflective-region rendering and surface normals. These results help in understanding how the method improves the reconstruction of reflective objects.
-
The paper includes a detailed discussion of its limitations. The authors discuss the ambiguity of uncertainty, inaccurate material boundaries in object-level masks, dependence on COLMAP and SAM, and limitations of a single cubemap in handling complex reflections. They also consider cases without strong reflectors and cases with changing illumination. This analysis helps clarify the conditions under which the method may fail.
Critical weaknesses
-
The introduction states that “most methods assume an infinite distance environment map” and motivates PCC as a way to address near-field reflection shifts. The limitation of direction-only environment maps is reasonable. However, SpecTRe-GS already uses secondary ray tracing to capture nearby reflected objects, and Ref-GS models spatially varying specular appearance. Therefore, near-field reflection modeling has already been considered in prior work. Methods that explicitly trace rays through scene geometry also account for local geometry and visibility more directly than a proxy box, although this does not necessarily mean better reconstruction results. Since PCC itself is an established technique, it remains unclear to me what advantage its use here provides over these existing solutions.
-
Table 4 shows the benefits of UGMS and PCC within the proposed pipeline, while Table 6 compares different mask generation methods. However, these experiments do not sufficiently distinguish whether the improvement over other baselines mainly comes from better masks, stronger metallic supervision, or the reflection rendering method. In particular, it remains unclear whether a ray-tracing-based reflection model, such as those used in SpecTRe-GS or MaterialRefGS, could achieve similar or better results under the same masks, supervision, and otherwise matched settings. If the advantage mainly comes from more accurate masks and stronger supervision, the evidence for the reflection rendering formulation as a separate methodological contribution would be limited. The paper’s contribution would then rest primarily on the mask generation and supervision pipeline.
-
The mask generation pipeline relies on a heuristic connection between high uncertainty and reflective regions. However, uncertainty is learned from reconstruction error and is not specific to reflections. The visibility-based weighting is intended to suppress unreliable candidates that appear in only a few views, but it does not distinguish the causes of high uncertainty. In real captures, exposure or white balance variations, changing illumination or shadows, and persistent errors in camera poses or geometry can also produce reconstruction inconsistencies. These effects may occur in regions that remain visible across many views and therefore would not necessarily be suppressed by the visibility weighting. It remains unclear to me how reliably the selected prompts identify reflective objects under these conditions. Although the authors acknowledge related limitations, the current evaluation provides limited direct evidence of mask accuracy and robustness across more diverse real scenes. Together with W2, this becomes particularly relevant: if the performance gains mainly come from the masks and their associated supervision, the reliability of this heuristic mask generation pipeline would be central to the paper’s main contribution.
Questions for authors
-
Regarding Weakness 1, I would suggest revising the introduction to acknowledge existing work on near-field reflections, including SpecTRe-GS and Ref-GS. It would help to distinguish the limitations of direction-only environment maps from the capabilities of methods that already model spatially varying reflections, and clarify what remains unresolved by these approaches.
-
Regarding Weakness 2, could the authors compare PCC with a scene-based ray-tracing alternative while keeping the UGMS masks, metallic supervision, geometry representation, material parameterization, and training settings fixed where possible? This experiment would help determine whether the advantage comes primarily from the masks and supervision or from the reflection rendering formulation.
-
Regarding Weakness 3, additional mask generation results on a broader range of real captures would strengthen the paper, particularly in scenes with competing sources of reconstruction inconsistency. Qualitative mask visualizations and, where annotations are available, quantitative mask accuracy would help assess the robustness of the uncertainty-guided selection beyond the current examples.
4 comments
Response to Reviewer ncs9 (2/2, W2, W3, Q2, and Q3)
(Continued from the previous comment.)
(W2, Q2) Controlled comparison of PCC with scene-based ray tracing
We compared PCC with the scene-based ray tracing formulation used in MaterialRefGS within our Gaussian inverse rendering pipeline. All variants used the same UGMS masks, metallic supervision, geometry representation, material parameterization, and training settings.
On IndoorRef12, PCC (Ours) achieved 33.72 dB on full images and 23.82 dB on reflective objects. Scene-based ray tracing (MaterialRefGS) achieved 31.14 dB and 20.82 dB, respectively (Table R6). This comparison shows that the reflection rendering formulation contributes to the improvement under same masks and supervision.
Video 4 on our anonymous rebuttal page LINK shows that some reflected details in the ray tracing results are blurred or misplaced. One possible explanation is that secondary rays hit floaters or inaccurately reconstructed surfaces before reaching the intended geometry.
Table R6. Controlled comparison of reflection models within our Gaussian inverse rendering pipeline on IndoorRef12. Note that all three reflection models are evaluated using the same UGMS masks, loss weights, and training protocol.
| Reflection model | Full image PSNR (dB) ↑ | Reflective object PSNR (dB) ↑ |
|---|---|---|
| Direction-only environment map | 33.12 | 22.70 |
| Secondary ray tracing (MaterialRefGS) | 31.14 | 20.82 |
| PCC (Ours) | 33.72 | 23.82 |
(W3-1) Uncertainty and the limits of visibility weighting
The authors agree that competing sources of reconstruction inconsistency can produce high uncertainty across many views and lead to prompts on non-reflective regions. In our controlled test, small camera pose perturbations had a limited effect on mask accuracy (Table R8). Larger pose errors and persistent illumination changes remain limitations (Appendix K.1, K.2).
On IndoorRef12, high uncertainty was 12.5 times as frequent on reflective objects as in the background, and all 96 selected prompts fell on reflective objects (Table S3). In the following (W3-2, Q3-1), we examine prompt selection and mask accuracy on a broader range of real captures with competing sources of reconstruction inconsistency.
(W3-2, Q3-1) Additional mask generation results on a broader range of real captures
We evaluated mask generation on three new real-world captures containing metallic and dielectric reflective objects. We used automatic exposure and white balance and estimated camera poses with COLMAP. Fig. 2 on the LINK shows the uncertainty maps, selected prompts, and resulting masks for two of the three captures.
As shown in Table R7, the generated masks achieved IoUs of 0.874 for Desk, 0.907 for TV, and 0.963 for Helmet, with a mean IoU of 0.915.
Table R7. UGMS mask IoU on three real captures. Mask IoU is measured against manual annotations on sampled test views.
| Scene | Reflective objects | Material | Mask IoU ↑ |
|---|---|---|---|
| Desk | Chrome, bell | Metal | 0.874 |
| TV | TV | Dielectric | 0.907 |
| Helmet | Helmet shell | Dielectric | 0.963 |
| Mean | 0.915 |
(W3-3, Q3-2) Quantitative mask accuracy for the robustness of the uncertainty-guided selection
We perturbed the training data for four IndoorRef12 scenes, testing ±20% exposure and white balance variations and 0.05° camera rotation noise separately (Table R8). We measured how well the uncertainty separates reflective objects from the background using AUROC. Exposure and white balance variations reduced AUROC from 0.948 to 0.906, while pose noise yielded an AUROC of 0.945. Mask IoU was 0.898 with the original data, 0.901 with exposure and white balance variations, and 0.884 with pose noise. Fig. 3 on our anonymous rebuttal page LINK shows the uncertainty maps and the resulting masks under each perturbation. Exposure and white balance variations raised the uncertainty almost uniformly over the whole image, and pose noise raised it along edges. In both cases, the uncertainty remained highest on the reflective objects, and the resulting masks were nearly unchanged.
Table R8. Uncertainty separation (AUROC) and mask accuracy of UGMS under exposure, white balance, and camera pose perturbations on four IndoorRef12 scenes (Rabbit, Chick, Apple, Knight).
| Condition | AUROC ↑ | Mask IoU ↑ |
|---|---|---|
| Original | 0.948 | 0.898 |
| Exposure and white balance (±20%) | 0.906 | 0.901 |
| Pose noise (0.05°) | 0.945 | 0.884 |
Comment by Reviewer ncs9
Thank you for your thorough response. My concerns have been adequately addressed, and I appreciate the additional clarifications and experiments. I am raising my score to 3 accordingly.
We sincerely thank the reviewer for carefully considering our response and for raising the score. We will incorporate the clarifications and additional experiments into the final version.
Response to Reviewer ncs9 (1/2, W1 and Q1)
The authors would like to thank the reviewer for the constructive comments. We address each comment below.
All figures and videos referred to below are available on our anonymous rebuttal page (https://iclr2027-rebuttal-paper53498.netlify.app).
(W1-1, Q1-1) Acknowledge existing works on near-field reflections in the introduction
As per the reviewer’s suggestion, the authors will revise the introduction to distinguish the limitations of direction-only environment maps from the capabilities of methods that already model spatially varying reflections. We will also clarify what remains unresolved by these approaches and explain why we use PCC to address it. The revision is as follows.
(Before)
Second, most methods assume an infinite distance environment map, where lighting depends only on direction. While this is reasonable outdoors, it breaks down indoors because dominant reflectors such as walls and furniture lie at finite distances. As a result, direction only lighting fails to capture spatially varying near-field reflection shifts, leading to flattened highlights and misaligned reflection anchoring across views (Fig. 1: Right).
(After)
Second, indoor reflections often come from nearby structures such as walls and furniture. Therefore, rays leaving different surface points in the same direction can hit different parts of the scene. A direction-only environment map misses this variation, which can lead to misaligned reflections across views (Fig. 1: Right). Recently, SpecTRe-GS and MaterialRefGS model near-field reflections by tracing secondary rays through the reconstructed scene, so geometry errors along those rays can misplace reflected content. Ref-GS represents spatially varying specular appearance through learned features. The remaining challenge is to model near-field reflection shifts explicitly with less reliance on detailed scene geometry. We address this by integrating parallax-corrected cubemaps (PCC) into Gaussian inverse rendering, using a coarse proxy of the surroundings to adjust the cubemap sampling direction as the reflection point moves.
We will incorporate this revision into the introduction of the final version.
(W1-2, Q1-2) What remains unresolved by existing approaches and Advantage of PCC
As stated in the revision above, the remaining challenge is to predict where nearby structures appear in the reflection as the reflection point moves, with less reliance on detailed scene geometry.
SpecTRe-GS and MaterialRefGS trace secondary rays through the reconstructed scene to evaluate local geometry and visibility more directly. If a surface along the reflection path is missing or inaccurately reconstructed, the ray can miss that surface or intersect it at the wrong location.
Ref-GS captures spatially varying specular appearance through learned features, while its far-field lighting is sampled by reflection direction and roughness. The relationship between reflector distance and reflection shift is therefore implicit in the learned appearance.
The advantage of PCC in our Gaussian inverse renderer is that it models this relationship explicitly from a coarse representation of the surroundings. Two rays with the same direction can intersect the proxy at different locations when they originate at different surface points. PCC uses those intersections to adjust the cubemap sampling direction and approximate where reflected content should appear.
For qualitative comparisons, in the indoor scenes shown in Video 3 on our anonymous rebuttal page LINK, our method better preserves the placement of reflected details across views, while the evaluated ray tracing baselines produce blurred or misplaced reflections. In addition, SpecTRe-GS takes a binary mask of the highly specular region as input. On the other hand, our method localizes the reflective regions automatically.