acceptodds
Withdrawn from ICLR 2027

MAGS: Mask Augmented Gaussian Splatting for reflective scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Novel view synthesis of reflective scenes remains challenging because different combinations of geometry, materials, and illumination can explain similar observed appearances. This ambiguity can lead to inconsistent material estimates within regions that share a common material, potentially degrading reflection rendering at novel viewpoints. We introduce Mask-Augmented Gaussian Splatting (MAGS), a framework that improves reflective novel view synthesis through mask-guided material regularization. Building on a physically based 2D Gaussian Splatting framework, MAGS uses segmentation masks as regional priors and penalizes deviations of rendered material attributes from their mask-level means. The resulting gradients propagate through differentiable rasterization to the underlying Gaussian material parameters, encouraging regional material consistency while allowing rendered appearance to vary across viewpoints. Experiments demonstrate competitive performance on the real-world Ref-Real benchmark and the best average PSNR, SSIM, and LPIPS among the compared methods on the synthetic ShinyBlender benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaussian Splatting, Inverse Rendering, Mask-guided Regularization, Material Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2o9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MAGS, a method for improving novel view synthesis in reflective scenes by encouraging spatial consistency in estimated material properties. Building upon SpecGloss-GS, a physically based rendering framework using 2D Gaussian Splatting, the proposed method introduces a mask-guided material regularization loss. Specifically, SAM2 segmentation masks are used to define regions within which rendered material attributes, including albedo, roughness, and metallic values, are encouraged to remain consistent by penalizing their deviations from the regional mean. This regularization is incorporated into the differentiable rendering pipeline to optimize the underlying Gaussian material parameters. The method is evaluated on the synthetic ShinyBlender and real-world Ref-Real datasets, with comparisons against several Gaussian Splatting-based approaches. Experimental results show modest improvements over SpecGloss-GS in novel view synthesis quality, achieving the best average PSNR, SSIM, and LPIPS on ShinyBlender and competitive performance on Ref-Real.

Critical strengths
  1. The paper addresses the ambiguity in material estimation, which is a fundamental challenge in inverse rendering. Different combinations of material properties and illumination can produce similar appearances, making it difficult to recover consistent material attributes from multi-view images. The proposed mask-guided material regularization is an intuitive and reasonable strategy for mitigating material estimation ambiguity. By encouraging consistent material properties within segmented regions, the method introduces additional constraints that help reduce the solution space of this under-constrained optimization problem.

  2. The experiments cover ShinyBlender and Ref-Real and include comparisons with relevant methods, such as SpecGloss-GS, MaterialRefGS, and EnvGS. The proposed method achieves the best average PSNR, SSIM, and LPIPS on ShinyBlender, demonstrating modest but measurable improvements over its direct baseline.

Critical weaknesses
  1. The proposed method largely builds upon the existing SpecGloss-GS framework, with the main contribution being a relatively straightforward mask-guided L1 regularization loss. Moreover, prior work such as MaterialRefGS has already explored material consistency constraints and the use of SAM2 masks for material supervision. The paper does not sufficiently clarify its technical novelty or advantages over these existing approaches.

  2. The improvements over the direct baseline, SpecGloss-GS, are relatively small, with average PSNR gains of only 0.17 dB on ShinyBlender and 0.05 dB on Ref-Real. Furthermore, MAGS underperforms MaterialRefGS on the average metrics of the real-world benchmark and even degrades performance on some individual scenes. These results raise questions about the practical significance and robustness of the proposed regularization.

Questions for authors

The proposed method assumes that segmentation masks correspond to regions with homogeneous material properties. However, the paper does not investigate whether more accurate material-based segmentation would further improve reconstruction quality. Could the authors compare SAM2-generated masks with ground-truth material segmentation masks on synthetic scenes, if available? Such an experiment would help validate the core assumption and clarify the extent to which the method's performance depends on the alignment between segmentation boundaries and actual material boundaries.

Reviewer_ExAdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to use segmentation results from SAM2 and estimated albedo and normal maps from off-the-shelf diffusion models to guide 2D GS reconstruction of a reflective scene. They use SpecGloss-GS as the optimization pipeline and conduct experiments on real and synthetic datasets.

Critical strengths
  • Mask-guided optimization is simple and practical.
  • Evaluation covers both real and synthetic data.
Critical weaknesses
  • The technical contribution is limited; they directly use segmentation maps from the off-the-shelf large model SAM2 to guide the optimization process, with a loss that penalizes deviations of materials from their regional means. A similar idea is already present in MaterialRefGS. The contribution of this work needs clarification.

  • The improvements are marginal. In Table 1, MaterialRefGS is mostly the best instead of "Ours". In Table 2, PSNR improves by less than 0.2 compared to the baseline SpecGloss-GS. These mixed results do not establish that the proposed method has a meaningful improvement.

  • Experimental results are limited. More real-world data should be included. For example, data from EnvGS for reflective scenes and data from MipNerf360 for general scenes. This helps to understand how robust the proposed method is.

Questions for authors

Please see the weakness.

Reviewer_oEnROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

MAGS addresses novel view synthesis of reflective scenes by adding mask-guided material regularization to SpecGloss-GS. It retains the underlying 2D Gaussian representation, physically based deferred renderer, and normal and diffuse-color priors. The proposed loss penalizes the L1 deviation of rendered material attributes from their mean within each SAM2 mask, with gradients propagated to the Gaussian parameters through rasterization. Experiments cover six synthetic ShinyBlender scenes and three real Ref-Real scenes. MAGS achieves the best reported average PSNR, SSIM, and LPIPS among the compared methods on ShinyBlender, while improvements over SpecGloss-GS on Ref-Real are small.

Critical strengths

The paper is clearly written and easy to follow. The motivation, pipeline overview, and loss formulation make it straightforward to understand the proposed regularizer and how it fits into the existing rendering framework.

Critical weaknesses
  1. The methodological novelty is limited, and the paper provides little additional insight into material regularization. MAGS retains the Gaussian representation, physically based rendering pipeline, lighting model, and normal and diffuse-color priors of SpecGloss-GS. The main addition is a SAM2-guided loss that penalizes deviations of rendered material attributes from their regional means. This is a narrow extension, and the paper provides limited evidence or analysis to establish its value beyond the existing framework. In particular, it does not examine which material channels benefit from the constraint, how the segmentation affects the outcome, or when the assumption of uniform material becomes inappropriate. Together with the modest gains over SpecGloss-GS, these gaps leave the added contribution insufficiently substantiated.

  2. The current experiments do not provide sufficient evidence that the added constraint yields reliable improvements. Compared with SpecGloss-GS, the average PSNR gains are only 0.17 dB on ShinyBlender and 0.05 dB on Ref-Real, with small changes in the other metrics. The gains are also scene-dependent: Coffee performs worse on all three metrics, including a 0.39 dB decrease in PSNR. The visual comparisons in Figures 3 and 4 show broadly similar results to SpecGloss-GS, without a clear and consistent improvement in reflection quality.

Questions for authors

Could the authors provide additional controlled experiments or targeted analyses to establish the effectiveness of the proposed regularizer beyond SpecGloss-GS? In particular, which material channels benefit from the constraint, and how does the choice of segmentation regions affect the results? Such evidence would help explain the scene-dependent gains and failures, and clarify when the regional material-consistency assumption is useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.