acceptodds
Withdrawn from ICLR 2027

RT-GS: Gaussian Splatting with Reflection and Transmittance Primitives

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Gaussian Splatting is a powerful tool for reconstructing diffuse scenes, but it struggles to simultaneously model specular reflections and the appearance of objects behind semi-transparent surfaces. These specular reflections and transmittance are essential for realistic novel view synthesis, and existing methods do not properly incorporate the underlying physical processes to simulate them. To address this issue, we propose RT-GS, a unified framework that integrates a microfacet material model and ray tracing to jointly model specular reflection and transmittance in Gaussian Splatting. We accomplish this by using separate Gaussian primitives for reflections and transmittance, which allow modeling distant reflections and reconstructing objects behind transparent surfaces concurrently. We utilize a differentiable ray tracing framework to obtain the specular reflection and transmittance appearance. Our experiments demonstrate that our method successfully produces reflections and recovers objects behind transparent surfaces in complex environments, achieving significant qualitative improvements over prior methods where these specular light interactions are prominent.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel view synthesis, 3D Gaussian splatting, differentiable ray tracing, transparent object reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bACFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a unified framework that integrates a microfacet material model and ray tracing to jointly model specular reflection and transmittance in 3D Gaussian Splatting (3DGS). It uses separated Gaussian primitives for reflections and transmittance. This allows modeling distance reflections and reconstructing objects behind transparent objects. A differentiable ray tracer is used to obtain the reflection and transmittance appearance.

Critical strengths
  1. Microfacet-weighted deferred shading is coupled with differentiable ray tracing over dedicated reflection and transmittance Gaussians in a unified GS-based pipeline.

  2. It separates transmittance Gaussian primitives to model the interiors of transparent objects.

Critical weaknesses
  1. The main contribution of the proposed method is coupling microfacet-weighted deferred shading with 3D Gaussian splatting. However, this is not new. At least the following paper has ready adopted the idea in reconstructing shiny objects. Shuichang Lai, Letian Huang, Jie Guo, Kai Cheng, Bowen Pan, Xiaoxiao Long, Jiangjing Lyu, Chengfei Lv, Yanwen Guo, Glossygs: Inverse rendering of glossy objects with 3d gaussian splatting, IEEE Transactions on Visualization and Computer Graphics, 2025.

  2. The proposed method in this paper doesn’t perform better than existing methods. For instance, From the visual comparison in Figure 5, RT-GS does not perform better than EnvGS. Neither method reconstructs the reflections on the car well, and both still have a noticeable gap compared with the ground truth. In addition, from the quantitative comparison in Table 1, the metrics of RT-GS and EnvGS are very close, almost within the range of numerical error.

  3. For the comparison with NU-NeRF in Figure 6, the proposed method doesn’t have any advantage. However, it is quite strange that Table 2 shows RT-GS achieving much higher metrics than NU-NeRF. From Figure 6, this seems to be because NU-NeRF does not reconstruct the background well. However, such a comparison seems somewhat unfair, since both methods target transparent object reconstruction rather than background reconstruction.

  4. There’re many loss terms in the final loss function in Equation (18). How are the weights determined? Do different scenes require different weights?

  5. No video is provided to show the performance of the proposed method.

Questions for authors

Why are the metrics of the proposed method considerably better than those of NU-NeRF, yet the visual results show no obvious improvement?

1 comment

W1. We agree that combining microfacet shading with Gaussian splatting is not itself new. Our contribution is using microfacet-based weighting to couple three Gaussian representations—base, environment, and transmission Gaussians—within a unified rendering framework, jointly modeling direct appearance, reflection, and transmission.

W2. We agree that RT-GS offers only modest improvements over EnvGS on reflective scenes, and both retain visible errors relative to the ground truth. The close metrics in Table 1 do not establish a substantial advantage in this setting. Our main improvement compared to EnvGS is on refractive scenes, where explicitly modeling transmitted appearance yields clearer visual and quantitative gains over EnvGS.

W4. The weights are chosen empirically to balance photometric reconstruction and auxiliary regularization. The loss weights are manually specified and shared across scenes. The only scene-specific adjustment is the perceptual loss weight, which is increased from the default 0.01 to 0.1 for the spheres scene. All other weights remain unchanged.

W5. The videos have been added to the Supplementary Material.

Q1 and W3. The selected visual examples do not fully reveal the differences in environment reconstruction and fine details, where NU-NeRF performs less well. NU-NeRF also excludes total internal reflection (TIR) regions from rendering, which can affect the quantitative comparison. We therefore additionally compare both methods within identical transparent-region masks and further exclude NU-NeRF’s TIR regions from both methods’ evaluation.

Our method achieves better PSNR, SSIM, and LPIPS on three of the four scenes, including both scenes evaluated with TIR regions excluded. BallStatue is the only scene with lower PSNR and SSIM, although our LPIPS remains better. This is consistent with the limitation of our straight-through approximation: its glass enclosure has non-negligible thickness, while NU-NeRF explicitly models refraction through thick objects. These regional comparisons provide stronger evidence of improvements that may be difficult to distinguish in the selected visualizations.

The results have been added in Appendix A.7.

Reviewer_SM6jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes RT-GS, a Gaussian Splatting framework that both considers reflection and transmittance at the object level. The method uses three separate Gaussian primitives for diffuse, reflection, and transmittance components, and combines them using a microfacet material model. It also introduces mesh-guided ray tracing to model first- and second-bounce transmission, allowing the reconstruction of objects inside transparent surfaces and the background behind them.

Critical strengths

S1. Joint modeling of reflection and transmittance. I find the overall framework reasonable and intuitive. The use of a microfacet material model provides a way to combine the different Gaussian components while considering their material properties.

S2. Mesh-guided transmission. I think the mesh-guided ray tracing is an interesting part of this work. By considering both first- and second-bounce transmission, the method can reconstruct objects inside transparent surfaces as well as the background behind them. The results show that this approach works reasonably.

Critical weaknesses

W1. Regarding transmission model. The method assumes infinitely thin transparent surfaces and does not consider refraction. I understand that this simplifies the problem, but I wonder why this assumption is made when the method mainly focuses on object-level transparent reconstruction. Refraction seems important for many transparent objects, and this limitation deserves more discussion.

W2. Limited analysis of intermediate results. The extracted mesh plays an important role in guiding the transmission rays, but its reconstruction quality is not clearly demonstrated. Also, since the method explicitly models first- & second-bounce transmission separately, I would like to see how these components look individually rather than only seeing the final rendered images.

W3. Missing closely related work. I find that GLINT[1] (CVPR 2026) is not discussed in this paper even though the paper is closely related. GLINT also models reflection and transmittance using separate Gaussian representations and secondary ray tracing. Although GLINT focuses on scene-scale reconstruction, making a direct comparison somewhat difficult, the overall ideas seem closely related. I think the authors should clearly explain how their method differs from GLINT and what the main technical novelty is across the paper.

[1] "GLINT: Modeling Scene-Scale Transparency via Gaussian Radiance Transport", CVPR 2026

Questions for authors

Q1. Why does the method assume straight-through transmission without considering refraction? How would this assumption affect the reconstruction of transparent objects with non-negligible thickness?

Q2. Could the authors show the extracted meshes used for mesh-guided ray tracing? I am curious how accurately the front and back surfaces are reconstructed.

Q3. Could the authors separately visualize the first- and second-bounce transmission results? I think this would help better understand what each component actually reconstructs.

Q4. Could the authors discuss the differences between the proposed method and GLINT (CVPR 2026)? While a direct quantitative comparison may be difficult, I think GLINT is closely related to this work and should be discussed.

3 comments

Q1. We adopt straight-through transmission as a practical approximation motivated by the prevalence of thin transparent structures in real scenes, such as windows and glass panels. For thin objects with approximately parallel interfaces, the lateral ray displacement may be small enough to neglect. However, this approximation becomes less accurate for thick or curved transparent objects, where refraction can substantially alter ray paths. It may therefore introduce errors in reconstructed geometry and transmitted appearance. Explicit refractive ray tracing would be needed to handle these cases more faithfully.

Q2. The front and back surfaces of the extracted mesh are added to Appendix A.5 in the paper.

Q3. The first- and second-bounce transmission results are added in Appendix A.6 in the paper.

Q4. While both RT-GS and GLINT employ decomposed Gaussian representations to model reflection and transmission, they differ in their modeling approaches and primary objectives.

First, GLINT uses a diffusion model to generate diffuse albedo maps, which serve as an appearance prior for identifying transparent regions. Combined with the depth discrepancy between interface and transmission representations, these maps are used to generate pseudo-masks for transparency supervision. In contrast, RT-GS uses GroundingDINO and SAM2 to identify transparent objects.

Second, GLINT explicitly separates rendering into an opaque branch and a transparent branch, blending their contributions according to the predicted transparency. In its transparent branch, the final appearance is composed only of reflected and transmitted radiance, without an intrinsic surface-color contribution. In contrast, RT-GS uses a microfacet material model to handle specular reflection and jointly models reflection and transmittance through differentiable ray tracing.

Finally, RT-GS employs a TSDF mesh extracted from diffuse Gaussians to provide geometric guidance during reconstruction, whereas GLINT focuses on recovering consistent interface geometry through depth and normal supervision.

Overall, RT-GS emphasizes microfacet-based reflection and geometry-guided transmittance reconstruction, while GLINT focuses on transparent-interface reconstruction, diffusion-based transparency priors, and separate opaque and transparent rendering branches.

I appreciate the authors' detailed response.

I think the validity of the straight-through transmission assumption might depend on the surface geometry, thickness, IOR, and camera viewing angles.

If transparent objects (e.g., water bottles, glass bowls, etc.) are filled with water, they can exhibit significant refraction. In particular, NU-NeRF also includes such scenarios, including nested transparent materials. Have the authors evaluated the proposed method under these conditions, or do they plan to extend it to handle them?

Since the proposed method traces transmission rays through the front and back surfaces while maintaining the original camera ray direction, I think the reconstruction could be highly sensitive to strong refraction.

I understand that your approach intentionally makes this simplifying assumption to avoid such challenging scenarios. However, I believe the generalizability of your method depends on how well it handles different transparent materials.

In addition, have you evaluated the geometric accuracy of the extracted surfaces, for example using Chamfer distance or EMD on scenes with ground-truth geometry? I found some qualitative results, but since your method explicitly uses TSDF fusion to extract meshes for guiding transmittance rays, I could not find a corresponding quantitative evaluation of the extracted geometry.

Thank you for the detailed feedback. We agree that the straight-through approximation depends on geometry, thickness, IOR, and viewing angle. Thinness alone does not guarantee its validity. Our current results therefore do not establish generalizability to strongly refractive materials. Extending the method to trace refracted rays and track medium transitions across interfaces is an important direction for future work.

Recovering a high-quality mesh is not our primary objective. The mesh is extracted midway through training and then fixed. It provides approximate origins for rays leaving the back surface and helps constrain the depth of transmission Gaussians within the transparent object. Under our straight-through assumption, these tasks require approximate geometric guidance rather than highly accurate refractive interfaces. The color results from the first and second transmission stages support the usefulness of this design on the evaluated scenes.

We did not quantitatively evaluate the extracted meshes because we could not find publicly available ground-truth geometry for the evaluated dataset.

Reviewer_L17nOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes RT-GS which include 3 kinds of gaussian primitives to try to address some specular and semi-transparent cases. It includes diffuse, reflection, and transmittance gaussian primitives. The most novel thing is transmittance gaussian primitive which use a TSDF mesh extracted from the diffuse gaussians. First hit transmittance ray will keep the camera ray direction under a thin shell assumption. They also use GroundingDINO and SAM2 to get the mask for transparent objs. Experiments covers 7 cases.

Critical strengths
  1. Compared with EnvGS, which is the most related work, this work introduced three kinds of gaussian primitives to represent the scene. And this work tries to address semi-transparent cases using transmittance ray.

  2. The writing is pretty clear and the work is easy to follow.

Critical weaknesses
  1. The results is too incremental compared with EnvGS. For example, in case Toycar of table 1,LPIPS of EnvGS is 0.206, LPIPS of your work is 0.205. This difference is pretty small so that it's difficult to say your work is better than EnvGS. And If we take a look of Figure 5, in the red square part, we can see the reflection of something, the difference is also very minor.

  2. You said that "transnittance rays keep the camera ray direction", however, in Eq. 11, it's written as a specular BTDF with a refracted direction ω_t and a Dirac delta, which the method never uses.

  3. You said you use a full microfacet model, I feel like this is overclaimed. For example, each pixel traces a single mirror-reflection ray. In this case, the term D·G·F/(4 cosθ_i cosθ_o) reduces to a per-pixel scalar weight on C_r. And roughness cannot produce glossy blur without cone tracing or multiple samples.

  4. For this kind of work, the best way to evaluate is to offer nvs video in your sup_mat. If we just take a look of several views of a case, it's diffucult to determine your performance.

Questions for authors
  1. If possible, could you please show the nvs videos for 7 cases compared with GT & EnvGS? If so and I feel like your results is better then EnvGS, I'm glad to increase your scores.

  2. How sensitive is the method to errors in the SAM2 masks and the TSDF mesh? How do the baselines perform if given the same masks?

1 comment

Q1 and W1. The NVS videos have been added to the Supplementary Material. Our main improvement compared to EnvGS is on refractive scenes, where explicitly modeling transmitted appearance yields clearer visual and quantitative gains over EnvGS.

W2. The Dirac delta is handled analytically, so it need not appear explicitly in the code. However, our implementation uses a straight-through transmission approximation rather than the Snell-refracted direction in Eq. (11). We will revise the equation and text to make this distinction explicit.

W3. We agree that “full microfacet model” overstates our implementation. We tested multiple reflection directions sampled according to roughness, but observed little improvement at increased computational cost. We therefore use a single mirror-reflection ray with microfacet-based weighting. In this approximation, roughness modulates the reflection weight but does not explicitly produce glossy blur. We will revise the wording and clarify this limitation.

Q2. In Appendix A.3, we discussed the robustness of our method. For other baselines, they did not model the transmittance part. TransparentGS utilizes the masks for transparent objects, and we discuss it in Appendix A.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.