acceptodds
Under review as a conference paper at ICLR 2027

LoFT-GS: Factorized Local Transport for Relightable Gaussian Splatting

16 views

Abstract

Relightable Gaussian splatting combines explicit reconstruction with fast rendering under novel views and illumination. However, a single visibility value per Gaussian cannot preserve the spatial distribution of occlusion within its footprint. When a cast-shadow boundary crosses a Gaussian, illuminated and occluded regions are collapsed into a shared shadow response. We introduce LoFT-GS, a locally factorized relighting representation that addresses this limitation by recording visibility on a local texture grid within each 2D Gaussian during light-space splatting. A range-conditioned shadow formulation combines this local visibility with a Gaussian-level range predictor. More generally, LoFT-GS separates spatially varying relighting factors from shared Gaussian-level computation. Local texels represent visibility and diffuse appearance and modulate specular amplitude and sharpness, while geometry, base directional responses, and neural predictions remain shared per Gaussian. Experiments across diverse scenes demonstrate improved relighting fidelity over the evaluated baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3DGS, Relighting, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2bPqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LoFT_GS, a locally factorized relighting representation that encodes visibility on a local texture grid within each 2D Gaussian during light-space splatting. This addresses the limitation of using a single visibility value per Gaussian, which cannot represent the spatial distribution of occlusion within its footprint. A range-conditioned shadow formulation is designed to combine the spatially varying local visibility with a Gaussian-level range predictor. Experiments across several scenes demonstrate the improved relighting fidelity over the evaluated baselines.

Critical strengths
  1. This paper is well written and easy to following. The idea of enhancing the per-Gaussian visibility with local textural representations is practical.
  2. A range-conditioned shadow formulation is designed to combine texel-level light-space visibility with a Gaussian-level range predictor, ensuring spatially varying shadows.
  3. Experimental results across 21 scenes show that the proposed method outperforms state-of-the-art methods.
Critical weaknesses
  1. The idea behind the proposed method is actually quite simple and straightforward. It improves existing relightable 3D Gaussian methods (e.g., GS^3, RNG, and SSD-GS) by incorporating a local texture per-Gaussian. For other shading formulations, they are quite similar to existing methods.
  2. Like some of other methods, the proposed method can only handle a single point/directional light. This is OK, since relighting 3DGS scenes with a single point/direction light is still challenging. However, I still hope new methods can handle general lighting conditions. I would see some more discussions.
  3. Why use different numbers of Gaussian primitives for different methods when evaluating runtime performance in Table 3? This seems to make the comparison unfair.
Questions for authors

Why use different numbers of Gaussian primitives for different methods when evaluating runtime performance in Table 3? This seems to make the comparison unfair.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2bPq

We thank the reviewer for their time and careful reading of our paper. We also appreciate your suggestions on lighting conditions and efficiency comparisons, which we address below.


1. More general lighting conditions

Thank you for this suggestion. We have extended LoFT-GS to environment-map relighting and added comparisons with GI-GS and SVG-IR in Table 4 and Figure 6. Specifically, we approximate the environment map as a set of directional lights, render each direction separately using our trained model, and sum the results weighted by the environment light intensities. Therefore, this extension requires no model retraining.

We find that LoFT-GS remains effective under environment lighting. In particular, when the environment map contains strong directional components, LoFT-GS recovers shadow placement, shadow intensity, and specular details closer to the ground truth, as highlighted by the red boxes in Figure 6.

However, per-direction rendering introduces computational overhead. In Supplementary Section D, we show the rendering results under different sample counts: shadow details are well recovered only after increasing the sample count to 512, which requires evaluating many single-light responses. Therefore, although our method can already handle environment lighting in this way, how to support general lighting conditions more efficiently is worth exploring in future work.


2. Efficiency comparison under the same budget

We agree that using the same Gaussian count gives a fairer measure of rendering overhead.

We sampled 200k and 300k Gaussians from the Lego model trained by GS³ as the common input for all four methods, and added the results to Table 3. All methods use the exact same Gaussian positions, scales, orientations, and opacities, along with their respective randomly initialized networks and native rendering pipelines. Although LoFT-GS increases per-frame runtime by about 37% on average compared to GS³, it is still faster than RNG and SSD-GS.

Meanwhile, we also retain the original Table 3 results based on actual training, because reducing the number of Gaussians while maintaining reconstruction quality is itself an important direction for improving efficiency, as also studied in works such as Mini-Splatting, LightGaussian, and Reduced-3DGS (Fang & Wang, 2024; Fan et al., 2024; Papantonakis et al., 2024). Using the same densification settings as GS³, LoFT-GS achieves higher reconstruction quality with 27–50% fewer Gaussians, indicating that our local representation enhances the expressive power of individual Gaussians. As a result, the practical rendering speed of our trained models is comparable to GS³, and even faster on some scenes.


Reference

1. Guangchi Fang and Bing Wang. Mini-Splatting: Representing Scenes with a Constrained Number of Gaussians. In European Conference on Computer Vision (ECCV), 2024.

2. Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024.

3. Panagiotis Papantonakis, Georgios Kopanas, Bernhard Kerbl, Alexandre Lanvin, and George Drettakis. Reducing the Memory Footprint of 3D Gaussian Splatting. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 7(1), 2024.

Reviewer_YpxfOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a local representation for Gaussian-based relighting to improve shadows and appearance details within each Gaussian. Instead of assigning a single visibility value to the entire Gaussian, the method computes visibility on a local grid and refines it using a Gaussian-level network. Per-texel diffuse albedo and specular amplitude and sharpness allow each Gaussian to represent local color changes and highlight variations that a single set of appearance parameters cannot capture. Experiments on real and synthetic scenes show clearer shadow boundaries, texture details, and highlights, with an average PSNR improvement of about 1.05 dB over GS3 across 16 scenes and comparable rendering speed on the reported efficiency scenes.

Critical strengths

LoFT-GS computes visibility on a local grid within each Gaussian, allowing it to represent shadow boundaries that a single visibility value would blur. Local diffuse albedo and specular controls also capture fine texture and highlight variations. The results show clear improvements in shadow boundaries, surface details, and highlights.

The method achieves the best average PSNR on the test sets, improving over GS3 by about 1.05 dB. The component ablations and grid-resolution experiments support the use of local factors. On the scenes in Table 3, it achieves these improvements with fewer Gaussians and rendering times close to GS3.

Critical weaknesses

The “Gaussian-level only” comparison in Table A3 covers only two scenes, so the benefit of the local representation needs to be tested more broadly. The efficiency evaluation reports rendering speed but lacks a training-time comparison.

Questions for authors
  1. Please extend the “Gaussian-level only” comparison in Table A3 to more test scenes.

  2. Please report training time alongside rendering speed for LoFT-GS, GS3, and SSD-GS, with the hardware and resolution used.

Please discuss the differences from TextureSplat, which also models local material variation within each Gaussian.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ypxf

We thank the reviewer for their time and careful reading of our paper. We also appreciate your suggestions on experiments and related work, which we address below.


1. Extending the Gaussian-level-only comparison

Thank you for this suggestion. Our previous Gaussian-level-only experiment covered only two scenes, which was indeed insufficient. We have now extended this experiment to all 16 main evaluation scenes, added the per-scene results in Table A2, and included the corresponding averages in Table 2 of the main text.

After extending to all scenes, the Gaussian-level-only variant achieves an average PSNR of 33.02 dB, outperforming GS³ (32.66 dB) by approximately 0.36 dB. One possible reason is that GS³ is built on 3DGS, whereas our framework is based on 2DGS. By computing ray-splat intersections directly on 2D Gaussian planes, 2DGS avoids the approximation errors introduced by local affine projection in 3DGS. This may be even more important in relighting, because beyond reconstructing appearance from camera views, occlusion must also be computed from the light source's viewpoint, where projection and depth inconsistencies can further affect shadow placement and stability as the light source moves.

On this basis, after adding the local representation, the average PSNR of the full model is further improved to 33.72 dB, which is 0.70 dB higher than Gaussian-level-only, and SSIM and LPIPS improve as well.


2. Adding training time and efficiency comparison

We have added the training time of each method in Table 3, and reported the GPU and image resolution used.

In addition, for a fairer comparison of computational efficiency, we kept the results of the models obtained from normal training of each method, and also added experiments under the same number of Gaussians. This allows us to evaluate both the per-frame rendering cost under the same Gaussian budget, which reflects the overhead of the rendering pipeline itself, and the practical rendering speed after normal training, which also reflects how compactly each method represents the scene.


3. Distinction from TextureSplat

Thank you for pointing out TextureSplat. We carefully read this work and recognize its contribution in improving reflective appearance by modeling local materials and normals within individual Gaussians. We apologize for omitting this related work in our initial draft, and have added the citation and discussion in the revised manuscript.

Both works use local representations within Gaussians, but they start from different problems. TextureSplat focuses on the spatial variation of materials and normals within a Gaussian. In contrast, LoFT-GS addresses the issue that as the light source moves, different regions within the same Gaussian footprint may be lit and occluded differently, which is lost when sharing a single visibility value per Gaussian.

Based on this, we design a relighting representation combining the Gaussian level and texel level: shared direction-dependent computations and neural predictions are done at the Gaussian level, while spatial variations in visibility and reflection response are modeled at the texel level. This preserves fine details while keeping rendering overhead low through shared computation.

Thank you for the additional experiments and clarifications. The revised evaluation is thorough, and your response addresses my main concerns.

Thank you for taking the time to review our updates. We’re glad the additional experiments and clarifications addressed your main concerns.

Reviewer_DZJvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a method for improving relightable Gaussian splatting by modeling spatially-varying appearance within individual Gaussians using local texel grids. A range-conditioned shadow formulation is also proposed to predict a shadow range per Gaussian, which is combined with the local visibility. The reported results are better than those of the baselines, and the rendering efficiency remains close to that of GS$^3$.

Critical strengths
  1. The paper is well organized and the presentation is clear.

  2. The motivation for modeling visibility within each Gaussian is convincing. A single visibility value per Gaussian cannot preserve the spatial distribution of occlusion within its footprint, which is addressed by the proposed local texture grid and range-conditioned shadow formulation.

  3. The proposed local texture grid can also reproduce fine-scale diffuse textures and specular highlights by allowing diffuse albedo and specular amplitude and sharpness to vary locally.

Critical weaknesses
  1. The abstract and introduction only mention the limitations of modelling visibility, but the proposed method also improves details in diffuse and specular appearance. A more complete discussion of the motivation in the abstract and introduction could help readers better understand the scope and benefits of the proposed method.

  2. Although the proposed method improves shadow details, the shadow boundaries still seem less accurate than those in the ground truth. The pixel-level shading methods such as IRGS can render accurate shadows. The paper could discuss the possibility of rendering shadows at the pixel-level on the rasterized G-buffer.

Questions for authors
  1. SVG-IR, which is one of the baselines, was originally designed for inverse rendering under environment lighting, but this paper employs the point light source. The paper and the appendix do not provide details about its adaptation to the point light source. Could the authors clarify how it is adapted to the point light source used in this paper?

  2. Table 1(a) reports a PSNR of only 22.40 dB for RNG on the Nefertiti scene, substantially below the results of the other methods. Could the authors give a qualitative comparison of this scene and explain why this result is very low?

  3. The rendered shadow boundaries still appear less accurate than those in the ground truth. Have the authors considered pixel-level shading? If so, what are the reasons for not adopting it?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DZJv

We thank the reviewer for their time and careful reading of our paper. We also appreciate the specific questions and suggestions about the baseline and shadow technique, which we answer below.


1. Choice of SVG-IR

We apologize for not sufficiently explaining why we selected SVG-IR. Due to space constraints, we did not discuss this baseline choice in detail in the main text. We kept SVG-IR’s original lighting model and learned one environment map shared by all images within each scene. Previous works, like SSD-GS, included GI-GS to demonstrate the necessity of modeling local point-light transport, and additionally compared with original 3DGS to illustrate the challenges in conventional reconstruction methods on this task. Therefore, we also wanted to include a representative environment-light inverse-rendering method in our evaluation.

We selected SVG-IR because it is a more recent approach than earlier baselines like GS-IR and GI-GS, and also improves representational capacity by modeling spatially varying material properties within individual Gaussians.


2. RNG results on Nefertiti

Thank you for pointing out the anomaly on Nefertiti. Actually, we also found this anomaly before, and we checked the camera poses but found no obvious errors. As RNG was the only method that showed a clear reconstruction problem on this scene at that time, we attributed the problem to a training collapse.

This time, we carefully checked every step and found that the error actually occurred during initialization, which had been overlooked previously. Because RNG initializes with a random point cloud around the origin while the camera poses of Nefertiti are offset from the origin, it led to collapse.

We have now completed the retraining, and the results look normal. We also checked other samples and confirmed that no similar problems exist. RNG's PSNR improved from 22.40 dB to 34.04 dB. We have updated Table 1 and the corresponding averages. After this correction, LoFT-GS still achieves an average PSNR improvement of 1.88 dB over RNG across all 16 scenes. We apologize for not identifying the cause earlier.


3. Pixel-wise shadows

We strongly agree with the reviewer's suggestion regarding pixel-wise shadows. We have also considered IRGS in our earlier explorations. IRGS traces rays from pixel-wise shading positions using 2D Gaussian ray tracing to compute visibility and indirect illumination, allowing a more complete treatment of light transport. However, this ray-tracing approach substantially increases rendering time. As reported in their paper, it takes approximately one second to render a complete image on an RTX 3090 with 256 ray samples. Therefore, we did not adopt this approach, as we aim to maintain real-time rendering efficiency.

In our early experiments, we also trained a pixel-wise shadow-mapping variant. We have added these results to Section 4.5 and Figure 7. Shadow mapping can produce sharp boundaries through depth comparisons, but we found that errors in the reconstructed Gaussian depth led to shadow misalignment. Even when the boundary was correct, geometric visibility alone did not fully explain the observed shadow intensity, which also depends on indirect and ambient illumination.

But we still believe that this direction remains worth exploring. Like through more accurate geometry reconstruction to make shadow computation more precise, or conversely, using shadows to refine the geometry. Meanwhile, to improve the actual shadow rendering, we could incorporate explicit ambient lighting, implicit modules, or even hybrid mesh-3DGS representations. For instance, GHPT (Bo et al., 2026) combines Gaussian-rendered G-buffers with mesh-based ray tracing to compute visibility and indirect illumination. These methods are both feasible and interesting.


Reference

1. Jinyang Bo, Fan Dou, Wenrui Quan, Shangxun Liu, Yang Xu, Yuhe Zhang, Kang Li, and Guohua Geng. GHPT: Real-Time Relightable Gaussian Splatting using Hybrid Path Tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

Thank you for the clarification and explanation. The responses address my concerns, and I maintain my rating (3 - Weak acceptance).

We’re glad our responses addressed your concerns. Thank you for your continued support for acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Reviewers,

Thank you for your careful reading and thoughtful comments. Your questions and suggestions have helped us explain LoFT-GS more clearly and strengthen the experimental evaluation. We have revised the manuscript and address each review in its own thread.

Major revisions and new results are highlighted in blue. Section, table, and figure numbers in our replies refer to the revised PDF.

Main Changes

  • Expanded Gaussian-level-only ablation. We extended this experiment to all 16 main scenes to better evaluate the contribution of our local representation. (Table 2 and Table A2)

  • Added training time and controlled efficiency comparisons. We report training times and compare the four methods using the same Gaussian geometry at budgets of 200k and 300k Gaussians. (Table 3)

  • Added environment-map relighting experiments. We compare LoFT-GS with GI-GS and SVG-IR, and provide the rendering setup and sampling study in the supplement. (Sec. 4.4, Table 4, Figure 6, and Appendix D)

  • Added pixel-wise shadow-mapping experiments. We include visual comparisons and discuss the observed differences in shadow placement and intensity. (Sec. 4.5 and Figure 7)

  • Corrected the RNG result on Nefertiti. We identified an initialization mismatch, retrained the model after correcting it, and updated the corresponding results and averages. (Table 1)

We hope these revisions and our rebuttal address your concerns, and we would be glad to provide further analysis if needed during the remaining discussion period. Sorry again for the extra message, and thank you for your time.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.