acceptodds
Withdrawn from ICLR 2027

3DIS: Real-Time Hybrid Relighting via 3D Importance Sampling from 3DGS Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D Gaussian Splatting (3DGS) supports high fidelity real time scene rendering, but its baked appearance representation makes it difficult to achieve harmonious illumination for inserted meshes. We propose a scene-level lighting estimation framework that recovers a reusable, spatially varying illumination representation from an existing 3DGS scene, enabling real-time and illumination-consistent rendering of inserted objects. Instead of predicting lighting independently for each view or object pose, our method estimates a compact virtual-point-light configuration from multi-location HDR observations of the 3DGS scene. To make this estimation both spatially stable and source-aware, we introduce a mixed 3D importance sampling strategy that allocates point lights across scene coverage, HDR radiance importance, and semantic light-source regions such as windows and lamps. The resulting point-light configuration captures both dominant direct illumination and distributed environmental light, enabling inserted meshes to be rendered consistently to both the 3DGS environment and the viewing direction (like specular reflection), even the inserted mesh is moved/manipulated. We also construct a paired synthetic benchmark comprising five scene–object configurations with matched object-free and object-present renderings for quantitative evaluation of insertion appearance. Experiments on multiple indoor scenes demonstrate that our method provides a practical solution for illumination-consistent mesh insertion and interactive relighting in reconstructed Gaussian environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Rendering, Illumination estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wJXbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a framework for estimating reusable, spatially-varying illumination from an existing 3DGS scene to enable illumination-consistent object insertion. The method generates RGB-D panoramas at multiple locations, reconstructs HDR observations using PanoLight, and estimates a compact set of VPLs through a combination of spatial coverage sampling, HDR importance sampling, and semantic source sampling. The estimated VPLs are further optimized and subsequently used for position-conditioned hybrid rendering.

Critical strengths
  1. The proposed pipeline integrates spatial HDR observations, semantic light source priors, mixed VPL sampling, and hybrid rendering. The separation between offline estimation and online rendering is appropriate for real-time applications.

  2. Combining spatial coverage sampling, HDR importance sampling, and semantic source sampling is a reasonable strategy for allocating a limited number of VPLs to both dominant direct illumination and distributed environmental light.

Critical weaknesses
  1. Although real-time relighting is claimed, the paper does not report any performance statistics such as fps and frame time. Please provide a timing breakdown for cubemap refreshing and hybrid rendering. Additionally, evaluate how the number of VPLs affects the performance.

  2. Line 186 mentions "indirect and environment falloff terms" and a "semantic-VPL shadow cubemap," but neither is explained in the main text.

  3. Only selecting a dominant VPL as the representative hard shadow caster is oversimplified, which may degrade the realism in the scenes with complex lighting.

  4. Line 358 claims that ComGS is based on 3D Gaussians, but ComGS is actually based on 2D Gaussians. Moreover, the objects inserted using ComGS appear to lack cast shadows in the qualitative comparisons, which should be supported in ComGS.

Questions for authors
  1. What is the rendering frame rate or frame time?

  2. What is the computational overhead of refreshing the local radiance cubemap when the mesh moves?

  3. What are the "indirect and environment falloff terms" and "semantic-VPL shadow cubemap" in line 186?

  4. In line 273, why is the captured surrounding 3DGS scene required to be modulated using the estimated scene illumination?

  5. Why the SynCom dataset provided by ComGS is not included in the evaluation?

  6. Why do the objects inserted using ComGS appear to lack cast shadows in the qualitative comparisons?

Reviewer_Uet2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses illumination-consistent insertion of textured meshes into existing 3DGS scenes. Rather than estimating a view- or location-specific HDR environment map per query, the authors estimate a reusable, scene-level set of ~100 virtual point lights offline: a 3D probe grid samples RGB-D panoramas from the Gaussian scene, PanoLight lifts them to HDR, semantic light sources are detected and tracked with DEVA and lifted via depth, and VPL positions/intensities are optimized against the HDR observations with a mixed 3D importance-sampling initialization plus energy/color regularizers. At runtime, a hybrid renderer on a consumer GPU (RTX 3060) shades the inserted mesh with position-conditioned VPL lighting, a dynamic radiance cubemap for reflections, and a single dominant-VPL shadow cubemap, composited with the original 3DGS render. On a new paired benchmark, the method beats DiffusionLight, LiMo, and ComGS on average.

Critical strengths
  1. The right problem decomposition for interactive use. A persistent, position-queryable scene illumination representation — estimated once, then reused as the object moves — is exactly what AR-style insertion needs, and it cleanly separates this paper from per-query HDR estimators (DiffusionLight, LiMo). The shadow ablation concretely demonstrates the value of the scene-level treatment.
  2. Practical, reproducible engineering with a useful benchmark. The offline/online split, consumer-GPU rendering, controlled protocol, and a paired object-free/object-present benchmark make the work easy to build on.
Critical weaknesses
  1. Thin and fragile quantitative evidence. Five scenes and 30 images, no variance, uneven per-scene wins, and the strongest claims (real-time, generalization) are supported anecdotally. As presented, the evidence supports "promising pipeline" more than "demonstrated improvement."
  2. Novelty is mostly combinatorial, and key baselines are missing. PanoLight + DEVA + importance sampling + VPL rendering are all prior art; the contribution is their integration. That can be fine, but then the comparison bar matters more — and the closest insertion-specific method and any relightable-3DGS baseline are absent.
Questions for authors
  1. Report FPS and per-frame cost breakdown (VPL shading, cubemap refresh, compositing) on the RTX 3060.
  2. How sensitive are results to the number of VPLs (e.g., 50/100/200) and to the probe-grid density?
  3. How are probe locations sampled to guarantee coverage near typical insertion spots, and what happens when the object is placed far from all probes?
Reviewer_Zz5AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper estimates spatial lighting representation from an existing 3D Gaussian Splatting scene for mesh-based object insertion. The paper proposes a virtual point light (VPL) optimization framework with 3d importance sampling. Beforehand, the framework uses PanoLight to reconstruct HDR observations at spatial probes and jointly fits the positions and RGB intensities of virtual point lights (VPLs). VPL initialization combines spatial coverage, HDR importance, and semantic source sampling, while semantic priors constrain the optimization to represent light sources. Optimized VPL is combined with dynamic reflection cubemap and representative-source shadows to render inserted meshes. Evaluation on includes a paired synthetic insertion benchmark, qualitative examples, and component ablations, showing relatively realistic lighting generation on inserted objects.

Critical strengths

A useful formulation for reusing illumination from existing 3DGS scenes. Estimating a VPL field enables inserted objects to move through a reconstructed scene without repeating HDR estimation at every new location during object insertion. This may enhanced applicability of 3DGS scenes on various applications involving dynamic interaction and editing.

Critical weaknesses
  1. The evaluation does not sufficiently isolate lighting reconstruction accuracy or the contribution of the proposed sampling. Object-insertion quality is relevant to lighting reconstruction of the scene, but final rendered images alone do not establish the accuracy of the recovered scene lighting. Additional analysis is needed to distinguish approximation of PanoLight's predicted HDR observations from recovery of the underlying illumination, and to analyze how well the field generalizes to locations not used for fitting.

  2. The sampling and optimization design is insufficiently explained in the main paper. Sec. 3.4 the probe-radiance prediction and HDR reconstruction loss, and explicitly states that VPL positions and intensities are jointly optimized. However, understanding the full framework requires Appendix A.1, which introduces numerous additional constraints without proper link with the main paper. The notation also changes (e.g., e_i in the main paper to I_i in the appendix for the RGB intensity), making hard to understand the whole optimization process

Questions for authors
  1. Clarification of fitting and rendering. How are initial VPL intensities and 3D anchors constructed, and how are the scene-specific sampling budgets and hyperparameters selected? Is the reflection cubemap maintained in linear HDR space?

  2. Direct validation of the spatial lighting field. Could the authors compare VPL-predicted incoming radiance with PanoLight HDR observations at both probes and held-out locations, and with ground-truth incoming radiance if it is available in the synthetic scenes? The indirect illumination visualization from FlashCache[1] together with lighting errors would be an example for the visualization.

  3. Clarification and validation of the sampling ablation. In table 2, What exactly replaces mixed VPL, and which semantic components are removed in each ablation? In addition, visualizing the initial and optimized VPL positions and intensities with and without the proposed sampling would help establish would help to understand the benefit of proposed sampling method.

[1] Vervin et al. "Flash Cache: Reducing Bias in Radiance Cache Based Inverse Rendering", ECCV 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.