acceptodds
Withdrawn from ICLR 2027

HyGI: Neural Global Illumination for Dynamic Hybrid Gaussian-Mesh Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Relighting dynamic scenes composed of 2D Gaussians and triangle meshes requires computing global illumination, i.e., light transport between these hybrid assets, to produce plausible shading, shadows, and indirect illumination effects.Existing relightable Gaussian methods primarily address relighting individual Gaussian assets, rather than modeling dynamic light transport between Gaussians and triangle meshes. Although a few real‑time global illumination techniques target such hybrid scenes, they often introduce severe visual artifacts due to the heuristic approximations required by vanilla real‑time renderer. Instead, we present HyGI, a neural global illumination renderer for hybrid Gaussian–mesh scenes with dynamic objects, lights, and views. Given a scene's Gaussian and mesh assets with physically based materials, HyGI takes an inexpensive stochastic G-buffer and light-space shadow maps as scene and shadow cues, then aggregates these spatially coherent information for efficient relighting with a kernel-predicting layer guided by Gaussian footprint descriptors, and further end-to-end predicts direct shading, visibility, and indirect illumination with light encoders and separate decoders. Our method targets interactive light transport between Gaussian and mesh assets and successfully produces the high visual quality of physically-based rendering in real-time. Across twelve scenes, HyGI improves valid-surface PSNR by 1.8–9.0dB over the real-time hybrid renderer 3DGS-GI and by 2.6–11.2dB over relightable Gaussian methods on static scenes, at 29–59ms per frame.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Rendering, Global Illumination, Gaussian Splatting, Dynamic Scene Relighting, Real-Time Rendering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_17BaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HyGI trains a separate lighting model for each scene containing Gaussian and mesh assets. A hybrid path tracer provides supervision. At runtime, learned filters process stochastic geometry, material, and shadow inputs before networks predict direct lighting, visibility, and indirect lighting. Evaluation covers changing views, lights, and object configurations, with higher reported valid-surface PSNR than 3DGS-GI at a higher per-frame cost.

Critical strengths

The paper reports improvements over 3DGS-GI in rendering global illumination for hybrid Gaussian–mesh scenes with moving objects and lights.

Critical weaknesses
  1. Limited methodological contribution. HyGI builds on LightFormer’s light and shadow encoders and NeLiF’s shading decoders and learned shadow filtering. It combines these modules with existing stochastic rasterization and kernel prediction. The contribution mainly consists of adapting and integrating these techniques for hybrid Gaussian–mesh rendering, with limited new technical insight.

  2. Insufficient experimental validation. All settings with moving objects compare only against 3DGS-GI. RNG and GS³ cover only static geometry and reconstruct their own assets. The input-processing ablations are limited to two static-geometry scenes. The comparisons and ablation studies do not adequately establish the advantages of the proposed integration or isolate the contribution of its Gaussian-specific adaptations.

Questions for authors
  1. What is the main technical advance over LightFormer and NeLiF, and which controlled experiments demonstrate its contribution?

  2. Can you provide additional relevant comparisons or controlled variants to substantiate the claimed advantages in scenes with moving objects, beyond the comparison with 3DGS-GI?

  3. Figure 1 does not clearly convey the method and needs substantial revision.

Reviewer_BPhfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents HyGI, a neural global illumination renderer for dynamic scenes containing Gaussian surfels and triangle-mesh assets with physically based materials. The method generates stochastic G-buffers and light-space shadow cues, filters the noisy inputs using kernel-predicting networks guided by Gaussian footprint descriptors, and predicts direct shading, visibility, and indirect illumination for shading. The renderer is trained for each scene, which is then used to directly inference global illumination instead of simulating light transport. Experiments on twelve scenes evaluate changes in objects, lighting, and viewpoints show robustly enhanced performance despite increase rendering cost. Multiple ablation studies show that HyGi can support multiple types of Gaussian surfels, and provide analysis of its descriptor based filtering approach.

Critical strengths
  1. A useful design for handling noisy stochastic Gaussian inputs. Stochastic rasterization reduces the cost of generating scene inputs but introduces sampling noise. Using Gaussian footprint descriptors to guide kernel-predicting filters is a well-motivated way to retain information about the primitives that individual stochastic samples do not adequately convey. Table 3 supports the usefulness of learned filtering and Gaussian footprint descriptors, particularly at low sample counts.

  2. Broad evaluation with generally strong rendering quality. The experiments cover different combinations of object, light, and camera motion, as well as multiple Gaussian surfel assets and both hybrid and all-Gaussian scenes, showing superior performance in most cases compared to existing work.

Critical weaknesses
  1. Training configuration is unclear. The distribution of training configurations is insufficiently specified. In particular, the ranges and sampling procedures for object poses, light configurations, and camera viewpoints, as well as their separation from evaluation configurations, remain unclear. These details are important for determining whether the dynamic-scene test results demonstrate interpolation within the training distribution or generalization to more challenging unseen configurations.

  2. The computational analysis does not establish the full cost of applying training for each scene. The paper reports inference times and some runtime breakdowns, but does not report training time, memory consompution or the cost of generating path-traced supervision. Such analysis is necessary to provide trade-off in training strategy and direct path tracing (e.g., 3DGS-GI).

Questions for authors
  1. Related work. I belive MCNeRF[1] provides relevant context on stochastic filtering with footprint descriptors from the radiance fields, thus worth citing

  2. Preparation and training cost. What are the per-scene times for reference generation and GI-network training, and the associated hardware and peak memory usage compared to existing methods?

  3. Filter support. Based on my knowledge, Monte Carlo denoising for rendered results or G-buffers require more larger kernel. Could the authors provide a kernel-size ablation, for example comparing 3×3, 5×5, and larger kernels? Also, it would be better to compare with non-learned alternatives such as joint bilateral filteirng or non-local means filtering.

  4. Partitioning Pyramids. How does the partitioning pyramid work in lines 243-246? Does the filteirng and partioning work similar with NPPD[2]

[1] Gupta et al. "MCNeRF: Monte Carlo Rendering and Denoising for Real-time NeRFs", SIGGRAPH ASIA 2023

[2] Balint et al., Neural Partitioning Pyramids for Denoising Monte Carlo Renderings, SIGGRAPH 2023

Reviewer_q6BKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HyGI is a per-scene neural renderer for global illumination in scenes that mix Gaussian (surfel or volumetric) assets with triangle meshes, under a moving camera, a moving point light or environment map, and moving objects. Supervision comes from a hybrid Monte-Carlo path tracer obtained by extending PTIR-GS to GFSGS surfels and meshes (Sec. 3.3, Eq. 3). At runtime the scene is rasterized stochastically (one accepted surfel per sample, Eq. 4) into a camera G-buffer and a light-space map; two kernel-predicting U-Nets filter the surface attributes and shadow cues, guided by Gaussian footprint descriptors; and a lighting network predicts unshadowed diffuse/specular direct shading, visibility (by learned filtering of the shadow-map occlusion) and indirect light from 500 virtual point lights (Eq. 1, Sec. 3.5). On twelve scene/light settings HyGI has the highest valid-surface PSNR in every scene, 1.8–9.0 dB above 3DGS-GI and 2.6–11.2 dB above RNG/GS³ on the five static scenes, at 29–59 ms per frame at 512² (Tables 1–2). Ablations cover asset geometry (Fig. 3), samples per pixel, filtering and descriptors (Table 3), the screen-space low-pass floor (App. A) and temporal behaviour in videos (Sec. 4.3, App. D).

Critical strengths
  • Input/supervision consistency is handled with care, and its importance is demonstrated. Sec. 3.4 orders fragments by the exact ray–surfel hit depth so that Eq. (4) reproduces ray-ordered compositing, and removes the rasterizer's low-pass floor. App. A shows that with the floor the silhouette coverage converges to about twice the reference value (0.94–0.95 vs. 0.46–0.50) and that removing it is worth 0.6–0.85 dB. A systematic mismatch of this kind would cap what any network could learn from the buffers, so this is a substantive rather than cosmetic part of the contribution.

  • Large and consistent improvements under a fair protocol. HyGI is best in valid-surface and receiver PSNR in all twelve scenes (Table 1), and Fig. 2 shows the shadow and detail differences that explain the numbers. The baselines are treated generously: 3DGS-GI gets 1 or 32 accumulated frames, whichever is better, plus environment importance sampling it does not natively have; RNG/GS³ get twice the training iterations; and the paper is explicit that RNG/GS³ reconstruct from images while HyGI receives the assets. Table 3 relates spp, filtering and descriptors to both accuracy and seed-to-seed noise σ, which is the right way to evaluate a stochastic input.

Critical weaknesses
  • The reported test frames were used for model selection. Appendix E states that the held-out frames "informed model selection" and "are validation rather than untouched test frames", and Sec. 4.2 reports, per scene, the better of the raw and EMA weights (3DGS-GI likewise gets its better accumulation setting on the same frames). Choosing between two checkpoints is probably a small effect, but the paper's main claim is generalization to new camera/light/object configurations of the trained scene, and the abstract's dB margins are stated as test results. A frozen-pipeline evaluation on configurations never looked at, and a statement of what else (architecture, losses, λ, spp, shadow-map resolution) was tuned on these frames, are needed to know how much of the margin survives.

  • The contribution relative to existing neural GI is not isolated, and the headline margins are PSNR-specific. Sec. 3.5 says the network combines kernel-predicting denoisers, LightFormer's light/shadow encoders and NeLiF's decoders and shadow filtering. The experiments, however, compare only with a non-neural real-time pipeline (3DGS-GI) and with image-based relightable Gaussians; the ablations remove inputs, not design choices. The Gaussian-specific component that is ablated — the footprint descriptors — adds 0.50 and 0.04 dB at 8 spp on the two scenes in Table 3 (and slightly lowers receiver PSNR on stone_plane, 28.76 → 28.64). An adaptation of LightFormer or NeLiF trained on the same hybrid buffers, references and budget would show whether the gains come from the hybrid input construction (which may well be the contribution) or from the network. Separately, on the static scenes where RNG and GS³ apply, the perceptual and HDR metrics do not favour HyGI: on Coffee dense and desk_chair, RNG/GS³ match or beat it in SSIM (Table 7) and HDR-FLIP (Table 10: 0.383 vs. 0.359/0.352; 0.494 vs. 0.466/0.432), so "2.6–11.2 dB" overstates the advantage on those scenes.

  • Sec. 3.2: "the o G opacity of 2DGS" — notation not defined.

  • Tables 2 and 11 largely duplicate each other; one could be dropped.

  • Several occurrences of "GS³relight", "GS³render", "GS³reconstruct" appear to be missing a space (possibly a typesetting artifact).

  • Per-scene training time (50k iterations, batch 4 at 512²) and reference-rendering time are not reported anywhere.

Questions for authors

Evaluation protocol (clarification and request; decisive for Weakness 1). Which choices were made using the held-out frames — only raw vs. EMA, or also architecture, losses, λ, spp, shadow-map resolution? After freezing all choices, can you report Table 1 on an independently sampled set of camera/light/object configurations for the main dynamic scenes? How much does the raw/EMA selection change the numbers?

Matched neural baseline (request; decisive for Weakness 2). What is required beyond the stochastic hybrid inputs and their filters relative to LightFormer/NeLiF? How does one of them perform when trained with the same hybrid G-buffers, shadow cues, reference images and iteration budget?

Per-scene cost (clarification; affects the practicality claim). What are the wall-clock costs of rendering the 800–1500 reference frames at 256–512 spp (Table 6) and of the 50k-iteration training, per scene? 3DGS-GI needs neither, so the comparison in Table 2 should be read together with this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.