acceptodds
Under review as a conference paper at ICLR 2027

TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructions

Abstract

We introduce TRON, a rendering framework that combines Gaussian ray tracing with neural rendering to enable realistic and controllable rendering of real-world 3D scenes under novel lighting, dynamic object motion, object insertion, and material editing. Prior approaches that rely solely on physics-based rendering of Gaussian representations struggle to achieve realistic relighting due to imperfections in reconstructed geometry, material estimates, and light transport estimation. At the same time, neural rendering methods often lack an explicit scene representation, limiting their ability to support interactive editing with fine grained manipulation. Instead, we leverage intrinsic decomposition priors from a learned inverse rendering model to regularize the material properties of a Gaussian field, and repurpose a ray tracer to provide radiometric guidance rather than final pixels. By treating this output as a structured 3D scaffold, we empower a lightweight neural renderer to bridge the domain gap between shading-model constrained estimates and photorealistic output. Our key insight is that the combination of explicit 3D knowledge with robust material priors provides speed and controllability, while neural rendering enables the synthesis of photorealistic images. To support real-world scenarios, we train our neural renderer with a multi-stage strategy consisting of large-scale pretraining and targeted fine-tuning on a newly constructed dataset of 2.1M rendered synthetic and real-world frames from 3D reconstructions. TRON outperforms Gaussian-based relighting methods in realism, and prior neural renderers in editability and speed. To the best of our knowledge, TRON is the first method to enable practical interactive applications in captured 3D environments, offering realistic appearance under dynamic geometric, lighting and material conditions.

Primary area:
generative models
Keywords:
neural rendering, relighting, editable reconstructions, gaussian fields, ray tracing, photorealism

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FTUAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRON combines an editable Gaussian scene with a neural renderer. After a change to the scene, a ray tracer produces a PBR image and an irradiance image. The neural renderer uses both to improve the final appearance. The paper trains this model on synthetic and real data, evaluates relighting quality and speed, and shows examples of lighting, material, and object edits.

Critical strengths

Combining Gaussian ray tracing with neural rendering is useful. The scene can be edited directly, and the irradiance input gives the neural renderer information about shadows.

The examples show clear improvements in long shadows and fine shadow details. The ablations of training stages and rendering inputs also help explain the method.

Critical weaknesses

Table 2 only evaluates the phase-1 model. The final model is used in the visual results, but its errors on this benchmark are not reported.

The reported 1.6 FPS is for single-frame rendering, while the final videos use offline processing of five frames. The cost of this mode is unclear.

The real-scene evaluation mainly measures visual quality. It does not directly show how accurately the output follows the requested lighting and material changes.

Questions for authors

Could you report Table 2 results for the final model, including errors in shadow regions?

What are the FPS and latency of the five-frame mode at the same resolution and on the same GPU?

Could you add a controlled test showing how well the output follows changes in light direction or material roughness?

2 comments

Q1. Could you report Table 2 results for the final model, including errors in shadow regions?

Thank you. We've come up with a way to compare shadows and highlights through estimated irradiance (Fig. 7, Tb. 3).

We did not add the RGB PSNR of the final model as it is not informative: after the domain shift occurs, the renderer tunes to realism at the expense of rendering exact synthetic objects (Figure 13). This is mentioned directly in the paper, line 421: "we deliberately measure the neural renderer immediately post phase 1, as afterwards we adapt it to the real domain (§4.2), which is our target setting".

Q2. What are the FPS and latency of the five-frame mode at the same resolution and on the same GPU?

Our paper presents an unoptimized neural renderer variant, which directly benefits from speed up techniques from the community. On an A6000, for example, with a drop-in replacement of an efficient VAE [1] and DLSS denoiser [2] (allows us to lower the shadow ray sample count), our neural renderer can process 5-frame chunks at 4.76 FPS (0.47 FPS reported in the paper), and 463ms first-frame latency (3.18s reported in the paper). Interactive mode runs at 7 FPS and 140ms first frame latency. Performance of both modes is doubled on L40S GPU with fp8 support.

Nothing needs to be retrained, and we happily included these speedups in our final revision, noting these are not our main contribution (Table 9 and Appendix G).

Note that the single step nature of our method makes it amenable to such optimizations, as for our unoptimized implementation 85% of the latency is spent on the VAE. Our pipeline processes each 5-frame chunk with a single forward pass, whereas the bidirectional generative baselines (DiffusionRenderer, UniRelight) process the entire video at once, using multiple denoising steps, making the DiT a bottleneck, rather than the VAE.

[1] TAEHV: Tiny AutoEncoder for Hunyuan Video, Boer Bohan 2025

[2] DLSS 4: Transforming Real-Time Graphics with AI, NVIDIA 2025

Q3. Could you add a controlled test showing how well the output follows changes in light direction or material roughness?

We added 2 qualitative figures showing a sweep of the light direction, and material editing. See Fig. 8 and Fig. 9.

Changes due to comments:

  • Shadow informed metrics: Fig. 7, Tb. 3, Section 5 (Relighting Synthetic Scenes)
  • FPS and latency: Appendix G, Tb. 9, Fig. 18, Section 5.3 (Interactivity)
  • Controlled sweeps on light direction / material: Fig. 8, Fig. 9

Thank you for your rich additional experiments for my comment and the other reviewers comment, which has illuminates most of my concerns. The clarification of my question 1 is plausible. Aditionally, the experiment comparing [1]hu and makes sense to me and explains to some extent why TRON get such good performance. Also, I am convinced by the speed now cause 7-15fps is a decent rate while the former one is a bit doudbful.

As considered by above reasons, I raise my score to 3 while conserving my confidence.

Reviewer_BhjyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRON combines Gaussian ray tracing with a single-step neural renderer for controllable rendering of reconstructed scenes. It uses intrinsic decomposition priors to estimate Gaussian geometry and materials. The ray tracer then produces PBR and irradiance buffers under the target lighting, which guide the neural renderer. Training proceeds through synthetic pretraining, real-scene adaptation, and temporal training on short clips. Synthetic experiments evaluate material decomposition and relighting quality. Real-scene experiments assess photorealism through VLM comparisons and report rendering speed. The paper also demonstrates object insertion and dynamic shadows. Supplementary videos show material editing.

Critical strengths
  1. The qualitative comparisons show more natural lighting and surface appearance than the Gaussian baselines and the ray-traced input.

  2. The system offers useful control over lighting, materials, and object placement or motion. Its fast ray-tracing preview supports iterative editing of captured scenes.

Critical weaknesses
  1. The scope of editing under the original lighting is unclear. For example, object insertion into a captured scene requires matching its original illumination while preserving the surrounding scene's appearance. Errors in material and lighting estimates may change the PBR result, and neural rendering may introduce further changes. The applicable conditions and limitations need clearer discussion.

  2. The quantitative photorealism evaluations in Tables 3–5 rely on a single VLM evaluator. These scores provide limited statistical support for conclusions about human preference. I strongly recommend a user study with a sufficient number of human participants and diverse test cases.

Questions for authors

Clarification and evaluation requests:

  1. Which rendering and editing tasks are supported under the original scene illumination? In particular, can an inserted object be matched to that illumination while retaining the surrounding scene's appearance? Please clarify the applicable conditions and limitations.

  2. Could you add final-model results to Table 2 for a direct comparison with phase 1 and the qualitative results in Figures 10 and 11?

  3. TRON's outputs tend to appear overly bright across the qualitative examples. Please clarify this.

Presentation suggestions:

  • It would be helpful to state the checkpoint and inference mode in the relevant figure and table captions.
  • Merge Figures 10 and 11, which repeat the same TRON and GT examples, and use consistent background settings.
  • Figure 6 would be more informative if all methods used the same real scene, inserted object, viewpoint, and lighting.
  • The paper appears to focus more on relighting than reconstruction. Presenting the relighting results first would better reflect this focus. The material decomposition results in Table 1 could then provide supporting evidence.
  • Please mark the best value for each metric in Table 2 in bold.
  • Table 6 leaves a large blank area on the right. Using the full available width would improve the layout.
2 comments

Q1. Which rendering and editing tasks are supported under the original scene illumination?

Thank you for the remark. Motivated by your question, we added qualitative figures (Fig. 20, Appendix H and Fig. 9b) showing that fitted environment maps let TRON edit materials, relight and harmonize inserted objects with the original illumination through differential compositing [1].

Note that by default, TRON does not optimize for the capture light when reconstructing a new scene at test time. An environment map can be fitted to the reconstruction, but we used such fits only to create the neural renderer's training inputs for 955 DL3DV scenes. An environment map fitted by differentiable ray tracing only approximates the capture light. It recovers the overall lighting but misses compact sources such as a small, bright sun, so shadows come out soft or missing. This happens because reconstructions are incomplete, and exclude nearby objects that block light (ceilings, tree canopies, the area behind the camera), the environment map has limited resolution during optimization, and an environment map can only represent distant light, not nearby lamps.

[1] Rendering Synthetic Objects into Real Scenes. P. Debevec, SIGGRAPH 1998.

Q2. Could you add final-model results to Table 2 for a direct comparison with phase 1 and the qualitative results in Figures 10 and 11?

We did not add the RGB PSNR of the final model as it is not informative: Table 2 evaluates on our synthetic benchmark, whereas the final model is further trained on real captures. After this domain shift occurs, the renderer tunes to realism at the expense of rendering exact synthetic objects (Figure 13). This is mentioned directly in the paper, line 421: "we deliberately measure the neural renderer immediately post phase 1, as afterwards we adapt it to the real domain (§4.2), which is our target setting".

To provide further validation for the rebuttal, we've come up with a way to compare shadows and highlights through estimated irradiance (see new Fig. 7, Tb. 3).

Q3. TRON's outputs tend to appear overly bright across the qualitative examples. Please clarify this.

Our limitation section acknowledges global tone shifts (line 531), which stem from a scale ambiguity in real training data, and we will clarify the cause. For each real training scene, we fit an environment map so that its rendered irradiance matches an irradiance prior predicted by [2]. Since the true HDR scale is unknown, we also set the envmap's global intensity so that the rendered PBR buffer matches the captured photo RGB, as this is the only reference with a known appearance.

However, the PBR buffer is an unshadowed render, whereas the photo contains shadows, reflections and exposure adjustments. This skews the fitted intensity, so real training inputs may come out darker relative to their photos, and the neural renderer learns to brighten its output. At test time, when both PBR and irradiance are rendered consistently from an environment map with a known scale, it overcompensates and produces outputs that appear overly bright. Since the ambiguity lies in the neural renderer's training inputs, a possible remedy is an additional input cue that anchors the absolute intensity scale.

[2] RGB↔X: Image decomposition and synthesis using material- and lighting-aware diffusion models. Zeng et al., SIGGRAPH 2024

Presentation suggestions

We went ahead and incorporated some of the suggested changes already. To avoid distracting the other reviewers with big changes, we leave the remainder of changes for the final manuscript.

W1. User Study

We will make an honest effort to run a user study. We may not make it in time for the rebuttal deadline, in which case we'll include it in the final version of the manuscript.

Changes due to comments:

  • Editing under original illumination: Fig. 20, Appendix H and Fig. 9b
  • Shadow informed metrics: Fig. 7, Tb. 3, Section 5 (Relighting Synthetic Scenes)
  • Presentation suggestions: Fig. 13, Fig. 11, Tb. 7

Thank you for your detail calrification and added experiments in revisions.

  1. Editing under the original scene lighting (W1/Q1) and over-bright output (Q3). The authors’ newly added differential rendering experiments (Fig. 9b, Fig. 20, Appendix H) and their analysis of the low-frequency limitations of ambient light fitting and the ambiguity in exposure scales in the training data are very clear and effectively address my concerns.
  2. Quantitative evaluation on the synthetic benchmark (Reviewer FTUA W1/Q1; Q2). The newly added irradiance consistency metric (Table 3, Fig. 7) is an good evaluation approach. I also understand that there may not have been sufficient time to conduct a user study. Overall, the author’s response addressed my main concerns, and I have decided to maintain my original positive rating (Weak Acceptance).
Reviewer_goyROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TRON, a hybrid rendering framework that combines 3D Gaussian Splatting, ray-traced shading, and generative neural rendering to enable relighting and editing of captured 3D environments. The method first uses DiffusionRenderer to estimate per-view normal and material priors, which supervise a staged Gaussian reconstruction and material optimization process. It then generates physically based rendering (PBR) and ray-traced irradiance buffers from the reconstructed scene and feeds them into a Cosmos-based, single-step neural renderer to produce realistic RGB images. The neural renderer is trained in three phases, progressing from synthetic data to real-world adaptation and temporal fine-tuning.

The paper evaluates TRON on synthetic and real-world scenes, demonstrating relighting, material editing, and object manipulation. Its evaluation includes quantitative image-quality comparisons, qualitative relighting results, comparisons with existing neural relighting methods, and runtime measurements for both PBR-only preview and full-quality rendering. The paper aims to combine the explicit scene control afforded by 3D Gaussian representations and physically based rendering with the perceptual quality of generative neural rendering.

Critical strengths

S1. Physically grounded conditioning for generative neural rendering. The use of complementary PBR and ray-traced irradiance buffers to condition a single-step neural renderer is an interesting and potentially valuable design. The two buffers provide different types of information about material-dependent appearance and scene illumination, offering a structured interface between an editable 3D representation and generative image synthesis. As far as I am concern, I appreciate this technical contribution that taking these two Gaussian-generated buffers as input and provides a promising direction for combining explicit scene control with perceptually realistic rendering.

S2. Integration of explicit scene editing with neural rendering. TRON integrates Gaussian reconstruction, material parameterization, ray-traced rendering, and temporal neural rendering into a unified framework, demonstrating relighting, material editing, and object manipulation in captured scenes. This establishes the practical potential of the proposed hybrid approach and makes the work more than a standalone image-enhancement method. Although the advantages over existing rendering systems remain insufficiently established, this integration is a meaningful strength of the paper.

Critical weaknesses

W1. The necessity and advantage of neural rendering over existing physically based approaches remain insufficiently established. The paper provides a reasonable motivation for introducing a neural renderer, arguing that imperfect Gaussian reconstructions and approximate shading models limit the realism of physically based rendering. However, this motivation does not establish that generative neural rendering is necessary to address these limitations. Prior work, particularly Dynamic Global Illumination for Interactive Gaussian Splatting Scenes in Real Time (DOI: 10.1109/TVCG.2026.3689199), already demonstrates interactive rendering with dynamic global illumination (they seem to support a wider range of lighting conditions, rather than just environment maps), geometry, and material editing. Without a direct comparison against such methods, it remains unclear whether the perceptual improvements offered by TRON's neural renderer justify its substantial computational overhead. The paper therefore does not sufficiently establish the practical advantage of its central design choice over existing real-time rendering alternatives.

W2. The claim of interactive photorealistic rendering is not supported by the reported runtime. The paper reports 51.81 FPS for the PBR-only preview, but this mode omits the ray-traced irradiance buffer that provides additional visibility and dynamic-shadow cues. It therefore does not demonstrate the full appearance quality and lighting effects that motivate TRON. Meanwhile, the Gaussian reconstruction front-end primarily combines diffusion-derived normal and material priors with staged Gaussian optimization, which appears relatively incremental, if you claim that PBR-only preview is one of the method's provided results. When the neural renderer is enabled, the reported throughput drops to approximately 1.6 FPS, which is far below typical expectations for interactive rendering. Thus, the paper effectively demonstrates either a fast but visually limited preview or a more realistic but slow rendering mode, rather than achieving both properties in a single pipeline. This gap directly undermines the central claim of practical interactive photorealistic rendering.

Questions for authors

Q1. Motivation and advantage of the neural renderer. Given that Dynamic Global Illumination for Interactive Gaussian Splatting Scenes in Real Time (DOI: 10.1109/TVCG.2026.3689199) already supports interactive scene editing and dynamic global illumination, what specific limitations of existing physically based rendering methods does TRON address? Can the authors provide a direct comparison under matched scenes, hardware, and resolution to demonstrate that the neural renderer offers sufficient improvements in perceptual quality to justify its additional computational cost?

Q2. Interactive rendering and end-to-end runtime. Which operating mode supports the paper's claim of interactive photorealistic rendering? The reported 51.81 FPS corresponds to the PBR-only preview, which does not include the full irradiance-guided neural rendering pipeline, whereas the full-quality mode achieves only approximately 1.6 FPS. Can the authors provide a complete runtime breakdown and clarify the visual limitations of the PBR-only mode, particularly regarding visibility and dynamic shadows? How do these results substantiate the claim that TRON enables practical interactive applications while preserving photorealistic rendering quality?

Q3. You feature dynamic shadows in applications in the project page. I wonder how to animate the reconstructed armadillo? Is this "armadillo" model also based on Gaussian primitives?

1 comment

Q1 / W1. Necessity and advantage of neural rendering over existing physically based approaches

We have added a direct comparison with Hu et al. [1], on their setting and ours, and highlight the advantage of our neural renderer for real-world captures. Observe that [1] mention in their limitations that "it relies on well-formed 3D Gaussian models and strong assumptions". They assume that all materials are known a-priori, and that the Light Transport Equation operates on a good approximation of a surface. That limits their evaluation to compositions of multiple objects with known materials, rather than captured real scenes.

Neural rendering has powerful advantages for relighting in-the-wild 3D Gaussian reconstructions, because it can fix areas of imperfectly estimated surfaces, geometry, or materials. Such artifacts are inevitable for real-world captures. For example, in areas of poor camera coverage, reconstructed scenes may contain missing tree canopies, gaps in indoor walls or ceilings required for accurate ray tracing, and highly inaccurate "fuzzy" surface geometries even for floors and walls, making accurate computation of shadows and secondary rays impossible. The key idea behind TRON is that approximate geometry and estimated materials still provide a strong 3D-consistent cue to drive the neural renderer, which is able to fix appearance and mitigate these limitations of the original scene representation.

[1] Dynamic Global Illumination for Interactive Gaussian Splatting Scenes in Real Time, Hu et al., IEEE 2026

Q2 / W2. Which mode is interactive, and what is the full runtime?

Our pipeline is interactive because it processes each frame with a single forward pass, whereas the bidirectional generative baselines (DiffusionRenderer, UniRelight) process the entire video at once, using multiple denoising steps, leading to large latency of minutes.

The single step nature of our method makes it amenable to further optimizations, as now 85% of the latency is spent on the VAE. A drop-in replacement of an efficient VAE [2] and a DLSS denoiser [3] (allows us to lower the shadow ray sample count) can boost the end-to-end interactive mode from 1.6 FPS to 7-15 FPS, depending on the exact hardware. Nothing needs to be retrained, and we happily included these speedups in our final revision, noting these are not our main contribution.

We agree that the 51.81 FPS PBR-only preview does not demonstrate the full appearance quality of TRON. We reported it as an additional preview mode, rather than as evidence of the full pipeline's rendering performance. This part is now removed from the text to avoid confusion.

[2] TAEHV: Tiny AutoEncoder for Hunyuan Video, Boer Bohan 2025

[3] DLSS 4: Transforming Real-Time Graphics with AI, NVIDIA 2025

Q3. How is the armadillo animated?

We added additional dynamic-shadow implementation details to Appendix H.

Briefly, we reconstruct the armadillo as a Gaussian representation, and use a physically-based simulation method [4] to animate the Gaussians. We then stack the armadillo's Gaussians with the scene's Gaussians. For each frame, the physics simulation [4] outputs a transformation per dynamic Gaussian. The tracer then recomputes the PBR and irradiance buffers of the combined scene, so the irradiance carries the moving contact shadow. The neural renderer produces the final frame.

[4] Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation. Modi et al., ACM TOG SIGGRAPH 2024.

Changes due to comments:

  • Comparison with Dynamic Global Illumination for Interactive Gaussian Splatting Scenes in Real Time: Fig. 12, Appendix A.2 ("On the Importance of Neural Rendering")
  • Interactivity & optimized pipeline: Section 5.3 (Interactivity), Appendix G, Tb. 9, Fig. 18
  • Armadillo animation & physically based simulations: Appendix H

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal Summary

We thank the reviewers for their comments and constructive feedback.

In response to the reviewers' suggestions, we have updated the manuscript, with changes highlighted in blue. The key changes include:

  • Interactivity and runtime performance (goyR, FTUA): Section 5.3 now explains how our single-forward-pass architecture enables interactivity and further optimization. Minor, training-free architectural changes increase performance to 7-15 FPS. Appendix G details the optimized pipeline, while Tb. 9 and Fig. 18 show the performance-quality tradeoff.
  • Additional quantitative evaluation (Bhjy, FTUA): We introduce a shadow and highlight aware metric, with results demonstrating our framework's advantage in shadow accuracy and lighting control. Section 5.2 describes the evaluation, with qualitative results in Fig. 7 and quantitative results in Tb. 3.
  • Importance of Neural Rendering (goyR): We added a comparison with Dynamic Global Illumination for Interactive Gaussian Splatting Scenes (Fig. 12), highlighting the benefits of our framework's neural rendering approach. In addition, we included a discussion in Appendix A.2, "On the Importance of Neural Rendering", for scenes in-the-wild.
  • Editing under original scene illumination (Bhjy): We added a discussion on applying our method for editing under the original scene illumination with differential rendering (Appendix H). New editing results demonstrate Harmonization (Fig. 20) and Material Editing of base color (Fig. 9b).
  • Materials and light direction sweep (FTUA): We included additional sweeps that measure sensitivity to light direction (Fig. 8) and material editing (Fig. 9). Section 5.3 has been updated with references to the new editing figures.

Reviewers previously highlighted the core idea of conditioning a neural renderer on ray-traced PBR and irradiance buffers (goyR, FTUA), the practical value of directly editing lighting, materials and objects in captured scenes (goyR, Bhjy, FTUA), more natural lighting and appearance than Gaussian baselines (Bhjy), clear improvements in long and fine shadow details (FTUA), and informative ablations (FTUA).

We hope the rebuttal provides further insight into the strengths and weaknesses of our method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.