acceptodds
Withdrawn from ICLR 2027

VIDEORNG: RELIGHTABLE NEURAL 3D ASSET EXTRACTION FROM GENERATIVE VIDEO MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating realistic 3D assets remains constrained by classical representations: surface meshes and compact physically based materials struggle to express objects whose appearance arises from fine geometry or complex light transport, while existing Gaussian-splatting generators typically bake fixed illumination into appearance. We introduce VideoRNG, a two-stage method that generates complete, relightable 3D assets from a text prompt or a single image without prescribing a mesh or analytic material model. A video generator fine-tuned from a pretrained video diffusion backbone first produces a structured RGBA turntable sequence under a known pose, camera, and illumination protocol. A feed-forward large reconstruction model initialized from a pretrained video diffusion backbone then maps the video-VAE latent frames of selected turntable frames directly to millions of anisotropic Gaussians with learned neural appearance features; all of its parameters are fine-tuned for reconstruction. A shared directional decoder and stochastic Gaussian ray tracer turn this representation, which we call ray-traced Relightable Neural Gaussians (rtRNG), into radiance under novel viewpoints and illumination. To preserve high-frequency appearance while learning geometry, we introduce depth-anchored appearance training, and we make million-Gaussian reconstruction practical through foreground token selection and cost-aware distributed sampling. VideoRNG generates complex assets such as fur, feathers, and spider webs that can be interactively rendered, relit, and integrated into scenes. Our anonymous results page: https://videorng.github.io/

Primary area:
generative models
Keywords:
3D Asset Generation, Relightable Neural Gaussians, Large Reconstruction Models, Video Diffusion Models, Neural Rendering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qNe4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VideoRNG, a system for generating relightable 3D assets from a text prompr or a single image. First, a video diffusion model generates a turntable video under fixed camera and lighting protocol. Then, a subsequence feed-forward LRM predicts a Gaussian splat per foreground pixel, each with a neural appearance latent, that is decoded by a shared directional MLP and rendered with stochastic Gaussian ray tracing. The proposed contributions are: 1) a pipeline that generates relightable 3D assets, without constraining the geometry/material representations to coarse meshes and PBR, 2) an LRM that reconstructs relightable neural Gaussian splats in a feed-forward pass.

Critical strengths

Well-motivated problem setup: Generating relightable 3D assets with fur, feathers, and thin structures, which are hard to represent with mesh+PBR, is a meaningful direction. The proposed ray-traced Relightable Neural Gaussians (rtRNG) allow the assets to be integrated into a path tracer with inter-asset shadowing.

Critical weaknesses

1. Design choices not justified with insights and evidences

This reviewer understands that the overall pipeline works, but most components are existing techniques, and this reviewer could not find why each design choice should be the proposed one, at both the logical and experimental levels.

  • 1.1) Generative part: The main contribution seems to be the feed-forward LRM that reconstructs rtRNG, while the generator, a straightforward fine-tuning of a pre-trained video model, serves as a front end that canonicalizes camera and lighting. Using generative video as observations for neural materials was also proposed in [1]. Would it be better to reposition the contributions toward the reconstruction method? Related to this, why is the LRM restricted to the fixed turntable protocol, although it already takes per-token Plücker rays (Sec. 3.2) like prior LRMs that accept arbitrarily posed views (e.g., GS-LRM [2], LSRM [3])? If the fixed protocol is necessary because the LRM relies on the known studio illumination, then this reviewer thinks that this is a limitation of the design rather than of the task, as ReLi3D [4] already reconstructs relightable assets from posed views under unknown illumination. Also, this reviewer has concerns about how the LRM, trained only on clean renders, handles the lighting and cross-view inconsistency that might be inherent in the generated videos, and this has not been evaluated or discussed.
  • 1.2) rtRNG: rtRNG is mostly a combination of RNG [5] and stochastic Gaussian ray tracing [6], without discussion of any technical difficulty in combining them. Moreover, ray-traced Gaussians (e.g., 3DGRUT [7]) and relightable Gaussians with explicit appearance models (e.g., RGCA [8], R3DG [9], TRON [10]) already exist, and the renderer used [6] also supports relightable per-Gaussian shading. The paper does not discuss which effects rtRNG can represent but PBR Gaussians cannot, or the cost of encoding all appearance into a single latent (editability, interpretability). Since the baselines in Tables 2/3 are meshes with PBR or Gaussians with baked appearance, the gains cannot be attributed to rtRNG rather than to the pixel-aligned geometry. Missing comparisons include a PBR-head variant of the same LRM and renderer, per-object PBR-GS [9,11,12] and RNG [5] optimization on the generated frames (as in Appendix C.9; the known studio lighting is even favorable for PBR inverse rendering), and open-source relightable baselines (ReLi3D [4], Pixal3D [13]). The authors should also show cases where rtRNG represents effects that PBR Gaussians cannot (e.g., complex specular or anisotropic appearance on simple geometry, which Sec. 3.4 claims but never shows).
  • 1.3) Proposed components: The two components claimed as improvements are not sufficiently validated.
    • The transmittance blend (Eq. 7) is compared only against RNA's binary selection (Table 7), although continuous shadow conditioning was already explored in RNG [5]; comparisons with RNG-style continuous conditioning or simple multiplicative visibility are needed.
    • For depth anchoring, Table 5 isolates its effect (26.32 → 30.61 dB) but measures fitting on the same 50 assets, while Table 6 evaluates generalization but compares against "rendering loss only," which lacks depth supervision, so its gain (24.49 → 29.22 dB) mixes anchoring with depth supervision. A full-scale comparison between Eq. 4 and Eq. 5 is needed.
  • 1.4) Gaussian prediction architecture: Feed-forward Gaussian prediction has been explored with pixel-aligned depths along rays (GS-LRM [2], LGM [14]), 3D-anchored structured latents (TRELLIS [15]), and learnable Gaussian tokens (TokenGS [16]). The paper adopts the pixel-aligned design with only a brief rationale (avoiding free-space placement, Sec. 3.2). Notably, TokenGS argues that ray-aligned depth regression is suboptimal, and that decoupling Gaussians from pixels improves robustness to multi-view inconsistencies, yields more regularized geometry with fewer primitives, and enables TTO without degrading learned priors. These are exactly the issues observed here: duplicated geometry (Weakness 2), elongated splats (Weakness 3), and degraded TTO results (Appendix C.9). A discussion or comparison with these alternatives is needed.
  • 1.5) Input view variants: The RGB and orthographic six-view variants (Sec. 3.1) are mentioned without motivation or evaluation. What are the benefits of having orthographic six-view inputs, rather than turntable views?

Overall, the paper reads as a stack of existing components and variants rather than a coherent method with clear insights.

2. Weak relighting evidence and reconstruction fidelity

  • The LRM reproduces input views at only 22.11/24.84 dB (Fig. 12) and 22.15 dB (Table 4). Also, it often loses the specular highlights of the generated videos (e.g., "Witch's butter fungus", "Orca", and "Sunglasses" on the project page). This reviewer could also observe hallucinated and duplicated geometry/texture, such as overlapped textures on the "Rocking horse"'s face and three front legs in the side view.
  • The ground-truth comparisons on the project page and Fig. 9 only rotate the training environment map, yet PSNR still drops dramatically for assets with high specular effects. Also, the HDRIs are not shown, and seem to be using too simple ones, that are similar to the training lighting (studio lights with one or two directional highlights).
  • This reviewer suggests a controlled relighting evaluation in two settings: 1) feeding the GT turntable videos for unseen assets to the LRM, which isolates the relighting quality of rtRNG, and 2) running the full single-image pipeline from a rendered view of the GT asset, which can evaluate the whole system. In both settings, the GT assets should be rendered under multiple complex IBLs, and the LRM predictions should be relit under the same IBLs, preferably with zoomed-in comparisons of specular highlights. The same protocol should be applied to the PBR alternatives in Weakness 1.2, to justify the benefits of using neural Gaussians.

3. Missing geometry evaluation

Even though the paper claims to generate 3D "assets" and the predicted depths are directly measurable, no geometry metrics (depth, normal, or Chamfer error) are reported. This is critical for rtRNG, since the decoder encodes directions in each Gaussian's local frame and the transmittance is traced through the Gaussians, so incoherent geometry leads to misplaced highlights and shadows. The "Selenite towers" on the project page shows this issue: the generated views contain flat crystal faces and sharp edges, but the reconstruction result seems to be consists of thin, elongated Gaussian splats without coherent surfaces, and the satin sheen is lost. To justify the claim of "3D asset" creation, this reviewer recommends the authors to report depth/normal errors, Chamfer distance, normal visualizations, and an analysis of whether each Gaussian's local frame aligns with the surface normal.

4. Unfair evaluation setup

In Tables 2/3, VideoRNG receives 29 frames covering 360°, while baselines receive 1–16 views. Also, while LSRM [3] already made its PBR inverse rendering model publicly available, the authors choose to use its baked textures as albedo, which makes the overall comparison experiments unfair and carelessly designed. Moreover, the multi-view TRELLIS.2 is described only as "our fine-tuning," without any details. Extending TRELLIS.2 to multi-view is non-trivial, so authors should have provided details. Pixal3D [13] is an open-sourced, multi-view extension of TRELLIS.2, so this reviewer recommends the authors to properly cite the paper and include it as a baseline, rather than naively using own multi-view extension of TRELLIS.2.

5. Limited generalization analysis

Although the paper claims generation from a single image, all image-conditioned results use rendered inputs (the "input photo" in Fig. 8 is also a render). Results on real images, e.g., Stanford-ORB [17] or DTC [18] with GT environment maps, would be more compelling. Moreover, since TTO degrades novel views and lighting (Appendix C.9), the final quality seems to be bounded by the feed-forward prediction.

6. Issues with the paper's presentation

Figure 2 contains several presentation issues that look unprofessional and discounts paper's quality. This overall makes the paper look not ready to be properly reviewed, and even feel like AI-slop. Some examples: overlapping texts and shapes, meaningless bolds for "K" and "M", misaligned shapes, legends, barely readable labels in (b), and an MLP labeled “4L × 64” that contradicts the 9-layer, width-512 decoder in Sec. 3.4. This reviewer wonders whether generative AI tools have been used to create this figure, and if so, should this be disclosed in the AI use statement, which mentions only coding, prompt construction, and manuscript suggestions? Also, the title on OpenReview (“VideoRNG: Relightable Neural 3D Asset Extraction from Generative Video Models”) differs from the title in the PDF (“VideoRNG: Relightable Neural Gaussians from Generative Video Models”); which one is intended?


[1] Xue et al., VideoNeuMat: Neural Material Extraction from Generative Video Models, SIGGRAPH 2026

[2] Zhang et al., GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting, ECCV 2024

[3] Li et al., LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows, ECCV 2026

[4] Dihlmann et al., ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination, ICLR 2026

[5] Fan et al., RNG: Relightable Neural Gaussians, CVPR 2025

[6] Xu et al., Stochastic Ray Tracing for the Reconstruction of 3D Gaussian Splatting, CVPR 2026

[7] Wu et al., 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting, CVPR 2025

[8] Saito et al., Relightable Gaussian Codec Avatars, CVPR 2024

[9] Gao et al., Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing, ECCV 2024

[10] Perel et al., TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructions, arXiv 2026

[11] Liang et al., GS-IR: 3D Gaussian Splatting for Inverse Rendering, CVPR 2024

[12] Jiang et al., GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces, CVPR 2024

[13] Li et al., Pixal3D: Pixel-Aligned 3D Generation from Images, SIGGRAPH 2026

[14] Tang et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, ECCV 2024

[15] Xiang et al., Structured 3D Latents for Scalable and Versatile 3D Generation, CVPR 2025

[16] Ren et al., TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens, CVPR 2026

[17] Kuang et al., Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark, NeurIPS 2023

[18] Dong et al., Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset, CVPR 2025

Questions for authors

(References as in the Weaknesses section.)

Clarification requests

  1. Is the fixed turntable protocol necessary for the LRM (Weakness 1.1)? If so, is it because the LRM relies on the known studio illumination, and how robust is the LRM, trained only on clean renders, to deviations in the generated videos (lighting, cross-view inconsistency)?
  2. Which input variant (RGBA turntable, RGB turntable, or orthographic six views) and which decoder (9-layer shared, 6-layer globally distilled, or 4-layer per-asset) were used for each figure and table? For the orthographic variant, how is the lighting defined, and what are its benefits over orbital views (Weakness 1.5)? Please provide a clear insights on these design choices.
  3. How is the GT depth for depth anchoring defined for fuzzy or semi-transparent objects (fur, feathers, crystals)?
  4. How was the multi-view TRELLIS.2 fine-tuned (data, number of input views), and why was LSRM [3] relit with its textures as albedo instead of its publicly-available inverse-rendering variant (Weakness 4)?
  5. What are the trade-offs of the pixel-aligned design against alternatives such as TokenGS [16], especially regarding the duplicated geometry, weirdly enlongated splats from the results (Weakness 1.4)? Will 2DGS, triangle splatting, etc improve the geometric reconstruction for the assets?

Requested evidence that could change this reviewer's assessment

  1. A PBR-head variant of the same LRM and renderer, and per-object PBR-GS [9,11,12] and RNG [5] optimization on the same generated frames (Weakness 1.2). Please also show cases where rtRNG represents effects that PBR Gaussians cannot, e.g., complex specular or anisotropic appearance on simple geometry, which Sec. 3.4 claims but never shows.
  2. A controlled relighting evaluation (Weakness 2) in two settings: 1) feeding the GT turntable videos of unseen objects to the LRM, and 2) running the full single-image pipeline from a rendered view of the GT unseen object asset. In both, please relight the prediction under multiple complex IBLs and compare with the GT rendered under the same IBLs, with zoomed-in comparisons of specular highlights, for both VideoRNG and the PBR alternatives in Q7. Please also visualize the HDRIs used, preferably with complex ones that differ from the training lighting distribution.
  3. Geometry metrics against GT on the synthetic validation set, with normal visualizations, and an analysis of whether each Gaussian's local frame aligns with the surface normal (Weakness 3).
  4. A full-scale comparison between Eq. 4 (joint training with depth loss) and Eq. 5 (anchoring), and a comparison of the transmittance blend with RNG-style continuous shadow conditioning or simple multiplicative visibility (Weakness 1.3).
  5. Comparisons under matched input conditions, e.g., multi-view baselines (LSRM [3], ReLi3D [4], Pixal3D [13]) receiving the same number and coverage of input views as VideoRNG (Weakness 4).
  6. Results on real images, given the claim of generation from a single image. E.g., single photos from Stanford-ORB [17] or DTC [18] passed through the full VideoRNG pipeline and relit under diverse environment maps (Weakness 5).
Reviewer_BtUGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

VideoRNG generates relightable 3D assets from text or a single image. A fine-tuned video generator supplies turntable observations, which a video-VAE encoder and LRM convert into relightable neural Gaussians. Training fixes Gaussian centers at ground-truth depths for appearance learning while separately supervising predicted depths; inference uses predicted depths. A directional decoder and stochastic tracer support relighting and scene rendering. Evaluation includes text-conditioned generation, reconstruction of 50 synthetic objects from rendered views, and four complete single-image cases.

Critical strengths

This work proposes a sensible generation pipeline. The LRM is well matched to its controlled input setting: generated RGBA turntables, or the supported black-background RGB variant, make foreground selection and one ray-aligned neural Gaussian per foreground pixel a sensible allocation of reconstruction capacity. The training adopts several simple tricks to improve quality. The results on multiple complex assets are good.

Critical weaknesses

Applicability to real photographs remains unsubstantiated. The neural-Gaussian representation is particularly compelling for assets whose fine geometry or appearance is difficult to capture with conventional mesh/PBR models, such as the fur and web in Fig. 4. However, the practical significance of this advantage depends on whether these structures survive the transition from a real photograph with a background to the standardized observations required by the LRM. The reliance on RGBA turntables or the supported black-background RGB variant places a substantial burden on the generation stage: a background-containing RGB photograph must become foreground-separated observations while retaining delicate structures and soft boundaries. This matters because foreground masks determine which tokens and Gaussians are retained. The reported examples do not establish that the generator can perform this conversion reliably on real photographs. Consequently, it remains unclear whether the fine-structure advantage demonstrated under controlled observations extends to the claimed photograph-input setting.

Questions for authors
  • How are the ground-truth ray depths used for depth anchoring obtained for fur and other fine or partially transparent structures, and what geometry representation and rendering procedure are used to produce them?
  • How does the LRM represent surfaces invisible in all input views when each foreground pixel emits only one Gaussian constrained to its camera ray, and training fixes its center to the corresponding ground-truth depth?
Reviewer_jg31OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets 3D asset generation for objects whose appearance comes from fine geometry or complex light transport (fur, feathers, webs), which meshes with PBR materials and baked-lighting Gaussian generators handle poorly. VideoRNG first fine-tunes Wan2.1-14B on about 22K TexVerse turntable renders to produce an RGBA turntable video under a fixed camera and studio lighting. A video-DiT-based large reconstruction model then maps selected latent frames to one Gaussian per foreground pixel, each with a neural appearance latent decoded by a shared directional MLP and rendered with stochastic Gaussian ray tracing (rtRNG). Training anchors Gaussian centers at ground-truth depth while supervising predicted depth separately. Evaluation includes text-to-3D metrics on 105 prompts, a 505-person preference study, reconstruction from GT views on 50 complex objects under input and new lighting, and ablations on depth training and shading.

Critical strengths
  • The visual results are very strong, especially on fur, feathers and thin structures, which meshes with PBR materials and baked-lighting Gaussians handle poorly. This is a real gap, and the output can be path-traced with shadows and scene lighting.
  • The margins are large and consistent across many baselines (about +4.5 dB PSNR on GT-view reconstruction, and better text-to-3D metrics), backed by a 505-person preference study with paired tests and Holm correction.
Critical weaknesses
  • Limited novelty. The method is mostly a combination and scaling of existing work: RNG (per-object relightable neural Gaussians), a GS-LRM-style pixel-aligned Gaussian reconstruction model, stochastic Gaussian ray tracing, and a video-diffusion backbone. Depth anchoring is the main new training idea, and it is the least well supported (see below).

  • The evaluation seems to favor the method's core setting. (a) No relightable-asset baseline is compared (per-object RNG, RelitLRM, F-RNG and similar), although relightability is the core claim. The relighting baselines are mesh textures used as albedo, or LGM with baked light. (b) The GT-view evaluation uses 50 of the most complex TexVerse validation objects, rendered with the method's own training camera and lighting protocol, while baselines are trained on other data. The text prompts are written to favor fibrous and thin subjects. (c) The depth-anchoring study fits and evaluates on the same 50 assets. The full-scale test only compares against rendering-loss-only, not joint depth loss or frozen depth. (d) There are no real photographs, and the 'single image' inputs are synthetic renders. Relighting of generated assets has no ground truth and is judged qualitatively. (e) Zero-elevation turntables leave tops and bottoms unobserved, and this is not analyzed.

Questions for authors
  1. Can you compare against at least one relightable method (for example a per-object RNG fit on the same turntable, or RelitLRM / F-RNG if obtainable) under new HDRIs? This determines whether the relighting gains come from rtRNG or from weak relighting baselines.

  2. How do joint depth-loss and frozen-depth training compare to depth anchoring on unseen assets at full scale, and what is the PSNR drop from GT depth to predicted depth at inference? This tests whether the key training contribution generalizes.

  3. How do results change on out-of-distribution data (for example Objaverse or GSO objects, or real photos) that were not rendered with your training protocol? This separates method quality from domain match.

Convincing answers to these would move me toward acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.