- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents Luce, a framework for generating 3D objects with physically based rendering (PBR) materials from a single image. Luce represents each object using three geometrically independent sets of 3D Gaussians that encode albedo, metallic–roughness, and surface normals, respectively. A variational autoencoder (VAE) compresses these Gaussian sets into a unified structured latent representation. An image-conditioned rectified-flow model generates latents that are subsequently decoded into relightable 3D assets.
Critical strengths
The paper has two main strengths:
- Luce demonstrates strong generation performance on Toys4K and the authors’ benchmark of AI-generated images.
- Multi-layer image conditioning helps preserve fine surface details, including text and logos, improving fidelity to the input image.
Critical weaknesses
The paper has three main weaknesses:
-
Limited architectural contribution. The overall framework largely follows TRELLIS and TRELLIS.2. The sparse-structure VAE and flow are reused without modification. Although Luce changes the VAE’s input representation and spatial compression scheme, these adaptations appear incremental rather than substantial architectural advances.
-
Limited new methodological insight. Several central ingredients have clear precedents: MGM [1] explores Gaussian-based generation with explicit PBR attributes, and IDArb [2] uses the same albedo, metallic–roughness, and normal grouping for intrinsic decomposition. Multi-layer image features are also an established technique, as acknowledged by the authors. Beyond combining these ingredients, the paper provides limited analysis of why its particular representation is preferable to simpler alternatives, such as Gaussians with shared geometry and multiple PBR attributes.
-
Cross-modality geometric consistency remains a concern. MGM [1] shows that separately reconstructing PBR components can produce inconsistent geometry and degrade relighting quality (Fig. 6). Luce partially mitigates this issue by first generating a shared sparse voxel structure, which constrains all three Gaussian sets to a common spatial scaffold. However, I am concerned that the adopted voxel resolution is too coarse to sufficiently constrain these geometric differences, particularly around fine structures and object boundaries.
-
[1] Ye et al. Large Material Gaussian Model for Relightable 3D Generation, 2025. Paper.
-
[2] Li et al. IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations, ICLR 2025. Paper.
Questions for authors
In addition to the concerns raised above, I would like to clarify the source of the reported performance gains. The conditioning ablation shows substantial gains from multi-layer DINOv2 features, particularly on the AI-generated-image benchmark, where the single-layer Luce variant underperforms TRELLIS GS in CLIP score. What evidence isolates the contribution of the proposed PBR Gaussian representation and VAE adaptations from these conditioning gains? In particular, how does Luce compare with a TRELLIS-based baseline equipped with the same multi-layer conditioning?
4 comments
Response to Reviewer h94U (Part 2)
W2: Independent vs. shared geometry
Following the review's suggestion, we compared Luce with the shared alternative, one geometry carrying albedo, normal and metallic-roughness (MR), at matched storage and at matched Gaussian count. We did so at every stage: the Gaussian PBR fitting that produces the VAE's inputs, VAE reconstruction and generation (details in our response to DL8F):
| Variant | Fit albedo PSNR ↑ | Fit normal PSNR ↑ | Fit MR PSNR ↑ | VAE albedo PSNR ↑ | VAE MR PSNR ↑ | VAE normal PSNR ↑ | VAE color PSNR ↑ | Generation FID ↓ |
|---|---|---|---|---|---|---|---|---|
| Luce (independent) | 42.28 | 37.93 | 40.51 | 39.76 | 40.24 | 36.87 | 36.96 | 20.99 |
| Shared, matched storage | 41.42 | 37.81 | 40.74 | 38.49 | 39.17 | 35.70 | 35.82 | 22.79 |
| Shared, matched count | 41.66 | 38.02 | 40.97 | 39.55 | 40.48 | 36.61 | 36.62 | 22.22 |
At matched storage, using a separate Gaussian set for each PBR modality improves albedo reconstruction PSNR from 38.49 to 39.76 dB and lowers generation FID from 22.79 to 20.99. The shared set has a small advantage in fitted MR PSNR (0.23 dB). At matched Gaussian count, the shared set also achieves higher MR reconstruction PSNR (40.48 vs. 40.24 dB), while Luce retains lower generation FID (20.99 vs. 22.22). Our motivation is that fine albedo details, such as lettering, can occur where MR is nearly uniform. Separate sets let each modality place its Gaussians where it needs detail.
MGM and IDArb. Thank you for these pointers; we will add IDArb to the related work of the revised version and expand the discussion of MGM.
- MGM stores albedo, roughness, and metallic values on shared 2D Gaussian surfels, whose orientations determine their normals. Our ablation tests geometry sharing within Luce's own reconstruction pipeline. In this variant, each shared Gaussian carries an explicit shading normal that can differ from its geometric orientation.
- IDArb performs 2D multi-view intrinsic decomposition. It uses the standard albedo / MR / normal grouping but produces image-space buffers rather than a 3D generative representation.
- What is new in Luce's representation. To our knowledge, Luce is the first image-to-3D generative model to represent shading normals with a dedicated set of 3D Gaussians whose positions, scales, rotations, and opacities are parameterized independently of the albedo and metallic-roughness sets. We supervise the normal Gaussians using shading normals rendered with each asset's authored normal maps, allowing them to capture detail finer than the underlying geometry. At mesh export, we bake the decoded normals into a tangent-space normal map.
W3: Cross-modality consistency and voxel resolution
MGM's Fig. 6 shows how reconstructing PBR components separately can lead to differences in shape. In Luce, all three Gaussian sets are decoded from the same latent on a shared voxel grid. We supervise them with renders of the same asset from the same cameras. Each Gaussian's center offset is limited to ±0.75 voxel relative to its cell center.
To quantify agreement in foreground coverage, we measured pairwise and three-way intersection over union (IoU) across the rendered masks for albedo, shading normals, and metallic-roughness (each from its own Gaussian set, before compositing) after Gaussian fitting, VAE reconstruction, and generation. To stress-test agreement near the silhouette, we also computed three-way IoU within a 10-pixel band around the boundary of the three masks' union:
| Stage | Pairwise mask IoU ↑ (alb–nor / alb–MR / nor–MR) | Three-way mask IoU ↑ | Three-way boundary-band IoU (10 pixels) ↑ |
|---|---|---|---|
| Input fits (4 views at 1024²) | 0.9988 / 0.9991 / 0.9989 | 0.9984 | 0.990 |
| After the VAE (4 views at 1024²) | 0.9986 / 0.9987 / 0.9985 | 0.9979 | 0.989 |
| Generated, Toys4K | 0.9981 / 0.9984 / 0.9979 | 0.9973 | 0.987 |
| Generated, AI images | 0.9982 / 0.9982 / 0.9980 | 0.9972 | 0.984 |
Within the 10-pixel silhouette band, three-way IoU is 0.987 for generated Toys4K assets and 0.984 on the AI-image benchmark. Across 150 views of the fitted assets, doubling the grid resolution changes pairwise foreground IoU only slightly, from 0.9985–0.9990 at 128³ to 0.9987–0.9992 at 256³. We also compared the rendered depths of the fitted Gaussian sets inside the silhouette. At 128³, the mean depth difference ranges from 1.1 to 1.7 voxels across the three modality pairs, about 1% of the grid width.
We will add these results to the revised version.
Response to Reviewer h94U (Part 3)
W1: Architectural contribution
Luce reuses TRELLIS 2's sparse-structure stage to generate the voxel scaffold (Sec. 3.3). Our main contribution is the PBR representation generated on that scaffold. Albedo, metallic-roughness, and shading normals each have their own Gaussian set with independently parameterized geometry. Our VAE encodes all three sets into one structured latent, which a rectified flow generates from a single image. Our goal was to preserve fine detail through Gaussian fitting, VAE reconstruction, and generation. The ablations summarized below examine how fidelity changes when we share Gaussian geometry across modalities, move pooling earlier in the VAE, or condition the flow on features from a single image-encoder layer.
- Representation. Independent geometry per modality gives +1.3 dB albedo after the VAE vs. shared geometry at matched storage (W2).
- VAE. Moving spatial pooling from before the encoder to after it raises albedo reconstruction PSNR from 36.67 to 39.76 dB. Metallic-roughness and shading normals also reconstruct more accurately.
- Flow conditioning. With the same VAE, Luce scores 25.21 Toys4K FID using DINOv2's last layer and 20.99 using multiple layers (Table 4). Fig. 4 illustrates the difference on fine details such as lettering.
What each comparison tests (Toys4K):
| Factor | Experiment | Result |
|---|---|---|
| Full systems, single-layer conditioning | Luce (DINOv2) vs. TRELLIS 2 (DINOv3), same sparse-structure stage | FID ↓ 25.21 vs. 29.22 |
| Full systems, multi-layer conditioning | Adapted TRELLIS (518 × 518 input) vs. Luce (1036 × 1036 input) | FID ↓ 27.38 vs. 20.99 |
| Independent vs. shared geometry | Matched storage; a VAE and flow trained for each variant | Albedo PSNR ↑ 39.76 vs. 38.49; FID ↓ 20.99 vs. 22.79 |
| Multi-layer conditioning | Multi- vs. single-layer Luce (Table 4) | FID ↓ 20.99 vs. 25.21 |
| Fusion projector | Re-trained single-layer Luce with and without a 10.5M-parameter projector | FID ↓ 23.55 with projector, 24.39 without |
| Conditioning resolution after fine-tuning | Luce fine-tuned to 518 × 518 vs. Luce at 1036 × 1036 | FID ↓ 22.14 vs. 20.99 |
| Flow size (different training histories) | Luce 0.6B / 1.3B / 2.19B | FID ↓ 22.47 / 22.41 / 20.99 |
We hope these results are helpful. If they address the concerns raised in the review, we would be grateful if the assessment could be reconsidered, and we are happy to answer any further questions.
Thank you for the thorough rebuttal. The ablations and experiments address several of my concerns, and I am raising my rating. However, I still see the contribution primarily as an integration of PBR Gaussians into an established image-to-3D pipeline. Also, the ablations show a benefit from independent geometry, but the authors' claim that each modality allocates Gaussians where it needs more detail remains unverified.
Response to Reviewer h94U
Thank you for suggesting the TRELLIS comparison. We fine-tuned the official TRELLIS flow on our training data with the same multi-layer DINOv2 features used by Luce, at TRELLIS's 518 × 518 input resolution. This improved TRELLIS's Toys4K FID from 30.75 to 27.38, compared with 20.99 for Luce. To test the contribution of the Gaussian representation more directly, we compared two variants within Luce. At matched storage, Luce achieves 20.99 FID with a separate Gaussian set for each PBR modality, compared with 22.79 when all three modalities share one set.
Q: Luce vs. a TRELLIS-family baseline with the same conditioning
(a) Luce vs. TRELLIS 2 at comparable conditioning resolution. Table 4's single-layer Luce uses only the final DINOv2 layer on 1036 × 1036 input images. TRELLIS 2 uses DINOv3 on 1024 × 1024 input images for its latent stage. With single-layer conditioning, Luce retains its advantage over TRELLIS 2 on Toys4K FID (25.21 vs. 29.22) and KID×100 (0.101 vs. 0.165). Their AI-image CLIP scores are similar (0.808 vs. 0.811). The two models share TRELLIS 2's sparse-structure stage (Sec. 3.3) but differ in their image encoders, latent representations, and flows. To test the contribution of Luce's PBR Gaussian representation more directly, we compare modality-specific Gaussian sets with a single shared set below.
(b) TRELLIS with our multi-layer conditioning and our data, as suggested in the review. We trained it with the same layers at TRELLIS's 518 × 518 input resolution and the same data as Luce, starting from the official TRELLIS flow (best in bold, second best underlined):
Fine-tuning TRELLIS with our data and multi-layer conditioning improves its Toys4K FID from 30.75 to 27.38. With 518 × 518 conditioning for both models, adapted TRELLIS scores 27.38 FID (KID×100 0.076) and Luce scores 22.14 (0.051). On AI-image CLIP, adapted TRELLIS scores 0.849 and 518 × 518 Luce 0.839. The full 1036 × 1036 Luce model scores 20.99 FID and 0.852 AI-image CLIP. Within Luce, switching from single-layer to multi-layer conditioning lowers FID from 25.21 to 20.99 (Table 4).
(c) Decoder fidelity and VAE design. Fine-tuning the TRELLIS flow improves its image conditioning, but the resulting latents are still decoded by the original TRELLIS decoder. Detail loss is already visible in TRELLIS's reconstruction of the TV (Fig. 6), where the lettering is blurred, and its shaded-color reconstruction PSNR is 27.0 dB (Table 2). To test a specific VAE design choice within Luce, we moved pooling from the end of the encoder to the start. Albedo reconstruction PSNR fell from 39.76 to 36.67 dB, a drop of 3.1 dB. We report the full ablation in our response to DL8F.
On AI-image CLIP and Toys4K FID. As the review notes, single-layer Luce scores lower than official TRELLIS GS on AI-image CLIP (0.808 vs. 0.830), while achieving lower Toys4K FID (25.21 vs. 30.75). After fine-tuning TRELLIS with our data and multi-layer conditioning, its AI-image CLIP reaches 0.849, close to Luce's 0.852.
On the AI-image benchmark, we use CLIP to measure overall alignment between the input image and the rendered asset. In Fig. 17, Luce GS preserves more of the lettering in "GRIMNIR'S FORGE", while official TRELLIS GS scrambles the lettering despite a slightly higher CLIP score (0.845 vs. 0.842 for Luce GS). Figs. 11–18 (Appendix G) show the other comparisons with their CLIP scores. Thank you for prompting this discussion about how well image alignment scores reflect fine details such as lettering.