- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DSTex, a PBR material generation method conditioned on an existing mesh and a single reference image. Its core contribution is a dense decoding design within a surface material autoencoder: compact surface-anchor latents are expanded to more densely sampled locations on the mesh, and positional modulation constructs fine-grained features for predicting albedo, metalness, and roughness. This expansion occurs only during decoding, without increasing the diffusion model’s latent sequence length. Experiments on material reconstruction, image-conditioned generation, and relighting validate the design and demonstrate quality improvements at a fixed latent count.
Critical strengths
-
The core contribution is clear and directly supported by controlled ablations. Table 2(e) compares variants with and without the densifier using the same latent counts during training and inference. Adding the module increases albedo PSNR from 36.21 to 38.94 and metalness/roughness PSNR from 38.46 to 40.61, while approximately halving LPIPS in both cases. This provides reasonably convincing evidence that increasing the density of surface features during decoding improves material reconstruction.
-
Reconstruction gains carry over to generation. In Table 1, adding the densifier at the same latent count reduces relighting LPIPS from 0.0734 to 0.0663 and albedo UV-map LPIPS from 0.1833 to 0.1767. Although the improvements in generation are smaller than those in reconstruction, the results support the module’s practical value within the complete generation system.
-
The evaluation protocol is reasonably well specified and includes quantitative relighting validation. The paper evaluates the methods on 1,000 held-out assets, with all methods using the same meshes, UV layouts, and texture resolution. Relighting evaluation uses environment lighting different from that of the reference image and six matched viewpoints, allowing the generated materials to be assessed under changes in illumination. The paper also reports end-to-end runtime, diffusion sampling time, and key training settings, helping clarify the quality gains and their computational costs.
Critical weaknesses
-
The benefits of the specific dense decoding design have not been sufficiently distinguished from those of increased decoder capacity. The existing ablation effectively demonstrates the benefit of adding the densifier at a fixed latent count, but the module also increases decoding computation. Given prior work on scaling decoders, it remains unclear how much of the improvement comes from the proposed expansion to surface locations and positional modulation, and how much could be achieved using a wider or deeper standard decoder. A comparison with an alternative decoder under a similar computational budget would help clarify the contribution of the specific design.
-
The trade-off between quality and computational cost has not yet been fully analyzed. In Table 1, adding the densifier increases end-to-end runtime from 73.61 seconds to 99.42 seconds, an increase of approximately 35%. This does not undermine the claim that the method does not increase the diffusion latent count, but it indicates a substantial decoding cost. Although the paper reports overall runtime and diffusion sampling time, the specific costs of decoding and texture baking, peak GPU memory usage, and the relationship between quality and cost at different expansion factors remain insufficiently clear.
-
The stability of generated materials across lighting conditions and the sources of the improvements require further validation. The current relighting experiments provide encouraging evidence, but the lighting conditions used for quantitative evaluation remain limited. Evaluation at the generation stage focuses primarily on albedo textures and the final rendered appearance, whereas per-channel metrics for metalness and roughness are mainly reported in the reconstruction experiments. It is therefore difficult to determine whether the generation gains primarily stem from more accurate albedo or more plausible material parameters. Given that single-image material inference admits multiple solutions, stability under multiple novel lighting conditions would provide more convincing evidence than simply requiring recovery of a unique set of ground-truth parameters. Furthermore, the reference images currently consist mainly of rendered and generated images, and generalization to real photographic inputs remains insufficiently validated.
Questions for authors
- Under comparable decoding computation or end-to-end runtime, does the proposed densifier retain an advantage over a wider or deeper standard decoder? Alternatively, does DSTex retain its advantage when the latent budget is adjusted to yield a comparable runtime? Could the authors provide at least one representative comparison with a matched budget to distinguish the contribution of the specific expansion mechanism from that of increased decoder capacity?
- Under the default configuration of 32K latents and 8× expansion, how much time is spent on material decoding and texture baking, respectively, and what is the peak GPU memory usage? How do quality and computational cost vary across different expansion factors?
- Could the authors evaluate the existing test assets under several additional HDRI lighting conditions to assess the stability of the generated materials? While acknowledging that single-image material inference admits multiple solutions, could the authors also provide relevant metrics for metalness and roughness to help identify which material channels contribute to the relighting improvements?
5 comments
Response to Reviewer GgL7 (2/3): Runtime, GPU memory, and expansion costs
Continuing our response to Reviewer GgL7 (W2 / Q2). All references are to the revised manuscript.
W2 / Q2. Default decoding, baking, memory, and expansion costs.
For default f=8, 32K, Table 4 gives stage means over 1,000 assets on A100 SXM4 GPUs, from GLB loading through export, excluding model loading.
| Stage (f=8, 32K) | Time (s) |
|---|---|
| GLB loading and mesh preparation | 0.26 |
| Conditioning / geometry encoding | 1.04 |
| Diffusion sampling (25 steps, CFG 3) | 40.98 |
| Completion / material decoding | 2.98 |
| Texture baking (2K) | 8.47 |
| Textured GLB export | 1.76 |
| Total | 55.49 |
Runtime update. At 32K, densification increases end-to-end runtime from 45.25 to 55.49 s (+22.6%; Table 4), adding decoding and baking cost while sampling time stays similar. DM sampling dominates default runtime.
For the same f=8 model, 16K inference lowers end-to-end runtime and full-pipeline peak GPU memory (Tables 1 and 4). The peaks below are per process and include allocator reserve. Baking includes UV rasterization, per-texel queries, seam dilation, and host transfer.
| f=8 inference latents | Median peak (GiB) | Maximum peak (GiB) |
|---|---|---|
| 16K | 36.1 | 38.6 |
| 32K (default) | 38.2 | 42.4 |
Baselines were re-timed under the same GLB-to-export boundary with redundant I/O removed, keeping their required stages and default settings; Table 4 lists their stages (Appendix C). DSTex weights, sampling steps, and guidance are unchanged.
Expansion factor (Table 5). Reconstruction quality and costs use f=1/4/8 VAEs trained at 4K, with six layers of width 768. Table 1 reports generation quality and runtime for final 8K-trained f=1/8 models.
| 16K evaluation | f=1 | f=4 | f=8 |
|---|---|---|---|
| Albedo PSNR (dB) ↑ | 31.63 | 33.17 | 34.22 |
| MR PSNR (dB) ↑ | 33.81 | 34.59 | 35.21 |
| Decoding (ms) ↓ | 17.6 | 204 | 729.6 |
| 2K baking (s) ↓ | 1.06 | 2.39 | 4.30 |
| Decoding peak (GiB) ↓ | 0.83 | 2.0 | 3.50 |
32K inference latents, using the same 4K-trained models:
| 32K evaluation | f=1 | f=4 | f=8 |
|---|---|---|---|
| Albedo PSNR (dB) ↑ | 33.54 | 34.93 | 36.60 |
| MR PSNR (dB) ↑ | 34.83 | 35.84 | 37.59 |
| Decoding (ms) ↓ | 55 | 757 | 3030 |
| 2K baking (s) ↓ | 1.51 | 4.25 | 8.32 |
| Decoding peak (GiB) ↓ | 1.2 | 3.5 | 6.6 |
Larger expansion factors improve reconstruction PSNR at higher decoder cost. At similar decoding and baking costs, f=4 at 32K has higher reconstruction PSNR than f=8 at 16K, using twice as many latents.
These are decoding peaks, including weights; Table 4 separately covers the final 8K-trained generation models.
Interactive examples and supporting results: anonymous supplementary page.
Q3 / W3 continue in Part 3.
Response to Reviewer GgL7 (3/3): Materials, lighting, and real photographs
Continuing our response to Reviewer GgL7 (Q3/W3). All references are to the revised manuscript.
Q3(a). Varied reference lighting.
Table 1 now reports albedo and MR variance across six reference lightings on all 1,000 assets. For each asset, we compute variance at each texel and channel across the six predicted UV maps, then average over valid texels, channels, and assets (Appendix D). MR variance is averaged over the three-channel packed map, including the constant channel whose variance is zero. Lower values indicate more consistent predictions. At 32K, DSTex (f=8) has both the lowest variance and the lowest metallic/roughness errors among the evaluated methods (Q3(b)). Values below are ×10⁻³; bold marks the best result.
| Method | Albedo variance ↓ | MR variance ↓ |
|---|---|---|
| LumiTex | 4.12 | 17.46 |
| UniTEX | 6.35 | — |
| TRELLIS.2 | 3.05 | 19.76 |
| DSTex (f=1, 16K) | 2.95 | 15.26 |
| DSTex (f=8, 16K) | 2.48 | 12.60 |
| DSTex (f=1, 32K) | 2.74 | 13.50 |
| DSTex (f=8, 32K) | 2.36 | 11.15 |
Under the expanded lighting protocol, Table 1 averages quality against GT over six reference HDRIs shared by all methods. Relighting uses six other randomly sampled HDRIs and six matched views; relighting LPIPS is computed directly at 1024 × 1024. Figure 5 illustrates material consistency for one asset under four reference lightings; Figure 12 adds four assets under five lightings. Single-image material inference still admits multiple solutions.
Q3(b). Metallic and roughness accuracy.
Selected material results from Table 1, evaluated against GT maps on the shared UV atlas (MR LPIPS at 256 × 256):
| Method | Metallic MSE ↓ | Roughness MSE ↓ | MR LPIPS ↓ |
|---|---|---|---|
| LumiTex | 0.1402 | 0.0905 | 0.6844 |
| TRELLIS.2 | 0.1379 | 0.0939 | 0.6591 |
| DSTex (f=1, 32K) | 0.0580 | 0.0460 | 0.3648 |
| DSTex (f=8, 32K) | 0.0334 | 0.0282 | 0.3240 |
At the same 32K latent count, adding the densifier reduces metallic/roughness MSE by 42.4%/38.7% relative to f=1. These per-channel results complement the relighting metrics.
Q3(c). Which channels contribute to the relighting improvement?
To connect material accuracy to rendered quality, Table 7 decomposes the relighting LPIPS gap between f=1 and f=8 at 32K on the same 1,000 assets, using one reference image per asset under a single fixed HDRI. We render four material combinations: both maps from f=1, both from f=8, f=1 albedo with f=8 MR, and f=8 albedo with f=1 MR. Shared meshes and UVs allow direct map swapping without rebaking. For each asset, all combinations and GT share one randomly sampled relighting HDRI and six camera views; GT is only the scoring reference and is not mixed into the predictions.
Averaging each channel's LPIPS improvement over the two replacement orders gives its Shapley contribution; the two contributions sum to the total gap:
| LPIPS evaluation | Albedo share | MR share |
|---|---|---|
| 256 × 256 | 53% | 47% |
| 1024 × 1024 | 58% | 42% |
In this comparison, improved MR contributes alongside albedo to the lower relighting error.
W3. Real photographs and failure cases.
Figures 9, 13, and 14 provide real-photograph results using default f=8, 32K. Each photograph conditions texturing of a mesh generated from that photograph. These are qualitative results because GT materials are unavailable.
Figure 15 shows predicted roughness following a glazed kettle's floral pattern, although the glaze should have relatively uniform roughness, and a highlight remaining in a camera lens's predicted base color. Appendix D discusses possible training-data correlations and limited transparent-lens examples as explanations. The Limitations paragraph acknowledges lighting leakage and material ambiguity.
Response to Reviewer GgL7 (1/3): Decoder capacity and latent budget
We thank the reviewer for the questions. References are to the revised manuscript; W2 / Q2 and Q3 / W3 follow in Parts 2 and 3.
W1 / Q1. Densification versus decoder capacity and latent count.
At 4K inference latents, DSTex achieves higher held-out surface-query albedo PSNR than deeper/wider decoders at comparable forward decoding latency, with fewer parameters and lower peak decoding memory (Figure 10; Table 3). We select deeper/wider f=1 capacities to approximately match this latency. All four models train from scratch at 4K latents for 5M sample presentations with the same data, Muon and AdamW, and a constant learning rate of 10⁻⁴ (Appendix B.1). Figure 10 also shows lower final training texture loss for DSTex; its held-out curve uses 4,096 assets.
GenFirst also finds that deeper decoders need not improve reconstruction in its joint VAE–autoregressive training setting (Zheng et al., 2026; Appendix B.1).
Selected results from Table 3; memory includes decoder weights:
At 16K inference latents, the deeper/wider f=1 decoders improve albedo PSNR by 0.17/0.60 dB over the standard f=1 baseline (Table 2(a, e)). DSTex improves all six reconstruction metrics over both variants with fixed weights (Table 2(e, f)), with longer decoding and texture baking from its 128K decoder feature tokens (Table 3). Quality is evaluated on material-channel renders of 1,000 assets; PSNR is in dB.
On all 1,000 assets, densified 16K takes 24.67 s versus 45.25 s for unexpanded 32K, reducing end-to-end runtime by 45.5% (Table 1). Nine of eleven quality metrics numerically favor densified 16K, including material errors and relighting LPIPS; both CMMD scores favor unexpanded 32K. Each variant uses fixed VAE–DM weights across latent counts (Section 4.3).