acceptodds
Under review as a conference paper at ICLR 2027

DSTex: Dense Surface Decoding for Native PBR Texture Generation

Abstract

Given a mesh and a reference image, native physically based rendering (PBR) texture generation synthesizes relightable surface materials directly in 3D. Latent-based frameworks typically compress these materials with an autoencoder and learn to generate the resulting latents through diffusion. Reconstructing fine lettering and intricate patterns from compact latents remains challenging, while increasing their number raises diffusion costs during training and sampling. We present DSTex, a native PBR texture generation framework that improves reconstruction by expanding the decoder's spatial feature capacity at a fixed latent count. Its autoencoder, DS-VAE, encodes material point samples into compact anchor tokens and expands them at finer locations sampled from the known mesh. Shared feature projection and relative-position modulation adapt each latent's content to these locations, constructing fine anchor tokens for continuous material queries without adding diffusion tokens. An image- and geometry-conditioned rectified-flow model generates the compact latents, which DS-VAE decodes into PBR materials on the input mesh. Quantitative and qualitative evaluations demonstrate that DSTex improves material reconstruction at matched latent counts and outperforms the evaluated open-source baselines in PBR texture generation.

Primary area:
generative models
Keywords:
Native 3D Texture Generation, Latent Diffusion Model, PBR Material Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GgL7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DSTex, a PBR material generation method conditioned on an existing mesh and a single reference image. Its core contribution is a dense decoding design within a surface material autoencoder: compact surface-anchor latents are expanded to more densely sampled locations on the mesh, and positional modulation constructs fine-grained features for predicting albedo, metalness, and roughness. This expansion occurs only during decoding, without increasing the diffusion model’s latent sequence length. Experiments on material reconstruction, image-conditioned generation, and relighting validate the design and demonstrate quality improvements at a fixed latent count.

Critical strengths
  1. The core contribution is clear and directly supported by controlled ablations.
Table 2(e) compares variants with and without the densifier using the same latent counts during training and inference. Adding the module increases albedo PSNR from 36.21 to 38.94 and metalness/roughness PSNR from 38.46 to 40.61, while approximately halving LPIPS in both cases. This provides reasonably convincing evidence that increasing the density of surface features during decoding improves material reconstruction.

  2. Reconstruction gains carry over to generation.
In Table 1, adding the densifier at the same latent count reduces relighting LPIPS from 0.0734 to 0.0663 and albedo UV-map LPIPS from 0.1833 to 0.1767. Although the improvements in generation are smaller than those in reconstruction, the results support the module’s practical value within the complete generation system.

  3. The evaluation protocol is reasonably well specified and includes quantitative relighting validation.
The paper evaluates the methods on 1,000 held-out assets, with all methods using the same meshes, UV layouts, and texture resolution. Relighting evaluation uses environment lighting different from that of the reference image and six matched viewpoints, allowing the generated materials to be assessed under changes in illumination. The paper also reports end-to-end runtime, diffusion sampling time, and key training settings, helping clarify the quality gains and their computational costs.

Critical weaknesses
  1. The benefits of the specific dense decoding design have not been sufficiently distinguished from those of increased decoder capacity.
The existing ablation effectively demonstrates the benefit of adding the densifier at a fixed latent count, but the module also increases decoding computation. Given prior work on scaling decoders, it remains unclear how much of the improvement comes from the proposed expansion to surface locations and positional modulation, and how much could be achieved using a wider or deeper standard decoder. A comparison with an alternative decoder under a similar computational budget would help clarify the contribution of the specific design.

  2. The trade-off between quality and computational cost has not yet been fully analyzed.
In Table 1, adding the densifier increases end-to-end runtime from 73.61 seconds to 99.42 seconds, an increase of approximately 35%. This does not undermine the claim that the method does not increase the diffusion latent count, but it indicates a substantial decoding cost. Although the paper reports overall runtime and diffusion sampling time, the specific costs of decoding and texture baking, peak GPU memory usage, and the relationship between quality and cost at different expansion factors remain insufficiently clear.

  3. The stability of generated materials across lighting conditions and the sources of the improvements require further validation.
The current relighting experiments provide encouraging evidence, but the lighting conditions used for quantitative evaluation remain limited. Evaluation at the generation stage focuses primarily on albedo textures and the final rendered appearance, whereas per-channel metrics for metalness and roughness are mainly reported in the reconstruction experiments. It is therefore difficult to determine whether the generation gains primarily stem from more accurate albedo or more plausible material parameters. Given that single-image material inference admits multiple solutions, stability under multiple novel lighting conditions would provide more convincing evidence than simply requiring recovery of a unique set of ground-truth parameters. Furthermore, the reference images currently consist mainly of rendered and generated images, and generalization to real photographic inputs remains insufficiently validated.

Questions for authors
  1. Under comparable decoding computation or end-to-end runtime, does the proposed densifier retain an advantage over a wider or deeper standard decoder? Alternatively, does DSTex retain its advantage when the latent budget is adjusted to yield a comparable runtime? Could the authors provide at least one representative comparison with a matched budget to distinguish the contribution of the specific expansion mechanism from that of increased decoder capacity?
  2. Under the default configuration of 32K latents and 8× expansion, how much time is spent on material decoding and texture baking, respectively, and what is the peak GPU memory usage? How do quality and computational cost vary across different expansion factors?
  3. Could the authors evaluate the existing test assets under several additional HDRI lighting conditions to assess the stability of the generated materials? While acknowledging that single-image material inference admits multiple solutions, could the authors also provide relevant metrics for metalness and roughness to help identify which material channels contribute to the relighting improvements?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GgL7 (1/3): Decoder capacity and latent budget

We thank the reviewer for the questions. References are to the revised manuscript; W2 / Q2 and Q3 / W3 follow in Parts 2 and 3.

W1 / Q1. Densification versus decoder capacity and latent count.

At 4K inference latents, DSTex achieves higher held-out surface-query albedo PSNR than deeper/wider decoders at comparable forward decoding latency, with fewer parameters and lower peak decoding memory (Figure 10; Table 3). We select deeper/wider f=1 capacities to approximately match this latency. All four models train from scratch at 4K latents for 5M sample presentations with the same data, Muon and AdamW, and a constant learning rate of 10⁻⁴ (Appendix B.1). Figure 10 also shows lower final training texture loss for DSTex; its held-out curve uses 4,096 assets.

GenFirst also finds that deeper decoders need not improve reconstruction in its joint VAE–autoregressive training setting (Zheng et al., 2026; Appendix B.1).

Selected results from Table 3; memory includes decoder weights:

DecoderParameters4K decoding (ms)4K peak (GiB)
Deeper (52 × 768)491M52.61.35
Wider (6 × 6144)3.62B49.67.91
DSTex (f=8, 6 × 768)58M57.01.19

At 16K inference latents, the deeper/wider f=1 decoders improve albedo PSNR by 0.17/0.60 dB over the standard f=1 baseline (Table 2(a, e)). DSTex improves all six reconstruction metrics over both variants with fixed weights (Table 2(e, f)), with longer decoding and texture baking from its 128K decoder feature tokens (Table 3). Quality is evaluated on material-channel renders of 1,000 assets; PSNR is in dB.

16K evaluationDeeperWiderDSTex (f=8)
Albedo PSNR ↑31.8032.2334.22
Albedo LPIPS ↓0.17410.16670.1342
Albedo SSIM ↑0.88390.88950.9154
MR PSNR ↑33.4334.2535.21
MR LPIPS ↓0.07550.07170.0618
MR SSIM ↑0.89220.90100.9091
Decoding (ms) ↓143.4272.3729.6
2K baking (s) ↓1.071.094.30
Decoding peak (GiB) ↓1.6610.313.50

On all 1,000 assets, densified 16K takes 24.67 s versus 45.25 s for unexpanded 32K, reducing end-to-end runtime by 45.5% (Table 1). Nine of eleven quality metrics numerically favor densified 16K, including material errors and relighting LPIPS; both CMMD scores favor unexpanded 32K. Each variant uses fixed VAE–DM weights across latent counts (Section 4.3).

AuthorsOpenReview ↗

Response to Reviewer GgL7 (2/3): Runtime, GPU memory, and expansion costs

Continuing our response to Reviewer GgL7 (W2 / Q2). All references are to the revised manuscript.

W2 / Q2. Default decoding, baking, memory, and expansion costs.

For default f=8, 32K, Table 4 gives stage means over 1,000 assets on A100 SXM4 GPUs, from GLB loading through export, excluding model loading.

Stage (f=8, 32K)Time (s)
GLB loading and mesh preparation0.26
Conditioning / geometry encoding1.04
Diffusion sampling (25 steps, CFG 3)40.98
Completion / material decoding2.98
Texture baking (2K)8.47
Textured GLB export1.76
Total55.49

Runtime update. At 32K, densification increases end-to-end runtime from 45.25 to 55.49 s (+22.6%; Table 4), adding decoding and baking cost while sampling time stays similar. DM sampling dominates default runtime.

For the same f=8 model, 16K inference lowers end-to-end runtime and full-pipeline peak GPU memory (Tables 1 and 4). The peaks below are per process and include allocator reserve. Baking includes UV rasterization, per-texel queries, seam dilation, and host transfer.

f=8 inference latentsMedian peak (GiB)Maximum peak (GiB)
16K36.138.6
32K (default)38.242.4

Baselines were re-timed under the same GLB-to-export boundary with redundant I/O removed, keeping their required stages and default settings; Table 4 lists their stages (Appendix C). DSTex weights, sampling steps, and guidance are unchanged.

Expansion factor (Table 5). Reconstruction quality and costs use f=1/4/8 VAEs trained at 4K, with six layers of width 768. Table 1 reports generation quality and runtime for final 8K-trained f=1/8 models.

16K evaluationf=1f=4f=8
Albedo PSNR (dB) ↑31.6333.1734.22
MR PSNR (dB) ↑33.8134.5935.21
Decoding (ms) ↓17.6204729.6
2K baking (s) ↓1.062.394.30
Decoding peak (GiB) ↓0.832.03.50

32K inference latents, using the same 4K-trained models:

32K evaluationf=1f=4f=8
Albedo PSNR (dB) ↑33.5434.9336.60
MR PSNR (dB) ↑34.8335.8437.59
Decoding (ms) ↓557573030
2K baking (s) ↓1.514.258.32
Decoding peak (GiB) ↓1.23.56.6

Larger expansion factors improve reconstruction PSNR at higher decoder cost. At similar decoding and baking costs, f=4 at 32K has higher reconstruction PSNR than f=8 at 16K, using twice as many latents.

These are decoding peaks, including weights; Table 4 separately covers the final 8K-trained generation models.

Interactive examples and supporting results: anonymous supplementary page.

Q3 / W3 continue in Part 3.

AuthorsOpenReview ↗

Response to Reviewer GgL7 (3/3): Materials, lighting, and real photographs

Continuing our response to Reviewer GgL7 (Q3/W3). All references are to the revised manuscript.

Q3(a). Varied reference lighting.

Table 1 now reports albedo and MR variance across six reference lightings on all 1,000 assets. For each asset, we compute variance at each texel and channel across the six predicted UV maps, then average over valid texels, channels, and assets (Appendix D). MR variance is averaged over the three-channel packed map, including the constant channel whose variance is zero. Lower values indicate more consistent predictions. At 32K, DSTex (f=8) has both the lowest variance and the lowest metallic/roughness errors among the evaluated methods (Q3(b)). Values below are ×10⁻³; bold marks the best result.

MethodAlbedo variance ↓MR variance ↓
LumiTex4.1217.46
UniTEX6.35—
TRELLIS.23.0519.76
DSTex (f=1, 16K)2.9515.26
DSTex (f=8, 16K)2.4812.60
DSTex (f=1, 32K)2.7413.50
DSTex (f=8, 32K)2.3611.15

Under the expanded lighting protocol, Table 1 averages quality against GT over six reference HDRIs shared by all methods. Relighting uses six other randomly sampled HDRIs and six matched views; relighting LPIPS is computed directly at 1024 × 1024. Figure 5 illustrates material consistency for one asset under four reference lightings; Figure 12 adds four assets under five lightings. Single-image material inference still admits multiple solutions.

Q3(b). Metallic and roughness accuracy.

Selected material results from Table 1, evaluated against GT maps on the shared UV atlas (MR LPIPS at 256 × 256):

MethodMetallic MSE ↓Roughness MSE ↓MR LPIPS ↓
LumiTex0.14020.09050.6844
TRELLIS.20.13790.09390.6591
DSTex (f=1, 32K)0.05800.04600.3648
DSTex (f=8, 32K)0.03340.02820.3240

At the same 32K latent count, adding the densifier reduces metallic/roughness MSE by 42.4%/38.7% relative to f=1. These per-channel results complement the relighting metrics.

Q3(c). Which channels contribute to the relighting improvement?

To connect material accuracy to rendered quality, Table 7 decomposes the relighting LPIPS gap between f=1 and f=8 at 32K on the same 1,000 assets, using one reference image per asset under a single fixed HDRI. We render four material combinations: both maps from f=1, both from f=8, f=1 albedo with f=8 MR, and f=8 albedo with f=1 MR. Shared meshes and UVs allow direct map swapping without rebaking. For each asset, all combinations and GT share one randomly sampled relighting HDRI and six camera views; GT is only the scoring reference and is not mixed into the predictions.

Averaging each channel's LPIPS improvement over the two replacement orders gives its Shapley contribution; the two contributions sum to the total gap:

LPIPS evaluationAlbedo shareMR share
256 × 25653%47%
1024 × 102458%42%

In this comparison, improved MR contributes alongside albedo to the lower relighting error.

W3. Real photographs and failure cases.

Figures 9, 13, and 14 provide real-photograph results using default f=8, 32K. Each photograph conditions texturing of a mesh generated from that photograph. These are qualitative results because GT materials are unavailable.

Figure 15 shows predicted roughness following a glazed kettle's floral pattern, although the glaze should have relatively uniform roughness, and a highlight remaining in a camera lens's predicted base color. Appendix D discusses possible training-data correlations and limited transparent-lens examples as explanations. The Limitations paragraph acknowledges lighting leakage and material ambiguity.

Reviewer_UMidOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DSTex, which generates PBR materials (albedo, metallic, roughness) for a given mesh from a reference image. Its autoencoder compresses surface samples into anchor tokens and, at decoding, expands each token at additional mesh-sampled points using positional embeddings and offset-conditioned FiLM. This increases spatial decoding capacity without adding diffusion tokens. A rectified-flow transformer generates the anchor latents. On 1,000 held-out assets, DSTex outperforms LumiTex, UniTEX and TRELLIS.2 on all eight reported generation quality metrics.

Critical strengths
  • DSTex ranks first on all eight quality metrics in Table 1. Even without the densifier, it outperforms the evaluated baselines, and the visual comparisons show well-preserved fine details.

  • Table 2 shows that both the Surface Encoder and densification help. With 8K training and 32K inference anchors, densification improves albedo and MR PSNR by 2.73 and 2.15 dB and roughly halves both LPIPS scores, and the corresponding generators improve UV and relighting LPIPS by 3.6% and 9.7% (Table 1).

Critical weaknesses
  • My main concern is how much the paper adds beyond recent native texturing models. NaTex [1] already generates continuous texture and PBR fields from surface-associated latents with image and geometry conditioning. LaFiTe [2] and TRELLIS.2 [3] decode compact latents into denser features. DSTex’s redesigned surface encoder, joint albedo/MR latent and expansion at mesh-sampled points are real differences, and the ablations show that the encoder and densification improve performance. Still, I find the resulting advance too small for acceptance: in my view, the method mainly applies compact-to-dense decoding to surface anchors, and there is little evidence for an additional perceptual benefit from offset modulation or for a better quality and runtime trade-off than using more tokens.

  • In the reconstruction ablation (Table 2), most of the albedo LPIPS gain comes from expansion with absolute positional embeddings. Adding offset-conditioned FiLM improves PSNR and albedo SSIM, but barely changes albedo LPIPS and worsens MR LPIPS and SSIM, a trade-off the authors acknowledge in Section 4.2. So the ablation gives only weak support for a perceptual advantage from the offset modulation, the most specific part of the densifier.

  • The comparisons at fixed latent counts support the paper’s stated claim, but the practical advantage of densification over a larger latent count is not quantified. In this setup, using more tokens needs no retraining: the models with and without the densifier are both trained with 8K anchor tokens and run at 32K, and Figure 6 raises the count from 16K to 32K with fixed weights. At 32K tokens, densification increases total runtime from 73.61 to 99.42 s (Table 1), as Section 4.3 notes. Figure 6 shows both options for one asset, but dataset-level quality and runtime for the 16K settings are not reported.

[1] Lai et al., “NaTex: Seamless Texture Generation as Latent Color Diffusion”, CVPR 2026.

[2] Chen et al., “LaFiTe: A Generative Latent Field for 3D Native Texturing”, CVPR 2026.

[3] Xiang et al., “Native and Compact Structured Latents for 3D Generation” (TRELLIS.2), CVPR 2026.

Questions for authors

Could the authors report quality and end-to-end runtime on the full evaluation set for the two 16K settings in Figure 6, in particular the densified model at 16K against the model without the densifier at 32K? Comparing their quality–runtime trade-offs would help clarify the effect of densification relative to just using more tokens.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UMid (1/2): Latent budget and runtime

We thank the reviewer for the detailed assessment. All references below are to the revised manuscript.

Q1 / W3. Full-set 16K results and the quality–runtime trade-off.

Densified 16K reduces end-to-end runtime by 45.5% relative to unexpanded 32K, with lower material errors, lower relighting LPIPS, and lower full-pipeline peak GPU memory. Nine of eleven quality metrics numerically favor densified 16K; both CMMD scores favor unexpanded 32K. Both material-map variances are also lower.

Table 1 reports four settings: f=1 has no densifier; f=8 uses 8× expansion. All use 1,000 assets, six shared reference HDRIs, and six other randomly sampled relighting HDRIs. See Appendices B.2 and D for evaluation details. Times are means; per-process peak GPU memory includes allocator reserve (Table 4). Variance averages per-texel, per-channel variation across the six maps (Appendix D). Bold marks the best quality, variance, or time among these four settings.

Metricf=1, 16Kf=8, 16Kf=1, 32Kf=8, 32K
UV CLIP-FID ↓12.4611.5912.1211.56
UV CMMD ↓0.8720.8360.8240.819
UV CLIP-I ↑0.95280.95590.95400.9560
UV LPIPS ↓0.20250.19090.19650.1884
Relit CLIP-FID ↓9.979.749.769.65
Relit CMMD ↓0.6090.5970.5730.554
Relit CLIP-I ↑0.97420.97530.97510.9757
Relit LPIPS ↓0.08440.07510.08160.0721
Metallic MSE ↓0.06190.03920.05800.0334
Roughness MSE ↓0.04720.03160.04600.0282
MR LPIPS ↓0.36650.32580.36480.3240
Albedo variance (×10⁻³) ↓2.952.482.742.36
MR variance (×10⁻³) ↓15.2612.6013.5011.15
Total time (s) ↓17.9424.6745.2555.49
Peak GPU memory: median (GiB) ↓35.836.138.438.2
Peak GPU memory: maximum (GiB) ↓39.238.640.942.4

Within each variant, 16K and 32K use the same trained VAE and diffusion-model weights; only the inference latent count changes. Increasing latent count improves both variants, while densification retains its advantage at 32K. Section 4.3 discusses these complementary scaling choices.

Runtime update. We reviewed all evaluation pipelines and re-timed all methods over the same 1,000 assets using a shared GLB-loading-to-export boundary (Appendix C, Table 4). We removed redundant I/O while retaining required inference stages and default model settings, including MR generation for LumiTex. For DSTex, this addressed issues such as unused input-texture loading, repeated reads, an intermediate mesh re-export, and duplicate map writes; weights, sampling steps, and guidance are unchanged. At 32K, densification adds decoding and baking cost (+22.6% total runtime); sampling time stays similar. DM sampling is the main runtime bottleneck.

W1 and W2 continue in Part 2.

AuthorsOpenReview ↗

Response to Reviewer UMid (2/2): Contribution and FiLM trade-off

Continuing our response to Reviewer UMid (W1 and W2). All references are to the revised manuscript.

W1. Contribution relative to recent native texturing methods.

Sections 2 and 3 clarify the technical differences. NaTex uses geometry-conditioned VecSets for continuous color queries and scales detail by increasing the latent count. LaFiTe queries interpolated local decoder features through an MLP, while TRELLIS.2 interpolates decoded voxel material attributes.

DSTex constructs additional feature tokens at locations sampled directly from the known mesh and processes them with self-attention for global material queries. It expands decoder features after diffusion while keeping the diffusion latent count fixed throughout denoising. For PBR generation, NaTex separately trains an albedo-conditioned material DiT for two-stage generation; our complete native PBR framework jointly models albedo and MR in one latent set and diffusion model.

Code and pretrained weights for NaTex and LaFiTe were not publicly available at evaluation time, so we could not include them in the quantitative comparison (Section 4, Baselines).

All four controlled decoders train at 4K latents; deeper/wider capacities are selected for comparable forward decoding latency at 4K inference (Table 3). DSTex achieves higher held-out surface-query albedo PSNR at 4K with fewer parameters and lower peak decoding memory (Figure 10; Table 3). At 16K with fixed weights, it improves all six reconstruction metrics over both variants, with longer decoding and texture baking (Tables 2(e, f) and 3).

For the VAEs trained at 4K, densified 16K achieves higher albedo/MR reconstruction PSNR than unexpanded 32K (Table 5). For the final generation models trained at 8K, densified 16K has lower material errors and relighting LPIPS, shorter end-to-end runtime, and lower full-pipeline peak GPU memory than unexpanded 32K (Tables 1 and 4).

W2. Offset-conditioned FiLM is a trade-off.

We agree that the FiLM ablation shows a trade-off. With the encoder and expansion factor fixed (f=8), relative FiLM improves albedo/MR PSNR by 0.88/0.32 dB and improves albedo SSIM over absolute PE alone, with a small improvement in albedo LPIPS. Absolute PE yields better MR LPIPS and SSIM (Table 2(c, f)). Section 4.2 reports both the gains and the trade-off.

Interactive examples and supporting results: anonymous supplementary page.

Thank you for the additional experiments and clarifications. The comparison between the densified 16K model and the unexpanded 32K model addresses my concern about the practical benefit of the core contribution.

I understand how the method differs from individual prior methods, but I am still unsure how novel it is compared with existing work overall. The ablations show that densification helps under controlled settings, but they do not compare the proposed densification with prior expansion methods in the same setup. However, I recognize that such controlled comparisons with components from prior work are not the convention in this literature. Given that the paper presents a practically useful PBR generation pipeline and my other concerns have been addressed, I am raising my score.

As the authors note, some closely related methods could not be compared because their code and weights were unavailable. To help others reproduce the results and for future comparisons, do the authors plan to release the code and pretrained checkpoints?

Thank you for your constructive feedback and for recognizing the practical benefit of our approach. Yes, we plan to make our code and pretrained checkpoints publicly available for research use after acceptance, subject to completion of our internal review process. We appreciate your careful assessment and helpful suggestions throughout the discussion.

Reviewer_NmGXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors focus on the problem of PBR texture generation. Specifically, the authors propose a method called DSTex, which generates PBR materials for a given mesh from a reference image. The core insight of the proposed DSTex is to expand compact latent tokens into dense surface anchors using mesh-sampled positions and relative-position modulation. Comparisons with open-source baselines and ablation studies show that the proposed method improves reconstruction and generation quality with finer surface details.

Critical strengths
  1. The writing of this manuscript is good and easy to follow.

  2. The proposed method addresses the loss of fine texture details in compact latent representations. It increases spatial decoding capacity without increasing the diffusion token count.

  3. Quantitative results and visual comparisons show improvements in small lettering and decorative patterns. These details are important for producing high-quality textured assets.

Critical weaknesses
  1. In the proposed DSTex, the main contribution extends existing decoder expansion and token upsampling ideas to surface material generation. However, the feature projection, positional encoding, and FiLM modulation are very conventional components.

  2. Densification improves quality at a fixed latent count, but it also increases decoding computation and runtime. Comparisons with wider or deeper decoders at similar computational costs are missing. This makes it unclear whether the gains require the proposed design or simply more decoder computation.

  3. Quantitative evaluation uses reference images rendered under a fixed studio environment, while broader examples mainly use AI-generated references. Performance on real photographs with varied lighting is not established. This limits confidence in the method’s generalization to practical inputs.

Questions for authors
  1. What is the key technical difference between the proposed Anchor Densifier and prior decoder-side token expansion methods, such as MSVS-VAE?

  2. At similar inference time and memory usage, how does anchor densification compare with increasing the decoder’s width or depth?

  3. Please report separate generation metrics for the metallic and roughness channels. The current albedo and relighting metrics do not directly establish the accuracy of each predicted material channel.

  4. Please provide results on real photographs or reference images under varied lighting conditions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NmGX (1/2): Contribution and decoder capacity

We thank the reviewer for the constructive questions. All references below are to the revised manuscript.

W1 / Q1. Contribution and differences from MSVS-VAE.

We agree that feature projection, positional embeddings, and FiLM are standard operations. Our contribution is a decoding design that expands each latent into multiple surface feature tokens at locations sampled from the known mesh while keeping the diffusion latent count fixed. The extra feature tokens are constructed after diffusion, avoiding a longer sequence at every denoising step.

The differences concern feature placement, construction, and readout. MSVS-VAE hierarchically splits features, predicts child offsets, and aggregates multi-scale local neighbors for SDF queries. DSTex obtains fine feature locations directly from the given mesh. Shared projection, absolute position embeddings, and relative-offset FiLM construct the features; self-attention processes them for global material queries (Sections 2 and 3.2; Figure 3).

Table 2 isolates the densification gain: with absolute position embeddings alone, f=8 raises albedo/MR PSNR from 31.63/33.81 to 33.34/34.89 for 4K-trained VAEs at 16K inference latents (Table 2(a, c)). The deeper/wider f=1 decoders reach 31.80/32.23 albedo PSNR (Table 2(e); see NmGX W2 / Q2).

For the VAEs trained at 4K, densified 16K achieves higher albedo/MR reconstruction PSNR than unexpanded 32K (Table 5). For the final generation models trained at 8K, densified 16K has lower material errors and relighting LPIPS, shorter end-to-end runtime, and lower full-pipeline peak GPU memory than unexpanded 32K (Tables 1 and 4).

W2 / Q2. Comparison with deeper and wider decoders.

At 4K inference latents, DSTex achieves higher held-out surface-query albedo PSNR than deeper/wider decoders at comparable forward decoding latency, with fewer parameters and lower peak decoding memory (Figure 10; Table 3). We select deeper/wider f=1 capacities to approximately match this latency. All four models train from scratch at 4K latents for 5M sample presentations with the same data, Muon and AdamW, and a constant learning rate of 10⁻⁴ (Appendix B.1). Figure 10 also shows lower final training texture loss for DSTex; its held-out curve uses 4,096 assets.

GenFirst also finds that deeper decoders need not improve reconstruction in its joint VAE–autoregressive training setting (Zheng et al., 2026; Appendix B.1).

Selected results from Table 3; peak memory includes decoder weights:

DecoderParameters4K decoding (ms)4K peak (GiB)
Deeper (52 × 768)491M52.61.35
Wider (6 × 6144)3.62B49.67.91
DSTex (f=8, 6 × 768)58M57.01.19

At 16K inference latents, the deeper/wider f=1 decoders improve albedo PSNR by 0.17/0.60 dB over the standard f=1 baseline (Table 2(a, e)). DSTex improves all six reconstruction metrics over both variants with fixed weights (Table 2(e, f)), with longer decoding and texture baking from its 128K decoder feature tokens (Table 3). Quality is evaluated on material-channel renders of 1,000 assets; PSNR is in dB.

16K evaluationDeeperWiderDSTex (f=8)
Albedo PSNR ↑31.8032.2334.22
Albedo LPIPS ↓0.17410.16670.1342
Albedo SSIM ↑0.88390.88950.9154
MR PSNR ↑33.4334.2535.21
MR LPIPS ↓0.07550.07170.0618
MR SSIM ↑0.89220.90100.9091
Decoding (ms) ↓143.4272.3729.6
2K baking (s) ↓1.071.094.30
Decoding peak (GiB) ↓1.6610.313.50

Q3 and W3 / Q4 continue in Part 2.

AuthorsOpenReview ↗

Response to Reviewer NmGX (2/2): Materials, lighting, and real photographs

Continuing our response to Reviewer NmGX (Q3 and W3 / Q4). All references are to the revised manuscript.

Q3. Metallic/roughness accuracy and material consistency.

Table 1 reports material accuracy on the shared UV atlas (MR LPIPS at 256 × 256), averaged over 1,000 assets and six reference lightings:

MethodMetallic MSE ↓Roughness MSE ↓MR LPIPS ↓
LumiTex0.14020.09050.6844
TRELLIS.20.13790.09390.6591
DSTex (f=1, 32K)0.05800.04600.3648
DSTex (f=8, 32K)0.03340.02820.3240

DSTex outperforms the evaluated PBR baselines. At 32K, densification reduces metallic/roughness MSE by 42.4%/38.7% relative to f=1.

Table 1 also reports per-texel, per-channel variance across the six predicted maps, averaged over valid texels, channels, and assets (×10⁻³). MR includes its constant third channel (Appendix D).

MethodAlbedo variance ↓MR variance ↓
LumiTex4.1217.46
TRELLIS.23.0519.76
DSTex (f=1, 32K)2.7413.50
DSTex (f=8, 32K)2.3611.15

At 32K, densification improves both material accuracy and consistency across reference lightings.

W3 / Q4. Real photographs and varied reference lighting.

Figures 9, 13, and 14 show real-photograph results with default f=8, 32K. Each photograph conditions texturing of a mesh generated from it. These are qualitative results without GT material maps.

All methods use six shared reference HDRIs and six other randomly sampled relighting HDRIs (Table 1; consistency results in Q3). Figure 5 shows more consistent predictions, closer to GT than TRELLIS.2's, for one asset under four reference lightings; Figure 12 adds four assets under five lightings. Single-image ambiguity remains. Figure 15 shows failures: roughness follows a glazed object's floral pattern, and a highlight remains in a camera lens's base color.

Interactive examples and supporting results: anonymous supplementary page.

Dear Reviewer NmGX,

Thank you again for your constructive feedback. We hope our rebuttal addresses the points you raised. If you have any further questions, we’d be happy to answer them.

Thank you for the additional clarifications and experiments. The additional experiments on decoder capacity and real-world results have carefully addressed my main concerns. I encourage the authors to release the code and the pre-trained models. I will maintain my original score of 3 (weak acceptance).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of revisions and anonymous supplementary examples

We thank the reviewers and Area Chair for their careful reading and constructive suggestions. Changes in the revised manuscript are marked in blue. All section, figure, and table references in our responses refer to that manuscript, including its appendices.

The main updates are:

  • Contribution and prior work: Sections 2 and 3 clarify how the decoder expands each latent into multiple surface feature tokens at locations sampled from the known mesh, while keeping the diffusion latent count fixed. They also distinguish this design from prior token expansion and native texturing methods.
  • Decoder capacity: Section 4.2, Tables 2–3, and Figure 10 compare DSTex with deeper/wider decoders. All variants are trained at 4K latents; the deeper/wider capacities are selected for comparable forward decoding latency at 4K. Under this training setup, increasing decoder parameter count alone yields limited reconstruction gains. Densification outperforms both variants in held-out surface-query albedo PSNR at 4K and all six reconstruction metrics at 16K with fixed weights. At 16K, these gains come with longer decoding and texture baking.
  • Latent budget and computational cost: Section 4.3 and Tables 1, 4–5 add full-set 16K generation results, stage timings across methods, peak GPU memory, and expansion-factor comparisons. We re-timed all methods after reviewing their evaluation pipelines and removing redundant I/O, retaining required inference stages and default model settings. Appendix C specifies the shared timing boundary. Table 1 runtimes were updated (submission → revision, in seconds): LumiTex 190.4 → 104.84; UniTEX 83.5 → 71.30; TRELLIS.2 28.6 → 16.80; DSTex at 32K: f=1, 73.61 → 45.25; f=8, 99.42 → 55.49.
  • Expanded lighting evaluation: Table 1 evaluates all methods on the same 1,000 assets under multiple shared reference HDRIs, with relighting under other randomly sampled HDRIs (Appendices B.2 and D). These quality results replace those in the submission.
  • Material evaluation and real photographs: Table 1 adds separate metallic and roughness metrics and albedo/MR variance. Figures 5 and 9 and Appendix D provide reference-lighting comparisons, real-photograph results, and failure cases. Table 7 analyzes the albedo and MR contributions to relighting improvements.

An anonymous supplementary page provides interactive 3D examples and organizes the supporting evidence by reviewer. Our individual responses include selected result tables alongside the answers, with references to the complete evidence in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.