acceptodds
Under review as a conference paper at ICLR 2027

TaoTex: Boosting Texture Detail Fidelity for Native 3D Material Generation

Abstract

Recent 3D generation models can produce accurate geometries while still struggling to reconstruct detailed textures. We propose a diffusion-based native 3D material generation model TaoTex, which faithfully recovers intricate textures through tailored strategies and improvements. First, we develop a data construction agent to create high-frequency textured 3D assets to bridge the data gap in public datasets. Training with these data significantly enhances the ability of TaoTex to recover challenging details such as text and patterns. Second, we design a multi-level feature fusion (MLFF) module to adaptively integrate local and global features of the conditional input, providing more complete texture cues for the diffusion model and thereby enhancing reconstruction fidelity. To alleviate VAE reconstruction errors, we adopt a latent-to-pixel space loss transition, further improving the pixel-level details and generation quality. Finally, we scale TaoTex to multi-view inputs by incorporating learnable viewpoint embeddings, achieving accurate and consistent material reconstruction across views. Extensive experiments demonstrate that our method significantly outperforms existing approaches in preserving texture details in both single- and multi-view settings.

Primary area:
generative models
Keywords:
Material Generation, Diffusion, Generative Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_saCXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TaoTex addresses the loss of fine texture details in geometry-conditioned native 3D PBR material generation. Built on TRELLIS.2, it combines a pipeline that synthesizes 50K assets with simple geometry and rich textures, multi-level DINOv2 feature fusion, and a transition from latent diffusion supervision to rendered pixel-space losses. For up to four canonical input views, it adds learned viewpoint embeddings and a low-SNR-biased timestep schedule. Experiments report improved reference fidelity and clearer text and patterns, with component-removal ablations in the multi-view setting.

Critical strengths
  1. Reasonable insight and problem diagnosis. I agree with the observation that native 3D material generators struggle with detailed appearance on geometrically simple surfaces. Limited high-frequency training data, overly semantic image conditions, and material-VAE reconstruction errors are plausible contributing factors, and the method addresses each explicitly.
  2. Meaningful data construction and multi-view design. Synthesizing rich text and patterns on simple shapes targets a useful gap in existing data. The use of symmetric geometry to discourage inferring viewpoint solely from shape, together with explicit viewpoint embeddings, is also well motivated.
  3. Evidence that the combined improvements are useful. Table 3 reports decreases in albedo PSNR from 25.48 to 22.90 without HFT data, 23.40 without MLFF, and 24.61 without pixel supervision. The qualitative comparisons also show clearer fine details. These results support the practical value of the components, even though they do not establish the necessity of every design choice.
Critical weaknesses
  1. The distinct contribution of the central design choices is not sufficiently established. The component-removal ablations support the usefulness of HFT data, MLFF, and pixel supervision within the proposed system. However, they do not distinguish the benefit of MLFF's specific fusion design from other ways of retaining local image detail, or the benefit of two-stage latent-to-pixel supervision from that of adding pixel supervision itself. Combining established ideas can be a meaningful contribution, but the paper would be stronger with a clearer account of its technical advance and focused comparisons against straightforward alternatives under controlled data and training budgets.
  2. Potential orientation bias in UV-space texture synthesis is not sufficiently addressed. An IGM trained on natural images may favor a globally upright image composition when synthesizing textures within a UV layout. UV atlas axes, however, need not correspond to the upright direction of the object surface: charts may be rotated to improve packing efficiency, potentially by arbitrary angles depending on the packing procedure. The resulting textures could therefore exhibit a systematic orientation bias rather than merely occasional alignment defects. The paper acknowledges imperfect UV alignment and uses structured layouts and MLLM quality checks, but does not clarify whether these measures address this particular bias or whether any targeted mitigation was attempted. The HFT ablation supports the overall usefulness of the data, but does not establish that this potential bias is absent or inconsequential.
  3. The evidence does not fully establish the scope of the improvements. The 30-object texture-rich evaluation set is reasonable for a focused research study, and its small size is not by itself a major concern. The more important limitation is that component ablations are reported only for multi-view inputs, while the paper also emphasizes single-view generation. The reported averages also leave unclear how consistently the gains hold across objects. Per-object results and focused single-view ablations would clarify the scope of the claim without requiring a much larger benchmark.
Questions for authors

My provisional assessment is weak reject. I find the reported improvements credible; my reservation concerns the extent and characterization of the contribution. Focused clarification or evidence on the points below could strengthen my assessment; I do not expect every suggested experiment to be completed during rebuttal.

  1. Distinct contribution and conditioning alternatives. Which part of TaoTex constitutes the main technical advance over prior native-3D material generators? A simpler TRELLIS.2-based retraining baseline using the same training data, including HFT, would help contextualize the gains from the full recipe. For conditioning, how does MLFF compare with a simpler multi-layer fusion, or a reconstruction-oriented 2D image-VAE encoder? These are possible comparisons, rather than an assumption that VAE features are superior. Please clarify how training budgets are matched, including both steps and compute where per-step costs differ.
  2. Role of the supervision schedule. Does the 100K latent-to-100K pixel transition improve over latent-only training and joint latent/pixel supervision from the start under comparable budgets? Please clarify whether the existing no-pixel-loss ablation uses the same total training steps and report the compute implications of rendered supervision. Since the material VAE is frozen during DiT training, a matched ground-truth-material VAE reconstruction reference would also help contextualize the remaining detail errors, although reconstruction error alone would not establish a strict limit on what the fixed decoder can represent.
  3. Natural-image orientation bias in UV space. When the IGM synthesizes textures within the UV mask, could its natural-image prior favor text and patterns that are upright in the atlas, irrespective of the corresponding surface orientation? UV charts have no universal up direction and may be rotated for packing efficiency, including by arbitrary angles depending on the packing procedure. This raises the possibility of a systematic bias in the synthesized dataset. The paper acknowledges imperfect alignment, but have you investigated this specific orientation bias or attempted to mitigate it? In particular, do your structured UV layouts constrain chart orientations, and do the MLLM checks assess orientation relative to the object surface rather than only pattern sharpness and boundary alignment? Please clarify what was tried and any evidence of its effectiveness. Consistent image/material supervision from the same asset would not by itself rule out such a distributional bias.
  4. Scope of the improvements. Are the gains consistent across individual test objects, and do HFT, MLFF, and pixel supervision also improve the single-view setting? Per-object results and a focused single-view ablation would help assess whether the observed component benefits extend beyond the reported multi-view averages.
Reviewer_uY7cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TaoTex improves native 3D material generation through synthetic texture-rich data, multi-level image features, pixel-space supervision, and viewpoint embeddings for multi-view conditioning.

Critical strengths
  1. The texture improvements are visually convincing, especially for text and patterns.
  2. Table 1 reports substantial reconstruction gains.
  3. Table 3 supports the usefulness of the main components.
Critical weaknesses
  1. Lack of details for data construction agent, what prompts are used, how long does it take to produce a sample, what does the generated samples looks like (visual examples) ?
  2. Quantitative evaluation uses only 30 selected TexVerse objects, while TexVerse also contributes training data; the train/test separation is not specified.
Questions for authors
  1. The paper emphasis on text fidelity, it is recommended add some text-recognition metrics.
  2. What resolution does Latent-to-pixel Space Loss operate on?
  3. What is the criterion for chosen layers for Multi-level Feature Fusion?
Reviewer_3CBhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TaoTex is a native 3D diffusion model that generates PBR materials conditioned on a given geometry and one or more images, targeting high-frequency texture detail on smooth surfaces where geometry gives no cues. Contributions: (a) a data-construction agent — an MLLM (Qwen) writes and iteratively refines Blender scripts for simple-topology meshes, auto-UV-unwraps them, and an image generation model synthesizes UV-domain textures with PBR parameters prescribed by the MLLM, yielding a 50K high-frequency-texture dataset; (b) a Multi-Level Feature Fusion module that fuses four DINOv2 layers with SE-style channel attention to recover detail lost by deep semantic features; (c) a latent-to-pixel loss transition that supervises rendered albedo/roughness/metallic in pixel space to counter VAE compression error; (d) multi-view support via learnable viewpoint embeddings, symmetric-geometry training data, and low-SNR timestep sampling. On 30 texture-rich TexVerse objects, TaoTex substantially outperforms TRELLIS 2, UniTEX, Ink3D, and MaterialMVP in both single- and multi-view settings.

Critical strengths
  1. The data problem is correctly identified and directly solved. The claim that public 3D datasets lack high-frequency texture exemplars — and that this, not architecture, is the binding constraint on text/logo recovery — is convincingly supported: removing HFT data produces the single largest ablation drop (PSNR 25.48→22.90) and Fig. 4 shows text becoming illegible without it. The MLLM-in-the-loop procedural asset pipeline is a reusable recipe.
  2. Large, consistent empirical margins. +3.2 dB PSNR over TRELLIS.2 single-view and +5.5 dB over the best multi-view baseline, with LPIPS roughly halved, plus wins over commercial models qualitatively. For a fidelity-focused paper, the evidence matches the claim.
Critical weaknesses
  1. Evaluation scale and domain. The quantitative benchmark uses only 30 objects from a single source (TexVerse), and the HFT training domain consists of synthetic simple primitives (cans, cups, boxes). It is not established that gains transfer to complex organic shapes or to categories far from the HFT distribution; the multi-view test is only four canonical views.
  2. Missing comparison to the closest concurrent native method. NaTex is discussed in related work as the most similar native 3D approach (decoupled geometry/color dual-stream autoencoder) but appears in no quantitative table, so the paper's standing against the strongest native-3D alternative is unknown.
Questions for authors
  1. Why is NaTex not in Tables 1–2? Please add it or explain.
  2. How does performance degrade on objects whose geometry is estimated (e.g., from TRELLIS 2 geometry stage) rather than GT? The teaser uses generated geometry, but all quantitative results use GT.
  3. The HFT textures come from Qwen-Image-3.0-Pro — does the model then partially learn to reproduce IGM-specific artifacts, and does anything prevent leakage between HFT textures and the TexVerse test objects?
  4. Higher-resolution o-voxel training under memory constraints is recommended for discussion in the future work section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.