acceptodds
Under review as a conference paper at ICLR 2027

Does Native 3D Texture Generation Necessarily Require 3D Assets for Training?

Abstract

Native 3D texture generation synthesizes colors directly in 3D space for a given geometry, conditioned on multi-view reference images. It is generally believed that training such models requires large-scale, high-quality real 3D asset data, whose acquisition remains a long-standing and challenging problem. In this work, we propose Tex-Zero, demonstrating that a high-fidelity native 3D texture generation framework can be trained without 3D assets. Our key observation is that only high-quality and fine-grained color information is essential for 3D texture training, while the required geometric information is less critical and can be manually constructed rather than obtained from real 3D assets. This finding makes it possible to transform abundant, high-quality 2D images into effective training samples for 3D texture generation. Specifically, we convert high-quality 2D images into 3D training samples by representing each image as a plane in 3D space and applying patch-wise random rotations and aggregation to construct complex geometric structures. Using these constructed image data, we train the Tex-Zero VAE, which can reconstruct real 3D assets with high quality despite never observing them during training. Building upon the Tex-Zero VAE, we train the Tex-Zero DiT also exclusively on the constructed image data, where the conditioning 2D multi-view images are transformed into planes in 3D space and also encoded by the Tex-Zero VAE, thereby reducing the representation gap and improving generation quality. Extensive experiments show that Tex-Zero generates high-fidelity 3D textures with fine-grained details solely using images as training data, offering a promising perspective on the data paradigm for scaling 3D texture generation.

Primary area:
generative models
Keywords:
3D Texture Generation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_KGJMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly investigates whether a large number of 2D images can be used to train a 3D texture generation model, so as to reduce reliance on expensive 3D training assets. It constructs training samples by cropping images and transforming them in 3D space. Then it trains a shared VAE encoder for 2D images and 3D textures, and finally trains a DiT to generate surface textures for 3D objects. The main contribution is a paradigm that uses 2D images as a substitute for 3D texture assets, enabling a unified representation between the two.

Critical strengths
  1. The research problem carries good practical value and the method shows promising potential. It can leverage abundant 2D images as an alternative to 3D texture libraries. 2. The method is concisely designed, and its data construction pipeline can yield rich geometric structures. 3. By unifying the representations of 2D reference images and 3D textures, it reduces the representation gap between separate encoders and facilitates the transfer of fine-grained details.
Critical weaknesses
  1. The model takes geometry and reference views as inputs. Good results may primarily stem from transferring existing details from the reference images onto the surface. The generation quality for non-visible regions is not reported. 2. The description of training for shared 2D–3D features is insufficiently clear. Although experimental results are presented, the training sources for each component (e.g., VAE and DiT) are not explicitly stated. The reasons behind the performance improvements remain unclear. 3. Evidence for generalization on real-world geometry is limited. The test set is inadequately described in terms of categories and geometric complexity. While manual rotation can simulate different viewpoints and occlusions, it cannot account for textures on naturally curved surfaces.
Questions for authors
  1. It is recommended to add evaluations on the texture generation quality for invisible regions, as well as the texture synthesis results on real curved surfaces. 2. The training setup for the shared feature module should be elaborated further. Ablation experiments are also needed to verify that the performance gain indeed comes from the shared-feature design.
Reviewer_WkMZOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether native 3D texture generation can be trained without textured 3D assets. The main idea is to turn ordinary images into artificial 3D training samples by dividing them into patches and randomly rotating and arranging the patches in space. Using these samples, the authors train both a sparse color VAE and a conditional texture generator, with the same VAE encoding reference images and target surface colors. Although trained only on image-derived data, the model can generate RGB textures on real 3D object geometry. The reported experiments show competitive VAE reconstruction and better reference-appearance preservation than the compared generation baselines. Adding image-derived data also improves the models trained with 3D assets.

Critical strengths

The paper presents an interesting approach to reducing dependence on textured 3D training assets. Both the color VAE and the texture generator are trained on randomly arranged image patches and generalize to object surfaces at inference time, without requiring a new generation backbone.

The reconstruction and generation experiments demonstrate the feasibility of this approach. The proposed data pipeline improves VAE reconstruction. Training with both image-derived samples and 3D assets further improves the results, suggesting that the approach is also useful when textured 3D training data are available.

Critical weaknesses

Table 2 compares RGB predictions with a system that produces PBR materials, but does not explain whether the evaluated images use base color or full PBR shading. Without this detail and the construction of the target images, the large gap to TRELLIS.2 is hard to interpret.

The evaluation uses 160 generation samples and 200 reconstruction assets from internal test sets. The paper should release these sets and explain how the reference and target images were created. Errors on observed and unobserved surfaces should be reported separately to distinguish color transfer from texture completion.

The patch ablations only evaluate VAE reconstruction. They do not show how the choice of training geometry affects texture generation.

Questions for authors
  1. Table 2 does not explain how RGB outputs and PBR materials are compared. Are TRELLIS.2 outputs evaluated as base color or shaded renders? How are the reference and GT images created? Please use the same rendering settings for both methods, specify the TRELLIS.2 texture checkpoint. Please also compare with the released TRELLIS.2-based Pixal3D texture model.

  2. The test sets are internal. Please release the 160 generation samples and 200 reconstruction assets. If redistribution is restricted, provide source IDs and preparation scripts. Please provide more details about the test data and evaluation setup. Report observed and unseen surface errors separately, and explain how voxel colors are transferred to the original mesh.

  3. Model size and training budgets are not clear enough to assess the comparison. Please report VAE and texture-DiT parameter counts. For 2D-only, 3D-only, and mixed training, give the training steps, global batch size, hardware, data mixing ratios, and any differences in architecture or training budget. Please measure texturing time and peak memory against the baselines on the same hardware and resolution, including condition encoding and decoding. Report single-view and multi-view costs separately.

  4. The choice of rigid square patches is not well justified. Would triangulated surfaces or non-rigidly deformed patches better approximate real object geometry? Please compare these constructions with the current pipeline and report their effects on reconstruction and texture generation.

  5. Why encode the reference image as a plane rather than project it onto the mesh? Please compare the two using the same model and training settings.

Reviewer_G5RmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that native 3D texture generators can be trained without any textured 3D assets. The idea is that what the models really need is high-quality color signal. Geometry, by contrast, mainly controls how that signal is arranged in space and how it appears across views, so it can be built by hand.

Critical strengths
  1. The motivation for this paper is pratical and important. 3D data scarcity is a real bottleneck for 3D generation. Showing that abundant 2D images can stand in for textured assets, even partially, is a useful finding for the community.
  2. The unified 2D–3D latent space is an effective technical design. Encoding conditioning images as planes with the same VAE removes the encoder mismatch. Table 5 and Figure 7 show this matters a lot (LPIPS 0.0481 vs. 0.0835–0.1228). Image-derived training is what makes the shared VAE good enough at encoding images for this to work. This parallels the move to shared tokenizers in unified multimodal models, and it is a nice transfer of that idea to 3D.
  3. Informative ablations. Table 4 shows how global rotation, patch-wise rotation, and aggregation each contribute. Appendix A gives a reasonable, if informal, explanation of each, for example the zero gradient for out-of-plane kernel weights under fixed-plane training (Eq. 13–14).
Critical weaknesses
  1. The generation evaluation does not test the hard part of texturing, so the headline gains may mostly measure copy fidelity. In Appendix section C, it says metrics compare renders of the generated asset against “the corresponding ground-truth images.” It is not stated whether these are the same views used as conditioning. I think this paper needs to report extra information: (a) evaluation on held-out views not used for conditioning (b) results under sparse conditioning (1 - 3 views with varying elevation) (c) perceptual or consistency measures for unobserved regions, such as FID/KID/CLIP-similarity on novel views or a user study.

  2. The conceptual claim that geometry need not be semantically meaningful is only validated where it is least likely to fail. Appendix A.3 casts hidden-region completion as within-patch image outpainting. On real objects, however, occluded content depends on 3D semantics: symmetry (the left side mirrors the right), part identity (a mug’s interior, a character’s back or soles), material continuity across curved surfaces. None of these can be learned from randomly posed flat patches whose spatial neighbors in 3D carry unrelated image content. And this training data augmentation actively teaches that nearby surfaces may have discontinuous colors, which could cause seams or incoherent completions on real assets. With six orthographic views covering most of the surface, these issues are largely hidden. Please add: sparse-view experiments (the same as in point 1), assets with heavy self-occlusion.

  3. The paper’s own numbers suggest 3D data still wins in a controlled comparison, and this is not stated plainly. | Model | LPIPS ↓ | PSNR ↑ | SSIM ↑ | |---|---|---|---| | 3D-trained DiT, same architecture (Table 3) | 0.0302 | 36.23 | 0.980 | | Image-only Tex-Zero, six-view (Table 2) | 0.0340 | 35.68 | 0.983 |

If these share a protocol, the image-only model is slightly worse than its 3D-trained counterpart. Table 3 also appears to use the f16c32 VAE, while the main model uses f16c16, which needs clarifying. The VAE shows the same pattern. In Table 1, the Tex-Zero VAE trails the 3D-trained VAE on PSNR-PC at every compression ratio (−2.5, −1.1, −0.8 dB) and wins mainly on LPIPS. The abstract and introduction (“outperforming representative baselines trained on real textured 3D data”) imply image-only training matches or beats 3D training. In fact, it only beats different architectures with different data and resolution. A row in Table 2 with the controlled 3D-trained DiT is essential. The major claim of this paper seems to be overstated with experiment results shown in this paper. That is still a good paper, but the framing should match the evidence.

  1. Only internal data is compared. All 3D evaluation (200 assets for the VAE, 160 for the DiT) and the 1M-asset 3D training set are internal. Results on a public benchmark (Objaverse/GSO/Toys4K) are needed to make this paper more convincing and solid.

Objaverse: A Universe of Annotated 3D Objects (Objaverse, CVPR 2023)

Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items (GSO, ICRA 2022)

Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias (Toys4K, CVPR2021)

Questions for authors

Please refer to critical weaknesses.

Reviewer_PisBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Tex-Zero is a native 3D texture generation framework trained without real textured 3D assets. It constructs training samples by representing 2D images as colored planes, applying patch-wise rotations, and aggregating the patches into spatially complex configurations. A texture VAE learns to encode and reconstruct colors on these supports, while a conditional flow model generates texture latents using geometry and multiview image conditions. The shared VAE encodes both image conditions and target textures. The work offers an interesting alternative to asset-based training, although further clarification and controlled experiments are needed to establish reproducibility and explain the source of its advantages.

Critical strengths

An interesting and potentially scalable data strategy. The paper investigates whether abundant 2D images can replace real textured assets when training geometry-conditioned texture models. The demonstrated transfer from constructed samples to real surfaces is a meaningful result.

A coherent representation design. Encoding conditioning images and target textures with the same VAE provides a plausible way to reduce representation mismatch and preserve fine appearance details.

Useful supporting experiments. The paper includes preprocessing ablations, comparisons with same-architecture models trained on 3D assets, alternative conditioning strategies, and mixed 2D/3D training. These experiments provide initial support beyond comparisons with external baselines.

Critical weaknesses

Important details remain missing for independent reproduction. Appendix B provides substantial architectural and optimization information, but the complete training schedule is not clearly specified. In particular, the duration of the essential single-plane VAE pretraining stage, total training updates, effective global batch size, and overall compute budget are unclear. Naming the image datasets and their aggregate size also does not fully specify the training distribution.

The experiments do not fully isolate the benefits of the proposed data source and unified representation. The 2D and 3D settings differ in dataset size and perceptual-loss construction, while some conditioning comparisons change both the representation and training data. Consequently, the reported improvements cannot be attributed solely to replacing real assets with image-derived samples. The preprocessing ablations also focus on VAE reconstruction rather than directly establishing the contribution of each operation to final texture generation.

The evaluation provides limited evidence for unseen-surface completion and broader generalization. The evaluation emphasizes transferring appearance from supplied views onto known geometry and relies on internal test sets. This leaves uncertainty about completion of genuinely unobserved regions, robustness to imperfect multiview inputs, and independent verification.

Questions for authors
  1. Could the authors report the number of total training steps for VAE pretraining, subsequent patch-based VAE training, and DiT training? Please also specify the number of training resources and other training configurations for better reproducing this work.
  2. How many images from SA-1B, BLIP3o-60k, and ShareGPT-4o are retained after preprocessing and filtering? Are sources sampled proportionally to their sizes or using explicit mixture weights, and do the VAE and DiT use the same mixture?
  3. Could the authors independently vary the training data source and conditioning encoder while holding architecture, training compute, input resolution, and latent capacity fixed? Since the 2D and 3D settings compute perceptual losses differently, a common-loss comparison would also help separate the effect of the data from the supervision objective.
  4. Can the authors evaluate global rotation, patch-wise rotation, and aggregation using final DiT generation metrics in addition to VAE reconstruction?
  5. Could the authors separately report performance on observed and unobserved surface regions, and evaluate reduced view counts, camera perturbations, and inconsistent multiview conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.