acceptodds
Under review as a conference paper at ICLR 2027

Texture Space Material Diffusion

Abstract

We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.

Primary area:
generative models
Keywords:
Generative models, surface materials, content creation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_JsjyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PBR material reconstruction and generation in UV texture space. Given a mesh, a valid UV atlas, camera poses, and one or more RGB views, it rasterizes each view back into a sparse UV observation and stores world position and normal maps at the same texels. It then concatenates these observation frames and the target material frames in the latent/frame interface of a fine-tuned Wan2.1-1.3B video DiT. The model must demodulate lighting, merge observed evidence, and inpaint unseen texels to produce basecolor plus height/roughness/metallic maps. The use of a video backbone supplies pretrained multi-frame transformer and VAE priors; the generated object is not a temporal video. A 3D-aware RoPE helps connect UV-distant but surface-near texels, while progressive noise rolling and coverage-weighted expert fusion extend inference to higher resolutions and more views. Experiments cover multi-view reconstruction, relighting, and single-image/text-conditioned material generation. The shared UV output removes one source of cross-view bake/fusion inconsistency, but does not by itself guarantee seam-free chart boundaries or pixel-perfect projection.

Critical strengths
  1. Useful problem formulation. Generating directly in a shared UV atlas produces deployable PBR maps and avoids reconciling independently generated values for the same surface location. Extending this formulation from diffuse texture to basecolor, height, roughness, and metallicity is practically useful.
  2. Coherent use of video diffusion priors. The method adapts a pretrained video DiT to texture-space observations, uses world-position and normal conditions, and adds a 3D-aware RoPE to connect UV tokens that are distant in the atlas but close on the surface. The high-resolution and many-view extensions are also practically motivated. The high-resolution procedure is an adaptation of prior tiled/noise-rolling inference ideas, rather than an entirely new diffusion algorithm.
  3. Practical inference extensions and useful evaluation components. Relighting is a useful test of material estimation beyond matching input illumination. The progressive noise-rolling/tiled inference and coverage-weighted multi-view fusion are practical ways to extend a model trained at limited resolution and view count, although they build on prior tiled-diffusion ideas rather than constituting entirely new diffusion algorithms. The diffuse-only TEXGen comparison and runtime reporting are also helpful.
Critical weaknesses
  1. The evaluation does not use task-matched metrics for the two problem settings. The paper evaluates multi-view-to-material reconstruction/relighting and single-view-to-material generation in separate tables, but the multi-view setting is still assessed mainly through rendered-image reconstruction and relighting scores. Since densely covered multi-view inputs primarily test illumination removal and PBR-channel decomposition, this setting should be evaluated with channel/material-estimation metrics under the same protocol as decomposition methods. The single-view setting is a different material-generation problem and should retain generation metrics and generation baselines. Without this task-specific separation of metrics and comparisons, the reported numbers do not cleanly establish performance on either capability.
  2. The method is insufficiently positioned against the closest and strongest relevant work. SeqTex is a closely related use of video diffusion for UV texture generation with geometry-informed correspondence, but it is neither discussed nor shown in a qualitative comparison. SeqTex predicts RGB/albedo rather than full PBR, so it is not a fair quantitative baseline for full image-to-material generation; its UV/albedo and seam behavior is nevertheless directly relevant. The multi-view decomposition setting is not compared with task-matched methods such as RGB↔X or SuperMat, and the single-view full-material setting is not compared with recent methods such as NaTex and LumiTex. These omissions make the relative standing of the proposed method unclear.
  3. Transfer of video priors to fragmented UV layouts and cross-chart alignment are insufficiently validated. UV charts can be independently rotated, cut, and packed, so atlas adjacency and orientation need not reflect coherent surface structure or natural-image composition. Position/normal conditioning and 3D-aware RoPE are reasonable responses, and the paper includes a RoPE ablation, but it does not test robustness to alternative valid unwraps or directly measure cross-chart continuity. A shared atlas avoids independently generated view predictions, but does not itself ensure seam-free surface appearance. The paper also provides only limited controlled evidence for complex, multicolored patterns.
Questions for authors

My current assessment is weak reject. The following evidence could materially change my assessment; the main issue is whether the method is competitive on each task under a fair, task-matched protocol.

  1. Use task-matched metrics and baselines. For the densely covered multi-view setting, please evaluate illumination removal and PBR-channel/material decomposition with metrics and protocols used by RGB↔X, SuperMat, or comparable decomposition methods, using matched views, geometry, lighting, and outputs. For the single-view full-surface generation setting, please use generation metrics and compare with TRELLIS.2, NaTex, LumiTex, and other full-material methods under the same mesh, camera, and resolution. The two settings should not be judged by the same notion of success.
  2. Relationship to SeqTex. Please discuss SeqTex explicitly and include qualitative comparisons of its shared RGB/albedo output, UV correspondence, and seam behavior under the same mesh and input image. SeqTex does not predict roughness, metallicity, or the other full PBR channels, so it should be used as a qualitative baseline to compare the albedo itself.
  3. Robustness to different UV unwraps and seam alignment. If the same mesh and input images are paired with different valid UV maps—for example, UVAtlas, Smart UV Project, or a learned unwrap—does the method produce similar material quality? For patterns that continue across a UV cut, please show close-ups on both sides of the cut and report a metric (if possible) on the corresponding 3D surface points. This would clarify whether world position, normals, and 3D-aware RoPE actually compensate for the lack of natural 2D adjacency in UV space.
  4. Does the high-resolution procedure actually resolve small-chart ambiguity? Figure 8demonstrates the practical 2K-to-8K, more-view setting, but it reports only one set ofimage-quality numbers. Could the authors report PSNR/SSIM/LPIPS for the same batch of test assets under the 2K and 8K settings, together with the corresponding number of input views and surface coverage? This would clarify the quality change associated with the high-resolution procedure, including whether it improves only fine detail or also reduces errors caused by multiple UV islands falling within one attention patch. The current 4× upscaler comparison in Table S4 does not answer this question because it evaluates a separate upscaling task.
  5. More diverse materials and the role of geometric detail. Please show more examples with rich, multicolored materials and complex patterns, comparable in diversity to the teapot example, rather than mostly monochromatic or visually simple cases. In addition, can the authors test whether high-frequency texture and material details remain correctly aligned on low-frequency geometry, such as a smooth sphere? A controlled comparison between detailed and smooth versions of the same shape, with the same reference appearance or prompt, would help determine whether the method can align texture from image evidence when geometry provides little local structure. This would separate the contribution of geometric cues from the model's ability to generate and align appearance itself.
Reviewer_VBE3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper proposes a diffusion framework that operates entirely in texture space. It fine-tunes the Wan2.1 video DiT to generate complete PBR materials from multi-view images, a single image, text, or low-resolution material maps.
  2. The method uses known geometry, UV parameterization, world-space positions, and normals to maintain view-consistent textures, while supporting material reconstruction, text-guided generation, and inference with 8K textures and more than 100 views.
  3. The experiments show competitive results on synthetic material reconstruction, relighting, and generative material metrics.
Critical strengths
  1. The task is practically valuable and unifies multi-view material reconstruction, single-view/text-guided material generation, and high-resolution material inference.
  2. The combination of texture-space diffusion, coverage-aware expert aggregation, and 3D-aware RoPE is natural and makes good use of pretrained video diffusion priors.
  3. The experiments cover synthetic data, real DTC data, relighting, different numbers of input views, 8K inference, and individual material components.
Critical weaknesses
  1. The related work and experimental comparisons do not sufficiently cover recent 3D texture generation methods. For example, UniTEX (“UniTEX: Universal High Fidelity Generative Texturing for 3D Shapes,” CVPR 2026) also studies high-quality, view-consistent 3D texture generation and uses large-scale 2D Diffusion Transformer priors. The paper should better explain its relationship to and differences from such recent methods.

  2. The method is mainly built on the Wan2.1 video diffusion DiT, while most comparison methods use Stable Diffusion, Flux, or other architectures. Therefore, it is difficult to determine whether the performance gains come from the proposed texture-space representation, projection/fusion strategy, and 3D-aware RoPE, or mainly from the stronger prior of the video generation model. The paper includes ablations of standard RoPE, G-buffer conditioning, and 3D-aware RoPE, but does not compare different texture-space designs with the same Wan2.1 backbone, nor does it replace the backbone with an image diffusion or Flux-based model while keeping the rest of the pipeline fixed. At least one controlled comparison with the same backbone would help clarify the contribution of the proposed method.

  3. The training data mainly consists of synthetic renderings, while the real-world evaluation is small. More visual examples from real test cases would help support generalization across different objects, materials, lighting conditions, and capture conditions.

  4. The paper should provide more details about the CRF and exposure fitting procedure, including the fitting data, optimization range, and whether the same processing is applied consistently to all methods.

  5. The paper has practical value for complete PBR material generation and material reconstruction under unknown lighting, but its methodological novelty may need clearer definition. Prior work such as TEXGen already performs diffusion-based generation directly in UV space, while 3D positional encoding and noise rolling also build on existing techniques. The main advances therefore seem to lie in adapting a pretrained video DiT and combining multi-view observations, world-space positions, normals, and text conditions for material generation. The authors should clarify whether the method extends existing UV-space diffusion with multiple input conditions and material outputs, or introduces a new representation or generation mechanism, and explain its distinct technical contributions.

  6. Regarding the third claimed contribution, although the supplementary material includes an input-view-count ablation, the gains from increasing the number of views from 17 to 117 are very small. These results do not yet show a clear quality improvement from dense-view inputs. The authors should further explain the practical value of the 100+ view setting and distinguish the system’s ability to process more views from evidence that additional views substantially improve quality.

Questions for authors
  1. Can the authors provide a controlled comparison using the same backbone, such as Wan2.1, but with alternative texture-space designs? Alternatively, can the proposed pipeline be evaluated with an image-diffusion or Flux-based backbone while keeping the remaining components fixed? I recognize that such experiments may be difficult and require substantial computational resources.

  2. Can the authors provide additional real-world test cases covering more objects, material types, lighting conditions, and capture conditions?

  3. For the CRF and exposure fitting, what data are used for fitting, what parameters and optimization range are allowed, and is exactly the same procedure applied to every method?

Reviewer_Z3uBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses PBR material estimation for objects with known geometry and UV parameterization. Each input view is back-projected into a partially covered UV atlas, and a fine-tuned video diffusion transformer (Wan 2.1) takes these texture-space observations, together with rasterized world-space position and normal guides, as frames and generates complete base color, height, roughness and metalness maps. The same formulation supports single-view and text-conditioned generation with a 3D-aware rotary embedding, and a texture upscaler with noise rolling and coverage-aware aggregation scales inference to 8K textures and more than 100 views. With known geometry, the method clearly outperforms differentiable path tracing (DiffPT) in synthetic relighting (30.46 vs. 24.90 dB PSNR) and per-map accuracy, and it leads the generation baselines in the single-view and text-to-material settings.

Critical strengths
  • The core idea is simple and well motivated. Treating material estimation as conditional completion of a UV atlas makes the output view-consistent and seam-free by construction and directly usable in standard graphics tools, while a pretrained video prior can be reused with minimal architectural change.
  • The synthetic results are strong: relighting PSNR of 30.46 vs. 24.90 dB for DiffPT (Table 1), an even larger margin without CRF adjustment (29.89 vs. 21.14 dB, Table S1), and clear per-map gains in roughness (24.32 vs. 17.98 dB) and metalness (18.24 vs. 13.23 dB) (Table S3). The method also leads all baselines in Table 2.
  • A single model covers multi-view, single-view and text-conditioned generation, and scaling to 8K and 100+ views requires no retraining.
  • Reporting is thorough: results without CRF, a variant using LSRM geometry, runtime and memory (Table S2), RoPE and upscaler ablations, and per-object, per-view supplementary data.
Critical weaknesses
  1. Real-world evidence is limited. On DTC, materials are evaluated only under the original lighting, on the 17 input views of each of the eight objects (136 images), so relighting is never measured on real data; Figure 4 shows it only qualitatively. In this setting, DiffPT is better on all three metrics (30.67 vs. 29.03 dB PSNR, Table 1). Without the CRF adjustment, which is fitted per method and scene to the ground-truth images under the original lighting, the method ranks below both baselines on real data (22.66 vs. 24.56 dB for LSRM and 27.01 dB for DiffPT, Table S1).

  2. The benefit of texture space itself is not isolated. The paper attributes its gains to generating entirely in texture space, but the multi-view comparison is against DiffPT, which has no learned prior, and LSRM, which must also reconstruct geometry. These comparisons show the value of a learned prior with known geometry, not of texture space versus image space. An image-space variant with the same backbone and training data (per-view PBR prediction followed by baking into UV) would directly test the central claim.

  3. Robustness to reconstructed geometry is not supported. The paper states that the method can be made robust to inconsistent geometry (p.2), but with LSRM geometry its synthetic relighting drops from 30.46 to 23.71 dB, and on real data it falls below LSRM itself (27.01 vs. 27.75 dB, Table 1). Since the stated target application is scanned objects (p.2), this mainly calls for narrowing the claim.

  4. The "first" claim and related work. Material Anything (CVPR 2025) already refines PBR maps with a diffusion model in UV space, but the paper describes it only as an image-space method (p.4). SeqTex (arXiv 2025), which fine-tunes a video diffusion model to generate UV textures, and ARM (arXiv 2024), which back-projects input views into UV space before predicting materials, are not cited. MaterialFusion (3DV 2025) and IDArb (ICLR 2025), the most direct prior-based multi-view material estimators, are not compared, even though the synthetic test set comes from MaterialFusion's BlenderVault (p.6).

  5. Reporting inconsistencies. In Table S1, DiffPT's synthetic relighting SSIM and LPIPS (0.903 / 0.0911) are identical to those in Table 1, while its PSNR differs (21.14 vs. 24.90 dB), which suggests a copy error. The Table 1 caption states that synthetic metrics use eight views, whereas Table S3 uses 17 views, and the supplementary per-view data appear to contain 17 frames per object, whose mean matches Table S1 (30.17 dB).

Questions for authors
  1. Could you report relighting on DTC using its captured environment lighting and held-out views, with and without CRF adjustment, or results on Stanford-ORB?
  2. Is the synthetic evaluation performed on the input views? If so, could you report masked metrics on held-out views that include regions unobserved in the inputs? Are the eight test HDR probes disjoint from the 697 Poly Haven training probes, and was BlenderVault deduplicated against the training assets?
  3. The supplementary data for the single-view diffuse comparison appear to give nearly identical PSNR for TEXGen and your method (30.94 vs. 30.92 dB), whereas Figure 7 reports only CLIP-FID, CMMD and LPIPS. Could you report PSNR and SSIM there and discuss what your approach contributes beyond TEXGen's texture-space formulation?
  4. The supplementary resolution × view-count matrix shows no gain at 8K (29.03 vs. 28.97 dB on DTC) and +0.56 dB from 117 views at roughly 20× the cost. Where does 8K inference help, for example at higher rendering resolutions? Could you also ablate the coverage-aware top-2 aggregation, and explain how the Figure 8 numbers were computed, since we could not reproduce them from the per-object supplementary metrics?
  5. How sensitive is the method to the UV parameterization (for example, automatic versus artist-made atlases, or different island packing), given that a 16×16 patch can contain several atlas segments (p.9)?
Reviewer_aW48OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies PBR material reconstruction and generation for a given mesh with a valid UV parameterization. Posed images are projected into a shared texture space, and a pretrained Wan2.1-1.3B video diffusion transformer is fine-tuned to predict base color and packed height, roughness, and metallicity maps. World-position and normal maps provide geometry conditioning; single-image and text-conditioned variants additionally use a 3D-aware rotary positional embedding. A generative upscaler with noise rolling and coverage-aware aggregation of view batches extends inference to 8K textures and more than 100 views. Experiments cover 32 synthetic objects and eight real DTC objects, with additional generation, material-component, and ablation evaluations. The strongest quantitative evidence concerns synthetic relighting and image/text-conditioned generation; neural materials are demonstrated as a proof of concept.

Critical strengths

The paper projects multi-view observations into a shared UV space and fine-tunes a pretrained video diffusion model to predict complete PBR materials. The framework supports multiple input modalities, and the synthetic relighting and material-component evaluations show practical potential (Tables 1 and S3).

Critical weaknesses
  1. The contributions of individual mechanisms are not fully isolated. The RoPE ablation is limited to 512 x 512 textures and shows a modest incremental gain. Figure 5 does not isolate training corruptions, while Figure 8 changes both view count and resolution, obscuring the separate effects of noise rolling and coverage-aware aggregation. Controlled ablations would clarify the benefit of each design choice.
  2. Real-world evidence and scaling efficiency remain limited. Quantitative real-world evaluation covers only eight DTC objects, with DiffPT performing better under the original illumination; quantitative relighting gains are primarily established on synthetic data. The 117-view, 8K configuration takes 48 minutes 19 seconds and 48 GiB on a GB300 (Table S2), while Figure 8 shows improved LPIPS but slightly lower PSNR and SSIM. Claims about practical advantages should therefore be qualified.
  3. Coverage of closely related reconstruction baselines is incomplete. Table 1 does not include MatMart, which also supports multi-view PBR reconstruction and completion of unobserved regions. A comparison, if an implementation is accessible, would help establish the benefit of direct UV-space estimation and fusion. Otherwise, the authors should explain the constraint and clarify the methodological differences.
Questions for authors
  1. Attribution of scaling gains. Could the authors clarify which parts of the improvement in Figure 8 are attributable to additional observations, higher output resolution, and the proposed inference mechanisms? A small comparison on the existing subset with one factor held fixed, including a simpler aggregation alternative where available, would help. If these results are unavailable, please qualify the corresponding claims.
  2. Real-world failure modes. Section 4.1 attributes DiffPT's advantage under the original illumination to overfitting. Can an analysis of existing real examples support this explanation and identify residual errors in the proposed method, such as lighting baked into base color, specular inaccuracies, or pose/geometry misalignment? This would clarify how far the synthetic relighting gains transfer to real captures.
  3. Reproducibility and cost of view aggregation. Increasing the input from 17 to 117 views raises 2K reconstruction time from 130 to 2610 seconds (Table S2). What batch size, overlap, and number of expert evaluations produce this cost? Please specify how per-texel coverage weights are computed and how zero-coverage regions are handled. These details would clarify the algorithm and its practical scaling behavior.
  4. Comparison with MatMart. If an implementation is accessible, could the authors compare against MatMart [1] using the same geometry, input views, and evaluation protocol, reporting material-component accuracy, relighting quality, and runtime? If a reproducible comparison is not feasible, please explain the constraint and clarify the methodological differences.

[1] Wu, X., Zhu, P., Lyu, J., Liu, X., Guo, J., Guo, Y., ... & Lyu, C. (2026). MatMart: Material Reconstruction of 3D Objects via Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2336-2345).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.