acceptodds
Under review as a conference paper at ICLR 2027

Style2Shape: Image Style Guided 3D Shape Material Generation

Abstract

We present Style2Shape, a framework for generating editable physically-based rendering (PBR) materials for a 3D mesh from a single reference image. Compared with text-guided methods, images provide high-bandwidth appearance cues but introduce two key difficulties: (i) faithfully transferring fine-grained visual style while preserving the target shape's structure, and (ii) producing physically-plausible, standard PBR assets rather than implicit, non-editable representations. To address these challenges, we propose a hybrid material representation that couples a retrieved procedural material (for physically-consistent reflectance) with a generated texture (for rich, instance-specific details), combined via learnable blending. Our pipeline consists of three stages: (1) structure-guided appearance transfer to synthesize pixel-aligned supervision, (2) hybrid initialization via physics-based retrieval and texture generation, and (3) physics-based optimization that jointly refines procedural parameters, UV transforms, and blending weights through differentiable rendering. Experiments show that Style2Shape produces high-fidelity, physically-plausible materials exported as standard PBR maps, making the results directly compatible with existing rendering workflows.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
procedural materials, texture synthesis, 3D material generation, physically-based ren dering, differentiable rendering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aitvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Style2Shape transfers material appearance from a reference image to a target mesh. An image editing model first produces an image aligned with the target geometry. The method retrieves procedural materials, blends their albedo with generated textures, and optimizes lighting, UV transforms, material parameters, and blending weights through differentiable rendering. The outputs are PBR maps with editable procedural parameters. Experiments show transfers of reference colors and patterns, with quantitative results measuring agreement with the generated target images.

Critical strengths

The paper presents a pipeline for generating procedural materials for a target mesh from a reference image, combining material retrieval, generated albedo, and differentiable optimization. Several examples show good transfers of reference colors and patterns, and the resulting PBR maps can be rendered under different views and lighting.

Critical weaknesses

The methodological novelty is limited. MaPa already combines generated image supervision, procedural material initialization, and differentiable fitting. Style2Shape combines reference-image guidance with generated albedo, but does not compare with MaPa. A comparison using the same aligned target image would help establish the benefit of the proposed material representation. The pipeline appears to be an engineering integration of existing components, and the experiments do not establish a substantial contribution beyond this combination.

The component ablations in Figure 5 are limited to qualitative results on a chair and a bag. They do not establish the benefits of retrieval, generated albedo, and staged optimization across the test set, and the style loss is not ablated.

Only albedo is generated; roughness, metallicity, and normals come from the retrieved procedural material. The paper does not show whether these channels remain compatible with the generated patterns. Please add a controlled material-recovery experiment using a reference rendered from the same object with known materials, and report rendering errors under held-out views and lighting, especially for glossy and metallic surfaces.

The method depends on image editing and PBR map prediction, but does not test how errors from these models affect retrieval and the final materials. Please show the generated textures under repeated tiling to check for visible seams.

Questions for authors
  1. Please compare with MaPa using the same aligned target image. What changes to the material representation or optimization are responsible for the improvement? Please also discuss newer related work such as MultiMat and Generative Detail Enhancement for Physically Based Materials.

  2. Please report quantitative ablations of generated albedo, retrieval, staged optimization, view selection, and the style loss across the test set.

  3. Please evaluate material recovery on objects with known materials, using rendered reference images, and report rendering errors under held-out views and lighting, especially for glossy and metallic surfaces. Does the generated albedo remain consistent with the procedural roughness, metallicity, and normals? Please also show the generated textures under repeated tiling to check for visible seams.

  4. How sensitive is the result to errors in the edited target image and PBR map prediction? Please test an alternative image editor or controlled errors in the intermediate predictions, and show whether the optimization can recover.

2 comments

Thank you for the concrete suggestions on representation, component isolation, and material validation. We added quantitative component ablations and an original LumiTex comparison.

1. MaPa and the representation contribution. MaPa uses generated supervision and procedural material fitting. Its official project page and author repository currently provide no runnable implementation, so we could not perform the requested direct comparison under the same ItrI_{tr}. Our within-pipeline ablation measures the benefit of adding the generated-albedo branch to procedural fitting: mean PSNR improves by 2.392 dB and mean LPIPS decreases by 16.5%. PSNR and LPIPS improve on each tested pair under the same generated target.

2. Component ablations. We evaluate three reference–mesh pairs: a bag, a sofa, and a table. Configurations share targets, cameras, rendering settings, and 200 optimization iterations. Metrics against ItrI_{tr} are foreground PSNR, foreground-averaged SSIM, and VGG LPIPS with a common black background.

ConfigurationPSNR ↑SSIM ↑LPIPS ↓
Full method18.4490.55620.1218
Without generated albedo16.0570.51900.1459
Random material initialization17.5280.49830.1283
Joint optimization from the start16.9860.52390.1083
Without final-stage style loss17.9830.53740.1232

Generated albedo is removed throughout fitting; the style ablation removes its loss only in final material refinement while retaining UV alignment. Retrieval and the final-stage style term improve all three mean metrics. Staging improves PSNR/SSIM, whereas joint optimization has lower LPIPS. View selection is addressed by the paper's existing qualitative ablation.

3. Original LumiTex comparison. We run LumiTex's AL/MR generation and LVSM completion with the same original reference images and meshes. Both methods' exported materials are frozen and receive the same additional 50-step illumination-only fit to ItrI_{tr} with a common renderer, camera, and mask. Unlike the ablation table, this table therefore includes additional lighting fitting.

MethodPSNR ↑SSIM ↑LPIPS ↓
Style2Shape19.8340.59370.1161
LumiTex17.9520.61530.1369

In this target-image fitting evaluation, Style2Shape improves mean PSNR by 1.882 dB and reduces mean LPIPS by 15.2%; LumiTex has higher SSIM.

4. Material compatibility, intermediate errors, and tiling. Generated albedo and procedural roughness, metallicity, and normals are fitted through one PBR renderer. The graph still determines the available reflectance and normal variations: an added albedo pattern does not automatically introduce corresponding variations in other channels. The requested known-material recovery under held-out views and lighting has not been evaluated.

Inspecting the sofa intermediate target shows that white pillows become purple only after fitting; their shared material segment limits separate control. This observation locates an error downstream of editing, rather than measuring recovery from controlled prediction errors. Texture generation requests horizontal and vertical boundary continuity, and UV repetition tiles the resulting maps; these mechanisms alone do not establish seam-free output.

5. Related work. MultiMat synthesizes procedural graphs using visual/textual graph representations and constrained search; our method fits retrieved graphs together with generated albedo to target geometry. Generative Detail Enhancement for Physically Based Materials enhances existing asset appearance through multiview diffusion and inverse rendering. Our setting transfers a reference photograph to a target mesh while retaining editable procedural parameters alongside generated albedo.

I appreciate the new ablations and the comparison with LumiTex. The three examples help explain what the generated albedo branch adds to the pipeline. My assessment still rests on two points: the methodological step beyond existing procedural material fitting seems modest, and the evaluation leaves open how well the recovered materials render under new lighting. Given these remaining concerns, I will keep my score of 2.

Reviewer_WH3BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the task of generating editable PBR materials from a single reference image for 3D meshes that already possess UV mapping and material segmentation capabilities. The workflow first employs an image editing model to transfer the reference appearance from the reference image to a selected viewpoint on the target mesh; next, it retrieves a procedural material and generates segmented albedo textures; finally, it optimizes lighting, UV parameters, procedural material parameters, and the blending weights of the two albedo textures within differential rendering.

Critical strengths
  1. The output of method consists of standard PBR channels and editable materials,which is compatitable with 3D edit software and have practicle applications.
  2. The pipeline is complete with reasonable proedures, which includs three stage workflow decomposing PBR channels and lights from single RGB image. Based on this pipeline, reconstructing 3D assets can more easily get PBR textures. Innovation on the method side mainly shows up in optimizing strategies and designing objective functions, especially during the differentiable material optimization stage.
  3. The optimization objective in differentiable material optimization is clear and well-designed, particularlly it use the weighted squared Frobenius distance for more accurate estimation with reference.
  4. The experiments include cross-category examples, renderings from diverse perspectives and lighting conditions, ablation studies, and user preference studies; the authors explicitly list limitations such as UV mapping, segmentation, single-image ambiguity, and proprietary image editors.
Critical weaknesses
  1. The method mentioned in the article relies too much on basic technique RGB-X, and the follow-up innovations are somewhat lacking. Especially when the whole tech pipeline is a series of loosely connected steps, it's easy for errors to accumulate without being able to pinpoint them accurately, which reduces the method's stability and its ability to generalize across different lighting conditions.
  2. The whole method is over-rely on the optimization, from PBR-meterial retrival to differentiable PRB rendering, the optimization method might be inefficient, even optimization is hard to converge, particularly in noumous parameter involved. This also greatly weakens the method's theoretical innovativeness. Maybe some intermediate encoding methods for roughness and metallic textures can be used to compare the similarity between the target and source ones.
  3. There is insufficient evidence regarding the role and reproducibility of Stage 1. Structure-guided image editing constitutes a critical step in generating supervisory maps and segmenting textures, relying on proprietary image editing models (e.g., GPT-Image-1 cited in the paper). The appendix provides only brief prompts, without specifying the exact model version, sampling/randomization parameters, failure rates, or results from repeated experiments; the Data Availability section states "upon reasonable request." Furthermore, the ablation study did not remove or replace Stage 1, making it difficult to determine whether the observed effects stem from optimization within this method or from the prior knowledge of the powerful generative model.
Questions for authors
  1. Can you provide independent validation distinct from I_tr, such as measurement of material/BRDF data, real multi-view references, or cross-illumination material property consistency metrics?
  2. Please report the exact version of the Stage 1 image editing model, its generation parameters, and its reproducibility; also, what changes occur in the results if Stage 1 is omitted or replaced with a reproducible open-source model?
  3. Please reports of failure cases across different UV exposures, material segments, and thin/obstructed components.
1 comment

Thank you for recognizing the editable PBR outputs and the optimization design. We added controlled downstream ablations and clarify how retrieval and generated supervision enter the method.

1. Contribution of the downstream material representation. We keep the generated target fixed and vary the downstream material representation or optimization. On three reference–mesh pairs—a bag, a sofa, and a table—generated albedo improves mean PSNR by 2.392 dB and reduces mean LPIPS by 16.5% relative to procedural-only fitting. Both metrics improve on every tested pair. This measures the benefit of adding the generated-albedo branch under the same Stage-1 target.

All configurations use the same targets, cameras, rendering settings, and 200 optimization iterations. Metrics compare final renders with ItrI_{tr} using foreground PSNR, foreground-averaged SSIM, and VGG LPIPS with a common black background.

ConfigurationPSNR ↑SSIM ↑LPIPS ↓
Full method18.4490.55620.1218
Without generated albedo16.0570.51900.1459
Random material initialization17.5280.49830.1283
Joint optimization from the start16.9860.52390.1083
Without final-stage style loss17.9830.53740.1232

Retrieval improves all three mean metrics relative to random initialization. The style ablation removes only the final-refinement style term, retaining UV alignment; its mean gains are smaller. Staging improves PSNR and SSIM over joint optimization, while joint optimization yields lower LPIPS.

2. Retrieval and optimization. Our retrieval projects candidate roughness and metallicity maps onto the target geometry and ranks them by their screen-space discrepancy from the decomposed target channels. Retrieval does not run an optimization loop for every candidate. Only the selected materials enter fitting. The staged variant estimates lighting for 50 iterations, then fixes lighting for 50 UV and 100 material-refinement iterations. The joint variant updates lighting, UVs, and materials together for 200 iterations. These measurements establish the observed fitting tradeoff at a fixed iteration count.

3. Stage 1 and error localization. The supplement's “Additional Method Details” gives the viewpoint-selection rule and editing prompts, and “Additional Pipeline Analysis” shows intermediate outputs. The geometry rendering and reference photograph supply structure and appearance, respectively. The new ablations reuse a fixed cached target for each pair, isolating downstream changes from variation in target generation. This experiment assesses downstream contribution; it does not test editor replacement or repeated-generation reproducibility.

In the inspected purple-sofa case, white pillows remain white in ItrI_{tr} but become purple after fitting. This locates the mismatch downstream of image editing. Their shared material segment limits separate control of the pillows; thin and occluded regions likewise receive less direct supervision from the selected view.

4. Evidence against the original reference. The paper's blind study directly evaluates perceived similarity to the original reference photograph: 30 participants rated 54 cases on a five-point scale with randomized method order.

MethodMean reference-fidelity rating ↑
Style2Shape3.90
PSDR-Room3.37
Material Anything2.83
Material Palette2.27

This supports perceived reference fidelity. The requested BRDF or cross-illumination material-property validation remains a separate, untested question.

Reviewer_U31dOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Style2Shape, a method that combines generative capabilities with procedural material retrieval to generate physically based rendering (PBR) materials for a 3D mesh from a single reference image. The pipeline consists of three stages: appearance transfer from the reference image to the selected view, material initialization based on the generated images and a procedural material library, and differentiable material optimization.

Critical strengths
  • The idea that combines the strengths of generative methods and retrieval-based material query is critical to PBR generation.
  • This paper is well-written and easy to follow.
Critical weaknesses
  • This work is not novel enough. The overall pipeline and several key design choices are similar to those of MaPa [1], particularly the use of procedural material retrieval and differentiable rendering for material optimization.
  • Some experimental results are insufficient. For example, an ablation study on the style loss term would help justify its contribution. Regarding multi-view consistency, it would be valuable to include more challenging cases involving complex geometry, severe self-occlusion, and spatially varying material patterns.
  • Though aiming to combine the performance merits of both generative methods (for reference and initialization) and procedural materials, it does not appear to inherit their efficiency advantages. The pipeline involves multiple stages and requires per-instance test-time optimization. Despite this computational cost, the improvements over existing baselines are relatively modest in some cases, and the method's effectiveness on high-frequency or structured patterns may remain weak.
  • It lacks comparison with some competitive methods, including MaPa [1], LumiTex [2], IDArb [3].

[1] Zhang, Shangzhan, et al. "Mapa: Text-driven photorealistic material painting for 3d shapes." ACM SIGGRAPH 2024 Conference Papers. 2024.

[2] Bao, Jingzhi, et al. "Lumitex: Towards high-fidelity pbr texture generation with illumination context." International Conference on Learning Representations. 2026.

[3] Li, Zhibing, et al. "Idarb: Intrinsic decomposition for arbitrary number of input views and illuminations." International Conference on Learning Representations. 2025.

Questions for authors
  • How does the method ensure multi-view consistency in principle, given that only a single viewpoint is selected for appearance transfer and optimization? Is consistency primarily achieved by applying the same generated tileable texture to each material segment across different views? If so, does this implicitly assume that each segment has approximately homogeneous or repetitive material patterns, thereby limiting the method's applicability to objects with spatially varying textures or complex occlusions? It would be helpful to clarify how the method handles unseen regions and non-repetitive patterns.
  • How does the proposed hybrid representation handle material details that involve variations in multiple PBR channels? The generated texture contributes only to albedo, while roughness, metallicity, and normal maps are determined by the retrieved procedural material. Does this constrain the expressiveness of the method when the reference contains fine-grained geometric or reflectance variations that cannot be represented by the retrieved material? Additional examples or analysis of such cases would help clarify this limitation.
2 comments

Thank you for the careful review and for recognizing the value of combining generated appearance with procedural materials. We added quantitative experiments addressing the representation, style loss, and LumiTex comparison.

1. Contribution beyond procedural fitting. Our central addition is reference-specific generated albedo blended with a procedural base, TA,k=wkTA,kgen+(1−wk)TA,kprocT_{A,k}=w_kT_{A,k}^{gen}+(1-w_k)T_{A,k}^{proc}, with an optimized blend weight. This expands the available color and pattern representation while retaining editable procedural reflectance parameters. With the same Stage-1 target, adding this branch improves mean PSNR by 2.392 dB and reduces mean LPIPS by 16.5% over procedural-only fitting; PSNR and LPIPS improve on each of the three tested pairs.

We evaluate a bag, a sofa, and a table reference–mesh pair, holding the target, camera, renderer, and iteration count (200) fixed across configurations. Metrics are foreground PSNR, foreground-averaged SSIM, and VGG LPIPS with a common black background, measured against ItrI_{tr}.

ConfigurationPSNR ↑SSIM ↑LPIPS ↓
Full method18.4490.55620.1218
Without generated albedo16.0570.51900.1459
Without final-stage style loss17.9830.53740.1232

The style ablation removes the final material-refinement style term while retaining UV alignment. Keeping this term improves mean PSNR by 0.466 dB, and also improves mean SSIM and LPIPS.

2. MaPa, LumiTex, and IDArb. MaPa also uses procedural material fitting. Its official project page and author repository currently provide no runnable implementation, preventing a direct comparison. The procedural-only control above tests our added albedo representation within the same pipeline.

We additionally ran original LumiTex, including AL/MR generation and LVSM completion, with the same original reference images and meshes. For this comparison, both methods' exported materials are frozen and receive an additional, identical 50-step illumination-only fit to ItrI_{tr} under a common renderer, camera, and mask. This additional lighting fit distinguishes the following table from the ablation table.

MethodPSNR ↑SSIM ↑LPIPS ↓
Style2Shape19.8340.59370.1161
LumiTex17.9520.61530.1369

In this target-image fitting evaluation, Style2Shape gains 1.882 dB in mean PSNR and reduces mean LPIPS by 15.2%; LumiTex has higher SSIM. IDArb addresses intrinsic decomposition, corresponding to channel estimation in our pipeline rather than the complete reference-to-mesh material synthesis task.

3. Multiview consistency and channel expressiveness. Consistency comes from rendering one shared UV-space material asset across views. Unseen surfaces inherit their segment's material, making this construction suitable for approximately homogeneous or repetitive patterns within a segment. Unique localized patterns require corresponding spatial assignments. Generated albedo extends color and pattern detail; roughness, metallicity, and normals remain controlled by the selected procedural graph. Thus, fine reflectance or normal variations must be representable by that graph.

4. Optimization cost. Retrieval ranks projected roughness/metallicity maps without optimizing every candidate. The selected materials then undergo 50 lighting, 50 UV, and 100 material-refinement iterations. The representation ablation quantifies the benefit obtained within this iteration count. This is a per-asset fitting process; exported PBR maps support subsequent rendering without rerunning fitting.

Thank you for the additional experiments and clarifications. The generated-albedo ablation demonstrates a useful improvement, but my main concern about the limited methodological contribution beyond MaPa remains. The proposed extension is relatively straightforward, and the current evidence does not sufficiently establish its broader benefits. Overall, the rebuttal strengthens the empirical evidence but does not sufficiently resolve my concerns about novelty and validation. I therefore maintain my weak-reject rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.