acceptodds
Under review as a conference paper at ICLR 2027

Extracting Neural Materials from Images

Abstract

Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initial base color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural materials, Inverse rendering, Large reconstruction model, Test-time optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Df3ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents NeuMatEx, a two-stage framework for extracting expressive neural materials from multi-view images. To address the highly non-convex optimization landscape of neural material representations, a Large Material Reconstruction Model (LMRM) first predicts initial material parameters and their uncertainties. These predictions are then refined through test-time optimization using differentiable Monte Carlo inverse path tracing, with uncertainty-guided regularization to preserve reliable predictions and reduce baked-in lighting effects. By leveraging an expressive neural material basis, the framework captures complex appearance effects beyond conventional PBR, including clearcoat, dust, fuzz, and scattering, while producing materials compatible with real-time path tracing.

Critical strengths
  1. Elegant uncertainty-guided TTO regularization. The proposed regularization allows flexible updates in uncertain regions while preserving reliable initial predictions, helping mitigate domain shifts not fully covered by pretraining and reducing overfitting to the input images.
  2. Extensive ablations and strong empirical performance. The paper systematically examines material parameterization, LMRM initialization, and uncertainty-guided regularization, providing useful insights into their effects on material decomposition and rendering quality. Comparisons with PBR-based baselines on synthetic and real-world data further demonstrate the framework’s effectiveness in material reconstruction and relighting.
Critical weaknesses
  1. Substantial test-time computational cost. Test-time optimization takes approximately 15 minutes per object on a single NVIDIA A100 GPU, introducing considerable overhead beyond feed-forward prediction. This may limit scalability to large asset collections. A quality–runtime analysis would help assess whether the reconstruction gains justify the additional computation.
Questions for authors

Could the authors provide an end-to-end runtime comparison with the evaluated baselines on the same hardware, separately reporting preprocessing, feed-forward inference, and test-time optimization costs to clarify the quality–efficiency trade-off?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Df3R

We thank the reviewer for the constructive review and for acknowledging the uncertainty-guided regularization and the ablation results. We address the comments below and in the revised manuscript (changes highlighted in blue).

Approximately 30% lower TTO runtime after optimizing CUDA implementation.

During the rebuttal, we identified and optimized unnecessary gradient accumulation for the frozen triplane decoder in our custom CUDA backward pass, while preserving gradients to the optimized plane features. This reduces 1,000-step TTO from 15.30 to 10.73 minutes (29.8% less time), with negligible quality changes (below 0.003 dB in per-object RGB and base color PSNR); 200 steps take 2.15 minutes. All NeuMatEx TTO times below use the optimized implementation, detailed in Appendix C.2 of the revised manuscript.

W1. Computational cost and the quality-runtime trade-off.

Thanks for the suggestion. Our analysis shows that NeuMatEx achieves higher reconstruction quality than NVDiffRecMC++ with a comparable TTO budget on the same 40 test objects and a single NVIDIA A100 GPU.

MethodTTO time (min) ↓RGB PSNR (dB) ↑Base color PSNR (dB) ↑
NVDiffRecMC++3.4426.2524.89
Ours (TTO 200 steps)2.1534.3625.13

To examine the quality-runtime trade-off more broadly, we evaluated intermediate checkpoints and extended the analysis to 2,000 TTO steps on these test objects.

TTO stepsTTO time (min)RGB PSNR (dB) ↑Base color PSNR (dB) ↑
0 (LMRM only)0.0031.9923.82
500.5433.7924.98
1001.0834.1025.03
2002.1534.3625.13
5005.3734.5725.22
1,00010.7334.7825.30
2,00021.4734.8125.35

The results suggest that 200 steps, or 2.15 minutes of TTO, could be a useful sweet spot for efficiency and quality. At this point, RGB PSNR improves by 2.37 dB and base color PSNR by 1.31 dB over the feed-forward LMRM prediction. Compared with 1,000 steps, it uses approximately 80% less TTO time, at a cost of 0.42 dB in RGB PSNR and 0.17 dB in base color PSNR. Extending from 1,000 to 2,000 steps gives smaller additional gains.

Thanks to the reviewer’s suggestion, we identified this useful quality-runtime trade-off and can now offer a more practical choice of optimization budget. While 200 steps is a useful operating point on this benchmark, we have not validated it as a universal stopping rule. We have added this quality-runtime analysis in Table S1, Appendix C.2 of the revised manuscript.

Q1. Runtime comparison with the baselines on the same hardware.

We have collected a runtime breakdown on the same 40 test objects, using a single NVIDIA A100 GPU per object. We report preprocessing and feed-forward inference in seconds, and TTO and total time in minutes. Total time is the sum of these three stages.

MethodPreprocessing (s)Inference (s)TTO stepsTTO (min)Total time (min)
Hunyuan3D-2.15.50630.282--0.60
TRELLIS.20.29822.624--0.38
LSRM5.17029.085--0.57
LSRM++5.17029.0851,0003.484.05
NVDiffRecMC++0.1721.5241,0003.443.47
Ours (LMRM only)0.1191.514--0.03
Ours (TTO 200 steps)0.1191.5142002.152.18
Ours (TTO 1,000 steps)0.1191.5141,00010.7310.76

Preprocessing includes mesh/image loading, normalization, and cropping. Inference includes the vision encoders and material generation/decoding. Evaluation/visualization rendering is excluded. In the quality-runtime analysis above, stopping TTO at 200 steps (2.15 minutes) retains much of the quality improvement of the full optimization. Considering both material reconstruction quality and runtime relative to the compared methods, NeuMatEx offers a strong and practical solution for high-fidelity material extraction. We have added the runtime breakdown in Table S2, Appendix C.2 of the revised manuscript.

I thank the authors for their detailed response. While TTO does introduce some additional computational overhead, it is slightly lower than I had expected. I will keep my score unchanged.

We thank the reviewer for the thoughtful feedback, which motivated a more thorough analysis of the quality-runtime trade-off and improvements to our TTO implementation.

Reviewer_MjchOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents NeuMatEx, a method for reconstructing neural materials of 3D objects from multi-view images. A large material reconstruction model is trained to predict an initial material field and per-channel uncertainties using a triplane representation. Subsequent differentiable path tracing optimization further refines the materials, with uncertainty-weighted regularization constraining high-confidence predictions while allowing uncertain regions to change.

The main contribution of the paper is to enable image-based material recovery to be practically applied to richer material representations than the typical PBR models common in inverse rendering. The paper reports evaluations on synthetic data, component ablations, and qualitative results on real-world captured data.

Critical strengths
  1. Recovering complex, reusable materials from images is valuable for asset creation. The demonstrated results, including multi-layered specular responses, transparent coatings, and fuzz, are impressive. Being able to recover a local material representation that is compatible with conventional light transport is particularly appealing.

  2. Using a reconstruction model to provide initialization for a highly expressive but hard-to-optimize latent space is reasonable. The uncertainty-guided regularization also directly addresses a core ambiguity: improvements in image reconstruction quality do not necessarily imply improvements in material decomposition quality.

  3. The paper includes comparisons with PBR material extraction methods, ablations on initialization and regularization, and a variant using predicted geometry. The NVDiffRecMC++ baseline is especially relevant, as it demonstrate that replacing the neural PBR with traditional PBR results in significantly worse results.

Critical weaknesses

The main experimental setup uses known geometry, camera poses, and HDR environment lighting. The predicted-geometry experiments are valuable; however, Table 1 shows that, compared to the full setup using ground-truth geometry, both reconstruction and relighting quality degrade noticeably. And unknown lighting conditions are never tested.

The real-world experiments use "scanned meshes" and "measured lighting". The paper stays rather vague on this, and need to be more explicit on how meshes and environment lighting were acquired. Meshes from industrial scanner combined with professional light measurement would make the this system much less practical.

Questions for authors

Would be good to see the real performance of the pipeline if both geometry and light are unknown, e.g., the original setup of NVDiffRecMC

3 comments
AuthorsOpenReview ↗

Response to Reviewer Mjch

We thank the reviewer for the valuable feedback and for acknowledging the value of reusable neural materials and uncertainty-guided material decomposition. We address the comments below and in the revised manuscript (changes highlighted in blue).

W1 / Q1. Performance when both geometry and lighting are unknown.

Our known-geometry and known-lighting setup isolates the material-recovery problem, but these assumptions are not strict limitations, as our differentiable renderer supports full gradient flow to any scene parameter. We conducted additional experiments on all 40 test objects to examine recovery without GT lighting and without both GT geometry and lighting.

For unknown lighting, we initialize a uniform environment map and optimize lighting, material, and then both jointly. For unknown geometry, we use an LSRM-predicted mesh and keep it fixed during optimization. We follow the test set and evaluation protocol of Table 1(a) in the main paper; BC denotes base color.

MethodRGB PSNR (dB) ↑RGB LPIPS ↓RGB SSIM ↑BC siPSNR (dB) ↑BC PSNR (dB) ↑Relight PSNR (dB) ↑Relight LPIPS ↓Relight SSIM ↑
LSRM (Pred mesh, feed-forward)18.290.1900.87022.1512.5418.330.1900.873
LSRM++ w/ Pred mesh & GT lighting20.680.1510.89123.2116.4120.580.1550.889
(a) Ours w/ GT mesh & GT lighting34.780.0500.96331.4525.3033.130.0470.966
(b) Ours w/ Pred mesh & GT lighting24.480.0930.92127.0926.4226.230.0920.923
(c) Ours w/ GT mesh & Learned lighting27.480.0540.96131.3224.7231.250.0520.962
(d) Ours w/ Pred mesh & Learned lighting24.090.0970.91726.9823.7825.560.0950.920

For reference, we include LSRM and LSRM++ from Table 1(a). Both use predicted geometry: LSRM is feed-forward, while LSRM++ uses GT lighting during TTO.With the same predicted geometry, NeuMatEx with predicted mesh and learned lighting, (d), achieves favorable neural material extraction with 24.09 dB RGB PSNR and 25.56 dB relighting PSNR, exceeding LSRM++ by 3.41 dB and 4.98 dB, respectively. Specifically, setting (d) addresses the reviewer’s question about performance when both geometry and lighting are unknown, as in the original setup of NVDiffRecMC. NVDiffRecMC also optimizes geometry, whereas we keep the LSRM-predicted mesh fixed and optimize material and lighting.

Geometry errors still affect material recovery, but these results demonstrate the promise of using predicted geometry, with further improvements expected as multi-view geometry estimation advances. We have added our (a)-(d) results in Table 4 and the discussion in Sec. 6.4 of the revised manuscript.

W2. Acquisition of the scanned meshes and measured lighting in the real-world experiments.

Our meshes, camera poses, photographs, and environment maps come from Digital Twin Catalog (DTC) [r1]. DTC uses a Covision Media structured-light scanner for geometry, three DSLR cameras for approximately 120 views per object, and calibrated chrome-ball images to recover lighting.

Accurate geometry and lighting acquisition without specialized equipment remains a community-wide challenge. The experiments above provide an initial step toward reducing these acquisition requirements by demonstrating material recovery with predicted geometry and learned lighting. The predicted geometry could benefit from further mesh refinement. For lighting, DiffusionLight [r2] could provide approximate HDR initialization for subsequent material-lighting optimization. These are promising directions that would be valuable to explore in future work.

References

[r1] Dong et al., "Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset", CVPR 2025.

[r2] Phongthawee et al., "DiffusionLight: Light Probes for Free by Painting a Chrome Ball", CVPR 2024.

Thanks for the additional results and information. These resolve my concern and thus I remain positive on the paper. I recommend the authors to add qualitative comparison among results for 1) GT mesh + GT lighting, 2) estimated mesh + GT lighting, 3) GT mesh + estimated lighting, 4) estimated mesh + estimated lighting in the final version, to better demonstrate how estimated mesh/lighting affect the final relighting/material-recovery quality.

We thank the reviewer for the positive assessment and constructive suggestions, which helped us strengthen the evaluation of NeuMatEx beyond known geometry and lighting. We'll add more qualitative results in the final version.

Reviewer_VGZwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NeuMatEx, the first inverse rendering framework that reconstructs neural material parameters from multi-view images, compared to traditional inverse rendering frameworks that reconstruct PBR materials. The method leverages a two-stage approach: (i) firstly, it pretrains a large Transformer feed-forward model (LMRM) to predict an initial guess of the material (parameterized by a triplane) along with uncertainty; (ii) secondly, a test-time optimization procedure refines the triplane parameters using rendering loss and is regularized by uncertainty. Experimental results show higher rendering quality compared to PBR-based inverse rendering methods, especially in the appearance of complex SVBSDFs.

Critical strengths

This is a meaningful work -- the first inverse rendering for neural materials. The results clearly demonstrate that neural materials can capture much more complex SVBRDF appearance than traditional PBR materials, so the necessity of introducing neural materials is well justified. This enables inverse rendering of objects with materials much more complicated than what PBR models can capture.

Critical weaknesses
  1. The main concern that I have about this paper is its practicality. Specifically, the method requires a ground-truth HDR environment map as the lighting for the TTO stage, which is difficult to acquire in real-world scenarios. In addition, Table 1 shows a significant performance drop when using imperfect geometry compared to perfect geometry. This paper is more of a proof of concept than a practical tool.
  2. Comparison fairness. The experiment uses Yu et al. 2026's neural material dataset, which contains complex appearances that PBR material models are inherently unable to capture, while the proposed method uses the same neural material model. This creates a "home advantage" for the proposed method. I recommend that the authors include results on datasets rendered by PBR materials to see if the neural material model can still win.
Questions for authors
  1. The two-stage training of LMRM makes the model's predictions lie in different parameter spaces (PBR vs. NM). How does the training curriculum handle the transition from PBR stage to NM stage? Do they share the same MLP head, or is the pretrained PBR head discarded and replaced by a new NM head trained from scratch?
  2. In L194-195, the paper mentions "we encode F = 17 views, evenly spaced in a 360-degree orbit around the object, alongside the six canonical views". Is this multi-view setting fixed in all experiments? Does the method support other arbitrary view settings?
9 comments
AuthorsOpenReview ↗

Response to Reviewer VGZw (Part 1/2)

We thank the reviewer for the thorough review and for acknowledging the value of extracting neural materials from images. We address the comments below and in the revised manuscript (changes highlighted in blue).

W1. Practicality: ground-truth lighting and sensitivity to imperfect geometry.

We thank the reviewer for highlighting these assumptions. Our known-geometry and known-lighting setup isolates the material-recovery problem, but these assumptions are not strict limitations, as our differentiable renderer supports full gradient flow to any scene parameter. We conducted additional experiments on all 40 test objects to examine recovery without GT lighting and without both GT geometry and lighting.

For unknown lighting, we initialize a uniform environment map and optimize lighting, material, and then both jointly. For unknown geometry, we use an LSRM-predicted mesh and keep it fixed during optimization. We follow the test set and evaluation protocol of Table 1(a) in the main paper; BC denotes base color.

MethodRGB PSNR (dB) ↑RGB LPIPS ↓RGB SSIM ↑BC siPSNR (dB) ↑BC PSNR (dB) ↑Relight PSNR (dB) ↑Relight LPIPS ↓Relight SSIM ↑
LSRM (Pred mesh, feed-forward)18.290.1900.87022.1512.5418.330.1900.873
LSRM++ w/ Pred mesh & GT lighting20.680.1510.89123.2116.4120.580.1550.889
(a) Ours w/ GT mesh & GT lighting34.780.0500.96331.4525.3033.130.0470.966
(b) Ours w/ Pred mesh & GT lighting24.480.0930.92127.0926.4226.230.0920.923
(c) Ours w/ GT mesh & Learned lighting27.480.0540.96131.3224.7231.250.0520.962
(d) Ours w/ Pred mesh & Learned lighting24.090.0970.91726.9823.7825.560.0950.920

For reference, we include LSRM and LSRM++ from Table 1(a). Both use predicted geometry: LSRM is feed-forward, while LSRM++ uses GT lighting during TTO.With the same predicted geometry, NeuMatEx with predicted mesh and learned lighting, (d), achieves favorable neural material extraction with 24.09 dB RGB PSNR and 25.56 dB relighting PSNR, exceeding LSRM++ by 3.41 dB and 4.98 dB, respectively.

Geometry errors still affect material recovery, but these results demonstrate the promise of using predicted geometry, with further improvements expected as multi-view geometry estimation advances. We have added our (a)-(d) results in Table 4 and the discussion in Sec. 6.4 of the revised manuscript.

W2. Comparison fairness and evaluation on a PBR-rendered test set.

Thanks for the suggestion. We agree that evaluation on a PBR-rendered test set is a useful complement. The neural material basis of Yu et al. (2026) extends conventional PBR material appearance with additional specular effects; our goal is to recover both ordinary PBR-like appearance and more complex materials. To test the former directly, we evaluated NeuMatEx on 32 held-out PBR-rendered objects from the BlenderVault dataset [r1]. We follow the method configurations and evaluation protocol in Table 1 of the main paper.

We have added the PBR-rendered test-set results in Table 1(b) of the revised manuscript.

MethodRGB PSNR (dB) ↑RGB LPIPS ↓RGB SSIM ↑BC siPSNR (dB) ↑BC PSNR (dB) ↑Relight PSNR (dB) ↑Relight LPIPS ↓Relight SSIM ↑
Hunyuan3D-2.125.920.0480.94326.3124.6325.810.0480.943
TRELLIS.228.350.0400.95029.9428.3828.100.0400.950
LSRM20.670.0710.92827.0118.8620.590.0720.928
LSRM++25.950.0480.94927.6023.5925.020.0520.943
NVDiffRecMC++32.570.0210.97632.9531.9831.730.0220.972
NeuMatEx (Ours, LSRM geom.)29.720.0320.96230.5129.5328.610.0360.954
NeuMatEx (Ours, Full)34.040.0150.98233.0231.6732.950.0170.978

In these results, NeuMatEx improves RGB PSNR by +1.47 dB and relighting PSNR by +1.22 dB over NVDiffRecMC++, the strongest multi-view optimization-based PBR baseline. We have also added qualitative comparisons in Fig. 6 of the revised manuscript. Thus, the benefit is not restricted to targets generated with the neural material basis. The real-photograph DTC examples in Sec. 6.3 and Appendix B.5 provide complementary evidence beyond synthetic neural material test sets.

References

[r1] Litman et al., "MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors", 3DV 2025.

AuthorsOpenReview ↗

Response to Reviewer VGZw (Part 2/2)

Q1. How does the training curriculum handle the transition from PBR to NM? Is the pretrained PBR MLP head reused or replaced by a new NM head trained from scratch?

The transition from PBR to neural material (NM) prediction is implemented through weight transfer and replacement of the output projections in the MLP heads. We initialize the NM-stage backbone and the hidden layers of both the material and uncertainty MLPs from the PBR-stage checkpoint. Only their final output layers are replaced and newly initialized to accommodate the change from 5 PBR channels (3 base color, 1 roughness, and 1 metallicity) to 9 NM channels (3 base color and 6 neural specular latents). We then jointly fine-tune the backbone and both MLPs under NM supervision. We have added these implementation details to Appendix B.1 of the revised manuscript.

Q2. Is the input setting of 17 orbital views and six canonical views fixed in all experiments? Does the method support arbitrary view settings?

Yes, the LMRM uses the fixed view layout in our experiments. Our DTC experiments (Sec. 6.3 and Fig. 7) demonstrate how to use photographs that do not follow this preset layout. As described in Appendix B.5, we first fit 2D Gaussian Splatting to the captured photographs, then synthesize the views expected by LMRM for material initialization. During TTO, we optimize against the original captured photographs, using their camera poses, to avoid fitting artifacts from the synthesized views. Thus, the preset-view requirement applies to LMRM initialization, while TTO can use the original posed capture views.

Thank you for your detailed rebuttal. The additional experimental results address the two critical weaknesses I noted in the initial review.

One concern remains after reading the response to Q2. It confirms that the proposed framework only natively supports the preset-view setting, while applying it to arbitrary real-world photos requires an additional 2DGS fitting and novel view synthesis stage. Complex reflection details caused by materials may not be well captured by 2DGS, and this may compromise the fidelity of the synthetic views input to LMRM for material initialization.

Comments on the paper

1 comment

We thank the reviewers for their constructive comments.

Reviews

We appreciate the following positive feedback:

  • NeuMatEx is recognized as “the first inverse rendering for neural materials” and a “meaningful work” (VGZw).
  • “Recovering complex, reusable materials from images is valuable for asset creation.” (Mjch)
  • Neural materials capture “much more complex SVBRDF appearance than traditional PBR materials” (VGZw), with the NVDiffRecMC++ comparison further demonstrating their advantage over traditional PBR (Mjch).
  • “Using a reconstruction model to provide initialization for a highly expressive but hard-to-optimize latent space is reasonable.” (Mjch)
  • “Elegant uncertainty-guided TTO regularization” preserves reliable predictions while allowing flexible updates in uncertain regions (Df3R), addressing a core ambiguity in material decomposition (Mjch).
  • “Extensive ablations and strong empirical performance” provide useful insights and demonstrate effective material reconstruction and relighting on synthetic and real-world data (Df3R).

Rebuttal

We address the reviewers' comments through additional experiments and clarifications, summarized below and detailed in the individual responses.

  • We extend NeuMatEx TTO to jointly optimize material and lighting for predicted geometry or/and lighting cases. Even without GT geometry and lighting, NeuMatEx recovers relightable neural materials with favorable reconstruction and relighting quality (Table 4 and Sec. 6.4).
  • We evaluate NeuMatEx on 32 held-out PBR assets from the BlenderVault dataset, achieving the best RGB reconstruction and relighting metrics compared to PBR-native methods (Table 1(b) and Fig. 6).
  • We identify a practical quality-runtime trade-off for NeuMatEx TTO: 200 steps take 2.15 minutes and achieve 34.36 dB RGB PSNR, compared with 26.25 dB in 3.44 minutes for NVDiffRecMC++. We also optimized the CUDA implementation, reducing average TTO runtime by approximately 30% with negligible quality changes (Tables S1-S2, Appendix C.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.