- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents NeuMatEx, a two-stage framework for extracting expressive neural materials from multi-view images. To address the highly non-convex optimization landscape of neural material representations, a Large Material Reconstruction Model (LMRM) first predicts initial material parameters and their uncertainties. These predictions are then refined through test-time optimization using differentiable Monte Carlo inverse path tracing, with uncertainty-guided regularization to preserve reliable predictions and reduce baked-in lighting effects. By leveraging an expressive neural material basis, the framework captures complex appearance effects beyond conventional PBR, including clearcoat, dust, fuzz, and scattering, while producing materials compatible with real-time path tracing.
Critical strengths
- Elegant uncertainty-guided TTO regularization. The proposed regularization allows flexible updates in uncertain regions while preserving reliable initial predictions, helping mitigate domain shifts not fully covered by pretraining and reducing overfitting to the input images.
- Extensive ablations and strong empirical performance. The paper systematically examines material parameterization, LMRM initialization, and uncertainty-guided regularization, providing useful insights into their effects on material decomposition and rendering quality. Comparisons with PBR-based baselines on synthetic and real-world data further demonstrate the framework’s effectiveness in material reconstruction and relighting.
Critical weaknesses
- Substantial test-time computational cost. Test-time optimization takes approximately 15 minutes per object on a single NVIDIA A100 GPU, introducing considerable overhead beyond feed-forward prediction. This may limit scalability to large asset collections. A quality–runtime analysis would help assess whether the reconstruction gains justify the additional computation.
Questions for authors
Could the authors provide an end-to-end runtime comparison with the evaluated baselines on the same hardware, separately reporting preprocessing, feed-forward inference, and test-time optimization costs to clarify the quality–efficiency trade-off?
3 comments
I thank the authors for their detailed response. While TTO does introduce some additional computational overhead, it is slightly lower than I had expected. I will keep my score unchanged.
We thank the reviewer for the thoughtful feedback, which motivated a more thorough analysis of the quality-runtime trade-off and improvements to our TTO implementation.
Response to Reviewer Df3R
We thank the reviewer for the constructive review and for acknowledging the uncertainty-guided regularization and the ablation results. We address the comments below and in the revised manuscript (changes highlighted in blue).
During the rebuttal, we identified and optimized unnecessary gradient accumulation for the frozen triplane decoder in our custom CUDA backward pass, while preserving gradients to the optimized plane features. This reduces 1,000-step TTO from 15.30 to 10.73 minutes (29.8% less time), with negligible quality changes (below 0.003 dB in per-object RGB and base color PSNR); 200 steps take 2.15 minutes. All NeuMatEx TTO times below use the optimized implementation, detailed in Appendix C.2 of the revised manuscript.
Thanks for the suggestion. Our analysis shows that NeuMatEx achieves higher reconstruction quality than NVDiffRecMC++ with a comparable TTO budget on the same 40 test objects and a single NVIDIA A100 GPU.
To examine the quality-runtime trade-off more broadly, we evaluated intermediate checkpoints and extended the analysis to 2,000 TTO steps on these test objects.
The results suggest that 200 steps, or 2.15 minutes of TTO, could be a useful sweet spot for efficiency and quality. At this point, RGB PSNR improves by 2.37 dB and base color PSNR by 1.31 dB over the feed-forward LMRM prediction. Compared with 1,000 steps, it uses approximately 80% less TTO time, at a cost of 0.42 dB in RGB PSNR and 0.17 dB in base color PSNR. Extending from 1,000 to 2,000 steps gives smaller additional gains.
Thanks to the reviewer’s suggestion, we identified this useful quality-runtime trade-off and can now offer a more practical choice of optimization budget. While 200 steps is a useful operating point on this benchmark, we have not validated it as a universal stopping rule. We have added this quality-runtime analysis in Table S1, Appendix C.2 of the revised manuscript.
We have collected a runtime breakdown on the same 40 test objects, using a single NVIDIA A100 GPU per object. We report preprocessing and feed-forward inference in seconds, and TTO and total time in minutes. Total time is the sum of these three stages.
Preprocessing includes mesh/image loading, normalization, and cropping. Inference includes the vision encoders and material generation/decoding. Evaluation/visualization rendering is excluded. In the quality-runtime analysis above, stopping TTO at 200 steps (2.15 minutes) retains much of the quality improvement of the full optimization. Considering both material reconstruction quality and runtime relative to the compared methods, NeuMatEx offers a strong and practical solution for high-fidelity material extraction. We have added the runtime breakdown in Table S2, Appendix C.2 of the revised manuscript.