acceptodds
Withdrawn from ICLR 2027

PBRSense: Single-Image PBR Estimation with VLM-Conditioned Image Editor

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Recovering the PBR (Physically based rendering) material properties of an object from a single image is a long-standing problem in computer vision and graphics. Traditional methods typically require multi-view inputs or controlled capture conditions and may take hours to solve the inverse rendering problem. Recent data-driven methods based on diffusion models can produce plausible material maps for in-distribution inputs but often struggle when the input falls outside the training distribution. VLM-conditioned image editing models provide a pretrained image-conditioned generation interface with language-controlled target specification. Whether such models can be adapted to predict PBR maps for general objects while addressing these limitations has not been systematically studied. In this paper, we propose PBRSense, which adapts a pretrained VLM-conditioned image editing model for pixel-aligned prediction of albedo, metallic, roughness, and surface normal maps from a single object image. Three attribute branches share the DiT backbone and use different text prompts to specify their prediction targets. We freeze the inherited VLM and VAE, selectively fine-tune DiT blocks, and use an endpoint velocity objective for single-step prediction. We compare trainable DiT ranges under a fixed training budget to inform the fine-tuning configuration. On the combined evaluation of two held-out benchmarks, the adapted model without communication already outperforms the evaluated baselines across the reported attributes; communication provides smaller additional gains. Qualitative results on real photographs illustrate its use beyond rendered inputs. Finally, using the predicted material maps as priors improves the material estimates of a frozen 3D generation model without additional training.

Primary area:
generative models
Keywords:
inverse rendering, texture generation, image edit model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eYiPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a method called PBRSense for single-image prediction of albedo, metallic, roughness, and surface normals. Specifically, PBRSense is a pretrained VLM-conditioned image edirot. It uses a shared backbone with attribute-specific prompts, selectively fine-tunes a subset of DiT blocks, and adopts an endpoint velocity objective for single-step prediction. Lightweight cross-attribute communication modules provide additional refinement. The predicted material maps are also used to guide a frozen 3D material generator.

Critical strengths
  1. The writing of this manuscript is good and easy to follow. For example, the descriptions of the proposed method are clear.

  2. PBRSense outperforms the evaluated baselines on the combined DTC and PolyHaven-315 evaluation for albedo, metallic, roughness, and surface normals.

  3. Using PBRSense predictions to guide a frozen 3D material generator reduces material errors in both input and novel views under the evaluated setting. This strengthens the practical significance of the work by showing that the predicted maps benefit a downstream application beyond standalone 2D benchmarks.

Critical weaknesses
  1. The core idea of the proposed PBRSense mainly adapts an existing image editing model through attribute-specific prompts, selective fine-tuning, and single-step training. However, the architectural addition, cross-attribute communication, is not very new in generation tasks. In addition, the main architectural addition, cross-attribute communication, provides relatively modest gains over the adapted backbone (Table 1). Thus, the proposed PBRSense has limited methodological novelty.

  2. In this work, all experiments use the pre-trained FireRed-Image-Edit-1.1. However, it is not clear whether the proposed adaptation strategy generalizes to other image editing models or whether the strong results depend heavily on this particular pre-trained model. The proposed adaptation strategy should be plug-and-play.

  3. The layer-selection study and detailed communication-structure ablations use a 4,000-step training budget, whereas the main results use 20,000 steps. This weakens the evidence supporting the specific design choices, particularly because several reported differences are small and their stability across training runs is not established.

Questions for authors
  1. Table 2 compares selective and full fine-tuning at 4,000 steps. Does the same conclusion hold at the full 20,000-step training budget? In addition, what is the effect of single-step training compared with retaining the original multi-step training objective?

  2. Do you have evidence that the proposed adaptation strategy is effective with another pre-trained image editing model? Such evidence would materially strengthen the claim that the framework offers insights beyond adapting FireRed specifically.

  3. Do the preferred fine-tuning layer range and communication structure remain preferable at the full 20,000-step training budget? Are the relatively small gains from communication consistent across independent training runs?

  4. What are the inference latency and peak GPU memory usage of PBRSense at 1024 × 1024 resolution when predicting all attributes, and how do they compare with the strongest baselines on the same hardware?

Reviewer_kDRJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses single-image PBR estimation: given a single object image, it predicts albedo, metallic, roughness and normal maps, i.e., it recovers material properties. Its core contribution is to build on a pretrained VLM-conditioned image editing model, FireRed-Image-Edit-1.1, keeping the pretrained VLM as the conditioning pathway so that the semantic content of the input is better exploited, and to fine-tune this editing model into a PBR decomposition estimator. The paper also explores some engineering trade-offs, settling on fine-tuning DiT blocks 29–49, and inserts a cross-attribute communication module: the model predicts albedo, metallic/roughness and normals through three separate branches (actually three parallel predictions), and since these quantities are in principle mutually dependent, the module lets the branches exchange information. Experiments show that the method outperforms eight baselines on all reported metrics on two object-level benchmarks, and that its predictions can directly guide and correct the material sampling of a frozen Pixal3D model in a training-free manner. The authors also show results on real photographs that are visibly ahead of prior work; they attribute this to the VLM’s semantic conditioning, although the paper does not verify this attribution (see below).

Critical strengths
  1. The empirical results are strong, and the proposed model is clearly more capable than the previous baselines. The appendix documents the evaluation protocol and baseline settings in detail, so the comparison is credible and the system can be assessed as a reproducible whole. This is the main reason to support acceptance.
  2. In the training-free Pixal3D experiment (Table 3), the comparison with IDArb as the injected prior is informative: the baseline prior actually increases the input-view errors, which shows that what helps is the higher quality of this model’s predictions.
Critical weaknesses
  1. The paper's central motivation(§1, §2.1–2.2) is that VLM-conditioned editing models are a more generalizable foundation than SD-based backbones, but it is never tested. The paper only shows that the adapted system beats the baselines. Table 1 changes the backbone and the training data (as well as compute and resolution) simultaneously, and the sole control, raw FireRed-1.1, shows only that the zero-shot prior is weak for PBR, i.e., that the gains come from fine-tuning. Nothing separates the prior from the data: no same-data backbone swap, no VLM/instruction ablation, no data-scaling curve. The paper attributes the failures of SD-based methods to sensitivity to the training distribution, yet its own remedy is training on five asset libraries (sizes unreported), which by itself explains the better generalization. Moreover, I checked recent work and saw that Liu et al. 2026 (arXiv 2605.04566) report zero-shot NYUv2 normal errors of 17.69° for FireRed-Image-Edit, on par with the instruction-tuned Vision Banana (17.78°), whereas Table 1 leaves raw FireRed’s normals blank; how much of the reported 7.5° comes from the prior rather than from fine-tuning is therefore unknown.

  2. The methodological novelty is limited. The only new architectural component, the cross-attribute communication module, changes results by 0.1–0.2 dB and in the third decimal of RMSE (Tables 1 and 5, no variance reported), and Appendix C itself admits that these differences “do not establish the ranking”. The choice of fine-tuning blocks 29–49 (Table 2) is a four-configuration sweep at 4k steps with no analysis, and its selection rule is contradicted by the table itself (full fine-tuning is better on three of five metrics). Beyond “a large editing model fine-tuned on large data works”, the paper offers limited insight to the community.

Questions for authors
  1. Since the base model’s zero-shot normals appear to be strong, how much of the 7.5° normal error comes from the pretrained prior and how much from fine-tuning? Beyond the normals, I would like more convincing evidence on this point. I suspect that a large part of the strong results stems from the strength of the base model, I think it's fine in itself and will not reject the paper if it actually is, but I would like the authors to quantify it rather than leave it open. Not like just say we have a better model and it's mainly because of we do right task adaptation.

  2. The most central question concerns the VLM. I find it plausible that the VLM’s semantic understanding is helpful, and possibly very important, for this task; the real-photograph results, for example, are clearly better than the baseline. But the paper currently lacks a convincing ablation of the VLM, and I would like a more convincing discussion of its role.

  3. A possbile suggestion: comparing with another backbone trained on the same data, and/or a variant with the VLM conditioning removed or the instruction replaced, would give us some proof for both questions.

Reviewer_BFdFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In PRBSense a VLM-image editing model is conditioned to predict PBR attributes from a single image. It enables pixel-aligned material projection to 3D using the predicted PBR attributes as priors in material generation with PIXAL3D. The pipeline uses three PBR outputs generated by a shared DiT. The paper describes a selective fine-tuning of the VLM-based image-editing model for predictions. The results are quite nice, and there is sufficient evaluation. But at the core, the key contribution is showing that some fine-tuning can enable the VLM to perform the task.

Critical strengths
  • The produced material prediction performance shows quite some improvement compared to existing prediction frameworks.
  • The authors provide the necessary evaluations and ablations.
Critical weaknesses
  • The core approach is basically a task-specific mild rewiring of the used VLM followed by some small fine tuning and single-step learning.
Questions for authors

Please indicate how your main contribution (besides running experiments) is significantly more than just fine-tuning an existing model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.