acceptodds
Withdrawn from ICLR 2027

PhysProbe: Probing Physical-Map Recovery in Frozen Vision Generative Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Physical perception requires recovering the scene factors behind an image, beyond producing a plausible RGB transformation. Can frozen general-purpose image editors expose these factors through their native interfaces? We introduce PhysProbe, a protocol-aware framework that evaluates depth, surface normals, albedo, roughness, and metallicity against dense physical references. Each score describes a model-protocol pair with fixed inputs, output conventions, metrics, and aggregation. The results reveal three separations: local depth-boundary recovery does not imply accurate relative geometry; low full-map material error does not imply successful localization of sparse metallic regions; and additional context does not uniformly improve physical recovery. Specialists retain large advantages in relative geometry, surface orientation, and albedo. Gemini achieves the strongest reported roughness metrics under the evaluated protocol, while metallic evaluation reveals that full-map fidelity and sparse-region localization can favor different models. Normal exemplars and auxiliary cues can further change observed rankings across models, showing that physical recovery depends on both target properties and access protocols rather than a single model ordering. PhysProbe probes the physical-perception component of Physical AI while separating observed output fidelity from hypotheses about internal representations or training.

Primary area:
datasets and benchmarks
Keywords:
Vision Generative Models, Multimodal Learning, Benchmarking, Physical Scene Understanding, Image Editing, Inverse Rendering, Dense Prediction, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UVApOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PhysProbe, a protocol-aware benchmark for evaluating dense physical-map recovery in frozen general-purpose image editors. It assesses five physical properties—depth, surface normals, albedo, roughness, and metallicity—using controlled access protocols and dense ground-truth annotations. The evaluation covers nine image editors and specialized predictors across OpenRooms-FF, InteriorVerse, and the newly introduced PBR-Rooms dataset. The main findings reveal discrepancies between depth-boundary accuracy and global geometry, between full-map material errors and sparse-region localization, and between additional conditioning cues and actual performance improvements.

Critical strengths

S1. Valuable and well-defined benchmarking framework. PhysProbe provides a unified evaluation of five complementary physical properties, with carefully specified access protocols, output conventions, and aggregation rules. Evaluating model–protocol pairs rather than claiming to measure intrinsic physical understanding is a well-motivated choice that enhances the framework's validity and potential community value.

S2. Meaningful findings beyond conventional performance rankings. The distinction between local depth-boundary recovery and relative-depth accuracy is informative. More importantly, the metallic evaluation demonstrates that low full-map MAE can conceal poor material localization: an all-zero predictor achieves lower MAE than every evaluated model. These observations motivate more informative evaluation practices.

S3. Useful dataset and reproducibility resources. PBR-Rooms introduces 1,000 procedurally generated scenes and 39,000 rendered images with aligned geometric and material annotations. Its renderer-controlled material parameters and lighting variations provide valuable infrastructure for future physical-perception evaluations.

S4. Extensive diagnostics and transparent analysis. The paper investigates prompt sensitivity, exemplars, auxiliary priors, photometric conditions, and dataset-specific performance. Scene-cluster bootstrap intervals and specialist training-data audits further strengthen the empirical analysis. The authors also acknowledge important limitations rather than overstating causal or architectural conclusions.

Critical weaknesses

W1. Missing simple material-prior baselines for roughness. Table 5 shows that Gemini achieves better roughness RMSE than the evaluated specialists (0.291 versus 0.346 for Intrinsic Image Diffusion). However, without constant-value or material-category prior baselines, it remains unclear whether this advantage reflects image-conditioned physical estimation or typical-material statistics. Furthermore, both roughness specialists are evaluated on OpenRooms-FF despite using InteriorVerse supervision. These factors limit the interpretation of one of the paper's most notable positive findings.

W2. Incomplete surface-normal evaluation coverage. Only three of the nine general-purpose editors have complete, scoreable normal-prediction results. GPT-Image-2.0 and Gemini are excluded due to incomplete runs without documented completion rates or causes, while four others exhibit representation failures (Tables 31–32). Although excluding invalid outputs is reasonable, protocol completion itself is an important capability. Reporting completion rates alongside conditional prediction accuracy would make the comparison more representative and informative.

W3. Insufficient evidence for the stability of protocol-induced ranking changes. The prompt and exemplar diagnostics rely on one generation per query–setting pair. For example, normal exemplars reduce FireRed's [email protected] from 0.3192 to 0.1938 while improving GPT-Image-1.5 from 0.1826 to 0.2016. Similarly, Qwen's depth Spearman correlation drops from 0.944 to 0.467 with an exemplar. These observations are interesting, but scene-level bootstrapping does not isolate generation stochasticity. Repeated generations are needed to establish whether these effects are stable.

W4. Potential training-source confounding in albedo comparisons. DNF-Intrinsic was trained on InteriorVerse, which is also an official evaluation source. Its MAE differs substantially between InteriorVerse (0.056) and OpenRooms-FF (0.132), and exact scene-level overlap has not been fully audited. The paper appropriately discloses this issue and provides source-specific results, but the overall specialist advantage should be interpreted more cautiously until the overlap is verified.

W5. Depth Boundary F1 may be overly sensitive to spatial misalignment. The boundary metric requires exact adjacent-pixel matches without displacement tolerance, while editor-native outputs may differ in resolution and aspect ratio before resizing. Consequently, large Boundary F1 differences, such as Qwen's 0.016 versus Gemini's 0.322 despite similar AbsRel-AI, may partly reflect boundary shifts or smoothing. A tolerance-based evaluation would help establish the robustness of this finding.

Questions for authors

Q1. Regarding W1: Can the authors add constant and material-category prior baselines for roughness using training data independent of the evaluation scenes? Does Gemini retain its advantage on RMSE, Bin-BMAE, and Region-Macro MAE?

Q2. Regarding W2: What are the exact completion rates and failure reasons for GPT-Image-2.0 and Gemini under the normal protocol? Can the authors report protocol success rates separately from conditional prediction accuracy for all nine editors?

Q3. Regarding W3: Do the reported normal-ranking reversal and Qwen depth degradation persist across at least three independent generations per setting? Can paired confidence intervals for the protocol-induced performance differences be provided?

Q4. Regarding W4: Have the authors verified whether the retained InteriorVerse evaluation scenes overlap with DNF-Intrinsic's training split? How should the main albedo conclusions be interpreted after accounting for the already reported OpenRooms-FF-only results?

Q5. Regarding W5: Do the major depth-boundary performance differences persist when allowing a 1–3 pixel matching tolerance or controlling output aspect ratio and alignment?

5 comments
AuthorsOpenReview ↗

Response to Reviewer UVAp (Part 1): W1 – Roughness Prior Baselines

We thank the reviewer for the thoughtful assessment and constructive suggestions. We appreciate the positive evaluation of PhysProbe's benchmarking framework, empirical findings, and reproducibility resources. In response to the comments, we have added analyses of roughness priors, repeated-generation variability, and depth-boundary matching tolerance, and clarified the status of the normal evaluations and the potential training-data overlap affecting the albedo comparison. Our point-by-point responses follow.


W1. Roughness prior baselines and evaluation source

To address the request for constant-value and material-category roughness priors, we added four baselines to examine how Gemini compares with predictions based on global roughness statistics and material-category statistics.

Prior construction. The global Mean and Median baselines assign every pixel a single roughness value estimated from a separate OpenRooms-FF statistics pool. For each image, we compute the GT roughness mean or median over valid pixels, aggregate the image-level values within each scene, and then aggregate across scenes.

For the material-category baselines, we use DMS-46 (Upchurch and Niu, ECCV 2022), an existing RGB-based material segmentation model that predicts per-pixel labels over 46 material categories. DMS-46 predicts material labels rather than roughness. We construct DMS46 Mean and DMS46 Median by mapping each predicted material category to its category-specific roughness statistic.

We apply DMS-46 to the RGB images in the statistics pool to obtain the predicted material categories. For each category, we compute the GT roughness mean over its predicted pixels in each image, aggregate these image-level values within each scene, and then aggregate across scenes. DMS46 Median follows the same hierarchy, using the median at each aggregation level. Only images and scenes containing the category contribute to its statistics. At test time, DMS-46 predicts the material categories from RGB, and the corresponding category-specific roughness statistic is assigned to each pixel. The test images' GT material labels are not used.

The statistics pool contains 17,888 images from 982 scan IDs across 551 locations, after excluding the 48 held-out evaluation locations. It covers all 46 material categories; 44 categories appear in the 1,125 test images. We also inspected predictions on 12 test scenes under different rendering conditions. Major surfaces, including walls, wood, fabrics, and tiles, were generally assigned plausible categories, while confusion remained around furniture boundaries, small objects, and low-light regions.

All five methods were evaluated on the same 48 test scenes (1,125 images), using identical GT maps, masks, and valid regions. The results are:

Table 1. Comparison of Gemini and roughness-prior baselines on OpenRooms-FF.

MethodRMSE ↓Bin-BMAE ↓Region-Macro MAE ↓SSIM ↑
Gemini 3 Pro Image0.2911490.2512810.2578130.524555
DMS46 Mean0.2877420.2607620.2402860.562176
DMS46 Median0.2874750.2616630.2416130.555274
Global Mean0.2939250.2662410.2437640.571836
Global Median0.2937170.2681500.2438560.572314
AuthorsOpenReview ↗

Response to Reviewer UVAp (Part 2): W1 Continued and W2 – Statistical Results and Normal Evaluation

W1 (continued). Statistical comparison and evaluation source

We performed 10,000 paired scene-bootstrap resamples. For simultaneous inference, we considered all 16 prior–metric comparisons (four priors × four metrics) using the bootstrap maximum absolute standardized centered deviation. For each comparison, centered bootstrap deviations were standardized by its bootstrap standard error. We took the 95th percentile of the maximum absolute standardized deviation across the 16 comparisons as a common critical value, and constructed each simultaneous interval as the observed paired difference ± this critical value multiplied by the corresponding standard error.

We define Δ=Prior−Gemini\Delta=\mathrm{Prior}-\mathrm{Gemini}. For error metrics, a positive Δ\Delta favors Gemini; for SSIM, a positive Δ\Delta favors the prior. The strongest-prior comparisons for each metric are:

Table 2. Paired comparisons with the best-performing prior for each metric.

MetricBest-performing priorΔOrdinary 95% CISimultaneous 95% CI
RMSE ↓DMS46 Median−0.003673[−0.012492, +0.006249][−0.015708, +0.008362]
Bin-BMAE ↓DMS46 Mean+0.009480[+0.000595, +0.018200][−0.001839, +0.020800]
Region-Macro MAE ↓DMS46 Mean−0.017527[−0.024299, −0.009774][−0.026910, −0.008144]
SSIM ↑Global Median+0.047759[+0.038151, +0.056731][+0.035739, +0.059778]

Gemini has the lowest Bin-BMAE point estimate, with a lower point estimate than both global constant priors. However, its advantage over the DMS46 priors is not established by the simultaneous intervals. The RMSE difference from DMS46 Median is also not statistically significant. The simultaneous interval confirms lower Region-Macro MAE for DMS46 Mean than for Gemini, while Global Median achieves higher SSIM. The complete results for all 16 comparisons are provided in the supplementary material. Overall, Gemini's roughness performance depends on the evaluation metric rather than consistently exceeding the statistical priors.

Evaluation source and specialist comparisons. We use OpenRooms-FF for official roughness evaluation rather than InteriorVerse. As explained in Appendix B.2.1, our target-specific screening found that some InteriorVerse roughness maps contain appearance-correlated or coarsely segmented patterns whose correspondence to the intended physical parameters is ambiguous. We therefore exclude these maps from official roughness scoring.

Both roughness specialists use InteriorVerse material supervision (Appendix Table 12), while their predictions are evaluated on OpenRooms-FF. On this evaluation source, Gemini obtains an RMSE of 0.291, compared with 0.346 for Intrinsic Image Diffusion (Table 5). This is a cross-source transfer comparison under a common OpenRooms-FF protocol; the RMSE difference should therefore be read in that setting, rather than as a source-independent architectural advantage.


W2. Incomplete surface-normal evaluation coverage

We agree that the normal-evaluation coverage should be clearly explained. Among the nine evaluated editors, three (GPT-Image-1.5, Doubao Seedream 5.0, and FireRed-Image-Edit) have complete, scoreable normal-prediction results.

Four other models (FLUX.1-Kontext-dev, LongCat-Image-Edit, Qwen-Image-2.0, and Wan2.7-Image) produced outputs that could not be interpreted as valid dense normal maps under the RGB-encoded representation required by our protocol. We classify these cases as representation failures and do not assign scores to the invalid outputs. Representative examples are provided in Appendix Tables 31–32.

The remaining two models, GPT-Image-2.0 and Gemini 3 Pro Image, have incomplete full-scale three-exemplar evaluations because of time and API-budget constraints during the rebuttal period. They are therefore not included in the quantitative comparison.

If the paper is accepted, we will complete the remaining full-scale evaluations for GPT-Image-2.0 and Gemini 3 Pro Image and include their results in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer UVAp (Part 3): W3 – Repeated-Generation Results for Normal Prediction

W3. Stability of protocol-induced performance changes

Across repeated generations on the same 96 images per task–model experiment, the opposite normal-exemplar effects for GPT-Image-1.5 and FireRed persist, and Qwen's depth degradation also persists. However, the confidence interval for GPT's lead over FireRed with exemplars includes zero. We therefore find a significant change in their relative performance gap, but do not claim a statistically robust ranking reversal.

For each task–model experiment, we evaluated the same 96 images under all protocol settings. For depth prediction, both A0 and A1 were generated three times per image. For normal prediction, A1 was generated three times per image, while each of the three A0 exemplars (E1–E3) was independently generated three times per image. The three exemplar conditions were equally weighted when computing the A0 score, following the original evaluation protocol.

We estimated paired 95% confidence intervals using 20,000 two-level bootstrap resamples: scenes were resampled as clusters, and the three generations were resampled within each query–setting condition. Protocol settings remained paired at the scene/query level. For normal A0, E1–E3 were kept as separate, equally weighted exemplar conditions, with generations resampled within each query–exemplar condition.

Normal prediction

For normal prediction, A0 denotes the full prompt with three exemplars (E1–E3), while A1 denotes the full prompt without exemplars. Each exemplar in A0 was independently generated three times; the three exemplar scores are equally weighted and aggregated following the original protocol. A1 was also repeated three times.

The repeated-generation results for [email protected] are:

ModelSettingRun 1Run 2Run 3Mean ± SD
GPT-Image-1.5A0 (with exemplars)0.20160.20990.20550.2056 ± 0.0041
GPT-Image-1.5A1 (without exemplars)0.18260.15410.14770.1615 ± 0.0186
FireRed-Image-EditA0 (with exemplars)0.19380.19360.19210.1931 ± 0.0009
FireRed-Image-EditA1 (without exemplars)0.31920.35810.35120.3428 ± 0.0208

The individual exemplar results under A0 are:

ModelExemplarRun 1Run 2Run 3Mean ± SD
GPT-Image-1.5E10.20350.19010.18870.1941 ± 0.0081
GPT-Image-1.5E20.19450.23040.21210.2123 ± 0.0179
GPT-Image-1.5E30.20680.20910.21560.2105 ± 0.0046
FireRed-Image-EditE10.18900.18560.18400.1862 ± 0.0025
FireRed-Image-EditE20.18700.19070.18780.1885 ± 0.0019
FireRed-Image-EditE30.20540.20440.20430.2047 ± 0.0006

We calculate the exemplar effect for each model as [email protected] with exemplars minus [email protected] without exemplars.

ModelExemplar effect (with − without)Paired 95% CI
GPT-Image-1.5+0.0442[+0.0219, +0.0654]
FireRed-Image-Edit−0.1497[−0.1920, −0.1090]

Both confidence intervals exclude zero. Under the tested exemplar conditions, adding exemplars improves GPT-Image-1.5's normal prediction but degrades FireRed-Image-Edit's performance.

We further examined the change in the relative performance gap between the two models. We define the with-exemplar gap as GPT-Image-1.5's [email protected] minus FireRed-Image-Edit's [email protected] under A0, and the without-exemplar gap in the same way under A1. The change in gap is the with-exemplar gap minus the without-exemplar gap.

StatisticEstimatePaired 95% CI
With-exemplar gap (GPT − FireRed)+0.0125[−0.0159, +0.0419]
Without-exemplar gap (GPT − FireRed)−0.1814[−0.2241, −0.1376]
Change in gap+0.1939[+0.1554, +0.2329]

FireRed significantly outperforms GPT-Image-1.5 without exemplars. The GPT lead with exemplars is not statistically resolved, as the confidence interval for the with-exemplar gap includes zero. However, the positive change in the relative performance gap is statistically supported, indicating that exemplar conditioning substantially changes the relative performance of the two models.

The other normal metrics show the same directional trends. Across the three runs, GPT-Image-1.5's mean angular error decreases from 52.28 without exemplars to 46.69 with exemplars, while Acc@30 increases from 0.2474 to 0.3167. For FireRed-Image-Edit, mean angular error increases from 56.00 to 61.72, while Acc@30 decreases from 0.4133 to 0.2456. These are descriptive averages across the three runs.

Reviewer_YyVoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce a benchmark, in which they challenge recent state of the art general image generators to produce physical maps from a single RGB input image. The probed physical maps are depth, normal, albedo, roughness and metallic. Their key contribution is a collected and self rendered dataset wit physical maps and a evaluation protocol, which in simple terms is just utilising the image pair, the correct metrics and an instruction prompt.

Their key finding is that specialised models out perform general models. However for roughness Gemini scores better.

Overall this is a solid idea but to me this dataset and evaluation protocol, feels like standard practice that is already used for e.g. relighting reconstruction methods. Their findings are helpful and I see the amount of work that building a benchmark takes. However the complexity and design seems straightforward and the paper is heavily AI written, such that I lean towards a weak reject, rather then a accept direction. I'm happy to discuss with fellow reviewers.

Critical strengths

The overall goal, testing general generative methods if they have a spatial understanding and can produce outputs.

It answers a question, I did not know the answer to yet. Are recent generative models capable of producing coherent physical maps and the answer is no (except Gemini for roughness).

Critical weaknesses

This paper is heavily AI written and sometimes hard to follow, many unnecessary compound words. I would urge the authors to express some hard to read phrases in their own words e.g.

"For surface normals, visually similar maps can encode incompatible coordinate and RGB conventions. PhysProbe therefore provides three fixed, scene-disjoint RGB–normal exemplars to define the output convention rather than provide query-specific information. Each query is processed in three independent one-shot runs, one for each exemplar, and the official normal score averages these runs to reduce dependence on any single example."

The benchmark does not heavily change instructions or bring the methods to its limits to understand if they tuely have no understanding of this physical maps. It just test the default functionality and provides a dataset to test on.

Questions for authors

Your evaluation protocol is fairly simple and you care mostly about the output but does this conclude to that these models don't have an internal spatial understanding? E.g. VLM^3 shows that LLMs can produce better performance then all specialised methods.

What is with video models? They should maybe have even more spatial understanding, how do they perform?

Why do Gemini and GPT-Image 2.0 have incomplete normal runs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YyVo (Part 1): Clarity, Evaluation Scope, and Benchmark Contributions

We thank the reviewer for the constructive comments on the clarity, scope, and interpretation of our evaluation. We have revised the relevant descriptions and address the concerns below.

W1. Clarity and presentation

We agree that some descriptions in the original manuscript used unnecessarily complex expressions and compound terminology. We have reviewed and revised the relevant passages throughout the manuscript to make the experimental procedures and evaluation settings more direct and easier to follow. We have incorporated these changes into the revised manuscript, which we will submit alongside this rebuttal. For example, visually similar normal maps may encode surface directions using different coordinate systems and RGB encodings. To specify the required output convention, we use three fixed RGB–normal example pairs from different scenes. The same three pairs are shared across all evaluated editors and are not tailored to individual test images. Each query is evaluated separately with each pair, and the three resulting scores are averaged.

W2. Evaluation scope and benchmark contributions

We agree that recovering dense physical maps from RGB images using aligned ground truth and target-specific metrics is an established practice in inverse rendering (Li et al., 2020; Wang et al., 2021; Zhu et al., 2022). PhysProbe builds on this practice to investigate what physical information frozen, general-purpose image editors can recover through their native interfaces and how their performance compares with that of specialized predictors. Beyond overall performance comparisons, our diagnostic experiments reveal several distinctions that are not captured by a single score:

  • Local boundary accuracy does not imply accurate relative-depth estimation. Gemini achieves a depth Boundary F1 of 0.322, close to Lotus at 0.318, yet its AbsRel-AI remains substantially higher than that of Depth Anything 3 (0.153 vs. 0.035), even after scale, shift, and polarity alignment. Local depth-discontinuity recovery and overall relative-depth accuracy therefore capture different aspects of performance.
  • Similar full-map errors can coexist with different localization scores. In the paired PBR-Rooms lighting diagnostic, GPT-Image-2.0's metallic MAE rounds to 0.018 across all six conditions, close to the all-zero baseline (0.017), while its IoU ranges from 0.036 to 0.103. Scene, camera, and material assignments remain fixed while lighting changes. This shows that full-map MAE alone does not capture metallic-region localization.
  • Additional context can affect models differently. In the normal example diagnostic, adding RGB–normal examples improves GPT-Image-1.5's [email protected] from 0.1826 to 0.2016, but reduces FireRed's from 0.3192 to 0.1938, reversing their relative ordering. This demonstrates that the effect of additional context depends on the model. Our data sources provide complementary supervision for these comparisons. OpenRooms-FF provides depth, normals, albedo, and roughness annotations; InteriorVerse supplements depth, normals, and albedo; and PBR-Rooms (approximately 1,000 scenes and 39,000 images) provides metallic ground truth exported directly from the renderer's PBR material parameters, together with controlled lighting variations. This combination supports evaluation across five physical properties while addressing the lack of reliable metallic ground truth in the other retained sources. Our official evaluation uses fixed and disclosed settings to compare different editors. Separate diagnostics vary prompt wording, reference examples, auxiliary cues, and lighting conditions to examine selected sources of evaluation sensitivity. These diagnostics are not intended as per-model prompt optimization. Each score should therefore be interpreted under its specified evaluation setting. Taken together, PhysProbe provides an editor-oriented benchmark combining dense physical-map supervision, fixed evaluation settings, and targeted diagnostics to characterize how frozen general-purpose editors recover different physical properties under controlled inputs.
AuthorsOpenReview ↗

Response to Reviewer YyVo (Part 2): Further Questions on Spatial Understanding, Video Models, and Normal Evaluation

Q1. Does the evaluation establish a lack of internal spatial understanding?

Our results do not establish that general-purpose image editors lack internal spatial or physical understanding. PhysProbe measures whether frozen editors can produce accurate, spatially aligned physical maps through their native interfaces under specified instructions and input conditions. Low scores may reflect inaccurate scene estimation, difficulty interpreting the required physical-map representation, or difficulty expressing estimates as spatially aligned pixel values. These factors cannot be fully distinguished from final outputs alone. Our results therefore characterize observable prediction performance under the evaluated settings, rather than directly measuring internal representations. Our evaluation also goes beyond a single default prompt. We conduct targeted diagnostics on prompt wording, reference examples, auxiliary cues, and photometric conditions, which reveal that outputs can be sensitive to the tested conditions. We agree that VLM3\mathrm{VLM}^{3} highlights an important consideration: poor performance under our evaluated conditions should not be taken as evidence that general-purpose models cannot achieve strong 3D prediction. VLM3\mathrm{VLM}^{3} demonstrates such performance using a standard VLM architecture with supervised fine-tuning on 3D tasks. Its results are compatible with ours: VLM3\mathrm{VLM}^{3} studies performance after 3D-oriented training, whereas PhysProbe investigates what existing image editors can recover through their native interfaces without task-specific adaptation. This remains a practically meaningful question: can an existing image editor produce accurate physical maps from an RGB image without additional task-specific training? We have revised the Introduction and Conclusion to keep our claims focused on observable physical-map recovery rather than the full extent of internal spatial understanding.

Q2. How might video generation models perform?

Video generation models are a relevant extension. We have not evaluated them, so we cannot determine from our current results whether they would perform better on physical-map prediction. Are we correct in understanding that the reviewer suggests evaluating video generation models on the same task: predicting physical maps aligned with a single input RGB image? If so, is there a particular model the reviewer would recommend testing?

Q3. Incomplete normal evaluations

Among the nine evaluated editors, three (GPT-Image-1.5, Doubao Seedream 5.0, and FireRed-Image-Edit) have complete runs with valid normal-map predictions. Four others (FLUX.1-Kontext-dev, LongCat-Image-Edit, Qwen-Image-2.0, and Wan2.7-Image) failed to produce valid normal maps under the required setting and receive no official normal scores. The normal evaluations for GPT-Image-2.0 and Gemini 3 Pro Image remain incomplete because the available time and API budget during the rebuttal period did not allow us to complete the three-example evaluation. We therefore report no official normal scores for these two models. The revised manuscript clarifies these statuses and distinguishes incomplete evaluations from failures to produce valid normal maps. We thank the reviewer again for these comments, which have helped us improve the presentation and clarify the scope and contributions of PhysProbe.

AuthorsOpenReview ↗

Summary of Manuscript Revisions

For ease of review, all changes in the revised manuscript are highlighted in red. In addition to the responses above, we have made four main revisions:

  1. Clarifying the contribution of PBR-Rooms. We have expanded its description, highlighting its scale (approximately 1,000 scenes and 39,000 images), aligned annotations for five physical properties, metallic ground truth exported directly from the renderer's PBR material parameters, and controlled lighting variations. We also clarify its role as a complementary data source and its use in official metallic evaluation.
  2. Revising the Introduction and research positioning. We revised the opening paragraph to clarify our research question and motivate dense physical-map recovery as a way to examine the capabilities of frozen image editors. We also revised the discussion of prior adaptation-based work to distinguish task-specific prediction after adaptation from our focus on what frozen editors can recover through their native interfaces.
  3. Improving readability and terminology. We simplified complex sentences, reduced unnecessary compound expressions, and improved terminology consistency throughout the manuscript, making the experimental procedures and findings easier to follow.
  4. Clarifying tables and captions. We revised selected table titles, column headings, and notes to clarify dataset roles, evaluation coverage, model run statuses, and metric aggregation. In particular, we clarified normal-evaluation coverage and distinguished unfinished evaluations from cases in which models failed to produce valid normal maps.
Reviewer_i1sPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how well large image editing models can predict physical attributes (e.g., depth, normals, PBR) of indoor images. Several commercial APIs (Gemini, GPT Image, Seedream) are evaluated with carefully designed datasets, prompt protocols, and evaluation metrics.

Critical strengths
  1. This paper aims at a clear and meaningful task -- evaluating how pretrained image editing models perform on predicting physical maps given indoor scene images. Corresponding results are presented for further research and applications to draw insights from.
  2. Evaluation protocols are designed to identify the pros and cons of different models in terms of several aspects, such as local boundary accuracy, geometry accuracy, and PBR fidelity.
Critical weaknesses
  1. The experiments are kind of insufficient and not many insights are provided. I suggest the authors have a look at Gabeur et al. (2026), which not only studied the SOTA expert model of each modality, but also tuned a pretrained model to see the performance gap.

  2. This paper does not provide much technical innovation -- designing evaluation protocols and running the APIs may not reach the acceptance bar of a top venue like ICLR.

Questions for authors

Please address the aforementioned weaknesses, showing more valuable insights in this paper.

Comments on the paper

3 comments
Area_Chair_HcQkOpenReview ↗

Missing important related works

Dear authors,

Thanks for the rebuttal. Several related works are not discussed in the paper -- see the non-exhaustive list below:

  • Bhattad et al. -- StyleGAN Knows Normal, Depth, Albedo, and More, NeurIPS 2023.
  • Chen et al. -- Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model, arXiv:2306.05720, 2023.
  • Zhao et al. -- Unleashing Text-to-Image Diffusion Models for Visual Perception, ICCV 2023
  • Gan et al. -- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists, ICLR 2024
  • Luo et al. -- Readout Guidance: Learning Control from Diffusion Features, CVPR 2024
  • Lee et al. -- Exploiting Diffusion Prior for Generalizable Dense Prediction, CVPR 2024.
  • Du et al. -- Generative Models: What Do They Know? Do They Know Things? Let’s Find Out!, CVPR-W 2024,
  • Zhan et al. -- A General Protocol to Probe Large Vision Models for 3D Physical Understanding, NeurIPS 2024.
  • El Banani et al. -- Probing the 3D Awareness of Visual Foundation Models, CVPR 2024.
  • Ke et al. -- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation, CVPR 2024, and Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis, TPAMI 2025
  • Xu et al. What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?, ICLR 2025
  • Wiedemer et al. -- Video models are zero-shot learners and reasoners, arXiv 2025
  • Ramachandran et al. -- How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks, ICLR 2026

Best,

AC

Dear reviewers,

Thanks for your continued efforts. The authors have provided a rebuttal to the initial reviews. Please review it carefully, along with all other reviews and the corresponding author responses, and consider whether you have any further questions for the authors or whether the rebuttal has addressed your concerns. Also see the additional comment from the AC about several missing related works. There are only a few days remaining before the discussion period closes, so please use this time to seek any further clarifications or raise any additional questions about this paper.

Best,

AC

AuthorsOpenReview ↗

Expanded Related Work and Clarified Positioning of PhysProbe

Dear Area Chair and Reviewers,

Thank you for the helpful comments and for highlighting these important related works. We have expanded and reorganized the Related Work section to cover all the suggested works and clarify their relationships to PhysProbe.

  1. Intrinsic scene representations in generative models, including the work of Bhattad, Chen, and Du.
  2. Generative models for visual perception and dense prediction, including VPD, InstructCV, Readout Guidance, DMP, Ke et al.'s CVPR 2024 work on monocular depth estimation and its TPAMI 2025 Marigold extension, and the study by Xu et al.
  3. Probing and benchmarking physical understanding in foundation models. We discuss both representation-level probing, including the work of Zhan and El Banani, and task-level evaluation of model capabilities, including the studies by Ramachandran and Wiedemer.
  4. Specialized physical prediction and evaluation resources, covering task-specific predictors and reference datasets relevant to the evaluation of dense physical properties.

We also clarify the relationship between PhysProbe and Ramachandran et al. Their work evaluates multimodal foundation models on standard computer vision tasks, including depth and surface-normal prediction, through API-compatible prompting, and examines prompt sensitivity and native image-generation outputs. PhysProbe complements these evaluations by focusing on dense recovery of geometry and material properties through frozen general-purpose image editors. It covers five physical targets: depth, surface normals, albedo, roughness, and metallicity. By integrating target-specific evaluation protocols, aligned physical references, and complementary numerical and spatial metrics, PhysProbe examines different aspects of physical-map recovery, including sparse metallic-region localization.

Beyond the Related Work section, we have also revised the Introduction to clarify the motivation and evaluation questions of PhysProbe, and updated the Conclusion to better summarize the integrated evaluation methodology and the main findings. These changes align the manuscript's overall narrative with the expanded literature discussion and the refined positioning of our work. We provide the revised manuscript for reference.

These revisions connect PhysProbe more clearly to prior research on generative representations, their adaptation for perception, and the evaluation of foundation-model capabilities. We appreciate your suggestions, which have helped us improve both the literature coverage and the positioning of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.