- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces PhysProbe, a protocol-aware benchmark for evaluating dense physical-map recovery in frozen general-purpose image editors. It assesses five physical properties—depth, surface normals, albedo, roughness, and metallicity—using controlled access protocols and dense ground-truth annotations. The evaluation covers nine image editors and specialized predictors across OpenRooms-FF, InteriorVerse, and the newly introduced PBR-Rooms dataset. The main findings reveal discrepancies between depth-boundary accuracy and global geometry, between full-map material errors and sparse-region localization, and between additional conditioning cues and actual performance improvements.
Critical strengths
S1. Valuable and well-defined benchmarking framework. PhysProbe provides a unified evaluation of five complementary physical properties, with carefully specified access protocols, output conventions, and aggregation rules. Evaluating model–protocol pairs rather than claiming to measure intrinsic physical understanding is a well-motivated choice that enhances the framework's validity and potential community value.
S2. Meaningful findings beyond conventional performance rankings. The distinction between local depth-boundary recovery and relative-depth accuracy is informative. More importantly, the metallic evaluation demonstrates that low full-map MAE can conceal poor material localization: an all-zero predictor achieves lower MAE than every evaluated model. These observations motivate more informative evaluation practices.
S3. Useful dataset and reproducibility resources. PBR-Rooms introduces 1,000 procedurally generated scenes and 39,000 rendered images with aligned geometric and material annotations. Its renderer-controlled material parameters and lighting variations provide valuable infrastructure for future physical-perception evaluations.
S4. Extensive diagnostics and transparent analysis. The paper investigates prompt sensitivity, exemplars, auxiliary priors, photometric conditions, and dataset-specific performance. Scene-cluster bootstrap intervals and specialist training-data audits further strengthen the empirical analysis. The authors also acknowledge important limitations rather than overstating causal or architectural conclusions.
Critical weaknesses
W1. Missing simple material-prior baselines for roughness. Table 5 shows that Gemini achieves better roughness RMSE than the evaluated specialists (0.291 versus 0.346 for Intrinsic Image Diffusion). However, without constant-value or material-category prior baselines, it remains unclear whether this advantage reflects image-conditioned physical estimation or typical-material statistics. Furthermore, both roughness specialists are evaluated on OpenRooms-FF despite using InteriorVerse supervision. These factors limit the interpretation of one of the paper's most notable positive findings.
W2. Incomplete surface-normal evaluation coverage. Only three of the nine general-purpose editors have complete, scoreable normal-prediction results. GPT-Image-2.0 and Gemini are excluded due to incomplete runs without documented completion rates or causes, while four others exhibit representation failures (Tables 31–32). Although excluding invalid outputs is reasonable, protocol completion itself is an important capability. Reporting completion rates alongside conditional prediction accuracy would make the comparison more representative and informative.
W3. Insufficient evidence for the stability of protocol-induced ranking changes. The prompt and exemplar diagnostics rely on one generation per query–setting pair. For example, normal exemplars reduce FireRed's [email protected] from 0.3192 to 0.1938 while improving GPT-Image-1.5 from 0.1826 to 0.2016. Similarly, Qwen's depth Spearman correlation drops from 0.944 to 0.467 with an exemplar. These observations are interesting, but scene-level bootstrapping does not isolate generation stochasticity. Repeated generations are needed to establish whether these effects are stable.
W4. Potential training-source confounding in albedo comparisons. DNF-Intrinsic was trained on InteriorVerse, which is also an official evaluation source. Its MAE differs substantially between InteriorVerse (0.056) and OpenRooms-FF (0.132), and exact scene-level overlap has not been fully audited. The paper appropriately discloses this issue and provides source-specific results, but the overall specialist advantage should be interpreted more cautiously until the overlap is verified.
W5. Depth Boundary F1 may be overly sensitive to spatial misalignment. The boundary metric requires exact adjacent-pixel matches without displacement tolerance, while editor-native outputs may differ in resolution and aspect ratio before resizing. Consequently, large Boundary F1 differences, such as Qwen's 0.016 versus Gemini's 0.322 despite similar AbsRel-AI, may partly reflect boundary shifts or smoothing. A tolerance-based evaluation would help establish the robustness of this finding.
Questions for authors
Q1. Regarding W1: Can the authors add constant and material-category prior baselines for roughness using training data independent of the evaluation scenes? Does Gemini retain its advantage on RMSE, Bin-BMAE, and Region-Macro MAE?
Q2. Regarding W2: What are the exact completion rates and failure reasons for GPT-Image-2.0 and Gemini under the normal protocol? Can the authors report protocol success rates separately from conditional prediction accuracy for all nine editors?
Q3. Regarding W3: Do the reported normal-ranking reversal and Qwen depth degradation persist across at least three independent generations per setting? Can paired confidence intervals for the protocol-induced performance differences be provided?
Q4. Regarding W4: Have the authors verified whether the retained InteriorVerse evaluation scenes overlap with DNF-Intrinsic's training split? How should the main albedo conclusions be interpreted after accounting for the already reported OpenRooms-FF-only results?
Q5. Regarding W5: Do the major depth-boundary performance differences persist when allowing a 1–3 pixel matching tolerance or controlling output aspect ratio and alignment?
5 comments
Response to Reviewer UVAp (Part 2): W1 Continued and W2 – Statistical Results and Normal Evaluation
W1 (continued). Statistical comparison and evaluation source
We performed 10,000 paired scene-bootstrap resamples. For simultaneous inference, we considered all 16 prior–metric comparisons (four priors × four metrics) using the bootstrap maximum absolute standardized centered deviation. For each comparison, centered bootstrap deviations were standardized by its bootstrap standard error. We took the 95th percentile of the maximum absolute standardized deviation across the 16 comparisons as a common critical value, and constructed each simultaneous interval as the observed paired difference ± this critical value multiplied by the corresponding standard error.
We define . For error metrics, a positive favors Gemini; for SSIM, a positive favors the prior. The strongest-prior comparisons for each metric are:
Table 2. Paired comparisons with the best-performing prior for each metric.
| Metric | Best-performing prior | Δ | Ordinary 95% CI | Simultaneous 95% CI |
|---|---|---|---|---|
| RMSE ↓ | DMS46 Median | −0.003673 | [−0.012492, +0.006249] | [−0.015708, +0.008362] |
| Bin-BMAE ↓ | DMS46 Mean | +0.009480 | [+0.000595, +0.018200] | [−0.001839, +0.020800] |
| Region-Macro MAE ↓ | DMS46 Mean | −0.017527 | [−0.024299, −0.009774] | [−0.026910, −0.008144] |
| SSIM ↑ | Global Median | +0.047759 | [+0.038151, +0.056731] | [+0.035739, +0.059778] |
Gemini has the lowest Bin-BMAE point estimate, with a lower point estimate than both global constant priors. However, its advantage over the DMS46 priors is not established by the simultaneous intervals. The RMSE difference from DMS46 Median is also not statistically significant. The simultaneous interval confirms lower Region-Macro MAE for DMS46 Mean than for Gemini, while Global Median achieves higher SSIM. The complete results for all 16 comparisons are provided in the supplementary material. Overall, Gemini's roughness performance depends on the evaluation metric rather than consistently exceeding the statistical priors.
Evaluation source and specialist comparisons. We use OpenRooms-FF for official roughness evaluation rather than InteriorVerse. As explained in Appendix B.2.1, our target-specific screening found that some InteriorVerse roughness maps contain appearance-correlated or coarsely segmented patterns whose correspondence to the intended physical parameters is ambiguous. We therefore exclude these maps from official roughness scoring.
Both roughness specialists use InteriorVerse material supervision (Appendix Table 12), while their predictions are evaluated on OpenRooms-FF. On this evaluation source, Gemini obtains an RMSE of 0.291, compared with 0.346 for Intrinsic Image Diffusion (Table 5). This is a cross-source transfer comparison under a common OpenRooms-FF protocol; the RMSE difference should therefore be read in that setting, rather than as a source-independent architectural advantage.
W2. Incomplete surface-normal evaluation coverage
We agree that the normal-evaluation coverage should be clearly explained. Among the nine evaluated editors, three (GPT-Image-1.5, Doubao Seedream 5.0, and FireRed-Image-Edit) have complete, scoreable normal-prediction results.
Four other models (FLUX.1-Kontext-dev, LongCat-Image-Edit, Qwen-Image-2.0, and Wan2.7-Image) produced outputs that could not be interpreted as valid dense normal maps under the RGB-encoded representation required by our protocol. We classify these cases as representation failures and do not assign scores to the invalid outputs. Representative examples are provided in Appendix Tables 31–32.
The remaining two models, GPT-Image-2.0 and Gemini 3 Pro Image, have incomplete full-scale three-exemplar evaluations because of time and API-budget constraints during the rebuttal period. They are therefore not included in the quantitative comparison.
If the paper is accepted, we will complete the remaining full-scale evaluations for GPT-Image-2.0 and Gemini 3 Pro Image and include their results in the camera-ready version.
Response to Reviewer UVAp (Part 3): W3 – Repeated-Generation Results for Normal Prediction
W3. Stability of protocol-induced performance changes
Across repeated generations on the same 96 images per task–model experiment, the opposite normal-exemplar effects for GPT-Image-1.5 and FireRed persist, and Qwen's depth degradation also persists. However, the confidence interval for GPT's lead over FireRed with exemplars includes zero. We therefore find a significant change in their relative performance gap, but do not claim a statistically robust ranking reversal.
For each task–model experiment, we evaluated the same 96 images under all protocol settings. For depth prediction, both A0 and A1 were generated three times per image. For normal prediction, A1 was generated three times per image, while each of the three A0 exemplars (E1–E3) was independently generated three times per image. The three exemplar conditions were equally weighted when computing the A0 score, following the original evaluation protocol.
We estimated paired 95% confidence intervals using 20,000 two-level bootstrap resamples: scenes were resampled as clusters, and the three generations were resampled within each query–setting condition. Protocol settings remained paired at the scene/query level. For normal A0, E1–E3 were kept as separate, equally weighted exemplar conditions, with generations resampled within each query–exemplar condition.
Normal prediction
For normal prediction, A0 denotes the full prompt with three exemplars (E1–E3), while A1 denotes the full prompt without exemplars. Each exemplar in A0 was independently generated three times; the three exemplar scores are equally weighted and aggregated following the original protocol. A1 was also repeated three times.
The repeated-generation results for [email protected] are:
| Model | Setting | Run 1 | Run 2 | Run 3 | Mean ± SD |
|---|---|---|---|---|---|
| GPT-Image-1.5 | A0 (with exemplars) | 0.2016 | 0.2099 | 0.2055 | 0.2056 ± 0.0041 |
| GPT-Image-1.5 | A1 (without exemplars) | 0.1826 | 0.1541 | 0.1477 | 0.1615 ± 0.0186 |
| FireRed-Image-Edit | A0 (with exemplars) | 0.1938 | 0.1936 | 0.1921 | 0.1931 ± 0.0009 |
| FireRed-Image-Edit | A1 (without exemplars) | 0.3192 | 0.3581 | 0.3512 | 0.3428 ± 0.0208 |
The individual exemplar results under A0 are:
| Model | Exemplar | Run 1 | Run 2 | Run 3 | Mean ± SD |
|---|---|---|---|---|---|
| GPT-Image-1.5 | E1 | 0.2035 | 0.1901 | 0.1887 | 0.1941 ± 0.0081 |
| GPT-Image-1.5 | E2 | 0.1945 | 0.2304 | 0.2121 | 0.2123 ± 0.0179 |
| GPT-Image-1.5 | E3 | 0.2068 | 0.2091 | 0.2156 | 0.2105 ± 0.0046 |
| FireRed-Image-Edit | E1 | 0.1890 | 0.1856 | 0.1840 | 0.1862 ± 0.0025 |
| FireRed-Image-Edit | E2 | 0.1870 | 0.1907 | 0.1878 | 0.1885 ± 0.0019 |
| FireRed-Image-Edit | E3 | 0.2054 | 0.2044 | 0.2043 | 0.2047 ± 0.0006 |
We calculate the exemplar effect for each model as [email protected] with exemplars minus [email protected] without exemplars.
| Model | Exemplar effect (with − without) | Paired 95% CI |
|---|---|---|
| GPT-Image-1.5 | +0.0442 | [+0.0219, +0.0654] |
| FireRed-Image-Edit | −0.1497 | [−0.1920, −0.1090] |
Both confidence intervals exclude zero. Under the tested exemplar conditions, adding exemplars improves GPT-Image-1.5's normal prediction but degrades FireRed-Image-Edit's performance.
We further examined the change in the relative performance gap between the two models. We define the with-exemplar gap as GPT-Image-1.5's [email protected] minus FireRed-Image-Edit's [email protected] under A0, and the without-exemplar gap in the same way under A1. The change in gap is the with-exemplar gap minus the without-exemplar gap.
| Statistic | Estimate | Paired 95% CI |
|---|---|---|
| With-exemplar gap (GPT − FireRed) | +0.0125 | [−0.0159, +0.0419] |
| Without-exemplar gap (GPT − FireRed) | −0.1814 | [−0.2241, −0.1376] |
| Change in gap | +0.1939 | [+0.1554, +0.2329] |
FireRed significantly outperforms GPT-Image-1.5 without exemplars. The GPT lead with exemplars is not statistically resolved, as the confidence interval for the with-exemplar gap includes zero. However, the positive change in the relative performance gap is statistically supported, indicating that exemplar conditioning substantially changes the relative performance of the two models.
The other normal metrics show the same directional trends. Across the three runs, GPT-Image-1.5's mean angular error decreases from 52.28 without exemplars to 46.69 with exemplars, while Acc@30 increases from 0.2474 to 0.3167. For FireRed-Image-Edit, mean angular error increases from 56.00 to 61.72, while Acc@30 decreases from 0.4133 to 0.2456. These are descriptive averages across the three runs.
Response to Reviewer UVAp (Part 1): W1 – Roughness Prior Baselines
We thank the reviewer for the thoughtful assessment and constructive suggestions. We appreciate the positive evaluation of PhysProbe's benchmarking framework, empirical findings, and reproducibility resources. In response to the comments, we have added analyses of roughness priors, repeated-generation variability, and depth-boundary matching tolerance, and clarified the status of the normal evaluations and the potential training-data overlap affecting the albedo comparison. Our point-by-point responses follow.
W1. Roughness prior baselines and evaluation source
To address the request for constant-value and material-category roughness priors, we added four baselines to examine how Gemini compares with predictions based on global roughness statistics and material-category statistics.
Prior construction. The global Mean and Median baselines assign every pixel a single roughness value estimated from a separate OpenRooms-FF statistics pool. For each image, we compute the GT roughness mean or median over valid pixels, aggregate the image-level values within each scene, and then aggregate across scenes.
For the material-category baselines, we use DMS-46 (Upchurch and Niu, ECCV 2022), an existing RGB-based material segmentation model that predicts per-pixel labels over 46 material categories. DMS-46 predicts material labels rather than roughness. We construct DMS46 Mean and DMS46 Median by mapping each predicted material category to its category-specific roughness statistic.
We apply DMS-46 to the RGB images in the statistics pool to obtain the predicted material categories. For each category, we compute the GT roughness mean over its predicted pixels in each image, aggregate these image-level values within each scene, and then aggregate across scenes. DMS46 Median follows the same hierarchy, using the median at each aggregation level. Only images and scenes containing the category contribute to its statistics. At test time, DMS-46 predicts the material categories from RGB, and the corresponding category-specific roughness statistic is assigned to each pixel. The test images' GT material labels are not used.
The statistics pool contains 17,888 images from 982 scan IDs across 551 locations, after excluding the 48 held-out evaluation locations. It covers all 46 material categories; 44 categories appear in the 1,125 test images. We also inspected predictions on 12 test scenes under different rendering conditions. Major surfaces, including walls, wood, fabrics, and tiles, were generally assigned plausible categories, while confusion remained around furniture boundaries, small objects, and low-light regions.
All five methods were evaluated on the same 48 test scenes (1,125 images), using identical GT maps, masks, and valid regions. The results are:
Table 1. Comparison of Gemini and roughness-prior baselines on OpenRooms-FF.