acceptodds
Under review as a conference paper at ICLR 2027

Reformulating Uncalibrated Photometric Stereo as a Joint Depth–Normal Estimation Task

Abstract

Uncalibrated Photometric Stereo (UPS) is traditionally formulated as a surface normal estimation problem, with post-integration to reconstruct depth. This decoupled strategy ignores the intrinsic coupling between local shape (normal) and global geometry (depth), leading to inconsistent reconstructions and error accumulation. To address this, we reformulate UPS as a joint depth–normal estimation problem that enforces mutual geometric constraints. Our framework leverages CLIP as a stable semantic cues to resolve illumination ambiguity by providing pseudo-lighting cues. A plane-sweeping-like module first recovers a coarse depth map, which is then jointly refined with camera intrinsics. The refined depth and intrinsics are used to derive geometry-consistent surface normals, enabling a consistency-driven supervision that supports semi-supervised learning with limited synthetic data and bridges photometric and geometric reconstruction. Extensive experiments demonstrate that our method outperforms state-of-the-art UPS techniques and is competitive with calibrated and monocular depth estimation methods. Ablation studies confirm that CLIP provides robust semantic cues that effectively stabilize the reconstruction process.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
photometric stereo, normal estimation, deep learning, depth estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xBE4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper reframes Uncalibrated Photometric Stereo (UPS). Instead of the typical workflow first predicting surface normals and then computing depth via integration, it treats the task as a joint depth‑normal estimation problem. To start with, the method leverages CLIP to pull directional semantic cues from multi‑illumination images and builds a pseudo‑lighting representation. A photometric stereo network then outputs NphotoN_{photo}. Next, a plane‑sweeping‑like module builds a depth‑dependent photometric cost volume to get a rough coarse depth. In the second stage, a Depth Refine Module jointly predicts improved depth together with camera intrinsics. We compute NdiffN_{diff} using the resulting depth and intrinsics. Finally, we carry out joint depth‑normal learning by enforcing consistency losses among NphotoN_{photo}, NdiffN_{diff} and ground‑truth normals. The paper also does supervised fine‑tuning for depth on a small‑scale Blender synthetic dataset.

Critical strengths

The joint depth‑normal formulation makes good sense and is worthwhile. Depth is not just treated as some post‑processing step after normal estimation. Instead, the method uses geometric consistency between depth and normals to improve normal learning. On the DiLiGenT dataset, the full model achieves an average MAE of 4.4°, which is competitive against many existing UPS approaches. Performance degrades fairly gently even with fewer input images like K=8 or K=16. Full results on DiLiGenT‑Pi also show decent performance across multiple non‑Lambertian material categories. The ablation experiments target the core modules reasonably well. The authors remove depth supervision, CLIP, normal supervision, and the plane‑sweep component one by one. We can see each of these components contributes noticeably to both normal and depth results, rather than only reporting end‑to‑end performance numbers. The prompt design analysis further looks at directional accuracy and cross‑object statistical consistency, which is more persuasive than just comparing a handful of different prompts. For synthetic fine‑tuning and evaluation, basic data separation is done. The fine‑tune set (Ft) and the N&D evaluation set use different 3D object models. What’s more, Ft is rendered with EEVEE and N&D uses Cycles. This setup is better than training and testing on exactly the same object assets.

Critical weaknesses
  1. The geometric relationship for depth‑to‑normal shown in Equation (7) needs major clarification.The paper writes Ndiff=K−1[−∂xD,−∂yD,1]⊤∥K−1[−∂xD,−∂yD,1]⊤∥.N_{\mathrm{diff}} = \frac{ K^{-1}[-\partial_xD,-\partial_yD,1]^\top }{ \|K^{-1}[-\partial_xD,-\partial_yD,1]^\top\| }. It presents this formula as depth‑derived normals combined with the perspective camera model and predicted intrinsics. For a standard perspective depth map where P(u,v)=D(u,v)K−1[u,v,1]⊤P(u,v)=D(u,v)K^{-1}[u,v,1]^\top, surface normals are normally calculated from ∂uP×∂vP\partial_u P\times \partial_v P. This calculation produces extra terms related to pixel coordinates, depth values themselves, and focal length. Equation 7 in the main text looks more like an orthographic or heavily simplified gradient approximation, instead of a complete perspective formulation. Meanwhile, Appendix A.1 explicitly points out that under perspective projection, ∂xz≠−nx/nz\partial_xz\neq -n_x/n_z and ∂yz≠−ny/nz\partial_yz\neq -n_y/n_z. So there is a mismatch between Equation 7 in the main text and what is written in the appendix about perspective versus orthographic projection. This inconsistency needs to be spelled out more clearly.
  2. The experimental evidence for depth estimation is far weaker than that for normal estimation. The N&D Syn benchmark only contains 5 objects. And the depth evaluation on real‑world DTU data in Table 2 only covers Scan6 and Scan18. Yet the paper draws fairly strong conclusions about generalization to real‑world scenes and non‑point‑light setups based only on this small subset of DTU. By comparison, normal estimation gets thorough validation on DiLiGenT and DiLiGenT‑Pi. Depth evaluation lacks that same level of thoroughness.
  3. The physical explanation for the plane‑sweeping‑like formulation is not appropriate. Equation 1 reads s^i(p,dm)=si−αP(p,dm)∥si−αP(p,dm)∥.\hat{s}_i(p,d_m) = \frac{s_i-\alpha P(p,d_m)} {\|s_i-\alpha P(p,d_m)\|}. But the main body calls sis_i a light direction. If sis_i truly is a direction vector, subtracting αP(p,dm)\alpha P(p,d_m) from it does not match the standard geometric setup for incident light from a point light source. This expression makes more sense if sis_i stands for some kind of pseudo‑light position. On top of that, α\alpha is a learnable parameter. This term might actually behave more like a learned depth‑aware shading proxy. In their synthetic dataset setup, lights sit 40 meters away while objects are smaller than one meter. The authors note that lighting direction changes across each object are at most around 1.43°, so depth hardly affects lighting direction under this far‑field lighting approximation. I suggest the authors explicitly state whether sis_i represents a direction or a pseudo‑light position. It would help to add simple ablation tests, for example setting α=0\alpha=0, fixing α to a constant value, or feeding in ground‑truth light information. These ablations can help figure out whether depth improvements come from real depth‑driven photometric signals or mostly from learned network priors.
Questions for authors
  1. Could you give the full derivation for Equation 7, mentioned in Weakness 1?
  2. Could you explain Weakness 2, or provide additional supporting experiments?
  3. Could you go into detail about the issues laid out in Weakness 3?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer xBE4: Perspective Geometry, Depth Evaluation, and Lighting Ablations

We thank the reviewer for the geometric analysis.

(1) Perspective geometry. Eq. (7) requires the perspective depth and pixel-coordinate terms. For optical-axis depth D>0 and zero-skew K, let r=K⁻¹[u,v,1]ᵀ and P=Dr. Then

P_v=D_v r+D K^{-1}[0,1,0]^\top,$$ $$P_u\times P_v= \frac{D}{f_xf_y} \begin{bmatrix} -f_xD_u\\ -f_yD_v\\ D+(u-c_x)D_u+(v-c_y)D_v \end{bmatrix}, \qquad N_{\mathrm{persp}}= \frac{P_u\times P_v}{\|P_u\times P_v\|}.$$ Normal orientation follows the ground-truth convention ([Haefner et al., ICCV 2019](https://cvai.cit.tum.de/_media/spezial/bib/haefner2019iccv.pdf)). Table X1 validates the geometric correction: analytic error decreases substantially, with a small end-to-end accuracy trade-off under matched retraining. **Table X1. Perspective geometry and matched retraining** | Operator | Plane MAE ↓ | Sphere / cap / wave MAE ↓ | DiLiGenT MAE ↓ | DiLiGenT-Pi MAE ↓ | Depth δ₁ ↑ | |:---|---:|---:|---:|---:|---:| | Submitted Eq. (7) | 1.8 | 2.5 / 2.9 / 3.4 | 4.43 ± 0.06 | 8.33 ± 0.06 | 0.880 | | Full perspective | 0.5 | 0.7 / 0.8 / 1.0 | 4.53 ± 0.21 | 8.50 ± 0.20 | 0.863 ± 0.012 | *MAE is in degrees. Analytic checks use known depth and intrinsics; model metrics are mean ± sample standard deviation over three matched training runs. Depth uses the new 25-object set. Per-run δ₁ values for the submitted model were reported to two decimal places; all three runs were 0.88 at that precision. Tables X2–X3 use the submitted formulation; the perspective-retrained variant is reported here separately.* **(2) Depth coverage.** Table X2 expands evaluation to 25 new held-out objects and 20 DTU scenes, with gains over every compared baseline across all reported depth metrics. On the same synthetic objects, normal MAE is 7.9° versus 16.1° for LINO-UniPS and 19.7° for SDM-UniPS, extending the evidence for both reconstruction outputs. **Table X2. Expanded synthetic and real-world depth evaluation** | Evaluation set | Method | Input images | δ₁ ↑ | AbsRel ↓ | RMSE ↓ | |:---|:---|:---|---:|---:|---:| | New N&D Syn: 25 held-out objects | Depth Anything V2 | 1, oracle best of 24 | 0.375 | 0.175 | 0.165 | | | Metric3D | 1, oracle best of 24 | 0.412 | 0.152 | 0.148 | | | Video Depth Anything | 24 | 0.390 | 0.172 | 0.161 | | | **Ours, submitted formulation** | 24 | **0.876** | **0.061** | **0.045** | | DTU: 20 scenes | Depth Anything V2 | 1, oracle best of 7 | 0.51 | 0.13 | 51 | | | Metric3D | 1, oracle best of 7 | 0.62 | 0.11 | 42 | | | Video Depth Anything | 7 | 0.65 | 0.10 | 35 | | | **Ours, submitted formulation** | 7 | **0.76** | **0.08** | **32** | *The synthetic objects exclude training and original test assets. RMSE uses normalized depth units for N&D Syn and millimeters after median scaling for DTU. DTU stacks contain seven lighting conditions at one viewpoint; no DTU-specific fine-tuning is used. Video Depth Anything is evaluated on the fixed-view illumination stacks as a transfer baseline.* **(3) Pseudo-light geometry.** For learned descriptor s_i, α>0, and consistent coordinate scaling, Eq. (1) admits $$\hat s_i(P)= \frac{L_i^{\mathrm{pseudo}}-P}{\|L_i^{\mathrm{pseudo}}-P\|}, \qquad L_i^{\mathrm{pseudo}}=\frac{s_i}{\alpha}.$$ This parameterizes a learned depth-conditioned cost without requiring calibrated source positions. The far-field rendering setup has limited physical direction variation. Table X3 tests depth dependence while retaining learned embeddings and refinement. Setting α=0 costs 27 percentage points in refined δ₁. A fixed nonzero α recovers most performance; learning its scale and providing ground-truth lighting yield further gains. These controls establish the contribution of depth conditioning within the learned reconstruction system. **Table X3. Depth-conditioned lighting controls** | Configuration | DiLiGenT MAE ↓ | DiLiGenT-Pi MAE ↓ | Coarse δ₁ ↑ | Refined δ₁ ↑ | |:---|---:|---:|---:|---:| | α = 0 | 4.63 | 8.67 | 0.410 | 0.610 | | α = 0.1, fixed | 4.57 | 8.60 | 0.670 | 0.867 | | **Learned α, submitted model** | 4.43 | 8.33 | 0.687 | 0.880 | | Ground-truth lighting† | 4.43 | 8.33 | 0.727 | 0.903 | *Means over three training seeds; MAE is in degrees and depth uses the new 25-object set. The α controls retain learned depth embeddings and refinement. † Ground-truth lighting is privileged input.* The revision will incorporate the perspective formulation into Eq. (7)/Appendix A.1, clarify the pseudo-source interpretation of Eq. (1), and provide the complete DTU scene list and per-scene results. Additional experiments will prioritize key checks with the final perspective-consistency variant and common-subset comparisons, followed by raw cost-versus-depth curves and the α=0/no-depth-embedding control.

Thank you very much for your patient replies. I still have one question: could you add the experimental results of Depth Anything V3 and MoGe3 in the section of Expanded synthetic and real-world depth evaluation?

AuthorsOpenReview ↗

Additional Single-Image Baselines and Expanded Evaluation

We thank the reviewer for suggesting stronger single-image baselines. We have added Depth Anything 3 and MoGe-3 to both the synthetic and real-world evaluations. These newer models improve over the previous single-image baselines and provide a more stringent comparison for our method.

(1) Depth evaluation. Depth Anything 3 and MoGe-3 are evaluated in the single-image setting at their native inference resolution. All methods use identical masks, normalization, and alignment: depth is normalized to [0,1][0,1] on N&D Syn, while DTU RMSE is reported in millimeters after the same median-scaling protocol. Each single-image prediction uses one frame; oracle best-of-N selection chooses among N independently evaluated candidates, as specified in Table R1. Our method jointly processes the corresponding illumination stack.

Table R1. Extended depth evaluation

DatasetMethodInputsδ₁ ↑AbsRel ↓RMSE ↓
N&D Syn (25 objects)Depth Anything V21, best of 240.3750.1750.165
Metric3D1, best of 240.4120.1520.148
Video Depth Anything240.3900.1720.161
Depth Anything 31, best of 240.4210.1540.145
MoGe-31, best of 240.4630.1420.132
Ours (submitted)240.8760.0610.045
DTU (first 20 scenes)Depth Anything V21, best of 70.510.1351
Metric3D1, best of 70.620.1142
Video Depth Anything70.650.1035
Depth Anything 31, best of 70.720.0934
MoGe-31, best of 70.700.0934
Ours (submitted)70.760.0832

The 25 synthetic objects are disjoint from training and the original test set, with five objects per category. DTU stacks contain seven lighting conditions at a fixed viewpoint. Ours denotes the submitted formulation throughout.

On N&D Syn, MoGe-3 is the strongest baseline, improving δ1\delta_1 from 0.412 to 0.463 over Metric3D, while ours reaches 0.876 with lower AbsRel and RMSE. The DTU margin is smaller: Depth Anything 3 and MoGe-3 reach 0.72/0.70 versus our 0.76, with 34 mm RMSE versus our 32 mm. The new baselines narrow the margin on DTU, while our method retains the best results across all reported depth metrics on both datasets.

(2) Category-wise depth results. The improvements are not uniform across object categories, so we report the complete breakdown below.

Table R2. Depth δ1\delta_1 on N&D Syn by category

CategoryMetric3DVideo Depth AnythingDepth Anything 3MoGe-3Ours
CHAIR0.420.360.410.430.87
SHIP0.490.250.330.360.83
FOOD0.360.340.420.410.91
BALL0.520.610.570.650.91
WEAPON0.270.390.370.450.86
Mean of displayed category scores0.4120.3900.4200.4600.876

The last row averages the displayed two-decimal category scores. Overall evaluation scores are reported in Table R1.

MoGe-3 is the strongest baseline on BALL (0.65), where the gap to ours is smallest. Our method achieves the highest δ1\delta_1 in all five categories, including margins of 0.44–0.49 over the strongest baseline on CHAIR and FOOD. On SHIP, both new models fall below Metric3D (0.33/0.36 versus 0.49).

(3) Normal evaluation. MoGe-3 also predicts surface normals, allowing a direct normal comparison. We use an oracle best-of-7 protocol that is deliberately favorable to the single-image baseline: for each object, we report the lowest ground-truth normal MAE among seven independently evaluated candidate frames. Depth Anything 3 does not directly predict normal maps and is therefore omitted.

Table R3. Normal MAE on N&D Syn by category (degrees, lower is better)

CategorySDM-UniPSLINO-UniPSMoGe-3 (best of 7)Ours
CHAIR11.59.615.27.2
SHIP34.229.527.413.7
FOOD16.517.114.15.6
BALL6.85.46.45.1
WEAPON29.318.732.58.0
Mean19.716.119.17.9

Even with oracle selection, MoGe-3 obtains 19.1° mean normal MAE versus our 7.9°. Its closest result is on BALL (6.4° versus 5.1°), whereas WEAPON rises to 32.5°. Stronger single-image depth predictions therefore do not necessarily yield comparably accurate normals, especially for complex non-Lambertian objects.

Across these expanded comparisons, our joint multi-illumination method retains the best average depth and normal results. We will include the complete comparisons and evaluation protocols in the revision.

Thank you again for your reply and the newly added experimental results.

Reviewer_PN3pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an uncalibrated photometric stereo method that can perform both normal and depth estimation from images captured under different lighting conditions. The key contribution is a technical pipeline with crafted modules, including a plane-sweeping module and coarse depth reconstruction, tested on synthetic data and real-world benchmarks to achieve state-of-the-art performance.

Critical strengths
  1. The paper shows strong performance in normal estimation; it achieves the SOTA performance on the DiLiGenT and DiLiGenT-Pi benchmarks, as well as the SOTA performance on depth estimation on the proposed N&D Syn datasets.
  2. Using CLIP's features as the pseudo light direction is an interesting idea.
Critical weaknesses
  1. Joint depth-normal optimization that could boost the accuracy of both terms is a common practice in 3D vision. The paper seems to propose only a learning-based pipeline for a domain-specific problem rather than reformulating it.
  2. The depth estimation is not fair. It is unclear whether the authors used all DTU depth data for evaluation (only two are listed in the Table), which is questionable. Meanwhile, the N & D Sync dataset is quite small. The evaluation is insufficient to support it as a contribution comparable to normal estimation.
  3. The authors do not sufficiently ablate the effects of important modules. For instance,
  • For the coarse depth reconstruction module, how does the accuracy of the coarse depth affect the performance of the depth refinement module? How good is the robustness of the refinement module?
  • For the pseudo light vectors, simply replacing the light with a zero vector will definitely cause side effects on the normal estimation if the network is not retrained. Meanwhile, they contribute to both normal estimation and coarse depth reconstruction. It is unclear to me what its effects are on each term separately.
Questions for authors
  1. Why use orthogonal projection for coarse depth estimation and perspective projection for fine depth estimation?
  2. The ablation study on the depth range M is also necessary, as it affects the accuracy of the coarse depth.
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer PN3p: Ablations and Expanded Depth Evaluation

We thank the reviewer for the constructive comments.

(1) Contribution. We make depth an explicitly learned, geometrically coupled output of single-view UPS under unknown lighting. CLIP-conditioned costs link illumination changes to coarse geometry, while depth–normal consistency enables depth learning from normal annotations before depth-supervised fine-tuning. Removing it raises DiLiGenT-Pi MAE from 8.3° to 15.7° (submission Table 3). “Reformulation” denotes these changed outputs and learning objective.

(2) Depth evaluation. We add 25 held-out objects and 20 DTU scenes. Ours outperforms all compared baselines on every depth metric without DTU-specific fine-tuning (Table P1). Monocular baselines use oracle image selection; ours jointly processes the illumination stack.

Table P1. Extended depth evaluation

SetMethodInputsδ₁ ↑AbsRel ↓RMSE ↓
New N&D Syn (25 objects)Depth Anything V21, best of 240.3750.1750.165
Metric3D1, best of 240.4120.1520.148
Video Depth Anything240.3900.1720.161
Ours (submitted)240.8760.0610.045
DTU (20 scenes)Depth Anything V21, best of 70.510.1351
Metric3D1, best of 70.620.1142
Video Depth Anything70.650.1035
Ours (submitted)70.760.0832

New synthetic objects exclude training/original test assets. RMSE uses normalized units for N&D Syn and mm after median scaling for DTU. DTU uses seven lights per viewpoint. Video Depth Anything is a transfer baseline on fixed-view stacks.

(3) Coarse-depth quality. With frozen weights and fixed upstream features, refinement reduces error for every tested corruption (Table P2); the sweeps also quantify accuracy loss with severity.

Table P2. Refinement under coarse-depth corruption

PerturbationCoarse AbsRel ↓Refined AbsRel ↓Refined δ₁ ↑
Clean0.1050.0610.88
Gaussian σ=1% ΔD0.1060.0620.88
Gaussian σ=3% ΔD0.1100.0650.86
Gaussian σ=5% ΔD0.1160.0690.84
Gaussian σ=10% ΔD0.1400.0840.75
Spatial k=30.1090.0680.85
Spatial k=70.1220.0740.83
Spatial k=110.1380.0850.78
Bilateral σColor=100.1060.0650.87
Bilateral σColor=500.1080.0680.86
Bilateral σColor=900.1150.0710.85
Local block bias0.1310.0790.80

Frozen submitted model on the new 25-object set. Coarse error is measured after corruption; ΔD=[0,1].

(4) CLIP injection sites. All four configurations were separately trained with three seeds. Both sites improve refined depth; normal-network conditioning has the stronger effect on normal accuracy (Table P3), avoiding test-time zero-vector replacement.

Table P3. Separately trained CLIP configurations

C_NC_DDiLiGenT MAE ↓DiLiGenT-Pi MAE ↓Coarse δ₁ ↑Refined δ₁ ↑
005.7714.370.6070.717
104.508.770.6830.797
015.6313.570.7030.817
114.438.330.6870.880

C_N/C_D: CLIP in the normal network/depth-cost construction. Means over three seeds; MAE in degrees; depth on the new set.

(5) Projection models. Orthographic lifting constructs coarse hypotheses before intrinsics are available; refinement predicts depth and focal parameters. Full perspective consistency instead uses the cross product of tangent vectors from back-projected depth. Matched retraining shows a small end-to-end trade-off (Table P4).

Table P4. Matched depth-to-normal retraining

OperatorDiLiGenT MAE ↓DiLiGenT-Pi MAE ↓Depth δ₁ ↑
Submitted Eq. (7)4.43 ± 0.068.33 ± 0.060.880
Full perspective4.53 ± 0.218.50 ± 0.200.863 ± 0.012

Mean ± sample SD over three runs; depth on the new set. Submitted per-run δ₁ is 0.88 at two-decimal precision. P1–P3/P5 use the submitted formulation.

(6) Depth hypotheses. M samples the fixed interval [0,1]. M=64 matches M=128 in refined accuracy, while M=128 costs 23.5% more latency and 14.7% more memory (Table P5).

Table P5. Depth-hypothesis count and efficiency

MCoarse δ₁ ↑Refined δ₁ ↑Time (s) ↓Peak memory (GB) ↓
160.6230.8072.4525.4
320.6530.8402.6326.4
640.6870.8802.9828.5
1280.7000.8803.6832.7

Depth means over three seeds on the new set. Profiling: 16 images, 512×512, fp16, batch 1.

We will add common-subset mean/oracle-best results for K∈{2,4,12,24}, update Eq. (7)/Appendix A.1, and provide the DTU scene list and per-scene scores.

Thank you for the additional experiments and clarifications. I think my concerns are addressed. I will raise my score accordingly.

Reviewer_Db1COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a framework for predicting both normal and depth in Uncalibrated Photometric Stereo (UPS). Instead of integrating normal to get depth, the proposed framework couples depth and normal via a two-stage structure. The first stage is the reconstruction stage, where it uses CLIP features to get pseudo illumination cues, estimates a photometric normal map and adopts a plane-sweep-like cost volume to predict a coarse depth map. The second stage is geometry-aware learning stage, where it refines the coarse depth map together with camera intrinsics and adds geometric consistency supervision. The method is evaluated on DiLiGenT and DiLiGenT-Pi benchmarks. It achieves state-of-the-art UPS performance. It is also evaluated on depth accuracy and outperforms some mono depth estimation methods.

Critical strengths
  1. The paper reformulates UPS as a joint depth-normal estimation task, directly resolves the issue of post-hoc normal integration to get depth. Deriving differential surface normals from refined depth allows semi-supervised depth learning from a dataset with only normal ground-truth.

  2. The method achieves state-of-the-art quantitative accuracy on both DiLiGenT and DiLiGenT-Pi. It also outperforms single-view depth estimation methods like DepthAnything on DTU.

Critical weaknesses

The pseudo-calibration stage relies on CLIP classifying global directional brightness distributions into discrete directional text prompts. It's also shown in the failure case analysis that when scenes feature complex relief geometry, self-occlusion or mixed materials, the correlation between CLIP probability distribution and true lighting drops, leading to drops in normal and depth estimation performance.

Questions for authors

The framework contains two stages, if the first stage fails, e.g the coarse depth map is noisy, how robust is the second stage? How stable of the framework in general for non-lambertian surfaces and high-frequency multi-source lighting?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Db1C: Robustness and Multi-Source Illumination

We thank the reviewer for the constructive assessment.

(1) CLIP cues and non-Lambertian surfaces. CLIP supplies the full directional-prompt probability vector as continuous conditioning. The submitted DiLiGenT-Pi results provide material-level evidence across metallic, specular, translucent, and rough materials (Appendix A.4). Table D1 now directly perturbs the CLIP probabilities, quantifying how degraded lighting cues affect both reconstruction outputs.

(2) Depth-refinement robustness. With frozen weights and unchanged upstream features, refinement lowers absolute depth error across all tested corruption families (Table D1). At 10% Gaussian noise, refined AbsRel is 40% lower than that of the corrupted coarse input. The severity sweep also identifies the increasing sensitivity to larger errors.

(3) Multi-source illumination. Without multi-source retraining, the model retains δ₁=0.80 with eight simultaneous sources per image. Table D1 reports the corresponding normal and depth degradation.

Table D1. Frozen-model robustness to depth and illumination perturbations

ConditionNormal MAE ↓Coarse AbsRel ↓Refined AbsRel ↓Refined δ₁ ↑
Clean reference7.90.1050.0610.88
Gaussian noise: σ = 1% ΔD—0.1060.0620.88
Gaussian noise: σ = 3% ΔD—0.1100.0650.86
Gaussian noise: σ = 5% ΔD—0.1160.0690.84
Gaussian noise: σ = 10% ΔD—0.1400.0840.75
Spatial filtering: k = 3—0.1090.0680.85
Spatial filtering: k = 7—0.1220.0740.83
Spatial filtering: k = 11—0.1380.0850.78
Bilateral filtering: σColor = 10—0.1060.0650.87
Bilateral filtering: σColor = 50—0.1080.0680.86
Bilateral filtering: σColor = 90—0.1150.0710.85
Local depth-block bias—0.1310.0790.80
Simultaneous sources: Q = 28.20.1090.0720.86
Simultaneous sources: Q = 48.80.1140.0730.84
Simultaneous sources: Q = 89.40.1290.0790.80
Q = 4, source distance R = 10 m9.20.1230.0760.82
CLIP probability perturbation8.50.1100.0720.83

All metrics use the new 25-object test set and frozen submitted model; normal MAE is in degrees. Coarse error is measured after corruption. ΔD denotes the depth span of [0,1]; Q counts simultaneous sources per image. The clean reference uses Q=1, and source distance is R=40 m unless specified. — indicates unchanged upstream normals for depth-only corruption. The 10 m condition varies source distance.

For the revision, we will prioritize same-input UPS comparisons under multi-source illumination, followed by material/region diagnostics and an independent source-angular-extent control at fixed distance.

Reviewer_Db1COpenReview ↗

Reply to rebuttal

Thank you for the comprehensive rebuttal and for running the stress-test benchmarks in Table D1. My concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.