- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper reframes Uncalibrated Photometric Stereo (UPS). Instead of the typical workflow first predicting surface normals and then computing depth via integration, it treats the task as a joint depth‑normal estimation problem. To start with, the method leverages CLIP to pull directional semantic cues from multi‑illumination images and builds a pseudo‑lighting representation. A photometric stereo network then outputs . Next, a plane‑sweeping‑like module builds a depth‑dependent photometric cost volume to get a rough coarse depth. In the second stage, a Depth Refine Module jointly predicts improved depth together with camera intrinsics. We compute using the resulting depth and intrinsics. Finally, we carry out joint depth‑normal learning by enforcing consistency losses among , and ground‑truth normals. The paper also does supervised fine‑tuning for depth on a small‑scale Blender synthetic dataset.
Critical strengths
The joint depth‑normal formulation makes good sense and is worthwhile. Depth is not just treated as some post‑processing step after normal estimation. Instead, the method uses geometric consistency between depth and normals to improve normal learning. On the DiLiGenT dataset, the full model achieves an average MAE of 4.4°, which is competitive against many existing UPS approaches. Performance degrades fairly gently even with fewer input images like K=8 or K=16. Full results on DiLiGenT‑Pi also show decent performance across multiple non‑Lambertian material categories. The ablation experiments target the core modules reasonably well. The authors remove depth supervision, CLIP, normal supervision, and the plane‑sweep component one by one. We can see each of these components contributes noticeably to both normal and depth results, rather than only reporting end‑to‑end performance numbers. The prompt design analysis further looks at directional accuracy and cross‑object statistical consistency, which is more persuasive than just comparing a handful of different prompts. For synthetic fine‑tuning and evaluation, basic data separation is done. The fine‑tune set (Ft) and the N&D evaluation set use different 3D object models. What’s more, Ft is rendered with EEVEE and N&D uses Cycles. This setup is better than training and testing on exactly the same object assets.
Critical weaknesses
- The geometric relationship for depth‑to‑normal shown in Equation (7) needs major clarification.The paper writes It presents this formula as depth‑derived normals combined with the perspective camera model and predicted intrinsics. For a standard perspective depth map where , surface normals are normally calculated from . This calculation produces extra terms related to pixel coordinates, depth values themselves, and focal length. Equation 7 in the main text looks more like an orthographic or heavily simplified gradient approximation, instead of a complete perspective formulation. Meanwhile, Appendix A.1 explicitly points out that under perspective projection, and . So there is a mismatch between Equation 7 in the main text and what is written in the appendix about perspective versus orthographic projection. This inconsistency needs to be spelled out more clearly.
- The experimental evidence for depth estimation is far weaker than that for normal estimation. The N&D Syn benchmark only contains 5 objects. And the depth evaluation on real‑world DTU data in Table 2 only covers Scan6 and Scan18. Yet the paper draws fairly strong conclusions about generalization to real‑world scenes and non‑point‑light setups based only on this small subset of DTU. By comparison, normal estimation gets thorough validation on DiLiGenT and DiLiGenT‑Pi. Depth evaluation lacks that same level of thoroughness.
- The physical explanation for the plane‑sweeping‑like formulation is not appropriate. Equation 1 reads But the main body calls a light direction. If truly is a direction vector, subtracting from it does not match the standard geometric setup for incident light from a point light source. This expression makes more sense if stands for some kind of pseudo‑light position. On top of that, is a learnable parameter. This term might actually behave more like a learned depth‑aware shading proxy. In their synthetic dataset setup, lights sit 40 meters away while objects are smaller than one meter. The authors note that lighting direction changes across each object are at most around 1.43°, so depth hardly affects lighting direction under this far‑field lighting approximation. I suggest the authors explicitly state whether represents a direction or a pseudo‑light position. It would help to add simple ablation tests, for example setting , fixing α to a constant value, or feeding in ground‑truth light information. These ablations can help figure out whether depth improvements come from real depth‑driven photometric signals or mostly from learned network priors.
Questions for authors
- Could you give the full derivation for Equation 7, mentioned in Weakness 1?
- Could you explain Weakness 2, or provide additional supporting experiments?
- Could you go into detail about the issues laid out in Weakness 3?
4 comments
Thank you very much for your patient replies. I still have one question: could you add the experimental results of Depth Anything V3 and MoGe3 in the section of Expanded synthetic and real-world depth evaluation?
Additional Single-Image Baselines and Expanded Evaluation
We thank the reviewer for suggesting stronger single-image baselines. We have added Depth Anything 3 and MoGe-3 to both the synthetic and real-world evaluations. These newer models improve over the previous single-image baselines and provide a more stringent comparison for our method.
(1) Depth evaluation. Depth Anything 3 and MoGe-3 are evaluated in the single-image setting at their native inference resolution. All methods use identical masks, normalization, and alignment: depth is normalized to on N&D Syn, while DTU RMSE is reported in millimeters after the same median-scaling protocol. Each single-image prediction uses one frame; oracle best-of-N selection chooses among N independently evaluated candidates, as specified in Table R1. Our method jointly processes the corresponding illumination stack.
Table R1. Extended depth evaluation
| Dataset | Method | Inputs | δ₁ ↑ | AbsRel ↓ | RMSE ↓ |
|---|---|---|---|---|---|
| N&D Syn (25 objects) | Depth Anything V2 | 1, best of 24 | 0.375 | 0.175 | 0.165 |
| Metric3D | 1, best of 24 | 0.412 | 0.152 | 0.148 | |
| Video Depth Anything | 24 | 0.390 | 0.172 | 0.161 | |
| Depth Anything 3 | 1, best of 24 | 0.421 | 0.154 | 0.145 | |
| MoGe-3 | 1, best of 24 | 0.463 | 0.142 | 0.132 | |
| Ours (submitted) | 24 | 0.876 | 0.061 | 0.045 | |
| DTU (first 20 scenes) | Depth Anything V2 | 1, best of 7 | 0.51 | 0.13 | 51 |
| Metric3D | 1, best of 7 | 0.62 | 0.11 | 42 | |
| Video Depth Anything | 7 | 0.65 | 0.10 | 35 | |
| Depth Anything 3 | 1, best of 7 | 0.72 | 0.09 | 34 | |
| MoGe-3 | 1, best of 7 | 0.70 | 0.09 | 34 | |
| Ours (submitted) | 7 | 0.76 | 0.08 | 32 |
The 25 synthetic objects are disjoint from training and the original test set, with five objects per category. DTU stacks contain seven lighting conditions at a fixed viewpoint. Ours denotes the submitted formulation throughout.
On N&D Syn, MoGe-3 is the strongest baseline, improving from 0.412 to 0.463 over Metric3D, while ours reaches 0.876 with lower AbsRel and RMSE. The DTU margin is smaller: Depth Anything 3 and MoGe-3 reach 0.72/0.70 versus our 0.76, with 34 mm RMSE versus our 32 mm. The new baselines narrow the margin on DTU, while our method retains the best results across all reported depth metrics on both datasets.
(2) Category-wise depth results. The improvements are not uniform across object categories, so we report the complete breakdown below.
Table R2. Depth on N&D Syn by category
| Category | Metric3D | Video Depth Anything | Depth Anything 3 | MoGe-3 | Ours |
|---|---|---|---|---|---|
| CHAIR | 0.42 | 0.36 | 0.41 | 0.43 | 0.87 |
| SHIP | 0.49 | 0.25 | 0.33 | 0.36 | 0.83 |
| FOOD | 0.36 | 0.34 | 0.42 | 0.41 | 0.91 |
| BALL | 0.52 | 0.61 | 0.57 | 0.65 | 0.91 |
| WEAPON | 0.27 | 0.39 | 0.37 | 0.45 | 0.86 |
| Mean of displayed category scores | 0.412 | 0.390 | 0.420 | 0.460 | 0.876 |
The last row averages the displayed two-decimal category scores. Overall evaluation scores are reported in Table R1.
MoGe-3 is the strongest baseline on BALL (0.65), where the gap to ours is smallest. Our method achieves the highest in all five categories, including margins of 0.44–0.49 over the strongest baseline on CHAIR and FOOD. On SHIP, both new models fall below Metric3D (0.33/0.36 versus 0.49).
(3) Normal evaluation. MoGe-3 also predicts surface normals, allowing a direct normal comparison. We use an oracle best-of-7 protocol that is deliberately favorable to the single-image baseline: for each object, we report the lowest ground-truth normal MAE among seven independently evaluated candidate frames. Depth Anything 3 does not directly predict normal maps and is therefore omitted.
Table R3. Normal MAE on N&D Syn by category (degrees, lower is better)
| Category | SDM-UniPS | LINO-UniPS | MoGe-3 (best of 7) | Ours |
|---|---|---|---|---|
| CHAIR | 11.5 | 9.6 | 15.2 | 7.2 |
| SHIP | 34.2 | 29.5 | 27.4 | 13.7 |
| FOOD | 16.5 | 17.1 | 14.1 | 5.6 |
| BALL | 6.8 | 5.4 | 6.4 | 5.1 |
| WEAPON | 29.3 | 18.7 | 32.5 | 8.0 |
| Mean | 19.7 | 16.1 | 19.1 | 7.9 |
Even with oracle selection, MoGe-3 obtains 19.1° mean normal MAE versus our 7.9°. Its closest result is on BALL (6.4° versus 5.1°), whereas WEAPON rises to 32.5°. Stronger single-image depth predictions therefore do not necessarily yield comparably accurate normals, especially for complex non-Lambertian objects.
Across these expanded comparisons, our joint multi-illumination method retains the best average depth and normal results. We will include the complete comparisons and evaluation protocols in the revision.
Thank you again for your reply and the newly added experimental results.
Author Response to Reviewer xBE4: Perspective Geometry, Depth Evaluation, and Lighting Ablations
We thank the reviewer for the geometric analysis.
(1) Perspective geometry. Eq. (7) requires the perspective depth and pixel-coordinate terms. For optical-axis depth D>0 and zero-skew K, let r=K⁻¹[u,v,1]ᵀ and P=Dr. Then
P_v=D_v r+D K^{-1}[0,1,0]^\top,$$ $$P_u\times P_v= \frac{D}{f_xf_y} \begin{bmatrix} -f_xD_u\\ -f_yD_v\\ D+(u-c_x)D_u+(v-c_y)D_v \end{bmatrix}, \qquad N_{\mathrm{persp}}= \frac{P_u\times P_v}{\|P_u\times P_v\|}.$$ Normal orientation follows the ground-truth convention ([Haefner et al., ICCV 2019](https://cvai.cit.tum.de/_media/spezial/bib/haefner2019iccv.pdf)). Table X1 validates the geometric correction: analytic error decreases substantially, with a small end-to-end accuracy trade-off under matched retraining. **Table X1. Perspective geometry and matched retraining** | Operator | Plane MAE ↓ | Sphere / cap / wave MAE ↓ | DiLiGenT MAE ↓ | DiLiGenT-Pi MAE ↓ | Depth δ₁ ↑ | |:---|---:|---:|---:|---:|---:| | Submitted Eq. (7) | 1.8 | 2.5 / 2.9 / 3.4 | 4.43 ± 0.06 | 8.33 ± 0.06 | 0.880 | | Full perspective | 0.5 | 0.7 / 0.8 / 1.0 | 4.53 ± 0.21 | 8.50 ± 0.20 | 0.863 ± 0.012 | *MAE is in degrees. Analytic checks use known depth and intrinsics; model metrics are mean ± sample standard deviation over three matched training runs. Depth uses the new 25-object set. Per-run δ₁ values for the submitted model were reported to two decimal places; all three runs were 0.88 at that precision. Tables X2–X3 use the submitted formulation; the perspective-retrained variant is reported here separately.* **(2) Depth coverage.** Table X2 expands evaluation to 25 new held-out objects and 20 DTU scenes, with gains over every compared baseline across all reported depth metrics. On the same synthetic objects, normal MAE is 7.9° versus 16.1° for LINO-UniPS and 19.7° for SDM-UniPS, extending the evidence for both reconstruction outputs. **Table X2. Expanded synthetic and real-world depth evaluation** | Evaluation set | Method | Input images | δ₁ ↑ | AbsRel ↓ | RMSE ↓ | |:---|:---|:---|---:|---:|---:| | New N&D Syn: 25 held-out objects | Depth Anything V2 | 1, oracle best of 24 | 0.375 | 0.175 | 0.165 | | | Metric3D | 1, oracle best of 24 | 0.412 | 0.152 | 0.148 | | | Video Depth Anything | 24 | 0.390 | 0.172 | 0.161 | | | **Ours, submitted formulation** | 24 | **0.876** | **0.061** | **0.045** | | DTU: 20 scenes | Depth Anything V2 | 1, oracle best of 7 | 0.51 | 0.13 | 51 | | | Metric3D | 1, oracle best of 7 | 0.62 | 0.11 | 42 | | | Video Depth Anything | 7 | 0.65 | 0.10 | 35 | | | **Ours, submitted formulation** | 7 | **0.76** | **0.08** | **32** | *The synthetic objects exclude training and original test assets. RMSE uses normalized depth units for N&D Syn and millimeters after median scaling for DTU. DTU stacks contain seven lighting conditions at one viewpoint; no DTU-specific fine-tuning is used. Video Depth Anything is evaluated on the fixed-view illumination stacks as a transfer baseline.* **(3) Pseudo-light geometry.** For learned descriptor s_i, α>0, and consistent coordinate scaling, Eq. (1) admits $$\hat s_i(P)= \frac{L_i^{\mathrm{pseudo}}-P}{\|L_i^{\mathrm{pseudo}}-P\|}, \qquad L_i^{\mathrm{pseudo}}=\frac{s_i}{\alpha}.$$ This parameterizes a learned depth-conditioned cost without requiring calibrated source positions. The far-field rendering setup has limited physical direction variation. Table X3 tests depth dependence while retaining learned embeddings and refinement. Setting α=0 costs 27 percentage points in refined δ₁. A fixed nonzero α recovers most performance; learning its scale and providing ground-truth lighting yield further gains. These controls establish the contribution of depth conditioning within the learned reconstruction system. **Table X3. Depth-conditioned lighting controls** | Configuration | DiLiGenT MAE ↓ | DiLiGenT-Pi MAE ↓ | Coarse δ₁ ↑ | Refined δ₁ ↑ | |:---|---:|---:|---:|---:| | α = 0 | 4.63 | 8.67 | 0.410 | 0.610 | | α = 0.1, fixed | 4.57 | 8.60 | 0.670 | 0.867 | | **Learned α, submitted model** | 4.43 | 8.33 | 0.687 | 0.880 | | Ground-truth lighting† | 4.43 | 8.33 | 0.727 | 0.903 | *Means over three training seeds; MAE is in degrees and depth uses the new 25-object set. The α controls retain learned depth embeddings and refinement. † Ground-truth lighting is privileged input.* The revision will incorporate the perspective formulation into Eq. (7)/Appendix A.1, clarify the pseudo-source interpretation of Eq. (1), and provide the complete DTU scene list and per-scene results. Additional experiments will prioritize key checks with the final perspective-consistency variant and common-subset comparisons, followed by raw cost-versus-depth curves and the α=0/no-depth-embedding control.