acceptodds
Withdrawn from ICLR 2027

Toward A Better Understanding of Monocular Depth Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Monocular depth estimation is an important task with rapid progress, but how to evaluate it is not fully resolved, as evidenced by a lack of standardization in existing literature and a large selection of evaluation metrics whose trade-offs and behaviors are not fully understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making smooth surfaces bumpy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. All code, data, and tools will be open-sourced.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
monocular depth estimation, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uWfFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper advocates that the current depth estimation metrics can be further improved. It starts by analyzing the sensitivity of the current metrics to hand-crafted GT depth perturbations and proposes a new Normal-based metric and a compositional metric that is aligned with human preference. Despite the proposal of the metric and the corresponding new evaluations, the paper provides no conclusions about the metric.

Critical strengths

None

Critical weaknesses

First, the claim that human judgement is relatively more reliable is not convincing. Humans are better at perceiving depth edges and relative depth, yet are not sensitive to the scale of the depth or the absolute metric error. Introducing human judgement into the metric computations inevitably involves another inductive bias which does not guarantee a better evaluation. Moreover, human judgements vary from person to person.

Second, the so-called RelNormal is essentially a normal metric, which can provide hints of the surface estimation quality but can not measure the absolute depth at all.

Third, the paper's technical standing is extremely vague.

  1. While it advocates using human-based judgment, it says, "Note that we do not claim that human judgment is the gold standard or an ideal proxy for all or some downstream applications; it is simply a potentially useful reference."
  2. When it shows the performance under the newly proposed metric, it says, "It is important to note that this example does not show that SAWA-H is a better metric than AbsRel because that depends on the application; it simply illustrates their different behaviors."

These claims confuse me because if the aim is not to design a better metric, there is no reason for the paper to exist.

Those disclaimers may help AIs to improve scores but are a strong sign of rejection for a human reviewer.

Questions for authors

No

Reviewer_ZdcjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors argue that current monocular depth estimation evaluation uses many different metrics and the trade-offs of these evaluation metrics are not well understood. They propose the use of a sensitivity analysis. They generate perturbations of the ground-truth depth along several axes and measure how dtronly different metrics respond. They formalize the process under the use of an exchange rate. The experiments are conducted using humans in the loop

Critical strengths

The work presented by the authors is well motivated, and the point that MDE evaluations have accumulated a mixture of metrics results in difficult-to-interpret evaluations. The perturbation framework is well motivated and a useful contribution. The proposed evaluation framework that attempts to compare to human judgment is well motivated and tackles gaps in existing evaluation frameworks.

Critical weaknesses

There are two fundamental weaknesses I see in the proposed work.

  1. The proposed metrics that attempt to align with human judgment can be biased based on the participants. The work used a small number of evaluators and a small number of depth estimations to evaluate the metrics, which is not sufficient to make generalizable conclusions.
  2. The authors do not differentiate between metric depth estimation and relative depth estimation. In the case of metric depth estimation, there is no reasoning why the ground truth depth is not sufficient or why human judgment is required. In the case of relative depth estimation, the proposed metrics are mostly trying to capture which type of 3d deformation is less perceivable by humans. This metric is valid for specific types of applications but the authors have to explicilty state their assumptions and their specific use case.
Questions for authors

Please provide legends for the color coding in the tables, as well as explanations of what is assumed to be better or worse. The different red and green intensities are really hard to parse and quickly draw conclusions about the comperative quality.

Reviewer_41ZwOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what geometric errors monocular depth evaluation metrics capture and how their responses compare with human judgments. It proposes an exchange-rate framework based on derivatives with respect to perturbation intensity, studying six perturbation families under eight conditions. Experiments use 95 synthetic scenes, 5,320 perturbed depth maps, and judgments on 315 depth maps collected across five retained expert annotators. The analysis identifies a substantial mismatch between commonly used metrics and the measured human sensitivity to curvature, and illustrates geometric distortions that affine-aligned evaluation discounts. The authors introduce textureless relighting and projected-contour visualizations, a relative-normal metric, RelNormal, and Sensitivity Aligned Composition. Adding RelNormal improves the fitted sensitivity-profile similarity of SAWA-H from 0.88 to 0.97. Additional experiments report scores for ten depth models across nine benchmarks and verify the numerical accuracy of RelNormal sampling. These demonstrate broad applicability, while independent validation of human alignment remains limited.

Critical strengths
  1. The paper provides useful, interpretable diagnostics for understanding depth metrics. Controlled geometric perturbations reveal distinctions that aggregate benchmark scores obscure, including the coexistence of small pointwise errors and visibly incorrect surface geometry. Considering alignment choices alongside metric definitions is particularly valuable. This contribution can improve the interpretation of evaluation results even independently of the proposed composite metric.

  2. The analysis is accompanied by practical tools and a constructive response to the identified limitations. Textureless relighting and projected contours make geometric defects easier to inspect, while RelNormal supplies complementary surface-relation information. SAC gives an explicit objective for composing metrics toward a specified sensitivity profile. The implementation details and sampling-accuracy checks in Appendix E further support the practical usefulness of these components.

Critical weaknesses
  1. The connection between the formal sensitivity definition and its numerical estimator is insufficiently established.
    Equation (1) defines sensitivity through derivatives at zero, but the implementation estimates these derivatives by fitting a constrained quadratic over sampled perturbation intensities (§3, p.5). For threshold metrics such as δ1\delta^1, the error on a finite depth map can remain exactly zero throughout a neighborhood of an unperturbed prediction; a quadratic fitted over a wider interval can nevertheless yield a nonzero linear coefficient. Consequently, the estimator can measure finite-range response rather than the stated local derivative. The paper does not provide response curves, fit diagnostics, or sensitivity to the fitting interval sufficient to determine how this distinction affects Figure 4 and the resulting composition weights. This matters because the sensitivity profiles support the central quantitative conclusions.

  2. Improved alignment with independent human judgments remains unverified.
    The reported 0.97 similarity establishes successful matching of the estimated target sensitivity profile, but it is not an independent test of perceptual validity. The paper does not report held-out scene or annotator validation, or quantitative human assessments of actual model predictions with the composition weights fixed. Appendix B provides useful quality-control information, including shared quality-control examples and an average cross-annotator standard deviation of 0.206; however, most assignments are otherwise disjoint, and uncertainty in the sensitivity estimates and weights is not quantified. Figures 5, 20, and 21 show that the metric can evaluate models across datasets, but do not establish that its rankings better match independently collected human judgments.

Questions for authors
  1. Clarification and robustness analysis: How should Equation (1) be interpreted for threshold metrics with locally flat responses? Please provide the perturbation intensity grids and representative response curves, and assess whether the principal conclusions remain stable under different fitting intervals or local estimators.

  2. Assessment-changing validation: With SAWA-H weights fixed, does the improvement persist on held-out scenes or independently annotated model predictions? Even a scene-level holdout and leave-one-annotator-out analysis, with uncertainty estimates, would help distinguish stable alignment from fitting the particular annotation sample.

  3. Clarification: How were the labels for the 25 quality-control examples established, and how sensitive are the human sensitivity profiles to annotator filtering and the rule that a depth map is accepted only when every visualization is accepted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.