- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper advocates that the current depth estimation metrics can be further improved. It starts by analyzing the sensitivity of the current metrics to hand-crafted GT depth perturbations and proposes a new Normal-based metric and a compositional metric that is aligned with human preference. Despite the proposal of the metric and the corresponding new evaluations, the paper provides no conclusions about the metric.
Critical strengths
None
Critical weaknesses
First, the claim that human judgement is relatively more reliable is not convincing. Humans are better at perceiving depth edges and relative depth, yet are not sensitive to the scale of the depth or the absolute metric error. Introducing human judgement into the metric computations inevitably involves another inductive bias which does not guarantee a better evaluation. Moreover, human judgements vary from person to person.
Second, the so-called RelNormal is essentially a normal metric, which can provide hints of the surface estimation quality but can not measure the absolute depth at all.
Third, the paper's technical standing is extremely vague.
- While it advocates using human-based judgment, it says, "Note that we do not claim that human judgment is the gold standard or an ideal proxy for all or some downstream applications; it is simply a potentially useful reference."
- When it shows the performance under the newly proposed metric, it says, "It is important to note that this example does not show that SAWA-H is a better metric than AbsRel because that depends on the application; it simply illustrates their different behaviors."
These claims confuse me because if the aim is not to design a better metric, there is no reason for the paper to exist.
Those disclaimers may help AIs to improve scores but are a strong sign of rejection for a human reviewer.
Questions for authors
No