- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses layered depth estimation, which aims to predict multiple depth layers for regions where a single depth value cannot adequately represent the surface, such as glass. The paper points out that existing methods assume all layers are always present, predict depth for every layer, and then compute metrics on these predictions. As a result, they do not predict depth only where it is actually needed, and they are evaluated under an overly favorable setting. To address this, the paper uses a Bernoulli distribution to jointly model layer presence and assignment, so that depth is predicted only where layers actually exist.
Critical strengths
-
The paper clearly identifies a problem with existing methods. Existing methods predict many depths and are designed to achieve high scores as long as one of them matches the ground truth. This is far from the actual goal of predicting depth only where it is needed, and I appreciate that the paper targets this issue.
-
The method uses a Bernoulli formulation with a log-likelihood objective to enforce both the number of layers and their assignment to some extent. The paper also provides a theoretical explanation of why this works.
-
To address this issue, the paper modifies the benchmark, which helps reveal the limitations of existing methods.
Critical weaknesses
- Clarity. The paper is very difficult to follow, and many parts must be inferred, which can easily lead to misunderstanding. First, many terms can only be understood after reading the SeeGroup paper. In particular, “component” is a very general term, which makes it even more confusing. SeeGroup uses the term “Laplace component” to represent each layer when describing its module. This paper, however, uses terms such as “component” and “count” from the beginning without clear definitions, which makes it hard to read. I suggest using “layer” instead of “component” and clearly defining “count” as the number of layers from the start. In addition, the explanation of SeeGroup’s limitations is too condensed. My understanding has two parts:
- When multiple predictions are made across other components, the count can still match, so incorrect predictions can be counted as correct.
- Even if omitted layers are mispredicted, the error can remain small under the current metrics. This is written in a way that is hard to follow. It would help to explain it using Fig. 1 as an example.
-
Undefined terms. The term “presence credit” appears abruptly. It seems to refer to the presence probability. Since the paper is based on a Bernoulli distribution, it could simply be called a probability, or at least be defined once before use. It also appears without definition in the method section. Similarly, the sentence “Our analysis distinguishes expected count correction from the allocation of presence credit among components. Matching expected count can therefore leave assignment unresolved” feels somewhat redundant. My understanding is that the paper analyzes how matching the expected count differs from resolving the assignment. However, it is unclear what “count-regularized configuration” and “assignment supervision” mean.
-
LD-real and LD-syn are also introduced abruptly in the introduction. They should be referenced or defined beforehand. Overall, the writing is so condensed that misunderstandings arise easily, and I found it difficult to confirm whether I understood the paper correctly. I recommend revising the writing throughout.
-
Computational cost. Eq. 3 appears to be a Bernoulli-based formulation that considers all possible cases. I am curious whether this leads to a long computation time.
-
Performance compared with SeeGroup. The proposed method performs worse than SeeGroup, and the reason should be clarified. The two methods appear to use similar architectures and are trained on similar datasets, differing mainly in the loss. The drop in overall geometric performance is therefore disappointing.
-
Suggestion for an additional metric. Separately, the authors could propose an additional metric to further emphasize the importance of their evaluation. Point map prediction is commonly evaluated with accuracy and completion. The current evaluation only measures the error of the predicted depths closest to the GT. A metric that measures how close each predicted depth is to the GT depth would provide stronger justification for modeling presence.
Questions for authors
I just curious it is hard to read only for me or not. So I need to see the other reviewer's comments and rebuttal, then I'll do final decision.
2 comments
Author Reply to Reviewer MJG6 (1/2)
4. Joint Evaluation(W5, W6)
We thank the reviewer for the suggestion to add a joint metric of count and depth, instead of scoring those two separately. That joint further shows the advantage of our method: a model can look accurate on depth alone and still recover the wrong set of layers.
The comparison is the dense synthetic split. Every occupied ray has a layer count and a metric depth. The scores are defined before the table. Accuracy and AbsRel are in percent. Count MAE is in layers. Means and sample standard deviations use four seeds, except the depth-accuracy and Joint entries for ordered assignment and count regularization, which use three.
Absolute relative error. , averaged with equal weight over the four layer ranks. Native AbsRel uses the metric depth, with no scale and no shift. It is reported only for models that predict absolute depth. Relative models have no native scale, so that cell is blank. Aligned AbsRel applies one scale and one shift to every model, including the metric rows, fit per image and per rank.
Depth accuracy. is the fraction of matched depths whose aligned AbsRel is below , with . The match uses the aligned depths. A depth that has no target does not enter this fraction.
Count. Count accuracy is the share of occupied rays whose predicted layer count equals the annotated count. Count MAE is the mean absolute difference of those two counts, in layers.
Joint. A ray passes only when its count is exact and every visible layer has aligned AbsRel below . Depth accuracy can be high on the matched layers while this joint stays low, because a wrong count fails the ray.
| Method | Scale | AbsRel native | AbsRel aligned | [email protected] | [email protected] | Count acc | Count MAE | [email protected] | [email protected] |
|---|---|---|---|---|---|---|---|---|---|
| SeeGroup | Relative | -- | 14.01 | 54.24 | 80.56 | 2.42 | 2.756 | 0.33 | 1.16 |
| LaRI scenes | Relative | -- | 26.54 | 40.61 | 67.44 | 2.43 | 2.789 | 0.12 | 0.45 |
| LaRI objects | Relative | -- | 32.98 | 14.94 | 36.32 | 33.29 | 0.746 | 4.30 | 11.17 |
| World Tracing r69l | Relative | -- | 24.51 | 65.77 | 85.54 | 2.43 | 2.789 | 0.18 | 0.76 |
| World Tracing r75b | Metric | 51.51 | 33.31 | 31.05 | 59.86 | 2.43 | 2.789 | 0.06 | 0.39 |
| Depth stacking | Metric | ||||||||
| ExactMB | Metric | ||||||||
| MAP | Metric | ||||||||
| Ordered assignment | Metric | ||||||||
| Count regularization | Metric |
Takeaways:
- Why SeeGroup looked stronger (W5). The SeeGroup row is a released relative-depth model. Its aligned AbsRel (14.01) and [email protected] (80.56) look competitive because both ignore extra layers that miss every target. Once the count is required, SeeGroup is 2.42% on count accuracy and 1.16% on [email protected]. Depth stacking shows the same gap on our own network: [email protected], and [email protected].
- What modeling presence changes. Moving from depth stacking to ExactMB raises count accuracy from to and [email protected] from to , while aligned AbsRel stays near SeeGroup ( vs. 14.01). MAP ( [email protected]) lands next to ExactMB, so the main gain comes from learning absence rather than from summing versus picking one match.
- What pairing and count supervision add. Fixing canonical pairing (ordered assignment) lowers aligned AbsRel to and raises [email protected] to . Replacing the slotwise gates with one categorical count loss (count regularization) gives the best result on every column: aligned AbsRel (below SeeGroup's 14.01), native AbsRel , count accuracy , and [email protected] .
- Why Joint is needed alongside depth accuracy (W6). measures how close matched predictions are to the ground truth. adds exact count and requires every visible layer to lie below , which separates models that emit all slots from models that predict depth only where layers exist.
Please let us know if a point is still unclear.
Thank you!
Author Reply to Reviewer MJG6 (1/2)
Dear Reviewer MJG6,
Thank you for the careful reading. We first clarify some of the terms that may have been unclear, and then report one dense LayeredDepth-Syn comparison in which depth, count, and joint recovery are separate columns.
1. Layer, count, and the two metric failures (W1, W3)
A layer is one visible surface along a camera ray. The count is the number of those layers. The network has K=4 prediction slots. Each slot has a metric depth, a presence probability qj=σ(ej) read from the image, and a positive scale. A slot may be absent, so the number of present layers need not equal K. SeeGroup calls the same prediction a Laplace component. Here we say "layer."
Three test sets are used. LD-Syn is the 500-image synthetic validation split of LayeredDepth. Each occupied ray has a complete count of one to four layers, with metric depth on channels L1, L3, L5, and L7. LD-Real is the 300-image real validation split. MD-3K is 3,161 real point pairs labeled by material. The numbers below are LD-Syn, where the count and the depth are both annotated.
Figure 1 states the task. The through-glass ray has count two. Two failures follow from that ray. First, two emitted depths can match the count and still miss the glass, if they are assigned to the wrong surfaces. Second, emitting only the nearer depth makes absolute relative error look small, because that error averages only where a prediction and a target both exist. The omitted layer is never charged.
2. Presence probability, count regularization, and ordered assignment (W2)
qj is the presence probability predicted from the image. The submitted PDF uses "presence credit" for a different quantity. After the ground-truth layers are seen, ρj in Equation (4) is the posterior probability that slot j is matched to one of them.
Equation (7) separates the two events you asked about. The summed presence gradient equals E∣X∣−∣T∣. A shift of the presence logits can match the expected count and still leave the pairing free. The five training objectives differ on two axes: how slots are paired with target layers, and how the count is optimized. They are trained separately.
Ordered assignment and count regularization both fix the canonical near-to-far pairing. They differ in the count term. Ordered assignment uses an independent binary gate on each slot. Count regularization uses one (K+1)-way categorical cross-entropy for the count. ExactMB sums every injection. MAP keeps only the best injection. Both of those use the same per-slot binary presence. Depth stacking uses the canonical order and leaves every slot active.
3. Cost of Equation (3) (W4)
Equation (3) is a training loss. At K=4, a ray whose count is 4 sums 4!=24 matches. At inference we follow the same evaluation protocol as LayeredDepth.