acceptodds
Withdrawn from ICLR 2027

Depth Any Seen: Which Surfaces and How Far?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When several surfaces are visible along a ray, recovering visible 3D structure from one image requires jointly estimating their presence and metric depth. Depth Any Seen represents these surfaces as image-conditioned multi-Bernoulli depth sets, whose components each contribute one depth or remain absent. Its auxiliary-free Exact Multi-Bernoulli objective (ExactMB) learns depth and presence by marginalizing one-to-one assignments to complete, distinct targets. Our analysis shows that matching expected count can leave component–surface assignment unresolved. We extend real and synthetic layered-depth benchmarks to evaluate depth accuracy, recovered support, and overprediction. Compared to depth stacking, ExactMB reduces overprediction by a relative 88.2% on LD-Real and 80.5% on MD-3K while retaining most ordinal accuracy, with comparable conditional metric-depth error on LD-Syn. Further ablation studies show that ordered assignment improves depth-accurate recall and precision over marginalization, whereas the count-regularized configuration achieves higher deeper-rank precision than ordered assignment at lower recall. Our code will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
layered depth estimation, 3D from a single image, 3D scene understanding, 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MJG6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses layered depth estimation, which aims to predict multiple depth layers for regions where a single depth value cannot adequately represent the surface, such as glass. The paper points out that existing methods assume all layers are always present, predict depth for every layer, and then compute metrics on these predictions. As a result, they do not predict depth only where it is actually needed, and they are evaluated under an overly favorable setting. To address this, the paper uses a Bernoulli distribution to jointly model layer presence and assignment, so that depth is predicted only where layers actually exist.

Critical strengths
  • The paper clearly identifies a problem with existing methods. Existing methods predict many depths and are designed to achieve high scores as long as one of them matches the ground truth. This is far from the actual goal of predicting depth only where it is needed, and I appreciate that the paper targets this issue.

  • The method uses a Bernoulli formulation with a log-likelihood objective to enforce both the number of layers and their assignment to some extent. The paper also provides a theoretical explanation of why this works.

  • To address this issue, the paper modifies the benchmark, which helps reveal the limitations of existing methods.

Critical weaknesses
  1. Clarity. The paper is very difficult to follow, and many parts must be inferred, which can easily lead to misunderstanding. First, many terms can only be understood after reading the SeeGroup paper. In particular, “component” is a very general term, which makes it even more confusing. SeeGroup uses the term “Laplace component” to represent each layer when describing its module. This paper, however, uses terms such as “component” and “count” from the beginning without clear definitions, which makes it hard to read. I suggest using “layer” instead of “component” and clearly defining “count” as the number of layers from the start. In addition, the explanation of SeeGroup’s limitations is too condensed. My understanding has two parts:
  • When multiple predictions are made across other components, the count can still match, so incorrect predictions can be counted as correct.
  • Even if omitted layers are mispredicted, the error can remain small under the current metrics. This is written in a way that is hard to follow. It would help to explain it using Fig. 1 as an example.
  1. Undefined terms. The term “presence credit” appears abruptly. It seems to refer to the presence probability. Since the paper is based on a Bernoulli distribution, it could simply be called a probability, or at least be defined once before use. It also appears without definition in the method section. Similarly, the sentence “Our analysis distinguishes expected count correction from the allocation of presence credit among components. Matching expected count can therefore leave assignment unresolved” feels somewhat redundant. My understanding is that the paper analyzes how matching the expected count differs from resolving the assignment. However, it is unclear what “count-regularized configuration” and “assignment supervision” mean.

  2. LD-real and LD-syn are also introduced abruptly in the introduction. They should be referenced or defined beforehand. Overall, the writing is so condensed that misunderstandings arise easily, and I found it difficult to confirm whether I understood the paper correctly. I recommend revising the writing throughout.

  3. Computational cost. Eq. 3 appears to be a Bernoulli-based formulation that considers all possible cases. I am curious whether this leads to a long computation time.

  4. Performance compared with SeeGroup. The proposed method performs worse than SeeGroup, and the reason should be clarified. The two methods appear to use similar architectures and are trained on similar datasets, differing mainly in the loss. The drop in overall geometric performance is therefore disappointing.

  5. Suggestion for an additional metric. Separately, the authors could propose an additional metric to further emphasize the importance of their evaluation. Point map prediction is commonly evaluated with accuracy and completion. The current evaluation only measures the error of the predicted depths closest to the GT. A metric that measures how close each predicted depth is to the GT depth would provide stronger justification for modeling presence.

Questions for authors

I just curious it is hard to read only for me or not. So I need to see the other reviewer's comments and rebuttal, then I'll do final decision.

2 comments
AuthorsOpenReview ↗

Author Reply to Reviewer MJG6 (1/2)

Dear Reviewer MJG6,

Thank you for the careful reading. We first clarify some of the terms that may have been unclear, and then report one dense LayeredDepth-Syn comparison in which depth, count, and joint recovery are separate columns.


1. Layer, count, and the two metric failures (W1, W3)

A layer is one visible surface along a camera ray. The count is the number of those layers. The network has K=4K=4 prediction slots. Each slot has a metric depth, a presence probability qj=σ(ej)q_j=\sigma(e_j) read from the image, and a positive scale. A slot may be absent, so the number of present layers need not equal KK. SeeGroup calls the same prediction a Laplace component. Here we say "layer."

Three test sets are used. LD-Syn is the 500-image synthetic validation split of LayeredDepth. Each occupied ray has a complete count of one to four layers, with metric depth on channels L1, L3, L5, and L7. LD-Real is the 300-image real validation split. MD-3K is 3,161 real point pairs labeled by material. The numbers below are LD-Syn, where the count and the depth are both annotated.

Figure 1 states the task. The through-glass ray has count two. Two failures follow from that ray. First, two emitted depths can match the count and still miss the glass, if they are assigned to the wrong surfaces. Second, emitting only the nearer depth makes absolute relative error look small, because that error averages only where a prediction and a target both exist. The omitted layer is never charged.


2. Presence probability, count regularization, and ordered assignment (W2)

qjq_j is the presence probability predicted from the image. The submitted PDF uses "presence credit" for a different quantity. After the ground-truth layers are seen, ρj\rho_j in Equation (4) is the posterior probability that slot jj is matched to one of them.

Equation (7) separates the two events you asked about. The summed presence gradient equals E∣X∣−∣T∣\mathbb{E}|\mathsf{X}|-|\mathcal{T}|. A shift of the presence logits can match the expected count and still leave the pairing free. The five training objectives differ on two axes: how slots are paired with target layers, and how the count is optimized. They are trained separately.

SetupSlot pairingCount optimization
Depth stackingcanonical ordernone (all KK slots active)
ExactMBall injections (soft)per-slot binary presence
MAPbest injection (hard)per-slot binary presence
Ordered assignmentcanonical orderper-slot binary presence
Count regularizationcanonical orderjoint categorical count CE

Ordered assignment and count regularization both fix the canonical near-to-far pairing. They differ in the count term. Ordered assignment uses an independent binary gate on each slot. Count regularization uses one (K+1)(K+1)-way categorical cross-entropy for the count. ExactMB sums every injection. MAP keeps only the best injection. Both of those use the same per-slot binary presence. Depth stacking uses the canonical order and leaves every slot active.


3. Cost of Equation (3) (W4)

Equation (3) is a training loss. At K=4K=4, a ray whose count is 4 sums 4!=244!=24 matches. At inference we follow the same evaluation protocol as LayeredDepth.

AuthorsOpenReview ↗

Author Reply to Reviewer MJG6 (1/2)

4. Joint Evaluation(W5, W6)

We thank the reviewer for the suggestion to add a joint metric of count and depth, instead of scoring those two separately. That joint further shows the advantage of our method: a model can look accurate on depth alone and still recover the wrong set of layers.

The comparison is the dense synthetic split. Every occupied ray has a layer count and a metric depth. The scores are defined before the table. Accuracy and AbsRel are in percent. Count MAE is in layers. Means and sample standard deviations use four seeds, except the depth-accuracy and Joint entries for ordered assignment and count regularization, which use three.

Absolute relative error. AbsRel=∣d−y∣/y\mathrm{AbsRel}=|d-y|/y, averaged with equal weight over the four layer ranks. Native AbsRel uses the metric depth, with no scale and no shift. It is reported only for models that predict absolute depth. Relative models have no native scale, so that cell is blank. Aligned AbsRel applies one scale and one shift to every model, including the metric rows, fit per image and per rank.

Depth accuracy. Acc@τ\mathrm{Acc}@\tau is the fraction of matched depths whose aligned AbsRel is below τ\tau, with τ∈{0.10,0.25}\tau\in\{0.10,0.25\}. The match uses the aligned depths. A depth that has no target does not enter this fraction.

Count. Count accuracy is the share of occupied rays whose predicted layer count equals the annotated count. Count MAE is the mean absolute difference of those two counts, in layers.

Joint. A ray passes Joint@τ\mathrm{Joint}@\tau only when its count is exact and every visible layer has aligned AbsRel below τ\tau. Depth accuracy can be high on the matched layers while this joint stays low, because a wrong count fails the ray.

MethodScaleAbsRel native ↓\downarrowAbsRel aligned ↓\downarrow[email protected] ↑\uparrow[email protected] ↑\uparrowCount acc ↑\uparrowCount MAE ↓\downarrow[email protected] ↑\uparrow[email protected] ↑\uparrow
SeeGroupRelative--14.0154.2480.562.422.7560.331.16
LaRI scenesRelative--26.5440.6167.442.432.7890.120.45
LaRI objectsRelative--32.9814.9436.3233.290.7464.3011.17
World Tracing r69lRelative--24.5165.7785.542.432.7890.180.76
World Tracing r75bMetric51.5133.3131.0559.862.432.7890.060.39
Depth stackingMetric18.29±0.1418.29\pm 0.1412.96±0.0912.96\pm 0.0974.84±0.2474.84\pm 0.2492.00±0.0692.00\pm 0.063.67±0.163.67\pm 0.162.244±0.0312.244\pm 0.0311.65±0.161.65\pm 0.162.75±0.172.75\pm 0.17
ExactMBMetric19.20±0.2019.20\pm 0.2015.23±0.3815.23\pm 0.3873.85±0.2873.85\pm 0.2891.16±0.1391.16\pm 0.1388.17±0.4788.17\pm 0.470.194±0.0070.194\pm 0.00768.48±0.3668.48\pm 0.3682.35±0.4082.35\pm 0.40
MAPMetric18.73±0.2918.73\pm 0.2915.08±0.0715.08\pm 0.0773.21±0.1073.21\pm 0.1090.78±0.0990.78\pm 0.0988.42±0.1888.42\pm 0.180.183±0.0020.183\pm 0.00268.04±0.1368.04\pm 0.1382.28±0.2282.28\pm 0.22
Ordered assignmentMetric17.61±0.1517.61\pm 0.1512.32±0.0812.32\pm 0.0876.64±0.0576.64\pm 0.0592.64±0.0592.64\pm 0.0589.68±0.1389.68\pm 0.130.154±0.0020.154\pm 0.00270.13±0.0370.13\pm 0.0383.89±0.0283.89\pm 0.02
Count regularizationMetric17.23±0.0717.23\pm 0.0711.50±0.1011.50\pm 0.1077.32±0.0777.32\pm 0.0792.99±0.0492.99\pm 0.0494.99±0.0394.99\pm 0.030.067±0.0010.067\pm 0.00173.42±0.0773.42\pm 0.0788.46±0.0388.46\pm 0.03

Takeaways:

  • Why SeeGroup looked stronger (W5). The SeeGroup row is a released relative-depth model. Its aligned AbsRel (14.01) and [email protected] (80.56) look competitive because both ignore extra layers that miss every target. Once the count is required, SeeGroup is 2.42% on count accuracy and 1.16% on [email protected]. Depth stacking shows the same gap on our own network: 74.84±0.2474.84\pm 0.24 [email protected], and 1.65±0.161.65\pm 0.16 [email protected].
  • What modeling presence changes. Moving from depth stacking to ExactMB raises count accuracy from 3.67±0.163.67\pm 0.16 to 88.17±0.4788.17\pm 0.47 and [email protected] from 2.75±0.172.75\pm 0.17 to 82.35±0.4082.35\pm 0.40, while aligned AbsRel stays near SeeGroup (15.23±0.3815.23\pm 0.38 vs. 14.01). MAP (82.28±0.2282.28\pm 0.22 [email protected]) lands next to ExactMB, so the main gain comes from learning absence rather than from summing versus picking one match.
  • What pairing and count supervision add. Fixing canonical pairing (ordered assignment) lowers aligned AbsRel to 12.32±0.0812.32\pm 0.08 and raises [email protected] to 70.13±0.0370.13\pm 0.03. Replacing the slotwise gates with one categorical count loss (count regularization) gives the best result on every column: aligned AbsRel 11.50±0.1011.50\pm 0.10 (below SeeGroup's 14.01), native AbsRel 17.23±0.0717.23\pm 0.07, count accuracy 94.99±0.0394.99\pm 0.03, and [email protected] 88.46±0.0388.46\pm 0.03.
  • Why Joint is needed alongside depth accuracy (W6). Acc@τ\mathrm{Acc}@\tau measures how close matched predictions are to the ground truth. Joint@τ\mathrm{Joint}@\tau adds exact count and requires every visible layer to lie below τ\tau, which separates models that emit all KK slots from models that predict depth only where layers exist.

Please let us know if a point is still unclear.

Thank you!

Reviewer_UxreOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work studies multi-layer depth estimation task for transparent objects. It proposes ExactMB, a multi-Bernoulli likelihood that marginalizes over one-to-one assignments between predicted components and target depths. Experiments show substantial reductions in overprediction compare to baselines, with some loss of recall and ordering accuracy.

Critical strengths

The task is well motivated, as unused prediction slots in a fixed stack of depth maps can produce spurious geometry. The proposed multi-Bernoulli formulation addresses this issue by explicitly modeling surface presence, allowing multiple visible surfaces to coexist along the same camera ray while leaving unused components absent. Its effectiveness in reducing overprediction is well supported by the experiments, demonstrating the practical value of jointly modeling depth and surface presence. A further strength is that the method predicts these depths in metric scale, which makes this method an very useful improvement for real-world applications.

Critical weaknesses

The experimental presentation is difficult to follow. The main quantitative table primarily reports controlled internal ablations, while results for existing methods appear separately without the proposed method included alongside them. Furthermore, the two tables use different alignment for evaluation, preventing direct comparison of their reported numbers.

Questions for authors

Could the authors provide a comparison with existing methods under a consistent evaluation protocol? One way to compare metric-depth methods with relative-depth methods is to apply the same alignment to both. Could the authors evaluate the proposed metric-depth method after alignment and compare directly to baselines as well?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Uxre (1/2)

Dear Reviewer Uxre,

Thank you for pointing out this presentation issue in our experimental comparisons. We agree that separating our controlled variants from released methods, and using different depth-alignment settings across Tables 1 and 2, made the results unnecessarily hard to compare. Below, we clarify the original protocol, introduce a unified Joint metric alongside shared-alignment depth evaluation, provide the complete side-by-side tables across all released baselines and our setups, and outline a much cleaner four-question experiment delivery.


1. Unified Dense Comparison on LayeredDepth-Syn Under Shared Alignment (Q1, W1)

With the single exception of LD-Syn AbsRel, all metrics in the submitted Tables 1 and 2 were already evaluated without scale/shift alignment and are directly comparable across rows. The discrepancy arose only on LD-Syn AbsRel: because most released baselines predict relative depth while our models predict metric depth, Table 2 reported aligned AbsRel whereas Table 1 reported unaligned metric AbsRel.

Following your suggestion, we evaluate every method (both relative and metric) under the exact same per-image, per-rank least-squares scale-and-shift alignment on the 500-image dense LD-Syn split. To make the comparison even clearer, we group all metrics into Depth, Count, and Joint (Depth + Count) in one unified table:

  • Native / Aligned AbsRel (↓\downarrow, %): AbsRel=∣d−y∣/y\mathrm{AbsRel}=|d-y|/y averaged equally across the four layer ranks, scored where both prediction and target exist (-- indicates no native metric scale).
  • Depth accuracy (↑\uparrow, %): Acc@τ\mathrm{Acc}@\tau is the percentage of matched layers with aligned AbsRel<τ\mathrm{AbsRel}<\tau (τ∈{0.10,0.25}\tau\in\{0.10,0.25\}).
  • Count (↑\uparrow %, ↓\downarrow layers): exact occupied-ray layer-count accuracy and count MAE.
  • Joint (↑\uparrow, %): Joint@τ\mathrm{Joint}@\tau requires both exact layer count and aligned AbsRel<τ\mathrm{AbsRel}<\tau on every visible layer along the ray.
MethodScaleAbsRel native ↓\downarrowAbsRel aligned ↓\downarrow[email protected] ↑\uparrow[email protected] ↑\uparrowCount acc ↑\uparrowCount MAE ↓\downarrow[email protected] ↑\uparrow[email protected] ↑\uparrow
SeeGroupRelative--14.0154.2480.562.422.7560.331.16
LaRI scenesRelative--26.5440.6167.442.432.7890.120.45
LaRI objectsRelative--32.9814.9436.3233.290.7464.3011.17
World Tracing r69lRelative--24.5165.7785.542.432.7890.180.76
World Tracing r75bMetric51.5133.3131.0559.862.432.7890.060.39
Depth stackingMetric18.29±0.1418.29\pm 0.1412.96±0.0912.96\pm 0.0974.84±0.2474.84\pm 0.2492.00±0.0692.00\pm 0.063.67±0.163.67\pm 0.162.244±0.0312.244\pm 0.0311.65±0.161.65\pm 0.162.75±0.172.75\pm 0.17
ExactMBMetric19.20±0.2019.20\pm 0.2015.23±0.3815.23\pm 0.3873.85±0.2873.85\pm 0.2891.16±0.1391.16\pm 0.1388.17±0.4788.17\pm 0.470.194±0.0070.194\pm 0.00768.48±0.3668.48\pm 0.3682.35±0.4082.35\pm 0.40
MAPMetric18.73±0.2918.73\pm 0.2915.08±0.0715.08\pm 0.0773.21±0.1073.21\pm 0.1090.78±0.0990.78\pm 0.0988.42±0.1888.42\pm 0.180.183±0.0020.183\pm 0.00268.04±0.1368.04\pm 0.1382.28±0.2282.28\pm 0.22
Ordered assignmentMetric17.61±0.1517.61\pm 0.1512.32±0.0812.32\pm 0.0876.64±0.0576.64\pm 0.0592.64±0.0592.64\pm 0.0589.68±0.1389.68\pm 0.130.154±0.0020.154\pm 0.00270.13±0.0370.13\pm 0.0383.89±0.0283.89\pm 0.02
Count regularizationMetric17.23±0.0717.23\pm 0.0711.50±0.1011.50\pm 0.1077.32±0.0777.32\pm 0.0792.99±0.0492.99\pm 0.0494.99±0.0394.99\pm 0.030.067±0.0010.067\pm 0.00173.42±0.0773.42\pm 0.0788.46±0.0388.46\pm 0.03

(Four-seed mean ±\pm SD, except three seeds for Ordered assignment and Count regularization on Acc/Joint.)

Key takeaways from the unified LD-Syn table:

  • Direct aligned depth comparison: Under identical alignment, auxiliary-free ExactMB (15.23±0.38%15.23\pm 0.38\% aligned AbsRel, 91.16±0.13%91.16\pm 0.13\% [email protected]) is close to SeeGroup (14.01%14.01\% AbsRel, 80.56%80.56\% [email protected]) and substantially stronger than LaRI and World Tracing, while Ordered assignment (12.32±0.08%12.32\pm 0.08\%) and Count regularization (11.50±0.10%11.50\pm 0.10\%) achieve lower aligned error than all released methods alongside accurate native metric scale (17.23%17.23\% to 19.20%19.20\% vs. 51.51%51.51\% for metric World Tracing r75b).
  • Why Joint is essential: Conditional depth metrics (AbsRel, Acc@$\tau$) ignore extra predicted layers that have no ground-truth target. Thus SeeGroup and Depth stacking look strong on [email protected] (80.56%80.56\% and 92.00%92.00\%) yet collapse on [email protected] (1.16%1.16\% and 2.75%2.75\%) because they emit all slots almost everywhere (Count acc 2.42%2.42\% and 3.67%3.67\%). Modeling presence lifts [email protected] to 82.35±0.40%82.35\pm 0.40\% (ExactMB) and 88.46±0.03%88.46\pm 0.03\% (Count regularization).

AuthorsOpenReview ↗

Author Response to Reviewer Uxre (2/2)

2. Unified Real-Scene Transfer Comparison via Joint Count + Order (Q1, W1)

Putting all methods into one table with a Joint metric clarifies real-scene transfer in the same way (no depth alignment is needed here since evaluation uses discrete counts and ordinal relations). Below is the complete zero-shot comparison on MD-3K (3,161 real material-annotated pairs, where transparent points have 2 layers and opaque points have 1 layer; Joint requires both exact pair count pattern and correct depth order):

MethodOrder ↑\uparrow (%)Order$\mid$Count ↑\uparrow (%)Count acc ↑\uparrow (%)Count MAE ↓\downarrowOpaque OverPred ↓\downarrow (%)Joint ↑\uparrow (%)
SeeGroup83.68--0.002.307100.000.00
LaRI scenes63.14--0.002.317100.000.00
LaRI objects30.3427.4338.060.41380.5210.44
World Tracing r69l67.13--0.004.317100.000.00
World Tracing r75b65.83--0.004.317100.000.00
Depth stacking88.55±0.6188.55\pm 0.61--0.03±0.040.03\pm 0.041.975±0.0141.975\pm 0.01498.44±0.1998.44\pm 0.190.00±0.000.00\pm 0.00
ExactMB81.34±0.5681.34\pm 0.5693.39±0.8993.39\pm 0.8946.36±3.9446.36\pm 3.940.419±0.0460.419\pm 0.04619.18±3.8119.18\pm 3.8143.30±3.7743.30\pm 3.77
MAP81.60±1.6081.60\pm 1.6093.08±1.2793.08\pm 1.2747.68±2.7147.68\pm 2.710.399±0.0250.399\pm 0.02520.89±1.3920.89\pm 1.3944.38±2.6044.38\pm 2.60
Ordered assignment82.00±1.8082.00\pm 1.8091.26±0.5991.26\pm 0.5952.44±3.0852.44\pm 3.080.336±0.0220.336\pm 0.02221.63±1.1821.63\pm 1.1847.86±2.9247.86\pm 2.92
Count regularization80.80±0.1780.80\pm 0.1791.56±0.4291.56\pm 0.4265.86±1.2265.86\pm 1.220.213±0.0060.213\pm 0.0068.57±0.548.57\pm 0.5460.30±1.0960.30\pm 1.09
  • Order alone masks dense overprediction: SeeGroup (83.68%83.68\% Order) and Depth stacking (88.55%88.55\% Order) hallucinate extra layers on nearly every opaque point (100%100\% and 98.44%98.44\% OverPred), yielding 0.00%0.00\% Joint. Likewise, on LD-Real, empty-quadruple overprediction drops from 100%100\% (SeeGroup, LaRI scenes, World Tracing r69l) to 11.62±1.33%11.62\pm 1.33\% (ExactMB) and 7.43±0.64%7.43\pm 0.64\% (Count regularization).
  • Substantial Joint gains on real scenes: ExactMB reaches 43.30±3.77%43.30\pm 3.77\% Joint (with 93.39±0.89%93.39\pm 0.89\% Order|Count) and Count regularization reaches 60.30±1.09%60.30\pm 1.09\% Joint, compared with 0.00%0.00\% to 10.44%10.44\% across all released baselines.

3. Clearer Experiment Organization Around Joint Evaluation (W1)

To remove the disconnect between external baselines and internal ablations, we will adopt this unified layout in the final manuscript, leading directly with the unified baseline-and-method tables above and organizing Section 5 around four focused questions:

  1. (Q1) Real-scene joint recovery: Does evaluating layer count together with depth order change how methods compare on real scenes? (Unified real-scene table above, showing why Order alone hides count failure.)
  2. (Q2) Dense depth and count under shared alignment: Does improved joint recovery preserve accurate depths when all relative and metric models share the same alignment? (Unified LD-Syn table above, grouping Depth, Count, and Joint@τ\mathrm{Joint}@\tau.)
  3. (Q3) Slot pairing and count supervision: On a shared backbone, how much gain comes from modeling absence (Depth stacking →\to ExactMB / MAP) versus fixing canonical pairing (Ordered assignment) and joint categorical count supervision (Count regularization), and what deeper-layer precision-recall tradeoff do they introduce?
  4. (Q4) Downstream 3D utility: How do variable-count metric layers enable cleaner multi-surface 3D reconstruction, inserted-light rendering, and robot route screening without phantom background sheets?

Thank you again for the constructive suggestion. Please let us know if you would like any further details or comparisons!

Reviewer_u2ByOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses recovering multiple visible surfaces along a single viewing ray (e.g., through glass) — a case standard single-depth monocular models compress away. Depth Any Seen represents ray-adaptive depth sets via an image-conditioned multi-Bernoulli random finite set: each of K components independently emits one Laplace-distributed metric depth or stays absent (Bernoulli). Its core technical contribution is the Exact Multi-Bernoulli (ExactMB) likelihood, an auxiliary-free objective that marginalizes over one-to-one assignments to complete, distinct targets, coupling presence and metric depth in a single proper scoring rule. The authors prove (and analyze) that matching expected count (Σ_j ∂L/∂e_j = E|X|−|T|) leaves component–surface assignment unresolved, motivating study of count-regularized and ordered-assignment variants and their precision–recall trade-offs. They also extend LD-Real / MD-3K / LD-Syn into a joint depth–presence benchmark separating depth accuracy, recovered support, and overprediction. ExactMB cuts LD-Real overprediction from 98.4% to 11.6% and MD-3K from 98.5% to 19.2% versus depth stacking, retaining ordinal accuracy and conditional metric error. Applications to layered 3D reconstruction, relighting, and robot path screening are shown.

Critical strengths
  1. Principled formulation of an important, under-studied problem. Casting layered visible-depth recovery as joint depth–presence learning with a proper multi-Bernoulli set likelihood is elegant and well-motivated; the analysis distinguishing expected-count correction from assignment resolution is a real conceptual contribution, not just an engineering tweak.
  2. Rigorous, multi-seed evaluation with a meaningful new evaluation perspective. The paper convincingly shows that depth stacking can score 85% ordinal ACC yet 0% Joint ACC, demonstrating that ordering ≠ membership. The stratified, support-aware metrics and the ablation of emission/assignment choices are thorough and reproducible (public datasets, code/checkpoints promised, W&B logs).
Critical weaknesses
  1. Moderate architectural novelty — adaptation rather than a new framework. The multi-Bernoulli set-prediction machinery is adopted from Hess et al. (2022), and the decoder/initialization come from SeeGroup (Wen & Deng 2026) + Depth Anything V2. The advance is the principled application to dense layered depth plus ExactMB marginalization and the analysis; the architecture itself is not novel. A direct comparison against a standard (non-marginalized) Bernoulli/set objective on the same data/backbone is missing, so the marginal contribution of ExactMB per se is not fully isolated.
  2. Synthetic-only training and a weak primary baseline bound the practical claim. All task training uses 14,800 synthetic images; real benchmarks are synthetic→real transfer with moderate accuracy (LD-Real ordinal ACC 61.7%, Joint ACC ~42%). The headline contrast is largely against "depth stacking" (select all K slots, no absence supervision), which is a deliberately weak comparator; a stronger baseline adding a simple learned/thresholded existence gate on the same decoder would make the ExactMB gain more convincing. Real-world robustness beyond LD-Real/MD-3K annotations is not demonstrated.
Questions for authors
  1. To isolate ExactMB's contribution from merely applying an existing set-prediction recipe, please compare against a standard Bernoulli/set objective (e.g., MAP assignment or a naive existence-gated decoder) trained on the same data and backbone. How much of the overprediction reduction is the set formulation vs the assignment marginalization?
  2. Training is synthetic-only; real results are transfer with moderate accuracy. How does the method behave on in-the-wild real transparent scenes outside LD-Real/MD-3K, and have you tried mixed synthetic–real supervision (you list this as future work — is it central to the practical claim)?
  3. Capacity is K=4 surfaces per ray. Real nested-transparency cases exceed this. How does the method degrade when the true count > K, and is there a tractable path to larger K?
  4. Could you add a stronger baseline than depth stacking — e.g., the same SeeGroup decoder with a learned/thresholded existence gate — to show ExactMB beats a naive absence mechanism rather than only an all-slots selector?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer u2By (1/2)

Dear Reviewer u2By,

Thank you for your thoughtful review and constructive questions. Below we address your comments by topic, directly connecting our two-axis objective design and multi-seed results to each question.


1. Architectural Scope vs. Formulation Contribution (W.1)

We agree, and we do not claim a new network backbone or decoder architecture. All controlled variants share the metric Depth Anything V2 ViT-L backbone and SeeGroup's four-pass recurrent DPT decoder. While our set representation builds on the multi-Bernoulli RFS machinery of Hess et al. (2022), adapting it from sparse object boxes to dense, ray-wise visible layered depth introduces distinct modeling and analytical contributions:

  1. Dense ray-set formulation: Each prediction slot j∈{1,…,K}j \in \{1,\dots,K\} represents an optional visible surface along a camera ray with metric Laplace depth dj>0d_j > 0, scale βj>0\beta_j > 0, and Bernoulli presence qj=σ(ej)q_j = \sigma(e_j), rather than an unordered object proposal.
  2. Exact likelihood and ownership analysis: At ray capacity K=4K=4, we evaluate the exact sum over all K!/(K−M)!K!/(K-M)! one-to-one injections (ExactMB) without auxiliary heuristics, and prove via ∑j=1K∂L/∂ej=E[∣X∣]−∣T∣\sum_{j=1}^K \partial\mathcal{L}/\partial e_j = \mathbb{E}[|X|] - |\mathcal{T}| (Eq. 7) that matching expected ray count leaves symmetric slot-to-surface ownership unresolved.
  3. Joint depth-and-presence evaluation: We show that standard depth/ordinal metrics reward dense all-slot emission, and introduce joint count-and-geometry evaluation across LD-Syn, LD-Real, and MD-3K.

2. Stronger Existence-Gate Baselines & Isolating ExactMB vs. MAP (W.1, W.2, Q.1, Q.4)

The stronger baselines you requested in Q.1 and Q.4 ("the same SeeGroup decoder with a learned/thresholded existence gate" and "a standard non-marginalized Bernoulli/set objective") are already trained on the exact same backbone, 14,800 synthetic images, 800k-step schedule, and four seeds in our controlled suite. Specifically, our five objectives form a clean two-axis taxonomy over slot pairing and count optimization:

SetupSlot pairingCount optimizationRole in isolating contributions
Depth stackingcanonical ordernone (all KK slots active)All-slot metric regression baseline (no absence supervision)
Ordered assignmentcanonical orderper-slot binary presenceQ.4 baseline: Same SeeGroup decoder + learned/thresholded existence gate (qj>0.01q_j > 0.01)
MAPbest injection (hard)per-slot binary presenceQ.1 baseline: Non-marginalized Bernoulli set objective (single best 1-to-1 match)
ExactMBall injections (soft)per-slot binary presenceMinimal-assumption set NLL: Marginalizes over all valid 1-to-1 injections
Count regularizationcanonical orderjoint categorical count CEReplaces independent slot gates with a (K+1)(K+1)-way count head + prefix decode

(Note on released SeeGroup: its coverage loss Lcover=−∑j=1Klog⁡max⁡iℓj(ti)L_{\mathrm{cover}} = -\sum_{j=1}^K \log\max_i \ell_j(t_i) encourages spare slots to copy an existing target depth at the scale floor rather than become absent, so its scale-derived validity score saturates and thresholding at 0.010.01 still yields 100%100\% empty-query overprediction.)

Comparing all five controlled setups side by side across LD-Syn, LD-Real, and MD-3K (4-seed mean ±\pm SD) directly answers Q.1 and Q.4:

SetupLD-Syn Aligned AbsRel ↓\downarrowLD-Syn Count acc ↑\uparrowLD-Syn [email protected] ↑\uparrowLD-Real Empty-Q OverPred ↓\downarrowMD-3K OverPred ↓\downarrowMD-3K Joint ↑\uparrow
SeeGroup (released)14.012.421.16100.00100.000.00
Depth stacking12.96±0.0912.96\pm 0.093.67±0.163.67\pm 0.162.75±0.172.75\pm 0.1798.42±0.3598.42\pm 0.3598.44±0.1998.44\pm 0.190.00±0.000.00\pm 0.00
MAP (1-match gate)15.08±0.0715.08\pm 0.0788.42±0.1888.42\pm 0.1882.28±0.2282.28\pm 0.2212.12±1.0912.12\pm 1.0920.89±1.3920.89\pm 1.3944.38±2.6044.38\pm 2.60
ExactMB (all-match)15.23±0.3815.23\pm 0.3888.17±0.4788.17\pm 0.4782.35±0.4082.35\pm 0.4011.62±1.3311.62\pm 1.3319.18±3.8119.18\pm 3.8143.30±3.7743.30\pm 3.77
Ordered assignment12.32±0.0812.32\pm 0.0889.68±0.1389.68\pm 0.1383.89±0.0283.89\pm 0.0210.12±1.1910.12\pm 1.1921.63±1.1821.63\pm 1.1847.86±2.9247.86\pm 2.92
Count regularization11.50±0.1011.50\pm 0.1094.99±0.0394.99\pm 0.0388.46±0.0388.46\pm 0.037.43±0.647.43\pm 0.648.57±0.548.57\pm 0.5460.30±1.0960.30\pm 1.09
AuthorsOpenReview ↗

Author Response to Reviewer u2By (2/2)

  • Set presence vs. assignment marginalization (Q.1): Moving from Depth stacking to MAP drives the primary gain: LD-Real overprediction drops from 98.42%98.42\% to 12.12%12.12\%, MD-3K Joint rises from 0.00%0.00\% to 44.38%44.38\%, and LD-Syn [email protected] jumps from 2.75%2.75\% to 82.28%82.28\%. Moving from MAP to ExactMB yields smaller shifts in this primary recipe (OverPred 12.12%→11.62%12.12\% \to 11.62\% on LD-Real and 20.89%→19.18%20.89\% \to 19.18\% on MD-3K, with overlapping seed intervals; in the separate Fig. 3(b) recipe, MAP →\to ExactMB reduces MD-3K overprediction by −5.31±1.49-5.31\pm 1.49 points). Furthermore, early training dynamics (Fig. 2, steps 1.8k to 2.0k) show a distinct optimization mechanism: ExactMB matches expected count (∑jqj=1.175\sum_j q_j = 1.175 vs. target 1.2001.200) while keeping thresholded activation conservative (0.2740.274 vs. MAP's 1.1261.126) until slot ownership specializes.
  • Why ExactMB is the principled starting point (Q.4): Rather than claiming ExactMB numerically dominates gated canonical baselines, ExactMB serves as the minimal-prior, permutation-invariant reference loss. Our Eq. 7 analysis explains why ExactMB and MAP leave room for improvement: matching expected count does not resolve slot pairing. Fixing canonical pairing (Ordered assignment) and supervising count jointly (Count regularization) directly inject those priors, lifting LD-Syn [email protected] to 83.89%83.89\% and 88.46%88.46\% and MD-3K Joint to 47.86%47.86\% and 60.30%60.30\%.

3. Synthetic-Only Training and Real-World Generalization (W.2, Q.2)

  • Zero-shot real transfer in context: We train solely on 14,800 synthetic LayeredDepth-Syn images because dense, per-pixel metric ground truth for multiple occluded transparent surfaces does not yet exist at training scale for real scenes. While zero-shot real numbers (ExactMB 61.71%61.71\% LD-Real quadruple joint, 43.30%43.30\% MD-3K joint; Count regularization 60.30%60.30\% MD-3K joint) leave headroom, all released multi-layer baselines (SeeGroup, World Tracing, LaRI) score 0.00%0.00\% to 10.44%10.44\% on MD-3K Joint because they hallucinate second layers on opaque surfaces (80.5%80.5\% to 100%100\% OverPred). Moreover, among pairs where count is predicted correctly, ExactMB's conditional depth order (Order|Count) on MD-3K is 93.39±0.89%93.39\pm 0.89\%, showing that synthetic-to-real geometric ordering transfers well and the remaining real-world gap is primarily surface-presence calibration on challenging materials.
  • In-the-wild scenes beyond LD-Real / MD-3K: Appendix C.8 (Figs. S8 to S21) and the supplementary video evaluate our auxiliary-regularized qualitative model (qj>0.5q_j > 0.5) on diverse uncalibrated real photographs (glassware, windows, cluttered indoor scenes), supporting downstream 3D unprojection, inserted-light rendering, and route screening without phantom background sheets. Mixed synthetic and real training (e.g., combining synthetic dense layers with sparse real ordinal or mask labels) is a promising engineering direction, though not required to establish the core depth and presence formulation.

4. Behavior When True Count >K> K and Scaling Beyond K=4K=4 (Q.3)

  • Behavior at and above capacity K=4K=4: Under injective matching ϕ ⁣:{1,…,M}→{1,…,K}\phi\colon \{1,\dots,M\}\to\{1,\dots,K\}, a ray with M>KM > K has no valid injection unless truncated to the nearest KK visible surfaces (at most KK layers can ever be emitted). Because LD-Syn, LD-Real, and MD-3K annotate at most 4 layers, M>4M > 4 degradation is not quantitatively measured here. However, on rays right at the capacity limit (M=4M=4 on LD-Syn), we observe graceful precision-oriented behavior: Depth stacking achieves higher rank-4 recall (54.09±0.46%54.09\pm 0.46\%) only by emitting 4 layers almost everywhere (2.40±0.12%2.40\pm 0.12\% rank-4 native precision), whereas ExactMB (14.51±0.25%14.51\pm 0.25\% precision, 39.35±0.50%39.35\pm 0.50\% recall) and Count regularization (52.44±0.61%52.44\pm 0.61\% precision, 40.47±0.73%40.47\pm 0.73\% recall) selectively emit deeper layers only when supported by image evidence.
  • Tractable paths to larger KK: Exact enumeration scales as O(BHW⋅K!(K−M)!⋅M)O(BHW \cdot \frac{K!}{(K-M)!} \cdot M) during training only (4!=244! = 24 terms at K=4K=4; inference is a single feed-forward pass with O(Klog⁡K)O(K\log K) sorting). Scaling training to larger KK has three clean paths: (1) if targets are canonically ordered near-to-far, Ordered assignment and Count regularization already scale linearly in O(K)O(K) with strong accuracy; (2) for permutation-invariant set training, KK-best / Hungarian pruning (MAP already runs in polynomial time with similar aggregate accuracy) or Sinkhorn/elementary-symmetric-polynomial DP avoids K!K! enumeration.

Thank you again for your detailed feedback. We hope these controlled comparisons and clarifications fully address your concerns, and we welcome any further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.