acceptodds
Under review as a conference paper at ICLR 2027

Modeling Depth Ambiguity: A Mixture-Density Representation for Flying-Point-Free Depth Estimation

Abstract

Despite advances in depth estimation, flying points remain a persistent failure mode: near object boundaries, depth estimators often predict spurious 3D points in the empty space between foreground and background surfaces. We trace this artifact to a standard modeling choice: assigning each pixel a single depth hypothesis. At boundaries, a pixel can straddle a foreground and a background surface, so its true depth is ambiguous between the two. A model that predicts a single depth cannot keep both possibilities, so training instead pulls the prediction toward an intermediate depth that lies on neither surface. We address this with MDA, a mixture-density representation that lets the model predict multiple depth hypotheses and their associated probabilities for each pixel. Near boundaries, different hypotheses can align with different surfaces, and the decoded depth is selected from one of these hypotheses rather than placed in the empty space between them. Across different architectures, MDA substantially improves boundary reconstruction over each base model and largely removes flying-point artifacts even under severe input blur, while adding negligible runtime overhead. The same mixture-density framework naturally extends to transparent objects, where it predicts multiple depth layers at transparent pixels, and to sky regions, where a dedicated component separates the unbounded sky from finite-depth regions, producing flying-point-free skylines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Depth estimation, depth ambiguity, boundary reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uAB2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a mixture-density depth representation that explicitly models the depth ambiguity near the boundary. Experiments show the flying points are significantly reduced, and the quantitative errors such as AbsRel are also reduced.

Critical strengths
  1. From table 1, the method seems to be very effective, reducing Chamfer Distance (CD) significantly while increasing little computational budget.
  2. The method is well motivated and solves the transparent object depth and sky depth.
Critical weaknesses
  1. Limited novelty: the main idea of mixture depth density seems to be too similar to the prior work SMDNet [1]. Increasing the number of clusters or applying to different backbones seem to be trivial.
  2. During inference, the cluster index of each pixel is selected (by argmax) independently. Will this result in uneven and bumpy surfaces when nearby pixels are assigned into different clusters? It would be beneficial to add metrics such as smoothness during evaluation.

[1] SMD-Nets: Stereo Mixture Density Networks

Questions for authors
  1. Could the author explain more substantial difference with respect to SMDNet?
3 comments

We thank the reviewer for the comments. New results are in the revised PDF (new text in blue; new tables and figures in App. E).

Q1. Difference from SMD-Nets.

Thank you for pointing this out. We have discussed SMD-Nets in the introduction and related work. SMD-Nets models stereo disparity with a two-component mixture. Our work differs from and extends it in two ways:

  • Scaling to large pretrained depth models. We generalize the mixture to K components in modern, large feed-forward (DA3, VGGT) and diffusion-based (Marigold V2) depth estimators, showing that the representation can be applied effectively to different depth prediction models and tasks. To preserve the performance of these pretrained models, we design a staged finetuning recipe (App. B.2). Each of the K output branches is initialized from the pretrained weights plus small noise to break symmetry, and the model is unfrozen in three stages, from the new head to the backbone. Unfreezing all layers at once performs much worse (Table R5, App. E.5).
  • Beyond boundary ambiguity. SMD-Nets uses the mixture only to choose between foreground and background at boundaries. We extend it to two other cases (Sec. 4). On transparent objects, a pixel sees both the glass and the background, so several depth components can be active at once. Sky has infinite depth, so we add one extra depth component to model this unbounded depth.

With this recipe, about 11k finetuning steps on 4 GPUs (compared with 128 H100s used to train DA3) are enough to outperform the original pretrained models at boundaries. MDA also even outperforms methods designed specifically for this problem, such as PPD for flying points (Tables 1 and 4).

We refer to the blog [1] from Michael J. Black discussing the different forms that novelty can take. Our work fits the described "novelty". First, "a new use of an old method can be novel if nobody ever thought to use it this way": Mixture densities are well known, but to our knowledge they have not been used to remove flying points in modern depth models, or to represent co-existing transparent layers and unbounded sky depth. Second, "if a paper has a simple idea that works better than the state of the art, then it is most likely not trivial": MDA changes only the output layer, yet with simple finetuning it improves the boundary quality of state-of-the-art models.

DatasetScheduleCD ↓B-CD ↓
NRGBDAll layers at once14.536.5
NRGBDThree stages (ours)11.530.5
7ScenesAll layers at once25.042.0
7ScenesThree stages (ours)24.040.5
HiRoomAll layers at once16.033.0
HiRoomThree stages (ours)11.528.0

W2. Can per-pixel selection create uneven, bumpy surfaces?

Per-pixel selection causes only subtle bumps. A bump appears only when neighboring pixels on the same surface select components that predict different depths. In our experiments, we find that this is rare away from boundaries, because the components there almost always predict the same depth and it barely matters which one is selected. Only 0.2% of non-boundary pixels switch to a component whose depth differs by more than 1%. Fig. R1 (App. E.6) visualizes this on planar regions. In the zoomed-in normal maps, MDA is about as smooth as the pretrained model. The selected component sometimes changes inside a plane (e.g., the carpet in the third row). Because the components predict nearly the same depth there, the normals show no seam.

Quantitative results confirm this. Normal consistency (NC), the cosine similarity between predicted and GT surface normals, is sensitive to small bumps, so bumpy surfaces would lower it. MDA has higher NC than the single-head DA3 finetuned in the same setting (Table R3, App. E.3, also listed below). On VGGT, MDA achieves NC on par with or better than the single-head VGGT (Table 5):

NC ↑HiRoomNRGBD
Finetuned DA30.8710.897
DA3+MDA0.8780.912

References

[1] M. J. Black. Novelty in Science. Medium, 2022. https://medium.com/@black_51980/novelty-in-science-8f1fd1a0a143

Thank you again for your review. As the discussion period is ending soon, we wanted to check whether our response and the revised paper address your two concerns, the difference from SMD-Nets and whether per-pixel selection create uneven, bumpy surfaces. If anything is still unclear, we would be glad to clarify it in the remaining time.

Best regards,

The authors

Reviewer_uAB2OpenReview ↗

Thank you for your reply

Thank you for your reply. My concern regarding the uneven and bumpy surfaces has been addressed. However, I still find the novelty insufficient. A similar mixture density formulation was already proposed in SMD-Net. As noted in line 266 of your paper ("Our mixture-density depth representation is architecture-agnostic"), your proposed modifications for scaling to large pretrained depth models—such as the staged finetuning recipe—appear incremental. Therefore, I will maintain my rating.

Reviewer_KiceOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles a common issue among state-of-the-art depth estimation models: near boundaries, models predict flying points between foreground and background. It addresses this problem by changing the last layer(s) of depth estimation models to predict a mixture of multiple depth hypotheses, where different hypotheses can represent different layers in the scene.

Authors apply their method to DA3, VGGT, and Marigold V2 and demonstrate that the boundary error is largely reduced qualitatively and quantitatively. Moreover, they show that similar ideas can be applied to predict multi-layer depth on transparent objects and address sky ambiguity.

Critical strengths
  • This paper is well motivated. Flying points is a common issue among state-of-the-art depth estimation models. Predicting a mixture of multi-layer hypotheses and the NLL loss functions (e.g. Eq. (3)) are naturally extended from standard model architectures and training objectives.

  • The methodology is general in the sense that: 1. It can be applied to multiple model architectures, as it only changes the last layer(s) of a model. 2. Similar ideas can be applied to address multi-layer depth on transparent objects and address sky ambiguity.

  • The method is efficient and effective, as it achieves better boundary error with negligible inference time overhead. Moreover, since the method only replaces the last layer(s) of a model, the improvement in boundary error is achieved with small training overhead: finetune pretrained checkpoints for only 10k steps.

Critical weaknesses
  • Quantitative evaluation is lacking: 1. DepthPro[1] proposes a boundary accuracy metric, F1 score, which is highly related to the boundary issue that this paper addresses, while such evaluation results are not reported in the paper. 2. For monocular depth estimation, the paper only reports boundary quality (Table 4) and does not report performance of standard monocular depth evaluation metrics (e.g. AbsRel, delta, RMSE, etc.). It is unclear that the improvement in boundary of MarigoldV2 comes at the cost of standard depth performance. Moreover, Table 4 only reports results on Hypersim, which is in the training set of MarigoldV2 (Sec 4.1 of MarigoldV2 paper), zero-shot performance is also unclear.

  • It is unclear how to supervise MDA with more advanced loss functions. In the paper, authors only derive loss function based on pixel-wise difference between predicted and ground truth depth (Eq. (3)). However, many recent depth estimation methods use a weighted training loss of many different terms, e.g. normal loss (e.g. used by MoGe), loss on gradient (derivatives) of depth maps (e.g. used by DepthPro). Other terms might be helpful to make models good at certain aspects, e.g. surface normal quality. From the paper, it is unclear whether MDA can be adapted to be supervised on those loss terms.

[1] Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, Vladlen Koltun

Questions for authors
  • Could authors report the performance of their method on DepthPro’s boundary accuracy benchmark? For MarigoldV2, could authors report both boundary quality and standard monocular depth evaluation metrics on more benchmarks (e.g. those used in MoGe)?

  • Can authors describe how to supervise MDA with loss terms that do not directly compute pixel-wise difference between predicted and ground truth depth map, e.g. normal loss or loss on gradient of depth maps?

  • Do authors have any insight about how the model learns the desired behavior of predicting mixture of multi-layer depth near boundaries? In the derived loss function (Eq. (3)), the model is still supervised to fit a single depth value, ground truth depth, at every pixel. This does not directly lead to the model predicting multiple mixtures, where each mixture corresponds to one layer of objects (e.g. foreground or background).

2 comments

We thank the reviewer for the constructive comments. New results are in the revised PDF (new text in blue in App. E).

Q1. Boundary F1 of DepthPro, and metrics of Marigold V2 on more benchmarks.

(a) Boundary F1 (Table R2, App. E.2). For a fair comparison, we finetune DA3 and VGGT with the same training data and settings as MDA. MDA has higher F1 than these models at every tolerance, except for DA3 on NRGBD at the default 0 px tolerance. At a tolerance of t px, a predicted boundary pixel counts as correct if it lies within t px of the GT boundary. Note that F1 is only weakly related to flying points: a flying point at the right image location can still count as a correct edge pixel, even though it floats between two surfaces. The 3D boundary Acc directly penalizes such points, and MDA reduces it to less than half of DA3's on NRGBD (Table 1).

DatasetModel0 px ↑1 px ↑2 px ↑3 px ↑
NRGBDFinetuned DA30.3170.7270.7840.809
NRGBDDA3+MDA0.2950.7380.7980.822
SintelFinetuned DA30.1510.4290.5180.563
SintelDA3+MDA0.1650.5060.6180.669
DatasetModel0 px ↑1 px ↑2 px ↑3 px ↑
NRGBDFinetuned VGGT0.2530.6390.7350.748
NRGBDVGGT+MDA0.2600.7100.7840.811
SintelFinetuned VGGT0.1400.4210.5440.590
SintelVGGT+MDA0.1490.4770.6000.654

(b) Marigold V2 (Table R1, App. E.1). On five benchmarks, MDA achieves similar score with Marigold V2 in standard depth metrics. It lowers the boundary error (SEE) on Sintel, NRGBD, HiRoom, and iBims-1. All these benchmarks are zero-shot.

DatasetModelAbsRel ↓δ1 (%) ↑
NYUv2Marigold V20.0358998.01
NYUv2+MDA0.0356598.02
KITTIMarigold V20.0541197.41
KITTI+MDA0.0541397.47
ScanNetMarigold V20.0366397.86
ScanNet+MDA0.0364297.88
DIODEMarigold V20.0523796.99
DIODE+MDA0.0523997.01
NRGBDMarigold V20.0227198.91
NRGBD+MDA0.0227298.87
DatasetModelSEE₃ ↓SEE₅ ↓SEE₇ ↓
SintelMarigold V21.4501.4271.407
Sintel+MDA1.3941.3711.352
NRGBDMarigold V20.0900.0840.078
NRGBD+MDA0.0780.0710.066
HiRoomMarigold V20.0840.0790.075
HiRoom+MDA0.0780.0720.068
iBims-1Marigold V20.1440.1210.109
iBims-1+MDA0.1350.1130.101

Q2. Can MDA be supervised with more advanced losses, e.g., normal or depth-gradient losses?

Yes. The mixture NLL replaces only the per-pixel depth loss, so other loss terms, such as normal or depth-gradient losses, can still be added to the training objective. We compute these losses on the final depth map: at each pixel, this is the depth of the component selected by Eq. 4, which is exactly what MDA outputs at inference. The mixture NLL already handles depth discontinuities, so we mask out edge pixels and apply the additional losses only in smooth regions. Edge pixels are detected as depth jumps between neighboring pixels. As shown in Table R4 (App. E.4), finetuning with an added normal loss improves global normal consistency (NC), while the boundary metrics (B-Acc, B-CD) remain nearly unchanged:

DatasetModelB-Acc ↓B-CD ↓NC ↑
NRGBDDA3+MDA24.030.50.912
NRGBD+ normal loss22.030.00.923
7ScenesDA3+MDA36.040.50.811
7Scenes+ normal loss35.041.50.819
HiRoomDA3+MDA30.028.00.878
HiRoom+ normal loss31.530.00.885

In addition, our Marigold V2 variant is also already trained with an additional loss term, Marigold V2's SinkLoss (App. B.4).

Q3. How does the model learn multiple surfaces when each pixel is supervised with a single GT depth?

Multi-depth ground truth is not required. Each pixel is labeled with only one depth, and the supervision for multiple surfaces comes from the training set as a whole. A boundary pixel’s features mix cues from the surfaces on both sides, so the network cannot reliably tell which surface the pixel belongs to. Across the training set, pixels with such similar features are labeled with the foreground depth in some images and with the background depth in others (Sec. 1 & App. A.4). The network therefore sees essentially the same input paired with two distinct groups of depth labels. A single-head regressor would average the two groups, whereas the mixture fits each group with its own component (Fig. 14). See App. A.4 for full analysis and “Look Inside a Pixel” demo in supp material for visualization.

Thank you again for your review. As the discussion period is ending soon, we wanted to check whether our response and the revised paper address you concerns, regarding the additional evaluation results, supervision with more advanced loss functions, and the mechanism for learning multiple surfaces from a single GT depth map. If anything is still unclear, we would be glad to clarify it in the remaining time.

Best regards,

The authors

Reviewer_sx1EOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Modeling Depth Ambiguity (MDA), a lightweight mixture-density representation designed to address flying-point artifacts near depth discontinuities. Instead of predicting a single depth value per pixel, MDA predicts multiple depth hypotheses and their associated probabilities, allowing ambiguous pixels to select plausible surface depths rather than intermediate values. The method modifies the prediction head of existing depth estimators and can be integrated into different architectures, including DA3, VGGT, and Marigold V2. Experiments demonstrate substantial improvements in boundary reconstruction with minimal computational overhead.

Critical strengths
  • Well-motivated problem with a simple and effective solution.

The paper identifies a specific and persistent limitation of existing depth estimation methods and provides a clear explanation of how unimodal depth representations contribute to flying-point artifacts. The proposed mixture-density formulation is conceptually simple yet well aligned with the underlying problem, and the substantial improvements in boundary quality demonstrate its effectiveness.

  • Lightweight and architecture-agnostic design.

MDA requires only modifications to the depth prediction head, making it easy to integrate into existing depth estimation frameworks without redesigning their architectures. Its successful application to DA3, VGGT, and Marigold V2 demonstrates its flexibility and practical value. The negligible inference overhead further makes it attractive for real-world applications.

  • Convincing experimental validation.

The paper provides extensive comparisons across multiple datasets and architectures, including quantitative evaluations, qualitative examples, and controlled ablation studies. In particular, the ablation comparing multi-head unimodal prediction with mixture-density prediction helps isolate the contribution of the proposed representation. The extensions to transparent objects and sky regions further illustrate the broader applicability of modeling depth ambiguity.

Critical weaknesses
  • Trade-off between boundary quality and overall reconstruction accuracy.

Although MDA substantially improves boundary reconstruction, these gains do not consistently translate into improvements in overall 3D reconstruction. For example, Appendix Table 5 shows that DA3+MDA slightly underperforms the original DA3 on several reconstruction metrics, particularly on NRGBD and HiRoom. A more detailed investigation of whether the mixture representation introduces errors away from depth boundaries would help clarify this trade-off.

  • Limited physical modeling of transparent objects.

The proposed multi-layer depth representation assumes straight camera rays, overlooking refraction through transparent surfaces. This assumption may lead to inaccurate geometry reconstruction, particularly for curved or thick transparent objects. The authors should discuss this limitation and its impact on reconstruction accuracy.

  • Limited scope of the contribution.

MDA serves as a lightweight add-on to existing depth estimators rather than a standalone framework. Its effectiveness therefore depends on the underlying backbone and does not address other depth estimation errors, although its simplicity and plug-and-play design are valuable strengths.

Questions for authors
  1. Scope of the method: To what extent can MDA correct depth estimation errors beyond object boundaries, particularly when the underlying backbone produces inaccurate depth predictions?

  2. Refraction: How does the proposed multi-layer representation account for refraction through transparent objects? How does this assumption affect reconstruction accuracy for curved or thick glass?

  3. Overall reconstruction quality: What explains the degradation in some global reconstruction metrics despite substantial improvements in boundary accuracy (Appendix Table 5)?

3 comments

We thank the reviewer for the positive assessment and the insightful questions. New results are in the revised PDF (new text in blue; new tables and figures in App. E).

Q1. To what extent can MDA correct errors beyond boundaries, when the backbone is inaccurate?

MDA targets one specific type of error: depth ambiguity. When a pixel has several plausible depths, a single-depth model averages them into a depth that lies on no surface, whereas MDA represents each plausible depth with a separate component. This ambiguity also occurs away from object boundaries. On transparent objects, MDA predicts both the glass and the background behind it. For the sky, an extra component represents the infinite depth, which removes flying points along the skyline (Sec. 4, Fig. 6).

When the backbone is inaccurate, MDA helps if the backbone is uncertain, but not if it is confidently wrong. Under blur (Figs. 4 and 7), the backbone is uncertain about where the boundary lies. Unimodal baselines place these pixels between the two surfaces, whereas MDA still places each pixel on one of them, so the surface boundary stays clean. On thin structures (Fig. 12), the backbone is confidently wrong. All components make the same error, so the mixture cannot correct it. We have added this to the limitations (App. D).

Q2. How does the representation handle refraction, and how does this affect curved or thick glass?

Our representation does not model refraction. It follows the LayeredDepth ground truth, which defines each layer along the straight camera ray: the first layer is the glass surface, and the last layer is the background as if the glass were removed. The first layer is unaffected by refraction. For the last layer, refraction bends the ray, so the background seen through the glass appears shifted, and the network must still predict its depth along the straight ray. This shift is small for thin, flat glass but large for curved or thick glass, so the last layer is harder to predict there. The mixture itself does not assume straight rays and could be trained on refraction-aware labels if they become available. We added this discussion to the limitations (App. D).

Q3. Why do some global reconstruction metrics drop (Table 5)?

MDA substantially improves the boundary metrics over each base model (Tables 1 and 3). However, boundary points make up only a small fraction of all points (Sec. 5.1), so these gains barely affect the global Acc and Comp in Table 5. Three observations indicate that the drop comes from our finetuning setting rather than from the mixture representation:

  1. Controlled baseline (Table R3, App. E.3). : To isolate the effect of the mixture, we finetune DA3 with its original single head on the same data and schedule as MDA. This single-head model drops by a similar amount relative to the released DA3, and MDA is on par with or slightly better than it:
DatasetModelAcc ↓Comp ↓CD ↓NC ↑
HiRoomReleased DA311.08.09.50.919
HiRoomFinetuned DA315.09.0‾\underline{9.0}12.00.871
HiRoomDA3+MDA14.0‾\underline{14.0}9.0‾\underline{9.0}11.5‾\underline{11.5}0.878‾\underline{0.878}
NRGBDReleased DA310.011.010.50.948
NRGBDFinetuned DA311.0‾\underline{11.0}11.011.0‾\underline{11.0}0.897
NRGBDDA3+MDA11.0‾\underline{11.0}12.011.50.912‾\underline{0.912}
  1. Another backbone (Table 5). On VGGT, MDA improves Acc on all three datasets, with on-par or better scores. This indicates that the mixture itself does not degrade global reconstruction.
  2. Pipeline gap. The released DA3 was trained on 128 H100 GPUs for 200k steps, using an unreleased pipeline with teacher pseudo-labels on real data [1]. In contrast, we finetune on 4 RTX PRO 6000 GPUs for 11k steps using only synthetic data. Both smaller batches [2, 3] and a narrower training distribution [4] are known to degrade overall performance. We plan to close this gap with a pipeline closer to DA3’s when compute allows.

References

[1] H. Lin et al. Depth Anything 3: Recovering the Visual Space from Any Views. ICLR 2026.

[2] C. J. Shallue et al. Measuring the Effects of Data Parallelism on Neural Network Training. JMLR 2019.

[3] S. McCandlish et al. An Empirical Model of Large-Batch Training. arXiv:1812.06162, 2018.

[4] A. Kumar et al. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022.

Thank you again for your review. As the discussion period is ending soon, we wanted to check whether our response and the revised paper address your concerns, regarding the errors MDA can correct, refraction handling, and the reason for global reconstruction metrics drop. If anything is still unclear, we would be glad to clarify it in the remaining time.

Best regards,

The authors

Thanks for the detailed responses. My concerns are addressed, and I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.