acceptodds
Under review as a conference paper at ICLR 2027

EagleDepth: Efficient Fine-Grained Monocular Depth Estimation via Pixel Diffusion Decoder

17 views

Abstract

Recovering detailed geometry from high-resolution images is critical for precise perception of the surroundings and objects. However, existing methods which use latent-space modeling and VAE reconstruction can compromise geometric details. Furthermore, decoding from latent codes introduces substantial inference overhead. To address those issues, we present , an efficient framework for high-resolution monocular depth estimation that combines the geometric priors of latent diffusion with fine-grained pixel-space generation. Our key idea is to retain depth-aware latent representations as guidance while generating the final depth map directly in pixel space. We train the latent and pixel components sequentially: first, we fine-tune a pretrained latent diffusion model using paired RGB–depth supervision; then, we adapt a pretrained pixel diffusion decoder, PiD, to predict depth conditioned on the learned features. Training of the pixel component starts at 1024 resolution and continues across multiple resolutions up to 4K. The latent branch processes resized, lower-resolution RGB images, while the pixel branch generates depth at the target resolution, bypassing the original VAE decoder. This design preserves learned geometric knowledge without requiring the latent backbone to operate at the output resolution. On five commonly used depth estimation datasets and the high-resolution Synth4K dataset, our framework achieves state-of-the-art depth estimation performance, with faster inference and better preservation of fine structures and object boundaries.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Monocular Depth Estimation, Pixel Diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wD4GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tracles the problem of high-resolution depth estimation. The core idea is to retain depth-aware latent representations as guidance while generating the final depth map directly in pixel space. The pixel branch generating depth at the target resolution bypasses the original VAE decoder, which gets rid of substantial inference overhead. The trained model achieves SoTA performance on Synth4K.

Critical strengths
  • EagleDepth achieves the SoTA on the Synth4K benchmark.
  • Exploring to replace the VAE decoder with other advanced techique like PiD for depth is an interesting direction.
Critical weaknesses
  • Exploring to replace the VAE decoder with other advanced techique like PiD for depth is an interesting direction (It's a strength I listed above). But the solution is less informative. As far as I understand from the paper, the proposed solution is merely tuning the PiD for depth estimation without any depth specific adaptation. All claims are somehow overlapped with the PiD and I do not see much new knowledge that this paper can bring to the feild.
  • The authors use FLUX.2-klein4B (Labs, 2025) as the base latent and fine-tune PiD for diffusion decoder. FLUX.2-klein4B is a very strong base model and it's nothing related to the contributions in this paper. It will raise the concern that EagleDepth achieves the SoTA only becaue it uses a stronger model as the base model. As shown in Tab.4, after using a weaker latent model, the performance is no longer best.
  • The training data is also somehow different from previous work with more synthetic data. This can also lead to concerns about comparison.
Questions for authors

Please reply to my Critical Weaknesses. I would like to know if my judgement about the techique contribution is reasonable. I would also like to hear the authors' opinions about the influence of scaling up the base model capacity and data volumn. These points will influence my final grade.

2 comments
AuthorsOpenReview ↗

Response to Reviewer wD4G (denoted as R3) - Part 1

W3-1 Differences from PiD and the paper's contributions.

Response Thanks for your comment. As stated in the paper, our core design is to adapt a pixel diffusion decoder to decode low-resolution depth latents into detailed, high-resolution depth maps. Our work offers two main contributions:

  1. We demonstrate the effectiveness of pixel diffusion decoding for depth estimation, with particularly strong results in fine-detail recovery.

  2. We identify and address challenges in adapting pixel diffusion decoding to depth estimation (a key focus of your question). Specifically, our method differs from the original PiD in two aspects:

    (a) Patch continuity. Each patch token corresponds to a 16 × 16 pixel region that may span sharply different foreground and background depths (such depth transitions are often much more abrupt than the corresponding appearance changes in natural RGB images). Decoding these tokens without local spatial mixing can produce visible patch discontinuities near object boundaries, as shown in Figure 4. We introduce a continuity-promoting module to suppress these artifacts.

    (b) RGB input. In text-to-image generation, the pixel diffusion decoder takes noise as its pixel input and relies on high-resolution priors learned during training to supply fine details. For depth estimation, we instead use the high-resolution RGB image as the pixel input, allowing the decoder to also obtain fine-detail information directly from the observed image.

W3-2 SOTA performance may be attributable solely to the latent base model, FLUX.2-klein-4B.

Response Thank you for your comment. The pixel diffusion decoder complements the overall depth accuracy provided by coarse depth-latent predictions by recovering high-resolution details. The quality of the latent depth model is therefore important. We also acknowledge that our optimization of the latent model is an engineering contribution rather than a fundamentally new method. Nevertheless, EagleDepth's performance is not attributable solely to the latent model. Table 4 of the main paper demonstrates two benefits of pixel diffusion decoding:

  1. Across all three latent-model settings, pixel diffusion decoding consistently yields substantial improvements in overall depth accuracy and the fine-detail metrics.
  2. Fine-tuning FLUX.2 alone does not achieve SOTA performance, as shown by the Flux.2-Depth (pixel loss) row in Table 4.

Even with a weaker latent model such as Lotus-2, our method remains competitive. For example, in terms of overall depth accuracy, ours (Lotus-2) achieves a mean δ1\delta_1 of 79.3 over Synth4K-1/2, approaching PPD (83.3) and URGT (84.3) while outperforming all other baselines. In terms of fine-detail recovery, pixel diffusion decoding improves the mean boundary F1 from 17.0 to 21.1, outperforming PPD (16.7).

In addition, PPD, the strongest compared baseline, also relies on external pretrained models. We provide the following comparison using different external models:

MethodETH3DDIODESynth4K-1Synth4K-2
PPD (DA2)4.5 / 98.35.6 / 96.422.9 / 66.2 / 34.0 / 53.9 / 1.618.2 / 74.7 / 29.8 / 56.1 / 4.1
PPD (MoGe-2)3.2 / 99.14.1 / 97.715.9 / 79.4 / 29.4 / 62.3 / 15.011.7 / 87.3 / 24.5 / 65.0 / 18.4
Ours (Lotus-2)4.1 / 98.34.8 / 96.917.9 / 74.8 / 31.3 / 57.4 / 20.313.2 / 83.7 / 25.8 / 60.1 / 21.9
Ours (Flux.2)3.1 / 99.23.9 / 97.915.0 / 80.3 / 27.4 / 62.6 / 22.511.0 / 87.5 / 24.3 / 64.7 / 22.2

ETH3D and DIODE report AbsRel ↓ / δ1\delta_1 ↑; Synth4K reports AbsRel ↓ / δ1\delta_1 ↑ / AbsRelH\mathrm{AbsRel}^{H} ↓ / δ1H\delta_1^{H} ↑ / boundary F1 ↑.

AuthorsOpenReview ↗

Response to Reviewer wD4G (denoted as R3) - Part 2

W3-3 Differences in training data and the use of more synthetic data.

Response Thank you for your comment. Different depth estimation methods use different training datasets, and we seek a balance between training efficiency and performance. Among our training datasets, Rawmantic3D is the only one that we introduce to depth estimation for the first time, motivated by our high-resolution training objective. All other datasets have been used in previous depth estimation methods. As detailed in Appendix B, we filter and subsample the datasets to obtain 536K RGB–depth pairs, a training set smaller than that used by most methods (e.g. DA3 and MoGe-2).

To provide a fairer comparison, we include additional reduced-data training settings in the comparison below:

MethodTraining dataETH3DDIODE
Marigold74K6.5 / 96.010.0 / 90.7
Lotus-259K5.1 / 97.56.1 / 95.2
PPD125K3.2 / 99.14.1 / 97.7
MoGe-28.9M3.5 / 98.74.7 / 96.4
OursMarigold-like3.6 / 98.84.8 / 96.9
OursPPD-like3.4 / 98.94.1 / 97.9
Ours (1024)536K3.1 / 99.23.9 / 97.9

These results show that our method remains competitive when trained on the same amount of data.

We also need to emphasize that, although PPD itself is trained on relatively few samples, it relies on pretrained vision foundation models (DA2 and MoGe-2) to provide priors. These models have been trained on millions of images with depth or 3D geometry supervision. In contrast, our method only uses 536K RGB–depth pairs for depth/geometry supervision.

Q3-1 Technical contribution and the influence of scaling up base model capacity and training data volume.

Response Thank you for your thoughtful assessment. Our goal is to develop a depth estimation model that makes accurate predictions and generalizes well. This involves engineering choices about the training data and the base model. At the same time, our method offers some novel contributions and brings substantial performance gains, as discussed in our response to W3-1.

Regarding base model capacity and training data volume, we agree that scaling them up can improve performance, consistent with scaling laws. Nevertheless, our core contribution also plays a crucial role in achieving these results, as discussed in our responses to W3-2 and W3-3.

Reviewer_VygqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EAGLEDEPTH, a method for high-resolution relative depth estimation from a single image. A latent diffusion model first predicts coarse depth latents from a downsampled image, and a pixel diffusion decoder uses these latents together with the high-resolution RGB image to recover fine structures. Predicting depth directly in pixel space aims to avoid detail loss during VAE decoding. The decoder also mixes features across neighboring patches to reduce patch-boundary artifacts.

Critical strengths
  1. Writing: The paper is clearly written, and the motivation for separating coarse geometry from fine-detail recovery is easy to understand. Figure 2 and its caption explain the two stages, while the magnified comparisons help readers inspect thin structures and depth boundaries. The patch-size analysis also clearly shows the trade-off between speed and detail.

  2. Effective Design and Strong Boundary Results: The decoder ablations across different coarse predictors support the design. The method leads the compared approaches in high-frequency depth accuracy and boundary F1 across all five Synth4K subsets; on Synth4K-1, boundary F1 is 31.1 versus 20.9 for the strongest compared baseline. These gains also come with a substantial speedup over PPD: 1.48 versus 27.58 seconds per 3840×2160 image on the same RTX 4090 (Table 6).

Critical weaknesses

Limited 3D Comparisons: The depth and boundary metrics provide strong evidence of improved local predictions. However, the point clouds in Appendix Figure 5 show only the proposed method, with limited viewpoint changes. Without matched baseline and ground-truth views, it is difficult to judge whether the sharper boundaries also reduce flying pixels and improve surface shape. Since the paper focuses on geometric detail, I would like to see a few side-view comparisons with PPD that expose these differences.

The design is reasonable and clearly presented, and the boundary results and runtime improvement over PPD are strong. The existing quantitative evidence is enough for me to lean toward a Weak Accept, although I would still like to see the corresponding improvements in 3D.

Questions for authors

Could you provide matched side views of EAGLEDEPTH, PPD, and ground-truth point clouds around a few representative depth boundaries? Please use the same camera and alignment settings so that differences in flying pixels and surface shape are visible.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Vygq (denoted as R2)

W2-1 & Q2-1 More 3D comparisons

Response Thank you for the suggestion.

We have added more 3D point-cloud comparisons in Figure 6 and Figure 7 of the appendix in the revised manuscript. These results show that detailed depth maps help distinguish small objects and delineate clear object boundaries in the reconstructed point clouds, such as plant leaves in Figure 6, eagle wings and flower petals in Figure 7. Our method also produces more accurate overall depth estimates, with less object deformation and distortion and cleaner point-cloud reconstructions.

Reviewer_62y9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper suggests that latent diffusion depth estimation models lose fine details at high resolutions due to latent compression and VAE reconstruction, while pixel-space methods are expensive at such resolutions. It proposes a monocular depth estimation method that combines a latent diffusion model for coarse depth with a pixel diffusion decoder for fine details. Its contributions rely on two main components: (i) a latent diffusion model adapted to predict coarse depth from a downsampled image in a single step; and (ii) a pretrained pixel diffusion decoder adapted to generate the high-resolution depth map, guided by the coarse depth and the original image. The method is tested on five standard depth benchmarks and the high-resolution Synth4K benchmark.

Critical strengths
  • The proposed method recovers fine details at high resolution clearly better than the baselines, with the best boundary F1 and high-frequency metrics on all Synth4K subsets (Tab. 2-3), while remaining competitive on standard benchmarks (Tab. 1). It is also the fastest among the evaluated methods at 4K (Tab. 6).
  • The paper is clearly presented and provides enough implementation details to reproduce the method. The pixel decoder also improves different latent priors (Tab. 4), which highlights that the design is not tied to a specific backbone.
Critical weaknesses
  • All methods except InfiniDepth† are evaluated with native 4K inputs (Tab. 2), which is higher than the resolution most baselines were trained on. Note how InfiniDepth improves considerably when it is run at its intended input resolution (InfiniDepth†), even outperforming EagleDepth (MR) in overall AbsRel on three of the five subsets, while the proposed method still leads in the fine-detail metrics. It would be useful for the authors to also evaluate the other baselines at their native resolution with upsampling to 4K, and also for the runtime comparison (Tab. 6).
  • Additionally, EagleDepth (MR) is further trained on high-resolution synthetic datasets, while Synth4K is also synthetic. This step seems to account for a large part of the fine-detail gains (e.g., boundary F1 on Synth4K-1 improves from 22.5 to 31.1, Tab. 3), but it is not clear whether this comes from the different data distribution, the higher training resolutions, or simply the additional training steps. It would be useful for the authors to disentangle these factors (e.g., continuing training at 1024 on the original data for the same number of steps, or at 1024 on the high-resolution datasets downsampled).
  • The method mainly combines existing techniques, such as single-step adaptation of latent diffusion models and the pretrained PiD decoder. Supervising the decoded depth through the frozen VAE decoder in a single-step model (Eq. 5) has also been explored before (e.g., Garcia et al., WACV 2025). Additionally, the continuity-promoting module, which is the main novel architectural component, is only evaluated qualitatively (Fig. 4).
Questions for authors

In addition to the main weaknesses above, the following questions:

  • Could the authors also compare to Depth Pro, which also targets sharp high-resolution depth and whose boundary F1 metric is used in the evaluation? A comparison to patch-based methods would also be informative.
  • How much does the PiD pretraining matter for the fine-detail results (e.g., compared to training the same decoder from scratch)?
  • How are the predictions aligned to the ground truth for evaluation, given that the model predicts normalized log-depth?
  • The paper does not include a limitations discussion. In which cases does the method fail?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 62y9 (denoted as R1) - Part 1

W1-1 Compare with baselines at their native resolution.

Response Thank you for the suggestion.

Upsampling can affect evaluation results, particularly fine-detail metrics. To ensure a fair comparison in our high-resolution evaluation, we have all methods produce 4K depth maps without upsampling their predictions. Since some baselines may not have been trained on high-resolution data, we also report Ours (1024) in Tables 2 and 3. This variant is likewise trained only at low resolution and outperforms the compared methods.

We acknowledge that evaluating the baselines at their native resolutions may improve their performance. Following your suggestion, we conducted additional experiments using the input resolutions recommended by their official repositories:

MethodResolutionSynth4K-1Synth4K-2
MoGe-2630 × 112018.9 / 73.5 / 29.1 / 60.3 / 4.015.8 / 78.8 / 26.6 / 61.0 / 1.9
PPD672 × 118413.8 / 83.5 / 29.0 / 65.7 / 6.810.1 / 90.5 / 26.4 / 63.9 / 5.2
Ours720 × 128012.7 / 85.0 / 28.1 / 63.9 / 8.79.0 / 91.5 / 26.5 / 64.3 / 6.3

Within each dataset column, the five values are ordered as AbsRel ↓ / δ1\delta_1 ↑ / AbsRelH\mathrm{AbsRel}^{H} ↓ / δ1H\delta_1^{H} ↑ / boundary F1 ↑.

These results indicate that our method retains advantages under this evaluation setting, particularly in boundary F1, the fine-detail metric emphasized in our work.

Regarding the requested runtime comparison at native resolutions, we think that comparing runtimes across different input sizes would not provide a fair comparison. We have therefore not conducted this experiment at this stage, and we welcome further discussion on this point.

W1-2 What accounts for the fine-detail gains of EagleDepth (MR)

Response Following your suggestion, we conducted two additional experiments: (1) continuing training at 1024 on the original data for the same number of steps as Ours (MR), and (2) continuing training for the same number of steps with the MR data mixture, with the high-resolution datasets downsampled to 1024. The results are shown below:

SettingSynth4K-1Synth4K-2
Ours (1024)15.0 / 80.3 / 27.4 / 62.6 / 22.511.0 / 87.5 / 24.3 / 64.7 / 22.2
(1) Original data for the same steps14.9 / 80.3 / 27.4 / 62.6 / 22.910.8 / 87.7 / 24.2 / 65.0 / 23.0
(2) High-resolution datasets downsampled to 102414.6 / 81.0 / 27.1 / 63.2 / 20.110.8 / 87.9 / 24.3 / 65.1 / 20.6
Ours (MR)12.3 / 85.3 / 22.8 / 69.0 / 31.18.4 / 92.4 / 21.3 / 71.0 / 29.5

Within each dataset column, the five values are ordered as AbsRel ↓ / δ1\delta_1 ↑ / AbsRelH\mathrm{AbsRel}^{H} ↓ / δ1H\delta_1^{H} ↑ / boundary F1 ↑.

Neither control yields gains comparable to Ours (MR), indicating that training at higher resolutions plays a substantial role. High-resolution data contain rich fine-detail information, while exposure to high-resolution inputs during training also helps reduce the resolution mismatch between training and inference. Nevertheless, as shown by Ours (1024) in Tables 2 and 3 of the main paper, our model trained only at low resolution already outperforms the compared methods.

AuthorsOpenReview ↗

Response to Reviewer 62y9 (denoted as R1) - Part 2

W1-3 Combination of existing techniques; lack of quantitative results for the continuity-promoting module.

Response Thank you for your comment.

Combination of existing techniques. We acknowledge that our method builds in part on existing techniques. However, our core contribution is to use a pixel diffusion decoder to decode low-resolution depth latents into detailed, high-resolution depth maps. Optimizing the latent depth diffusion model serves to improve overall performance; we do not claim this adaptation as a novel contribution. Our work offers two main contributions:

  1. We demonstrate the effectiveness of pixel diffusion decoding for depth estimation, with particularly strong results in fine-detail recovery, as acknowledged by R1 and R2.

  2. We identify and address challenges in adapting pixel diffusion decoding to depth estimation (a key focus of your question). Specifically, our method differs from the original PiD in two aspects:

    (a) Patch continuity. Each patch token corresponds to a 16 × 16 pixel region that may span sharply different foreground and background depths (such depth transitions are often much more abrupt than the corresponding appearance changes in natural RGB images). Decoding these tokens without local spatial mixing can produce visible patch discontinuities near object boundaries, as shown in Figure 4. We introduce a continuity-promoting module to suppress these artifacts.

    (b) RGB input. In text-to-image generation, the pixel diffusion decoder takes noise as its pixel input and relies on high-resolution priors learned during training to supply fine details. For depth estimation, we instead use the high-resolution RGB image as the pixel input, allowing the decoder to also obtain fine-detail information directly from the observed image.

Quantitative results for the continuity-promoting module. This module primarily addresses patch discontinuities, which are visually apparent but can be difficult to capture with standard quantitative metrics. Following your suggestion, we provide additional quantitative results below:

SettingETH3DDIODE
w/o module3.3 / 98.94.0 / 97.7
w/ module3.1 / 99.23.9 / 97.9

Within each dataset column, values are ordered as AbsRel ↓ / δ1\delta_1 ↑.

Q1-1 Compare to Depth Pro and patch-based methods.

Response Thank you for the suggestion. In the main paper, we compared our method with nine baselines, several of which had already reported comparisons with Depth Pro. We therefore did not include a direct comparison with Depth Pro in the initial submission, and we now provide this additional comparison.

Comparison with Depth Pro. Depth Pro's official implementation uses a fixed internal inference resolution of 1536 × 1536, resizing inputs at other resolutions to this grid. Since resolution has a substantial effect on metrics, we evaluate Ours (MR) with the same 1536 × 1536 pixel-space input and output resolution. Predictions from both methods are evaluated on the original ground-truth grids using the same evaluation protocol.

MethodETH3DDIODESynth4K-1Synth4K-2
Depth Pro (1536 × 1536)5.0 / 96.74.9 / 96.414.3 / 82.2 / 28.0 / 63.1 / 12.79.5 / 89.0 / 25.0 / 63.2 / 7.7
Ours (MR, 1536 × 1536)3.3 / 99.14.0 / 97.812.7 / 85.1 / 25.7 / 66.0 / 14.19.0 / 91.9 / 25.0 / 66.2 / 11.2

Comparison with patch-based methods. URGT, which is already included in our main-paper comparisons (Tables 2 and 3), is a patch-based method. We additionally include PatchRefinerV2 [1] as another patch-based baseline.

MethodETH3DDIODESynth4K-1Synth4K-2
PatchRefinerV2-M [1]8.7 / 93.99.4 / 91.338.7 / 50.9 / 51.2 / 42.9 / 7.923.9 / 63.8 / 37.1 / 47.9 / 7.7
Ours (MR)3.1 / 99.13.9 / 97.912.3 / 85.3 / 22.8 / 69.0 / 31.18.4 / 92.4 / 21.3 / 71.0 / 29.5

ETH3D and DIODE report AbsRel ↓ / δ1\delta_1 ↑; Synth4K reports AbsRel ↓ / δ1\delta_1 ↑ / AbsRelH\mathrm{AbsRel}^{H} ↓ / δ1H\delta_1^{H} ↑ / boundary F1 ↑.

[1] Li, Zhenyu, et al. "Patchrefiner v2: Fast and lightweight real-domain high-resolution metric depth estimation." International Conference on Learning Representations.

AuthorsOpenReview ↗

Response to Reviewer 62y9 (denoted as R1) - Part 3

Q1-2 How much does PiD pretraining matter for the fine-detail results

Response PiD is pretrained on high-resolution images, allowing it to acquire priors for fine image details that benefit fine-detail recovery. Following your suggestion, we compare training the depth decoder with and without pretrained PiD weights, using the same training steps in both settings:

SettingETH3DDIODESynth4K-1Synth4K-2
w/o pretrain3.2 / 99.14.0 / 97.817.5 / 78.2 / 38.5 / 55.7 / 3.313.1 / 84.6 / 32.2 / 57.0 / 6.1
w/ pretrain3.1 / 99.23.9 / 97.915.0 / 80.3 / 27.4 / 62.6 / 22.511.0 / 87.5 / 24.3 / 64.7 / 22.2

ETH3D and DIODE report AbsRel ↓ / δ1\delta_1 ↑; Synth4K reports AbsRel ↓ / δ1\delta_1 ↑ / AbsRelH\mathrm{AbsRel}^{H} ↓ / δ1H\delta_1^{H} ↑ / boundary F1 ↑.

Q1-3 Log-depth alignment.

Response Log-depth representations have also been used in PPD and InfiniDepth. Following this log-depth alignment convention, we estimate a per-image scale and shift by least squares to align our normalized predictions with log⁡(1+dGT)\log(1+d_{\mathrm{GT}}) over valid pixels, then apply exp⁡(⋅)−1\exp(\cdot)-1 to recover depth before computing metrics.

Q1-4 Limitations discussion.

Response EagleDepth has two main limitations:

  1. Metric depth. EagleDepth is trained for relative depth estimation and does not directly predict depth in physical units. This limits its direct applicability to tasks requiring metric depth.
  2. Temporal consistency. As an image-based diffusion model, EagleDepth processes video frames independently without explicit temporal constraints. This can lead to slight variations in depth predictions between adjacent frames, perceived as temporal flicker.

Future work could extend this pipeline to metric depth estimation and temporally consistent video depth estimation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.