acceptodds
Under review as a conference paper at ICLR 2027

PXDepth: Learning Pixel-Space Features for Structure-Preserving Monocular Depth Estimation

Abstract

Recent monocular depth estimators achieve strong zero-shot generalization, yet often struggle to preserve fine-grained structures. We attribute this limitation to the prevalent combination of large-patch ViT encoders and convolutional decoders. Coarse tokenization can weaken pixel-level cues, while convolutional decoding struggles to model low-frequency regions and high-frequency details simultaneously. To address this issue, we propose PXDepth, a discriminative monocular depth model that separates global context modeling from pixel-space feature learning for depth prediction. Specifically, a large-patch ViT captures global scene context, while a Pixel-Space Depth Predictor composed of Context-Modulated Pixel Transformer blocks learns pixel-space features throughout depth estimation. This design preserves fine structures without sacrificing global depth consistency. Across diverse zero-shot benchmarks, PXDepth substantially improves local geometric fidelity while retaining competitive global depth accuracy and lower inference latency than multi-step generative methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Monocular Depth Estimation, 3D vision, Pixel-Space Feature

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k4BvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a discriminative single-pass model for monocular depth estimation. The idea is to separate two roles: a large-patch ViT gives global context, and a second network works on full-resolution pixel features, guided by that context, to predict the final depth. The authors say that standard ViT + convolutional decoder models lose fine structures because depth is rebuilt from low res features. They claim better local geometry than discriminative models, competitive global accuracy, and lower latency than generative models. The model is trained only on synthetic data and evaluated zero-shot on several benchmarks, with both global and boundary metrics.

Critical strengths
  • The problem is relevant (fine structures and flying points in point clouds), and the motivation is clear. The paper is well written and easy to follow.
  • The idea is simple and works in a single forward pass, with no iterative sampling. Thin structures (eg poles) look better in many qualitative examples.
  • The evaluation is wide: many baselines, two benchmarks, and boundary metrics also on real data.
  • The authors are honest in the appendix: they compare with the closest recent works (Moge3, SurGe), they show the center-crop comparison with PointDiT, and they show worse results where they exist (for example Moge2 on global accuracy). The limitations are stated.
  • Compared with MDA (a much larger model), the boundary quality is similar with lower latency. Compared with generative pixel-space methods, the latency is clearly lower at the tested resolution.
  • Training and evaluation protocols are described in detail and the authors promise code and weights.
Critical weaknesses
  1. The comparison with the baselines is not controlled. PXDepth is trained only on synthetic data with its own loss, and starts from moge2, while the baselines have other data and other training. The ablation (table 3b) has a controlled comparison with a conv decoder, but at small scale, and the gain is small. So it is not clear how much of the gain in table 1-2 comes from the architecture. HiRoom CD is 54.9 in tab.2 and 72.5 in table 3, so the training setup changes the results more than the decoder. I would like to see MOGe2 (conv decoder) trained with the same data, loss and budget, in the main tables.

  2. It is not shown that the proposed idea is what matters. The only baseline is the moge2 decoder, which is much cheaper, so a bigger model could do better for any design. There is no baseline with similar compute and a simple design (for example rgb-guided upsampling, or a decoder with full-resolution skip connections on the same ViT). There is also no test that separates "full-resolution features" from "more transformer layers after the ViT" (for example the same predictor on a low-resolution stream, or with blurred RGB input). In table 3c the three guidance variants may have a different number of parameters, and no parameter count is given. If so, the gain of the guidance can be only capacity.

  3. The efficiency claim is shown only at low resolution. Latency is measured at 518x518 (48.9 ms), but at 1022x770 it is 193 ms. I expect the cost of the attention grow with the square of the number of tokens, so 2K or 4K images may be very expensive. This is the setting where fine structures matter more. The model is also 2x times slower than Moge, and the generative baselines are tested only with 3-4 steps (what about 1?). The paper should show latency, memory and accuracy vs resolution for all methods, in the same conditions, and parameters and FLOPs for all methods.

  4. The comparison with PointDiT depends on the protocol. With the center crop protocolPointDiT-large is better than PXDepth on average. In tabel 1 PointDiT is resized without keeping the aspect ratio and loses. So the claim of better accuracy than PointDiT is not valid in the other protocol, and this result is only in the appendix. Also PXDepth starts from MoGe-2, while PointDiT uses DINOv3 features and is trained from scratch, and the DINOv3 variant of PXDepth is clearly worse.

  5. Global accuracy is below the closest model...PXDepth is below moge2 (its own initialization) on 6 of 8 datasets in delta1. The differences are larger on real driving data . This could be the cost of synthetic training, but the paper only says mog2 has more diverse data and does not test it. The word "competitive" is correct for a small gap but it should be discussed more.

  6. The boundary gain over the best baseline is small. against moge2, Infinidepth, PPD and PointDiT the gain is large, but against MDA mean CD is 70.6 vs 72.8, Acc is worse and on 7Scenes (the only real one) MDA is also better in CD. Since CD is the average of acc and completeness the CD win may come from completeness. Please report completeness.

  7. the metrics do not directly measure the main claim. The claim is about structural distortion and local geometry, but the paper reports global metrics and boundary acc/CD (after ICP, on GT edge pixels). As far as I know, recent works on the same problem also use boundary F1/SEE (see marigold v2), local relative error and normal error. Distortion of thin structures or smooth surfaces and flying pixels are not measured directly. The MDA protocol is fine as a starting point, but alone it is not enough.

Some considerations about the eval protocol:

  • The inference resolution of the proposed method for the accuracy results is not stated. If it is about 1022x770, the latency in table 2 is not the right one. It is also not clear if the predicted masks are used in the boundary evaluation (and the masks of the baselines).
  • The naming is inconsistent: the caption of fig1 calls InfiniDepth, MDA and PPD "structure-aware", but tables 1-2 put them in "Global Prediction" and only "Ours" in "Structure Aware". So the sentence "better global accuracy than structure-aware methods" compares the method with itself.
  • Fig. 1 does not include moge2 (better delta1 and faster) and mixes numbers from two benchmarks. PointDiT is missing in the qualitative figures. MDA has a strange outlier on DDAD (rel 21.8) that lowers its mean..please check.

Related work / comparisons issues:

  • The natural baselines for "low-resolution features to full resolution" (guided or joint upsampling, learned convex upsampling) are not discussed.
  • PointDiT, Moge3 and Surge should be in the main tables, not in the appendix (I do not penalize the last two if they are concurrent).

Some minor points:

  • Missing details: batch size, data augmentation, initialization of the modulation layers, size of the feed-forward layers, etc.
  • The paper says latent diffusion models lose structure because of the VAE, but no such model is compared. A recent latent-DiT depth model that the paper already cites (for example Marigoldv2 or Lotus2) would test this claim.
  • The context features come from the sum of 4 layers (D.1), but the main text does not say it.
Questions for authors
  1. To show that the gain comes from the design: MoGe-2 (conv decoder) and PXDepth trained with the same data, loss and budget, in the main tables. Also a baseline with similar compute and a simple design (RGB-guided decoder), and the same predictor with a low-resolution stream. What is the real source of the boundary gain?
  2. Efficiency in the settings where the method should help: latency, memory and accuracy vs resolution (up to at least 2K), parameters and FLOPs of all methods, same measurement protocol, and the generative methods with fewer steps.
  3. Metrics that measure the claim: boundary F1, local error or normals, flying pixels, and completeness separately. Does the gain over MDA stay?
  4. PointDiT with its own protocol in the main tables. Does the claim of better accuracy still hold, also with a DINOv3-initialized PXDepth?
  5. How do you explain the drop on real data (DDAD, KITTI) with respect to MoGe-2? Can a small amount of real or pseudo-labeled data fix it?
  6. At which resolution are the accuracy numbers computed, and are predicted masks used in the boundary evaluation?

My current score is weak reject, mainly because the gains are not clearly attributable to the design (data, loss, capacity), and because the efficiency claim holds only at low resolution. The idea is reasonable and the real-data boundary results are interesting. I can raise the score if the rebuttal gives the controlled comparison with MoGe-2 and the simple-design baselines at similar compute.

POST REBUTTAL: In light of the authors’ rebuttal and the additional clarifications provided, I have decided to raise my rating from Weak Reject to Weak Accept. The rebuttal satisfactorily addresses my main concerns.

10 comments
AuthorsOpenReview ↗

Response to Reviewer k4Bv (1/4)

We appreciate the time and effort you devoted to this detailed review. Your comments have helped us strengthen the architectural validation and provide a more comprehensive assessment of geometric fidelity and computational efficiency.

W1, W2 & Q1: Sources of Geometric Fidelity Improvements: Synthetic-only training, Full-resolution Skip Connections, or Model Capacity?

A: Thank you for suggesting this controlled experiment, which is essential for validating our core claim. We retrain MoGe-2 with a randomly initialized convolutional decoder (MoGe-2-Synth) to ablate the effect of synthetic-only training. We also compare PXDepth with a baseline combining the MoGe-2 backbone and a full-resolution RGB-guided convolution-based decoder, with a total parameter count comparable to that of PXDepth. The RGB-guided decoder adopts a U-Net-style architecture with RGB skip connections, while global context from the MoGe-2 backbone is injected into the bottleneck through adaptive normalization. Further details are provided in Appendix D.5 and Figure 10. Both baselines use the same data, loss functions, and training budget as PXDepth.

We evaluate Rel and δ1\delta_1 on the MoGe Benchmark, and Acc, CD, boundary F1, Local Rel, and Local δ0.01\delta_{0.01} on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER. Rel and δ1\delta_1 assess global depth accuracy. Acc and CD evaluate 3D boundary geometry, boundary F1 measures 2D boundary alignment, and Local Rel and Local δ0.01\delta_{0.01} assess local geometric fidelity. The results have been added to Table 6 of the revised paper.

As shown in Table R1 and Fig. 7, the RGB-guided convolution-based baseline improves local geometric fidelity, indicating that increased model capacity and RGB skip connections can benefit local structure preservation. However, structural distortions caused by convolutional decoding still remain. Similarly, synthetic-only training slightly improves MoGe-2's local structures but cannot fundamentally resolve these distortions. In contrast, PXDepth achieves better boundary and local geometric fidelity while preserving fine-grained structures. These results support the contribution of our pixel-space design beyond increased model capacity, full-resolution skip connections, or synthetic-only training.

Table R1. Controlled comparison of PXDepth with RGB-guided decoder baselines.

MethodRel ↓\downarrowδ1↑\delta_1\uparrowAcc (mm) ↓\downarrowCD (mm) ↓\downarrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrowParams (M)
MoGe-25.1395.5990.1102.62.1163.99330.90
MoGe-2-Synth5.9794.5498.9104.22.1165.29330.90
MoGe-2 backbone + RGB-guided decoder6.1094.6174.490.52.0766.60701.43
Ours5.7594.8466.580.61.9171.22686.97
AuthorsOpenReview ↗

Response to Reviewer k4Bv (2/4)

W3 & Q2: Efficiency and Model Capacity Comparison Across Different Resolutions and Generative Sampling Steps

A: Thank you for suggesting these comparisons, which help make our evaluation fairer and more comprehensive.

We report parameter counts, FLOPs, latency, and memory usage across different input resolutions and numbers of generative sampling steps on an RTX 5880, as shown in Table R2 and Table R3. PXDepth achieves lower latency than the multi-step generative baselines at both resolutions. The results have been added to Table 12 in Appendix F.2 of the revised paper.

Our current high-resolution training stage targets approximately 1K resolution, comparable to the setting used by PPD, while PointDiT operates at a resolution of 512 × 512. Scaling structure-preserving depth prediction to 2K and beyond remains challenging, particularly when balancing structural fidelity and computational efficiency. We plan to explore higher-resolution training and inference in future work.

Table R2. Efficiency comparison at 518 × 518 resolution on an RTX 5880.

CategoryMethodParams (M)FLOPs (T)Latency (ms)Memory (GiB)
Global AccuracyDA V2335.31.320.41.7
DepthPro952.019.2223.73.7
MoGe-2330.91.420.51.6
Structure AwareInfiniDepth307.21.661.61.9
PointDiT-L (Step=1)771.51.551.83.8
PointDiT-L (Step=2)771.52.276.03.8
PointDiT-L (Step=3)771.52.9104.63.8
PointDiT-L (Step=4)771.53.7125.53.8
PPD (Step=1)804.23.272.24.0
PPD (Step=2)804.25.7118.04.1
PPD (Step=3)804.28.3163.04.1
PPD (Step=4)804.210.8206.94.1
MDA1355.74.0100.16.3
Ours687.03.348.92.9

Table R3. Efficiency comparison at 1022 × 770 resolution on an RTX 5880.

CategoryMethodParams (M)FLOPs (T)Latency (ms)Memory (GiB)
Global AccuracyDA V2335.34.956.52.4
DepthPro952.019.2223.73.7
MoGe-2330.95.171.52.4
Structure AwareInfiniDepth307.25.3163.72.1
PPD (Step=1)804.215.6287.64.5
PPD (Step=2)804.228.4451.34.5
PPD (Step=3)804.241.2621.74.5
PPD (Step=4)804.253.9783.94.5
MDA1355.714.2222.98.6
Ours687.015.4190.03.5

W6, W7 & Q3: Metrics that measure the claim: boundary F1, local error or normals, flying pixels, and completeness separately.

A: We thank the reviewer for suggesting these additional evaluation metrics, which enable a more comprehensive assessment of local structural fidelity. Following this suggestion, we additionally report completeness, boundary F1, Local Rel, Local δ0.01\delta_{0.01}, and Normal Error, together with the original Acc and CD metrics.

Specifically, boundary F1 evaluates the alignment of depth discontinuities in 2D. Acc, completeness, and CD measure geometric fidelity near object boundaries. Local Rel and Local δ0.01\delta_{0.01} assess local geometric fidelity. Normal Error evaluates surface orientation consistency. These metrics are evaluated on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER.

As shown in Table R4, PXDepth achieves the best completeness, CD, boundary F1, Local Rel, and Local δ0.01\delta_{0.01}, while maintaining competitive Acc and Normal Error. These results provide more direct evidence that PXDepth better preserves local structures and reduces structural distortions. We have added the average results to Table 2 and the per-dataset breakdowns to Tables 19 and 20 of the revised paper.

Table R4. Comprehensive evaluation of boundary fidelity and local geometric fidelity on five datasets.

CategoryMethodAcc (mm) ↓\downarrowCompleteness (mm) ↓\downarrowCD (mm) ↓\downarrowF1 (r=1r=1) ↑\uparrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrowNormal (∘^\circ)
Global PredictionDA V2102.7131.3117.016.652.0661.9521.91
DepthPro99.9122.7111.315.982.0363.7824.14
MoGe-290.1115.1102.613.272.1163.9922.16
Structure AwarePPD80.2155.0117.619.442.0563.8437.38
InfiniDepth92.9140.2116.519.322.1760.2624.59
MDA64.7104.784.718.222.4763.2627.04
SurGe71.997.584.718.202.0369.6020.23
MoGe-379.6110.495.015.402.0166.4621.13
Ours66.594.780.622.591.9171.2222.04
AuthorsOpenReview ↗

Response to Reviewer k4Bv (3/4)

W4 & Q4: Comparison with PointDiT and initialization of PXDepth

A: Thank you for pointing out this issue, which helps us make a fairer comparison with PointDiT. We have moved the comparison with PointDiT under its official center-crop protocol from the appendix to Tables 3 and 4 in the main paper, and removed the PointDiT results obtained under the mismatched evaluation protocol from Tables 1 and 2. PointDiT-L is restricted to a fixed input resolution of 512 × 512, whereas PXDepth supports varying input resolutions and aspect ratios. Under this shared evaluation protocol, PXDepth achieves better global depth accuracy and local geometric fidelity on the MDA Benchmark, although PointDiT-L achieves slightly better average global depth accuracy on the MoGe Benchmark, as shown in Tables 3 (Table R5) and 4 (Table R6). Regarding initialization, MoGe-2 initialization accelerates convergence, but the DINOv3-initialized PXDepth also produces structure-preserving depth predictions after convergence.

Table R5. Center-crop comparison with PointDiT on the MoGe Benchmark.

MethodRel ↓\downarrowδ1↑\delta_1\uparrow
PointDiT-L (3 steps)5.295.3
Ours5.595.0

Table R6. Center-crop comparison with PointDiT on the MDA Benchmark.

MethodRel ↓\downarrowδ1↑\delta_1\uparrowAcc ↓\downarrowCD ↓\downarrowF1 ↑\uparrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrowLatency (ms) ↓\downarrow
PointDiT-L (3 steps)3.497.177.975.422.901.9468.44104.6
Ours3.097.556.663.327.721.8371.9648.9

W5 & Q5: How do you explain the drop on real data (DDAD, KITTI) with respect to MoGe-2? Can a small amount of real or pseudo-labeled data fix it?

A: We attribute this gap mainly to our synthetic-only training, while MoGe-2 benefits from more diverse training data. This is also supported by our controlled comparison in Table 6 (as shown in Table R7), where MoGe-2-Synth shows lower global depth accuracy than the original MoGe-2. Incorporating real-world or pseudo-labeled data is a promising direction to mitigate this gap, which we plan to explore in future work. We have further discussed the global accuracy gap between MoGe-2 and PXDepth in Section 5.3 of the revised paper.

Table R7. Effect of synthetic-only training on global depth accuracy and local geometric fidelity.

MethodRel ↓\downarrowδ1↑\delta_1\uparrowAcc (mm) ↓\downarrowCD (mm) ↓\downarrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrow
MoGe-25.1395.5990.1102.62.1163.99
MoGe-2-Synth5.9794.5498.9104.22.1165.29
Ours5.7594.8466.580.61.9171.22

Q6: At which resolution are the accuracy numbers computed, and are predicted masks used in the boundary evaluation?

A: Sorry for the confusion. In the original paper, we already specified the dataset-specific evaluation resolutions in Appendix E.2 (Table 10). We evaluate accuracy at these resolutions. For boundary evaluation, PXDepth and all baselines use ground-truth masks rather than predicted masks.

Reviewer_bJ6eOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the task of pixel-accurate monocular depth estimation. The main argument is that VAEs and ViT Encoders compress the high frequency information needed to pixel pefect reconstruct a scene without flying pixels. The paper proposes PXDepth which employs a pixel feature embedding togther with a context encoder that circumvents the compression problem of previous methods.

Critical strengths

The paper adresses an important problem in monocular depth estimation. The architectural idea is clear and the empirical results seem to be strong and are acompanied with useful ablations.

Critical weaknesses
  • The paper claims that the method is explicitly structure preserving, but there is no evaluation of "structure preserving" through out the experiments. The current evaluation metrics are only measuring the boundery fidelity on GT.
Questions for authors

Q1: The main argument of the paper is that converntional method loose information during the tokenization, and PXDepth directly preserves RGB pixel features. This is a long standing problem in computer vision: like FPN/U-Nets also solve this problem via skip connections and HRNets with parallel representations. So my question is: Does the gain speciifically come from the pixel-space transformer learning or simply because PXDepth feeds the decoder a high resolution RGB connection and with substantially more capacity/compute? I think a controled experuiment against higher resolution RGB/skip branch with comparable capacity and compute would significally strengthen the story of the paper. Q2: The model has 687 parameters, and the you report the efficiently as wall-clock inference speed. For a fair comparison, could you also add a FLOP comparison? Can you please clarify how much additional capacity the pixel space depth predictor introduces compared to the decode-only control?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bj6e (1/2)

Thank you for your thoughtful assessment of our work. Your suggestions have helped us clarify the contribution of our pixel-space design through controlled comparisons and a broader efficiency analysis.

W1: "Structure-preserving" metrics

A: Thanks for suggesting a more comprehensive evaluation. We additionally report the boundary F1 score (radius = 1) following DepthPro, together with Acc and CD. Moreover, to further evaluate the local structure of the proposed method, we additionally report Local Rel and Local δ0.01\delta_{0.01} following MoGe-3 on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER, as shown in Table R1.

Boundary F1 mainly evaluates the alignment of depth discontinuities in 2D, while the other metrics assess boundary geometry and local geometric fidelity. PXDepth achieves the best CD, boundary F1, Local Rel, and Local δ0.01\delta_{0.01} among the compared methods. These results support the effectiveness of PXDepth in preserving depth boundaries and local geometric structures. We have added the average results to Table 2 and the per-dataset breakdowns to Tables 19 and 20 of the revised paper.

Table R1. Extended evaluation of boundary fidelity and local geometric fidelity on five datasets.

CategoryMethodAcc (mm) ↓\downarrowCD (mm) ↓\downarrowF1 (r=1r=1) ↑\uparrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrow
Global PredictionDA V2102.7117.016.652.0661.95
DepthPro99.9111.315.982.0363.78
MoGe-290.1102.613.272.1163.99
Structure AwarePPD80.2117.619.442.0563.84
InfiniDepth92.9116.519.322.1760.26
MDA64.784.718.222.4763.26
SurGe71.984.718.202.0369.60
MoGe-379.695.015.402.0166.46
Ours66.580.622.591.9171.22

Q1: Sources of Geometric Fidelity Improvements: Pixel-Space Transformer, RGB Skip Connections, or Model Capacity?

A: Thank you for suggesting this controlled experiment, which is essential for validating our core claim. We compare PXDepth with a baseline combining the MoGe-2 backbone and a full-resolution RGB-guided convolution-based decoder, with a total parameter count comparable to that of PXDepth. The RGB-guided decoder adopts a U-Net-style architecture with RGB skip connections, while global context from the MoGe-2 backbone is injected into the bottleneck through adaptive normalization. Further details are provided in Appendix D.5 and Figure 10. We additionally retrain MoGe-2 with a randomly initialized convolutional decoder (MoGe-2-Synth) to ablate the effect of synthetic-only training. Both baselines use the same data, loss functions, and training budget as PXDepth.

We evaluate Rel and δ1\delta_1 on the MoGe Benchmark, and Acc, CD, boundary F1, Local Rel, and Local δ0.01\delta_{0.01} on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER. Rel and δ1\delta_1 assess global depth accuracy. Acc and CD evaluate 3D boundary geometry, boundary F1 measures 2D boundary alignment, and Local Rel and Local δ0.01\delta_{0.01} assess local geometric fidelity.

As shown in Table R2 and the point-cloud visualizations in Fig. 7 of the revised paper, the RGB-guided convolution-based baseline improves local geometric fidelity, indicating that increased model capacity and RGB skip connections can benefit local structure preservation. However, structural distortions caused by convolutional decoding still remain. Similarly, synthetic-only training slightly improves MoGe-2's local structures but cannot fundamentally resolve these distortions. In contrast, PXDepth achieves better boundary and local geometric fidelity while preserving fine-grained structures. These results support the contribution of our pixel-space design beyond increased model capacity, high-resolution skip connections, or synthetic-only training. The results have been added to Table 6 of the revised paper.

Table R2. Controlled comparison of PXDepth with convolution-based baselines.

MethodRel ↓\downarrowδ1↑\delta_1\uparrowAcc (mm) ↓\downarrowCD (mm) ↓\downarrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrowParams (M)
MoGe-25.1395.5990.1102.62.1163.99330.90
MoGe-2-Synth5.9794.5498.9104.22.1165.29330.90
MoGe-2 backbone + RGB-guided decoder6.1094.6174.490.52.0766.60701.43
Ours5.7594.8466.580.61.9171.22686.97
AuthorsOpenReview ↗

Response to Reviewer bj6e (2/2)

Q2: Computational Efficiency and Model Capacity

A: Thank you for suggesting these comparisons, which help make our evaluation fairer and more comprehensive. We report parameter counts, FLOPs, latency, and memory usage at different input resolutions on an RTX 5880, as shown in Table R3 and Table R4. PXDepth achieves lower latency than the multi-step generative baselines at both resolutions. Compared with the convolutional-decoder control, PXDepth introduces an additional 356M parameters. Nevertheless, PXDepth has fewer parameters than generative methods such as PPD (804.2M), while offering a favorable trade-off between local geometric fidelity and computational cost. The results have been added to Table 12 in Appendix F.2 of the revised paper.

Table R3. Efficiency comparison at 518 × 518 resolution on an RTX 5880.

CategoryMethodParams (M)FLOPs (T)Latency (ms)Memory (GiB)
Global AccuracyDA V2335.31.320.41.7
DepthPro952.019.2223.73.7
MoGe-2330.91.420.51.6
Structure AwareInfiniDepth307.21.661.61.9
PointDiT-L (Step=1)771.51.551.83.8
PointDiT-L (Step=2)771.52.276.03.8
PointDiT-L (Step=3)771.52.9104.63.8
PointDiT-L (Step=4)771.53.7125.53.8
PPD (Step=1)804.23.272.24.0
PPD (Step=2)804.25.7118.04.1
PPD (Step=3)804.28.3163.04.1
PPD (Step=4)804.210.8206.94.1
MDA1355.74.0100.16.3
Ours687.03.348.92.9

Table R4. Efficiency comparison at 1022 × 770 resolution on an RTX 5880.

CategoryMethodParams (M)FLOPs (T)Latency (ms)Memory (GiB)
Global AccuracyDA V2335.34.956.52.4
DepthPro952.019.2223.73.7
MoGe-2330.95.171.52.4
Structure AwareInfiniDepth307.25.3163.72.1
PPD (Step=1)804.215.6287.64.5
PPD (Step=2)804.228.4451.34.5
PPD (Step=3)804.241.2621.74.5
PPD (Step=4)804.253.9783.94.5
MDA1355.714.2222.98.6
Ours687.015.4190.03.5

Note that PointDiT-L only accepts inputs at a fixed resolution of 512 × 512.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer bJ6e,

Thank you again for your valuable feedback and thoughtful review.

We hope that our previous response has addressed your questions and concerns. We would greatly appreciate hearing your thoughts on our response. If you have any remaining questions or concerns, please feel free to let us know. We are always available and would be more than happy to promptly address them and provide any further clarifications.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_EBeqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies the loss of pixel-level cues caused by spatially compressed representations in existing transformer-based discriminative monocular depth estimation models. To address this issue, the authors propose a Context-Modulated Pixel Transformer that effectively utilizes pixel-level cues to improve geometric fidelity while largely maintaining depth estimation accuracy.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The proposed Context-Modulated Pixel Transformer block improves fine detail structure and local geometric fidelity with a moderate increase in inference time.
  3. The paper identifies the limitations of spatially compressed encoder representations in existing discriminative MDE models.
Critical weaknesses
  1. Compared with MoGe-2, PXDepth shows lower performance on most global depth metrics but improved geometric fidelity. This resembles the trade-off observed in MoGe-2 paper when training only on synthetic data. It is unclear how much of the improvement comes from the proposed architecture rather than the synthetic-only training.
  2. The point-cloud visualizations show strong geometric fidelity, but the depth maps in Figure 8 exhibit visible staircase artifacts. It is unclear how the point clouds were generated and visualized.
Questions for authors
  1. Could you provide edge-related metric (e.g. boundary F1 score proposed in Depth Pro) to further evaluate the artifacts observed in the depth maps?
  2. Could you provide the performance comparison of "DINOv3-initialized" and "MoGe-2-initialized" models in Table 3 on real-world datasets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EBeq (1/2)

We sincerely appreciate your careful review and constructive feedback. Your questions about synthetic-only training and geometric evaluation have helped us strengthen the evidence for the structural improvements of PXDepth.


W1: Effect of Synthetic-Only Training on Geometric Fidelity

A: Thanks for raising this important question. Local structural distortions in existing models, including MoGe-2, have also been observed in recent works such as PPD, SurGe, and MoGe-3.

To investigate the effect of synthetic-only training on geometric fidelity, we retrain MoGe-2 with a randomly initialized convolutional decoder using the same data, loss functions, and training budget as PXDepth. We evaluate global depth accuracy using Rel and δ1\delta_1 on the MoGe Benchmark. We also evaluate boundary F1 (radius = 1) following DepthPro, and Local Rel and Local δ0.01\delta_{0.01} following MoGe-3 on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER.

As shown in Table R1, synthetic-only training does not substantially improve MoGe-2’s local geometric fidelity. The qualitative comparisons in Fig. 7 show that local structural distortions persist in both MoGe-2 and MoGe-2-Synth. These results indicate that synthetic-only training and increased model capacity alone are insufficient to resolve these structural distortions and cannot fully explain PXDepth’s gains in structural fidelity. These results have been added to Table 6 and Figure 7 of the revised paper.

Table R1. Effect of synthetic-only training on geometric fidelity.

MethodRel ↓\downarrowδ1↑\delta_1\uparrowAcc (mm) ↓\downarrowCD (mm) ↓\downarrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrow
MoGe-25.1395.5990.1102.62.1163.99
MoGe-2-Synth5.9794.5498.9104.22.1165.29
Ours5.7594.8466.580.61.9171.22

W2: Staircase artifacts in point-cloud visualizations

A: Sorry for the confusion. For the KITTI examples in Figure 8 (now Figure 9 in the revised paper), both models produce depth maps at a resolution of 1610 × 490, which are then resized to the original image resolution of 1242 × 375. In the original figure, the left depth map was resized using bilinear interpolation, while the right one used nearest-neighbor interpolation, causing the visible staircase artifacts on the right. We have updated Figure 9 to use bilinear interpolation for both visualizations. We follow a standard pipeline for point-cloud visualization, back-projecting the predicted depth maps into 3D using the camera intrinsics and rendering the resulting points.

AuthorsOpenReview ↗

Response to Reviewer EBeq (2/2)

Q1: Edge-related metric

A: Thanks for suggesting a more comprehensive evaluation. We additionally report the boundary F1 score (radius = 1) following DepthPro together with Acc and CD. Moreover, to further evaluate the local structure of the proposed method, we additionally report Local Rel and Local δ0.01\delta_{0.01} following MoGe-3 on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER, as shown in Table R2.

Boundary F1 mainly evaluates the alignment of depth discontinuities in 2D, while the other metrics assess boundary geometry and local geometric fidelity. PXDepth achieves the best CD, boundary F1, Local Rel, and Local δ0.01\delta_{0.01} among the compared methods. These results support the effectiveness of PXDepth in preserving depth boundaries and local geometric structures. We have added the average results to Table 2 and the per-dataset breakdowns to Tables 19 and 20 of the revised paper.

Table R2. Extended evaluation of boundary fidelity and local geometric fidelity on five datasets.

CategoryMethodAcc (mm) ↓\downarrowCD (mm) ↓\downarrowF1 (r=1r=1) ↑\uparrowLocal Rel ↓\downarrowLocal δ0.01↑\delta_{0.01}\uparrow
Global PredictionDA V2102.7117.016.652.0661.95
DepthPro99.9111.315.982.0363.78
MoGe-290.1102.613.272.1163.99
Structure AwarePPD80.2117.619.442.0563.84
InfiniDepth92.9116.519.322.1760.26
MDA64.784.718.222.4763.26
SurGe71.984.718.202.0369.60
MoGe-379.695.015.402.0166.46
Ours66.580.622.591.9171.22

Q2: The performance comparison of "DINOv3-initialized" and "MoGe-2-initialized" models in Table 3 on real-world datasets

A: We additionally compare DINOv3-initialized and MoGe-2-initialized PXDepth on the real-world iBims-1 dataset, as shown in Table R3. We observe that MoGe-2 initialization leads to faster convergence, while both variants achieve comparable final performance in most metrics.

Table R3. Comparison of DINOv3 and MoGe-2 initialization on the real-world dataset.

MethodRelδ1\delta_1Acc (mm)CD (mm)
DINOv3-initialized3.6897.66135.76207.1
MoGe-2-initialized3.5197.81118.25208.99

Thank you for the responses of authors. My concerns have been adequately addressed, and I will maintain my original positive assessment of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely appreciate the time and effort devoted by the ACs, SACs, PCs, and all reviewers to evaluating our work. We are encouraged that our method "improves fine detail structure and local geometric fidelity" (EBeq), that "The architectural idea is clear and the empirical results seem to be strong" (bJ6e), and that "The idea is simple and works in a single forward pass" (k4Bv). Your constructive feedback has helped us strengthen the experimental validation and clarify the contributions of PXDepth.

In response to the reviewers' comments, we have provided additional experimental results and revised the manuscript accordingly. Our main additions are summarized below.

  1. Controlled comparisons to identify the source of geometric improvements. A key concern is whether PXDepth's geometric improvements come from its pixel-space design or simply from synthetic-only training, RGB skip connections, or increased model capacity. To address this, we conduct two controlled experiments using MoGe-2-Synth and a full-resolution RGB-guided convolutional decoder with comparable model capacity. The results show that synthetic-only training and RGB-guided convolutional decoding can improve local structures, but structural distortions still remain. In contrast, PXDepth better preserves fine-grained structures, supporting the contribution of our pixel-space design.

  2. Expanded efficiency analysis across resolutions and sampling steps. We reported FLOPs, latency, and memory usage across different resolutions and sampling steps, including per-dataset measurements at the actual evaluation resolutions.

  3. More comprehensive evaluation of local geometric fidelity. We supplement Acc and CD with boundary F1, Local Rel, Local δ0.01\delta_{0.01}, and Normal Error on 7Scenes, HiRoom, NRGBD, iBims-1, and HAMMER. PXDepth achieves the best CD, boundary F1, Local Rel, and Local δ0.01\delta_{0.01} among the compared methods, while maintaining Acc comparable to MDA. We also report Completeness separately for the original boundary evaluation, showing that PXDepth provides more complete geometric coverage of ground-truth boundary regions than MDA. These complementary evaluations provide stronger evidence of structural preservation beyond boundary geometry alone.

Together, these additions strengthen the evidence for our pixel-space design and provide a more complete assessment of its structural fidelity and computational cost. We hope they help clarify the contribution of PXDepth and address the reviewers' main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.