acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Grounded Video Depth Estimation

Abstract

Our goal is to infer temporally and geometrically consistent depth maps that lift monocular video clips into 3D. To this end, we develop , a geometry-grounded feed-forward framework that fills in three missing pieces in video depth estimation, , cross-frame modelling, geometric supervision, and data scaling. First, instead of most existing efforts relying on fixed-location feature aggregation or externally estimated correspondences, we propose to adaptively locate a sparse set of query-relevant features for efficient cross-frame information aggregation under camera and object motion. Second, we introduce to directly constrain intra-frame structure and inter-frame geometry after back-projection of depth maps. Third, to scale geometric supervision to unlabeled videos, we develop , which distills geometric knowledge from a reconstruction teacher to provide pseudo 3D supervision, yielding a more scalable solution. Extensive experiments on standard benchmarks demonstrate that G2 achieves state-of-the-art performance on both zero-shot video depth estimation and geometric reconstruction. We will release the code and model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Depth Estimation, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r14iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents G2, a geometry-grounded feed-forward framework for temporally consistent monocular video depth estimation. The framework introduces three components to improve cross-frame feature aggregation, geometric consistency, and training-data scalability.

First, Motion-Adaptive Implicit Alignment (MAIA) uses query-conditioned sparse sampling in a spatiotemporal feature volume to aggregate information across frames without relying on external optical flow or point tracking. Second, Explicit 3D Supervision (E3DS) introduces intra-frame point-map supervision and inter-frame projection losses to improve the geometric consistency of predicted depth maps. Third, Multi-View Geometric Distillation (MVGD) transfers geometric knowledge from a pretrained multi-view reconstruction teacher to unlabeled videos through pseudo 3D supervision and feature-level distillation.

The framework is initialized from Video Depth Anything and evaluated on zero-shot video depth estimation and geometric reconstruction benchmarks. The results demonstrate competitive depth accuracy, particularly under scale-invariant evaluation, and improved geometric reconstruction compared with several existing methods.

Critical strengths
  • The paper is well-written and easy to follow.
  • The paper addresses an important limitation of video depth estimation by improving cross-frame geometric consistency for downstream 3D applications.
  • The method achieves strong zero-shot depth and reconstruction performance, particularly on 7-Scenes and NRGBD.
  • The framework coherently combines adaptive temporal aggregation, geometric constraints, and teacher-guided learning.
  • Comprehensive component-level ablations clarify the effects of temporal aggregation, sampling strategies, geometric supervision, and distillation.
Critical weaknesses
  • MAIA has limited methodological novelty and closely resembles deformable attention. In particular, Deformable DETR (ICLR 2021) already introduced query-dependent sparse sampling around reference locations using learned offsets. MAIA appears to extend this formulation to a spatiotemporal feature volume, so a direct comparison with deformable attention adapted to the same video setting is needed.

  • The geometric novelty of E3DS is insufficiently justified. Because predicted and target point maps derived from depth share the same camera rays, the intra-frame point-map loss may largely reduce to geometry-weighted depth regression. Meanwhile, the inter-frame projection loss resembles established cross-view reprojection constraints. Comparisons with ray-weighted depth regression and standard reprojection losses would better isolate its contribution.

  • MAIA’s claimed ability to handle camera and object motion is not directly validated. Its offsets are predicted from feature embeddings, but the paper does not show whether they correspond to meaningful motion-displaced observations. Motion-stratified evaluations, learned-offset visualizations, and analyses of validity masks on independently moving objects would strengthen this claim.

  • Multi-View Geometric Distillation using the pretrained Pi3 teacher is not clearly distinguished from conventional pseudo-label training. The reported gains from teacher-guided geometry and feature distillation are modest, and a feature-distillation-only ablation is missing. Comparisons with simple pseudo-depth supervision, the frozen Pi3 teacher, and matched unlabeled-data baselines are needed.

  • The related-work positioning against recent multi-view geometry foundation models is incomplete. Depth Anything 3 (ICLR 2026), VGGT (CVPR 2025), and DUSt3R (CVPR 2024) are particularly relevant to spatially consistent depth, point-map estimation, and joint multi-view geometric reasoning. The paper should clarify how G2 differs from these models and what practical advantages its specialized video-depth architecture provides.

  • Computational and statistical analyses are insufficient. The paper should report MAIA’s runtime, peak memory, parameter overhead, and scaling with video length relative to the Video Depth Anything backbone. Multi-seed results, confidence intervals, or significance tests are also needed because several component-level improvements are relatively small.

Questions for authors
  • How does MAIA fundamentally differ from deformable attention, particularly Deformable DETR (ICLR 2021)? Could the authors compare it with a conventional deformable-attention module extended to the same spatiotemporal volume under matched training and computational settings?

  • Since back-projected point-map errors are determined by depth errors along known camera rays, what additional geometric information does the intra-frame point-map loss provide? Please compare it with ray-weighted depth regression using matched weights and robust penalties.

  • Do MAIA’s learned offsets represent meaningful cross-frame motion or scene-point correspondences? Offset visualizations, results stratified by static regions, moving objects, and camera motion, and statistics on dynamic pixels excluded by the validity mask would help.

  • Could the authors provide a feature-distillation-only ablation and compare MVGD with conventional pseudo-depth distillation using the same Pi3 teacher, unlabeled data, and training budget? Where applicable, please also report the frozen Pi3 teacher’s performance.

  • How does G2 compare with Depth Anything 3 (ICLR 2026) and VGGT (CVPR 2025) in geometric consistency, depth accuracy, and efficiency? If direct comparison is inappropriate, please clarify the differences in task assumptions and intended applications.

  • What parameter, FLOP, memory, and latency overhead does MAIA introduce over Video Depth Anything, and how does it scale with video length? Please also report confidence intervals or multi-seed results for the main comparisons and ablations.

9 comments
AuthorsOpenReview ↗

Responses to Reviewer r14i (1/6)

We thank Reviewer r14i for recognizing the importance of cross-frame geometric consistency, the strong reconstruction results on 7-Scenes and NRGBD, and the value of the component ablations. We appreciate your suggestions for more direct comparisons and for clarifying what each component contributes.


W1/Q1: "How does MAIA fundamentally differ from deformable attention, particularly Deformable DETR (ICLR 2021)? Could the authors compare it with a conventional deformable-attention module extended to the same spatiotemporal volume under matched training and computational settings?"

A1: Thank you for this valuable suggestion. Extending conventional deformable attention to a spatiotemporal feature volume provides a generic sampling mechanism, but it does not inherently address the geometric correspondence and spatiotemporal support required for video depth estimation. Directly applying such an extension therefore results in inferior performance. Our design builds on Deformable DETR [1] and DAT [2]. MAIA uses query-conditioned residual offsets to refine staggered spatiotemporal anchors, with content-based query–key aggregation over valid samples. We clarify the key design differences and provide the requested controlled comparisons below.

First, MAIA uses staggered spatiotemporal anchors to organize complementary spatial and temporal support. Within each attention head, the preset sampling locations share a fixed spatial radius and are distributed evenly across neighboring temporal frames, which are then refined by learned query-conditioned residual offsets (Appendix B.2, Lines 881–895, Eq. 18). By contrast, a 3D adaptation of DAT relies on a uniformly spaced spatiotemporal reference grid, whereas a 3D adaptation of Deformable DETR uses an initial radial sampling layout centered on the reference point. This prior ensures balanced coverage across multiple spatial directions and temporal strides, reducing reliance on a single spatiotemporal layout and avoiding rigid coupling between spatial angle and temporal offset.

Second, MAIA computes aggregation weights from the content of valid sampled features rather than from the query alone. Following the content-based aggregation principle in DAT, MAIA pairs query-specific offset prediction with attention weights derived from query–key similarity, and masks out sampling positions falling outside the video feature volume so that only valid samples contribute. In contrast, Deformable DETR predicts both sampling offsets and attention weights via linear projections of the query without comparing the query against the sampled keys. For video depth estimation, query–key similarity is crucial because it directly evaluates how each piece of sampled geometric evidence contributes to depth recovery. The controlled comparisons below evaluate the resulting module as a whole.

Third, to ensure a fair comparison, we extend Deformable DETR and DAT to 3D space and compare MAIA against them. All variants share the exact same configuration. We report scale-invariant (SI) aligned depth accuracy and TAE, followed by reconstrcuction metrics.

Attention variantBonn AbsRel ↓Bonn δ₁ ↑ScanNet AbsRel ↓ScanNet δ₁ ↑Sintel AbsRel ↓Sintel δ₁ ↑Bonn TAE (%) ↓Sintel TAE (%) ↓
MAIA(ours)0.05980.98000.07970.94450.34790.63212.74928.612
3D Deformable DETR0.07280.97530.08440.93800.41060.57402.89229.291
3D DAT0.07080.97470.08340.93940.39150.58832.91629.299
Attention variant7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
MAIA(ours)0.02200.01140.79480.05810.02060.9620
3D Deformable DETR0.02630.01200.78900.07490.02300.9546
3D DAT0.02720.01210.78270.08320.02320.9549

MAIA lowers SI TAE on Bonn and Sintel and improves all six reconstruction metrics relative to both 3D alternatives. Compared with 3D Deformable DETR, it lowers Bonn SI TAE by 4.9% and NRGBD accuracy error by 22.4%.

In the revised manuscript, Sec. 3.1 explains the learned offsets and aggregation over valid samples (Lines 164–195). Sec. 4.4 presents the matched Bonn and 7-Scenes comparison and the residual-offset ablation (Lines 452–471, Tables 4 and 5). Appendix D.3 gives the profiling setup and results (Lines 994–1003, Table 11).

[1] Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICLR 2021. [2] Vision Transformer with Deformable Attention. In CVPR 2022.

AuthorsOpenReview ↗

Responses to Reviewer r14i (2/6)

W2/Q2: "Since back-projected point-map errors are determined by depth errors along known camera rays, what additional geometric information does the intra-frame point-map loss provide? Please compare it with ray-weighted depth regression using matched weights and robust penalties."

A2: Thank you for asking about the supervision provided by the E3DS. For labeled data, predicted and target depths are back-projected along the same camera ray, so the resulting 3D point difference equals the depth difference scaled by ray vector. Applying the same robust penalty to each ray-scaled residual component gives the same objective when masking, normalization, and weights match. Therefore, the intra-frame supervision simply parameterizes depth supervision in camera coordinates without introducing additional geometric information. We adopt the point-map formulation to unify both intra-frame and inter-frame supervision within a common 3D representation. The inter-frame supervision projects the source prediction into the target view, where it is supervised by the target-frame ground truth. Standard reprojection instead compares projected depth with the target-frame prediction.

The E3DS and ray-weighted depth regression runs start from identical model weights and use the same robust penalty and inter-frame projection loss. The comparison varies only the formulation of the intra-frame supervision.

We report SI depth metrics and TAE, followed by reconstruction metrics.

Supervision settingBonn AbsRel ↓Bonn δ₁ ↑ScanNet AbsRel ↓ScanNet δ₁ ↑Sintel AbsRel ↓Sintel δ₁ ↑Bonn TAE (%) ↓Sintel TAE (%) ↓
E3DS(ours)0.05980.98000.07970.94450.34790.63212.74928.612
Ray-weighted depth regression0.06330.97820.07990.94390.34950.61782.76528.970
Standard reprojection loss0.06680.97750.08240.94210.36330.59912.80429.994
Supervision setting7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
E3DS(ours)0.02200.01140.79480.05810.02060.9620
Ray-weighted depth regression0.02310.01180.79420.05820.02120.9610
Standard reprojection loss0.02570.01210.79280.07560.02320.9611

E3DS lowers NRGBD completeness error to 0.0206 from 0.0212 with ray-weighted depth regression and 0.0232 with standard reprojection. E3DS also gives the lowest SI TAE on Bonn (2.749%) and Sintel (28.612%).

In the revised version, Appendix B.4 derives the ray-scaled point-map loss (Lines 915–929, Eq. 19), and Sec. 3.2 explains inter-frame supervision against target-view ground truth (Lines 223–231). Sec. 4.4 reports the comparisons with ray-weighted depth regression and standard reprojection (Lines 500–507, Table 7).


W3/Q3: "Do MAIA’s learned offsets represent meaningful cross-frame motion or scene-point correspondences? Offset visualizations, results stratified by static regions, moving objects, and camera motion, and statistics on dynamic pixels excluded by the validity mask would help."

A3: Thank you for asking how MAIA's learned offsets relate to cross-frame motion. For each clip, we visualize the final sampling locations for a single query in the second MAIA block, obtained by refining preset locations with query-conditioned residual offsets. In Fig. 4(a), several samples remain near the same junction of the Sydney Opera House sails as the viewpoint changes. In Fig. 4(b), several samples fall on or near the moving animal's torso in neighboring frames. These examples illustrate that MAIA can adaptively sample corresponding scene regions under camera and object motion.

In the residual-offset ablation, adding the learned offsets to the preset anchors increases Bonn's SI-aligned δ1\delta_1 from 0.967 to 0.980.

The E3DS validity mask filters rigid projections that disagree with the target geometry. Independent motion and occlusion can cause such inconsistencies. Estimating the fraction of dynamic pixels rejected by the mask requires separate pixel-level motion labels. We count rejections by the 3D consistency test over all in-bounds projections with valid source depth and valid target depth at the projected location.

DatasetRejection rate
TartanAir11.87%
MVS-Synth12.07%
Virtual KITTI10.98%
PointOdyssey14.76%

In the revised version, Appendix D.2 discusses MAIA sampling under camera and object motion (Lines 987–990, Fig. 4), and Sec. 4.4 reports the residual-offset ablation (Lines 464–471, Table 5). Appendix B.3 defines eligible projections and presents the manuscript's eight-frame mask rejection diagnostic (Lines 899–911, Table 9).

AuthorsOpenReview ↗

Responses to Reviewer r14i (3/6)

W4/Q4-1: "Could the authors provide a feature-distillation-only ablation and compare MVGD with conventional pseudo-depth distillation using the same π3\pi^3 teacher, unlabeled data, and training budget?"

A4-1: Thank you for requesting these controls. All five settings retain the same synthetic supervision. In the new comparisons, we match all the training configuration across variants. Synthetic-only continuation sets the distillation weights to zero. π3\pi^3 [3] depth distillation uses the teacher's depth pseudo-labels. MVGD feature distillation only uses the scene-level feature relation loss. Geometry-only distillation uses the teacher-guided geometric loss L3Dtea\mathcal{L}_{\mathrm{3D}}^{\mathrm{tea}}, which draws on teacher depth, point maps, and camera estimates. MVGD combines the feature and geometric losses.

We report depth accuracy, TAE and reconstruction metrics under scale-invariant alignment.

Supervision settingBonn AbsRel ↓Bonn δ₁ ↑ScanNet AbsRel ↓ScanNet δ₁ ↑Sintel AbsRel ↓Sintel δ₁ ↑Bonn TAE (%) ↓Sintel TAE (%) ↓
Synthetic-only continuation0.05550.98200.08330.93920.20100.68392.79030.144
π3\pi^3 depth distillation0.05580.98240.07380.95010.20590.68932.60633.206
MVGD feature distillation0.05520.98210.08220.94000.19800.69202.79132.569
Geometry-only distillation0.05190.98360.07750.94550.20080.68772.63528.563
MVGD(ours)0.05080.98440.07590.94920.19760.69952.53928.302
Supervision setting7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
Synthetic-only continuation0.02280.01140.79260.10570.06840.9465
π3\pi^3 depth distillation0.02270.01150.79160.05860.02710.9526
MVGD feature distillation0.02390.01190.79320.10730.06910.9469
Geometry-only distillation0.02230.01140.79510.05890.01930.9648
MVGD(ours)0.02150.01130.79390.05750.01990.9622

Compared with pseudo-depth distillation using the π3\pi^3 teacher, MVGD demonstrates superior performance on Sintel and NRGBD—reducing Sintel TAE by 14.8% and NRGBD accuracy error by 2.0%, while only slightly trailing π3\pi^3 depth distillation on ScanNet in AbsRel and δ1\delta_1. In addition, relative to synthetic-only continuation, MVGD reduces TAE on Bonn and Sintel, as well as accuracy error across the two reconstruction benchmarks.

In the revised version, Sec. 3.4 describes the data and schedule for second-stage training (Lines 296–308). Sec. 4.4 describes the synthetic-only continuation control and compares feature distillation, geometry-only distillation, and depth-map distillation with MVGD on Bonn and 7-Scenes (Lines 509–517, Table 8).

[3] π3\pi^3: Permutation-Equivariant Visual Geometry Learning. In ICLR 2026.


W4/Q4-2: "Where applicable, please also report the frozen π3\pi^3 teacher’s performance."

A4-2: We report the frozen π3\pi^3 teacher's depth results on Bonn and Sintel under the π3\pi^3 evaluation protocol, and its reconstruction results on 7-Scenes and NRGBD (mean / median). ScanNet is part of the teacher's training data, so we omit it from zero-shot comparison.

TeacherBonn AbsRel ↓Bonn δ₁ ↑Sintel AbsRel ↓Sintel δ₁ ↑
π3\pi^30.04970.97530.23390.6642
Teacher7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
π3\pi^30.016 / 0.0070.022 / 0.0110.689 / 0.7920.015 / 0.0080.013 / 0.0050.898 / 0.987

In the revised version, Appendix D.8 describes the Bonn and Sintel depth protocol (Lines 1107–1110) and the dense reconstruction protocol (Lines 1115–1119). The frozen π3\pi^3 teacher's depth and reconstruction results appear in Tables 17 and 18, respectively.

Reviewer_nww2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes G2, a video depth estimation framework that aims to improve cross-frame consistency and 3D geometry after back-projection. MAIA aggregates cross-frame features through learnable spatiotemporal sampling; E3DS supervises geometry with intra-frame point-map and inter-frame projection losses; MVGD uses a frozen Pi3 teacher to provide geometric pseudo-labels and feature-relation supervision on real unlabeled videos. The experiments cover three video depth datasets and two geometric reconstruction datasets, reporting competitive depth and reconstruction results. The reconstruction evaluation uses ground-truth camera intrinsics and extrinsics.

Critical strengths
  1. The supervision targets the 3D geometric quality of video depth predictions, with a clear motivation. Table 7 shows that both the intra-frame and inter-frame losses contribute; when combined, the 7-Scenes SSI/SI Acc. decreases from 0.031/0.029 to 0.025/0.022.

  2. The method achieves competitive results across multiple datasets. G2 obtains an Acc. of 0.057 on NRGBD under both SSI and SI alignment; Tables 4–5 also show gains from MAIA over standard temporal attention and fixed-anchor sampling.

Critical weaknesses
  1. The source of the temporal improvements is not sufficiently explained. In Table 2, G2's ScanNet SSI TAE is 0.795, worse than VDA's 0.549. The ablations in Tables 4–9 report only depth and reconstruction metrics, without TAE. It is therefore unclear how MAIA, inter-frame geometric supervision, and distillation each affect temporal consistency, and the degradation above is not explained.
  2. The gains from MVGD geometric distillation are not sufficiently separated from the effects of additional training on real videos. In Table 9, adding only teacher geometric supervision leaves all four reconstruction metrics unchanged from the no-distillation baseline at the displayed precision; adding feature distillation then reduces each of the two Acc. values by 0.001. The paper does not report the actual amount of unlabeled data or state whether the no-distillation baseline receives the same number of training updates. The evidence is therefore insufficient to determine whether the gains come from geometric distillation or additional data and training.
Questions for authors
  1. Why does G2 perform worse than VDA on ScanNet SSI TAE? Please explain the temporal degradation in this setting.
  2. Could you also report TAE in the MAIA, E3DS, and MVGD ablations to show each module's effect on temporal consistency? In particular, does the inter-frame projection loss improve temporal consistency?
  3. Does the no-distillation baseline in Table 9 receive the same number of training updates? Using the same real videos and training budget, how does supervision with only Pi3 depth pseudo-labels compare with MVGD? This would help distinguish the effects of geometric distillation from those of training with additional data.
6 comments
AuthorsOpenReview ↗

Responses to Reviewer nww2 (1/3)

We sincerely thank Reviewer nww2 for the constructive feedback and for recognizing the motivation for geometric supervision and the competitive depth and reconstruction results. We provide point-by-point responses below, with particular attention to the ScanNet temporal result, module-level temporal ablations, and the training controls for distillation.


W1/Q1: "Why does G2 perform worse than VDA on ScanNet SSI TAE? Please explain the temporal degradation in this setting."

A1: Thank you for highlighting this result. G2 builds on VDA and improves all other reported depth, 3D reconstruction, and TAE metrics relative to VDA. Our ScanNet analysis explains this exception through different gains from sequence-level alignment driven by distinct training objectives and dataset-specific error structures.

First, VDA and G2 are trained with different alignment objectives, which leads to markedly different gains under sequence-level SSI evaluation. Specifically, VDA uses scale- and shift-invariant depth supervision, whereas G2 supervises 3D geometry after aligning predicted depths with a single video-level scale (Sec. 3.2, Lines 199–206). G2 therefore learns to recover scene structure up to a global scale without requiring an additional shift correction. Re-evaluating the exact same ScanNet predictions under SI and SSI alignment gives:

MethodScanNet SI-TAE (%) ↓ScanNet SSI-TAE (%) ↓
VDA1.26130.5489
G2(ours)0.80850.7958

Under SI alignment, G2 reduces TAE by 35.9% relative to VDA. Introducing sequence-level shift correction in inverse-depth space reduces VDA’s TAE by 56.48%, compared with only 1.57% for G2, thereby reversing their ranking. Consistent with their training objectives, the median fitted shift normalized by mean ground-truth inverse depth is 0.4997 for VDA versus 0.0075 for G2. Because G2 already recovers global calibration up to scale, it requires minimal global shift correction, while SSI compensates for a substantial global inverse-depth offset in VDA. We further isolated this effect on a diagnostic subset of 19 valid scenes by holding the predictions fixed and interpolating between the SI- and SSI-aligned inverse depths. Moving from the interpolation midpoint to full SSI reduces VDA’s TAE from 0.8589% to 0.5843%, whereas G2’s TAE changes minimally from 0.8428% to 0.8422%, confirming that varying only the alignment reproduces the ranking reversal.

Second, alignment gains depend on the dataset-specific error structure of each model, which explains why this ranking reversal is specific to ScanNet. On ScanNet, SSI absorbs a substantial global inverse-depth offset in VDA and reverses the relative ranking, even though G2 outperforms VDA under SI alignment. In contrast, on Bonn and Sintel, G2 retains lower TAE under both SI and SSI, demonstrating that its temporal consistency improvements persist after global affine correction. Together with its consistent depth and 3D reconstruction improvements over VDA, these findings account for both G2’s broader gains and the specific ScanNet SSI-TAE exception.

In the revised version, Sec. 4.2 reports benchmark TAE under both alignments (Lines 361–373, Table 2). Appendix D.7 explains the ScanNet ranking reversal using fixed predictions, fitted shifts, and interpolation between SI and SSI alignment (Lines 1063–1070 and 1087–1102, Tables 15 and 16).

AuthorsOpenReview ↗

Responses to Reviewer nww2 (2/3)

W1/Q2: "Could you also report TAE in the MAIA, E3DS, and MVGD ablations to show each module's effect on temporal consistency? In particular, does the inter-frame projection loss improve temporal consistency?"

A2: Thank you for requesting this comparison. We report TAE on Bonn and Sintel under both SI and SSI alignment. The newly added TAE results are summarized below.

MAIA variantBonn SI TAE ↓Sintel SI TAE ↓Bonn SSI TAE ↓Sintel SSI TAE ↓
MAIA(ours)2.749228.61252.434452.3244
3D Deformable DETR2.891529.29112.458054.5374
3D DAT2.916329.29912.516954.4353
E3DS variantBonn SI TAE ↓Sintel SI TAE ↓Bonn SSI TAE ↓Sintel SSI TAE ↓
E3DS(ours)2.749228.61252.434552.3244
Ray-weighted depth regression2.765428.96992.460852.5142
Standard reprojection loss2.804129.99402.454954.6815
MVGD variantBonn SI TAE ↓Sintel SI TAE ↓Bonn SSI TAE ↓Sintel SSI TAE ↓
MVGD(ours)2.539028.30252.378451.0244
Synthetic-only continuation2.790030.14382.455051.5959
π3\pi^3 depth distillation2.606133.20572.422353.8353
MVGD feature distillation2.790832.56922.456151.8651
Geometry-only distillation2.635028.56302.431251.2845

MAIA lowers TAE on both Bonn and Sintel relative to both 3D deformable-attention variants under both alignments. MVGD gives lower TAE than synthetic-only continuation and each individual distillation variant on both datasets under both alignments.

With the intra-frame point-map loss enabled in both runs, adding the inter-frame projection loss lowers Bonn TAE from 2.7670% to 2.7492% under SI and from 2.4509% to 2.4345% under SSI. E3DS also gives the lowest TAE on Bonn and Sintel among the three supervision settings under both alignments.

In the revised version, Sec. 4.4 reports Bonn TAE under SI and SSI for MAIA (Table 4), E3DS (Tables 6 and 7), and MVGD (Table 8). The E3DS ablation discusses the effect of the inter-frame projection loss (Lines 473–476 and 495–499) and the comparison with standard reprojection (Lines 500–507).

AuthorsOpenReview ↗

Responses to Reviewer nww2 (3/3)

W2/Q3: "Does the no-distillation baseline in Table 9 receive the same number of training updates? Using the same real videos and training budget, how does supervision with only π3\pi^3 depth pseudo-labels compare with MVGD? This would help distinguish the effects of geometric distillation from those of training with additional data."

A3: Yes. In the revised distillation ablation (Table 8), the no-distillation baseline (Synthetic-only continuation) receives the exact same number of training updates and synthetic-data exposure as other settings. Across all compared settings, we match the training budget and the unlabeled real-video batches. Both teacher-supervised variants use the same frozen π3\pi^3 [1] teacher on the identical set of unlabeled real videos. π3\pi^3 depth distillation trains solely with depth pseudo-labels, whereas MVGD combines teacher-guided geometric supervision with feature distillation.

The unlabeled pool contains 50K videos, with 29.6K from OpenVid-1M and 20.4K from Kinetics-700-2020. In each epoch, we sample one clip per selected video, yielding 6,679 clips in total. (3353 from OpenVid-1M and 3326 from Kinetics-700-2020).

Below, we report depth accuracy, TAE, and reconstruction metrics under scale-invariant alignment:

Supervision settingBonn AbsRel ↓Bonn δ₁ ↑ScanNet AbsRel ↓ScanNet δ₁ ↑Sintel AbsRel ↓Sintel δ₁ ↑Bonn TAE ↓Sintel TAE ↓
Synthetic-only continuation0.05550.98200.08330.93920.20100.68392.79030.144
π3\pi^3 depth distillation0.05580.98240.07380.95010.20590.68932.60633.206
MVGD(ours)0.05080.98440.07590.94920.19760.69952.53928.302
Supervision setting7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
Synthetic-only continuation0.02280.01140.79260.10570.06840.9465
π3\pi^3 depth distillation0.02270.01150.79160.05860.02710.9526
MVGD(ours)0.02150.01130.79390.05750.01990.9622

MVGD improves 12 of 14 SI metrics relative to π3\pi^3 depth distillation. It improves depth accuracy and TAE on Bonn and Sintel and all six median reconstruction metrics. Depth-only distillation gives lower AbsRel and higher δ1\delta_1 on ScanNet.

MVGD also lowers Bonn and Sintel TAE and improves all median reconstruction metrics relative to synthetic-only continuation under the same update budget. Together, the comparisons show the gains of MVGD over both controls.

In the revised manuscript, Sec. 3.4 describes the distillation data and training schedule (Lines 296–308). Sec. 4.4 describes the schedule-matched synthetic-only continuation and compares it with π3\pi^3 depth-map distillation and MVGD on Bonn and 7-Scenes (Lines 509–517, Table 8).

[1] π3\pi^3: Permutation-Equivariant Visual Geometry Learning. In ICLR 2026.


Summary of Responses to Reviewer nww2

Your feedback helps us make the temporal and distillation evidence more explicit.

  1. The ScanNet SSI temporal result (Sec. 4.2, Table 2) and its alignment diagnostics (Appendix D.7, Tables 15 and 16).
  2. Bonn TAE for the MAIA, E3DS, and MVGD ablations, including the effect of the inter-frame projection supervision (Sec. 4.4, Tables 4 and 6–8).
  3. The unlabeled video pool and training schedule (Sec. 3.4) and comparisons with synthetic-only continuation and π3\pi^3 depth-only supervision (Sec. 4.4, Table 8).

We use red to mark the corresponding changes in the revised manuscript. Thank you again for your helpful comments, and we welcome further discussion.

Reviewer_xn8vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents G2, a feed-forward framework for monocular video depth estimation that aims to improve both temporal consistency and the geometry recovered from predicted depth. Building on pretrained Video Depth Anything, the method introduces three components: Motion-Adaptive Implicit Alignment (MAIA), which aggregates features through query-conditioned sparse spatiotemporal sampling; Explicit 3D Supervision (E3DS), which combines intra-frame point-map and inter-frame projection losses; and Multi-View Geometric Distillation (MVGD), which transfers geometric pseudo-labels and frame-level feature relations from a frozen Pi3 teacher to unlabeled videos. Training combines labeled synthetic datasets with unlabeled real videos. Evaluation covers depth accuracy and temporal consistency on Bonn, ScanNet, and Sintel, together with geometric reconstruction on 7-Scenes and NRGBD using ground-truth cameras. The method achieves the best reported average depth ranks under both SSI and SI alignment, with particularly strong reconstruction results under SI alignment.

Critical strengths

1.The gains extend beyond conventional depth metrics. In Table 3, G2 reduces SI-aligned reconstruction accuracy error from 0.030 to 0.021 on 7-Scenes and from 0.095 to 0.057 on NRGBD relative to GeometryCrafter, the strongest competing method on these metrics. Table 1 also reports a Sintel SI-aligned δ1 of 0.700 versus 0.594 for GeometryCrafter. These results support the practical value of learning depth that preserves geometry without requiring an additional shift correction.

  1. Tables 4–8 separately examine temporal aggregation, learnable sampling offsets, sampling density, and geometric supervision. In particular, Table 7 shows that combining intra-frame and inter-frame supervision improves reconstruction accuracy from 0.031/0.029 to 0.025/0.022 under SSI/SI alignment, while Bonn depth accuracy changes little. This supports the paper’s central observation that standard depth accuracy alone does not fully characterize recovered geometric quality.
Critical weaknesses
  1. The methodological contribution is insufficiently distinguished from existing building blocks. MAIA’s query-conditioned offsets, interpolated feature sampling, and sparse attention closely resemble established deformable-attention mechanisms, but the paper does not adequately explain or experimentally isolate the distinction. Table 4 compares against conventional temporal attention rather than a closely matched deformable-attention alternative. Moreover, when target point maps are obtained by back-projecting ground-truth depth with the same intrinsics, the intra-frame point residual is a depth residual multiplied by the camera ray. Its benefit therefore needs to be distinguished from changing the depth-loss parameterization and weighting. These issues leave uncertainty about which technical advances, beyond their integration, account for the improvements.

  2. The evidence for scalable geometric distillation is substantially weaker than the framing suggests. MVGD is presented as one of three central contributions, yet its ablation is restricted to 7-Scenes reconstruction and Bonn depth accuracy. Section 4.4 reports only 0.5%–2.4% reductions in reconstruction errors and 0.28/0.44 percentage-point improvements in Bonn δ1. These gains may be useful, but their robustness is unclear without repeated runs or uncertainty estimates. The paper also omits the actual number of unlabeled training clips, data-scale experiments, and a matched teacher-depth-only distillation baseline. Consequently, it establishes a possible benefit from additional teacher supervision more clearly than it establishes the claimed resolution of the data-scaling bottleneck.

Questions for authors
  1. What specifically distinguishes MAIA from applying deformable attention to a spatiotemporal feature volume? Can the authors provide a comparison with a closely matched deformable-attention baseline and report inference time and peak memory at the same resolution and clip length?

  2. Interpretation of intra-frame supervision: For labeled data, the point-map residual appears to satisfy Pt,u−Pt,utar=(Dt,u−Dt,utar)(Kttar)−1u~P_{t,u}-P^{tar}_{t,u}=(D_{t,u}-D^{tar}_{t,u})(K^{tar}_t)^{-1}\tilde{u}. How should the improvement from this loss be interpreted relative to camera-ray-dependent depth supervision? Is there evidence separating this effect from the contribution of inter-frame geometric constraints?

  3. Distillation evidence: How many unlabeled clips are actually used, and are the Table 9 variants matched for training steps and synthetic-data exposure? How does MVGD compare with using only the same teacher’s depth pseudo-labels? Evidence on variability or different unlabeled-data amounts would help determine whether the reported gains support the scalability claim.

  4. Are SI/SSI alignment parameters fitted per frame, per clip, or per complete sequence, and is alignment performed in depth or inverse-depth space? Please also specify inference clip length, resolution, overlapping-window handling, and the visibility/occlusion rules used for reconstruction and TAE. These choices are consequential for assessing temporal and geometric consistency.

8 comments
AuthorsOpenReview ↗

Responses to Reviewer xn8v (1/6)

We sincerely thank Reviewer xn8v for the constructive feedback and for recognizing the improvements in recovered geometry and the value of examining depth accuracy together with reconstruction quality. We provide point-by-point responses below, focusing on the methodological distinctions, the evidence for distillation, and the evaluation protocol.


W1/Q1-1: "What specifically distinguishes MAIA from applying deformable attention to a spatiotemporal feature volume?"

A1-1: Thank you for the thoughtful question. Extending existing deformable attention to a spatiotemporal feature volume provides a general sampling mechanism, but this extension alone does not explicitly address the spatial and temporal support needed for video depth estimation, resulting in inferior performance. Our design builds on DAT [1] and Deformable DETR [2]. MAIA uses query-conditioned residual offsets to refine staggered spatiotemporal anchors, with content-based query–key aggregation over valid samples.

First, MAIA uses staggered spatiotemporal anchors to organize complementary spatial and temporal support. Within each attention head, the preset sampling positions share a spatial radius and are distributed evenly over a neighboring temporal range. The construction is detailed in Appendix B.2 (Lines 881–895, Eq. 18). For comparison, our 3D adaptation of DAT uses a uniformly spaced spatiotemporal reference grid, while our 3D adaptation of Deformable DETR uses a radial sampling layout centered on the reference point. This prior provides support across multiple spatial directions and temporal offsets. Its staggered arrangement is designed to reduce dependence on a single spatiotemporal layout and avoid imposing a fixed association between spatial angle and temporal offset.

Second, MAIA combines query-specific sampling with content-aware aggregation. DAT shares sampling locations across queries and computes aggregation weights from query–key similarity. Deformable DETR predicts sampling offsets and attention weights for each query through linear projections, without using the sampled keys to compute those weights. MAIA combines query-conditioned residual offsets with weights computed from the query and sampled keys. The residuals adapt the preset positions to the input, and query–key similarity determines the relevance of the sampled content. In addition, MAIA excludes positions outside the video feature volume, so that only valid samples contribute to aggregation. The contribution therefore concerns the structured spatiotemporal prior and its combination with adaptive sampling and content-based weighting.

In the revised version, Sec. 3.1 explains the learned offsets and aggregation over valid samples (Lines 164–195).

[1] Vision Transformer with Deformable Attention. In CVPR 2022.

[2] Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICLR 2021.

AuthorsOpenReview ↗

Responses to Reviewer xn8v (2/6)

W1/Q1-2: "Can the authors provide a comparison with a closely matched deformable-attention baseline and report inference time and peak memory at the same resolution and clip length?"

A1-2: Thank you for this valuable suggestion. We conducted a controlled comparison against a closely matched deformable-attention baseline at the same resolution and clip length, demonstrating that our method achieves superior performance with lower inference time.

First, we adapt DAT and Deformable DETR into 3D variants for a fair comparison with MAIA on the same spatiotemporal feature volume. The comparison is designed to match the backbone, input resolution, clip length, sampling-point budget, and training configuration. We report scale-invariant (SI) aligned depth accuracy and TAE, followed by reconstruction metrics on the reconstruction benchmarks.

Attention variantBonn AbsRel ↓Bonn δ₁ ↑ScanNet AbsRel ↓ScanNet δ₁ ↑Sintel AbsRel ↓Sintel δ₁ ↑Bonn TAE (%) ↓Sintel TAE (%) ↓
MAIA(ours)0.05980.98000.07970.94450.34790.63212.74928.612
3D Deformable DETR0.07280.97530.08440.93800.41060.57402.89229.291
3D DAT0.07080.97470.08340.93940.39150.58832.91629.299
Attention variant7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
MAIA(ours)0.02200.01140.79480.05810.02060.9620
3D Deformable DETR0.02630.01200.78900.07490.02300.9546
3D DAT0.02720.01210.78270.08320.02320.9549

Compared with 3D Deformable DETR, MAIA lowers Bonn AbsRel from 0.0728 to 0.0598 and Bonn SI TAE by 4.9%. It also lowers accuracy error by 16.3% on 7-Scenes and 22.4% on NRGBD. Overall, compared with both 3D DAT and 3D Deformable DETR, MAIA demonstrates consistently superior performance in depth accuracy, temporal coherency, and 3D reconstruction.

Second, we measure efficiency with one 32-frame clip at 518×518518\times518 in FP32 on the same GPU, using five warm-up runs and ten timed runs per variant.

Attention variantTotal parameters (M)Peak memory (MiB) ↓Median latency (ms/clip) ↓
MAIA(ours)435.368636.24445.77
3D Deformable DETR435.558638.54470.05
3D DAT434.808634.14789.90

Across all three variants, total parameters and peak memory remain closely matched. In terms of speed, MAIA achieves the lowest median latency, slightly outperforming 3D Deformable DETR while reducing latency by 7.2% compared with 3D DAT. Together with the accuracy gains above, MAIA delivers consistently superior depth, temporal, and 3D reconstruction performance with no extra parameter or memory overhead and even faster inference.

In the revised version, Sec. 4.4 reports the matched Bonn and 7-Scenes comparison (Lines 452–463, Table 4), and Appendix D.3 gives the 32-frame profiling setup and results (Lines 994–1003, Table 11).

AuthorsOpenReview ↗

Responses to Reviewer xn8v (3/6)

W1/Q2: "Interpretation of intra-frame supervision: For labeled data, the point-map residual appears to satisfy Pt,u−Pt,utar=(Dt,u−Dt,utar)(Kttar)−1u~P_{t,u}-P^{\mathrm{tar}}_{t,u}=(D_{t,u}-D^{\mathrm{tar}}_{t,u})(K^{\mathrm{tar}}_t)^{-1}\tilde{u}. How should the improvement from this loss be interpreted relative to camera-ray-dependent depth supervision? Is there evidence separating this effect from the contribution of inter-frame geometric constraints?"

A2: Thank you for highlighting this point. For labeled data, since predicted and target depths are back-projected along the same camera ray, the 3D point error reduces to the depth error scaled by that ray vector. Applying the same robust penalty to each ray-scaled residual component gives the same objective when masking, normalization, and weights match. The intra-frame supervision therefore parameterizes depth supervision in camera coordinates. We use the point-map formulation to express both intra-frame and inter-frame supervision in a common 3D representation. The inter-frame supervision projects the source prediction into the target view, where it is supervised by the target-frame ground truth. Standard reprojection instead compares projected depth with the target-frame prediction.

The E3DS and ray-weighted depth regression runs start from identical model weights and use the same robust penalty and inter-frame projection loss. The comparison varies only the formulation of the intra-frame supervision.

We compare E3DS with ray-weighted depth regression and standard reprojection on SI TAE and reconstruction metrics.

Supervision setting7-Scenes Acc. ↓7-Scenes Comp. ↓7-Scenes NC ↑NRGBD Acc. ↓NRGBD Comp. ↓NRGBD NC ↑
E3DS(ours)0.02200.01140.79480.05810.02060.9620
Ray-weighted depth regression0.02310.01180.79420.05820.02120.9610
Standard reprojection loss0.02570.01210.79280.07560.02320.9611
Supervision settingBonn TAE (%) ↓Sintel TAE (%) ↓
E3DS(ours)2.74928.612
Ray-weighted depth regression2.76528.970
Standard reprojection loss2.80429.994

E3DS gives the lowest SI TAE on both Bonn and Sintel among the three supervision settings, while ray-weighted depth regression achieves close yet slightly inferior performance.

In the revised version, Appendix B.4 derives the ray-scaled point-map loss (Lines 915–929, Eq. 19), and Sec. 3.2 explains inter-frame supervision against target-view ground truth (Lines 223–231). Sec. 4.4 reports the loss ablation and the comparisons with ray-weighted depth regression and standard reprojection (Lines 473–476 and 495–507, Tables 6 and 7).


W2/Q3-1: "Distillation evidence: How many unlabeled clips are actually used, and are the Table 9 variants matched for training steps and synthetic-data exposure? "

A3-1: Thank you for this important question.

First, the unlabeled pool contains 50K videos, with 29.6K from OpenVid-1M and 20.4K from Kinetics-700-2020. In each epoch, we sample one clip per selected video, yielding 6,679 clips in total (3353 from OpenVid-1M and 3326 from Kinetics-700-2020).

Second, all variants are strictly matched for both total training steps and synthetic-data exposure in the revised Table 8. Under this controlled budget, all our main conclusions remain unchanged.

In the revised manuscript, Sec. 3.4 describes the 50K-video pool and the training schedule (Lines 296–308). Sec. 4.4 specifies that synthetic-only continuation follows the same second-stage schedule (Lines 509–511, Table 8).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Responses

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback. We have provided detailed point-by-point responses and marked all corresponding changes in red in the revised manuscript. Below, we summarize the key aspects of our work, the reviewers’ comments, and our responses.


Summary of This Work

The paper introduces G2, a feed-forward framework to enhance temporal consistency and 3D geometric recovery in monocular video depth estimation (Reviewer xn8v). Motion-Adaptive Implicit Alignment (MAIA) aggregates cross-frame features via sparse sampling (Reviewer r14i), Explicit 3D Supervision (E3DS) constrains intra- and inter-frame geometry (Reviewer nww2), and Multi-View Geometric Distillation (MVGD) extends geometric supervision to unlabeled videos through a frozen π3\pi^3 teacher (Reviewer xn8v). Overall, G2 achieves the best reported average depth rank under both scale-invariant (SI) and scale-and-shift-invariant (SSI) alignment, with gains extending beyond conventional depth metrics to 3D geometric reconstruction (Reviewer xn8v).


Summary of Reviewers’ Comments and Our Responses

Reviewer xn8v (Rating 3 & Confidence 3)

Reviewer xn8v highlights G2’s gains in recovered geometry and the ablations showing the value of evaluating reconstruction alongside depth accuracy. The main concerns relate to: i) MAIA’s distinction from deformable attention and its comparative accuracy and efficiency (q1), ii) the interpretation of intra-frame supervision and the contribution of inter-frame constraints (q2), iii) the distillation controls and evidence for robustness and scalability (q3), and iv) the inference and evaluation protocols (q4).

In response, we i) clarify MAIA’s design and benchmark its accuracy and efficiency against matched baselines, ii) explain the intra-frame loss in terms of depth supervision and isolate the inter-frame contribution, iii) document the unlabeled data and add matched controls, data-scale comparisons, and multi-seed analyses, and iv) clarify the inference, alignment, and evaluation protocols.

Following our response, Reviewer xn8v states that the supplementary experiments and clarifications address their main concerns and raises their score accordingly.

Reviewer nww2 (Rating 2 & Confidence 3)

Reviewer nww2 acknowledges the clear motivation for 3D supervision and G2’s competitive depth and reconstruction results. The main concerns relate to: i) G2’s temporal-consistency gap on ScanNet under SSI alignment relative to VDA (q1), ii) the contribution of each component to temporal consistency (q2), and iii) whether MVGD’s gains can be separated from the effects of additional data and training (q3).

In response, we i) analyze how alignment affects the temporal comparison, ii) report temporal ablations across components and isolate the effect of inter-frame supervision, and iii) specify the unlabeled data and compare MVGD with matched no-distillation and depth pseudo-label controls.

Reviewer r14i (Rating 2 & Confidence 3)

Reviewer r14i recognizes G2’s improvements in cross-frame geometric consistency, strong zero-shot depth and reconstruction results, and comprehensive component ablations. The main concerns relate to: i) MAIA’s novelty relative to deformable attention (q1), ii) E3DS’s contribution relative to ray-weighted depth regression and standard reprojection (q2), iii) evidence for motion-aware sampling and validity-mask behavior (q3), iv) distillation controls and frozen-teacher performance (q4), v) G2’s positioning relative to multi-view reconstruction models (q5), and vi) MAIA’s computational overhead and scaling, along with statistical support for the reported gains (q6).

In response, we i) explain MAIA’s design and add matched attention comparisons, ii) clarify the supervision objectives and evaluate alternative losses, iii) visualize sampling under motion and analyze mask rejections, iv) add controlled distillation ablations and evaluate the teacher model's performance, v) clarify task assumptions and compare depth accuracy, reconstruction quality, and model size, and vi) profile computational overhead across clip lengths and report statistical checks on comparisons.


Further details are provided in our point-by-point replies, with all revisions highlighted in red throughout the updated manuscript. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.