acceptodds
Withdrawn from ICLR 2027

Towards Consistent Video Geometry Estimation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This work presents ViGeo, a feed-forward foundation model for recovering spatially dense and temporally consistent geometry from video sequences. Built upon a plain transformer architecture without task-specific architectural modifications, ViGeo supports streaming, full-sequence, and long-video inference within a unified model. The key design is dynamic chunking attention, which exposes the model to both bidirectional and causal temporal contexts during training and allows it to adapt its attention pattern at test time without retraining. To improve supervision quality, we further introduce a completion-based data refinement framework. This framework trains a video depth completion teacher that conditions on sparse and noisy annotations and exploits video/multi-view context to produce dense, temporally coherent, and geometrically reliable training targets. Beyond depth and point maps, ViGeo also predicts surface normals within the same framework. Trained solely on public datasets, ViGeo achieves state-of-the-art performance across online, offline, and long-video depth estimation, surface normal estimation, and video point map estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3d reconstruction, depth completion, geometry estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8C99OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ViGeo is a feed-forward model that predicts per-frame depth, normals and point maps from video. Its ViT-G, initialized from Depth Anything 3, uses dynamic chunking attention (DCA) across frames: full attention within a temporal chunk, causal across chunks, with chunk partitions randomized in training. One checkpoint therefore runs offline, streaming, or chunk-wise with a KV cache. Real-data labels are refined by a multi-frame depth completion teacher. Evaluation spans video, long-video and monocular depth, point maps and normals.

Critical strengths
  1. With negligible overhead, one unified checkpoint handles offline, streaming, and long-video inference—matching offline full-attention training (Sintel Rel 0.291 vs. 0.289) and surpassing online causal training (0.322 vs. 0.330, Table 5). This eliminates the conventional need to train dedicated models for each regime, such as VGGT and StreamVGGT.
  2. The appendix covers latency, chunk-size robustness and DCA post-training on other models.
Critical weaknesses
  1. Questionable reliability of the state-of-the-art comparisons.

    • Inconsistent baseline results: Under the same alignment protocol, the reported Sintel scores for Pi3, VGGT, and StreamVGGT are noticeably worse than those in their original publications (e.g., Pi3 degrades from 0.233 to 0.281), even though their Bonn and KITTI metrics match. Notably, Sintel is the exact benchmark where ViGeo claims its largest margin of improvement.
    • Selective reporting: VGGT-Ω actually outperforms ViGeo on the Bonn dataset (0.043 vs. 0.046, as shown in Table 15), yet it is omitted from the main comparison tables.
  2. The contributions are incremental, which limits significance.

    • DCA (block-causal attention trained with random chunk sizes) is essentially U2/WeNet's dynamic chunk training from speech recognition, and MUSt3R (CVPR 2025) already runs one network offline and online. Neither is cited.
    • In the ablations, DCA and the refined labels each change Rel by at most 0.008, so the gains likely come mainly from the DA3-Giant initialization and scale.
Questions for authors
  1. How was Sintel evaluated, and why do the baselines' Sintel numbers differ from published ones while Bonn and KITTI match? Which checkpoint produced each table (Table 8 uses an updated model)?
  2. In which coordinate frame are point maps evaluated in Table 4 (world-frame outputs of VGGT, StreamVGGT and STream3R would absorb pose error), and how are the multi-frame point clouds in Fig. 6 registered without predicted poses?
  3. What data trains the completion teacher, and does it overlap with the datasets it relabels?
  4. Minor suggestion (does not affect my rating): the related work could also discuss other feed-forward models that run online by reusing cached features of past frames, such as MUSt3R (CVPR 2025) and MoRe (CVPR 2026).
Reviewer_WnNeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a feedforward consistent depth estimation method that uses heterogeneous attention strategies (bidirectional, chunk-based, and causal) during training. To improve the accuracy in general, the paper also introduces a data refinement pipeline that uses a depth-completion method to generate more accurate pseudo ground truth in the real-world domain where it's difficult to get annotations. According to the provided table, the method shows state-of-the-art accuracy. Also, the proposed training scheme (mixing bidirectional, chunk, and causal during training) shows its strengths -- it outperforms baselines that are trained exclusively for bidirectional (full attention) or causal.

Critical strengths
  • Clearly written

    The paper is easy to follow: equations are straightforward, figures help understand the method better, and the ablation study validates its proposed ideas on dynamic chunking and data refinement framework.

  • Good ablation study

    The main question was whether a method trained on dynamic chunking can give a trade-off and underperform baselines that are trained on either bidirectional or causal, because the network can be optimized to perform best for either bidirectional or causal. Table 5 clears up the concern; the proposed dynamic chunking gives overall best/competitive methods, showing that diversifying these attention patterns actually helps the network generalize better to both online (bidirectional) and offline (causal) settings. This is an insightful data point.

Critical weaknesses
  • Many critical elements are missing in the main paper

    The paper does not include many critical elements and may require substantial rewriting or reformatting. The details are as follows:

    • Missing surface normal comparison (the paper claims it as a contribution). No qualitative result on surface normal provided.
    • Tables 2, 3, and 4 miss many recent state-of-the-art methods, e.g., VGGT-Omega, FastVGGT, CUT3R, TTT3R, etc. They are all published methods that appeared before the paper submission deadline and should be fairly compared.
    • Missing related work section: The paper loosely discusses related work in Sec.1 (Introduction) and Sec. 2 (Method). It would be good to discuss previous methods more thoroughly as a separate section (or subsection) to clearly understand where the paper is positioned in the literature.
    • Missing implementation details: what training datasets are used, how much computational cost is required, etc.
  • Slightly overclaimed title -- keyword on "Consistent"

    It was a bit confusing that the title does not clearly reflect the paper's main contributions. The main contributions are about dynamic chunking and the training data refinement frameworks, but the title sounds very general (ie, towards consistent video geometry estimation). It is unclear if it's okay to claim this general title when one achieves 'better accuracy on benchmark datasets', also without showing actual evidence/analyses on consistency.

  • Missing analyses on the depth completion teacher

    Though it's another half of the paper's contribution, detailed analyses are missing: how accurate is the pseudo ground truth generated by the teacher? Can it measure the accuracy on any existing benchmark datasets that are not used for training the teacher, even on a synthetic dataset? How much computational cost does it take to generate the pseudo ground truth and train the teacher model? Also, when training the model with this teacher model, what qualitative improvement does the model show that may not be reflected in the metrics?


Weaknesses and concerns outweigh the strengths -- I rate the paper as Weak Rejection.

Questions for authors
  • Reference style needs to be revised, probably by using \citep{} or \citet{}, instead of \cite{}.

  • At L58, what does it mean by fixed temporal access pattern and rely on future frames for fullsequence reasoning? Especially, it's not so clear what future frames mean, where offline methods assume all frames are already captured and available.

  • In Figure 3, the attention masks of causal attention and dynamic chunking attention show the same block-causal methods. Doesn't causal attention need to show the attention map with ∣Ll∣=1|L_l| = 1, according to line 191?

Reviewer_NMfTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ViGeo, a unified video geometry estimation model that jointly predicts depth, point maps, and surface normals. It has two core components:

  1. Dynamic chunking attention: bidirectional attention within chunks and causal attention across chunks allow one model to support offline, streaming, and long-video inference.
  2. A video depth-completion teacher: sparse, noisy LiDAR/SfM annotations are used to generate denser and more stable pseudo-labels.

The paper evaluates depth, point maps, and normals on Sintel, Bonn, KITTI, HAMMER, and NYUv2, reporting strong results.

Critical strengths
  1. The problem has practical value. Existing models usually target either offline or streaming inference. ViGeo aims to unify both settings in one model, with clear application value.

  2. Dynamic chunking attention is simple and effective. Table 5 shows that the full-attention model degrades substantially when switching from offline to online inference, while dynamic chunking performs well in both modes. For example, on KITTI, online Rel is 0.055 for dynamic chunking, compared with 0.148 for the model trained with full attention.

  3. The task coverage is broad. Beyond depth, the paper evaluates point maps and normals. The appendix also includes cross-model transfer and varying-chunk-length experiments.

Critical weaknesses
  1. Recent relevant baselines are not fully covered.

    The paper compares against VGGT, Pi3, DA3, StreamVGGT, Stream3R, DepthCrafter, and FlashDepth, but omits recent video depth estimation or feed-forward reconstruction models such as GemDepth and VGGT-Omega. This does not invalidate ViGeo's results, but weakens the claims of state-of-the-art performance and efficiency advantages. Comparisons with newer methods would broaden and strengthen the conclusions.

  2. The TAE metric is insufficiently defined.

    Tables 16–18 use TAE to evaluate temporal consistency, but the appendix does not provide its full formula or computation procedure.

  3. The source of ViGeo's advantage on a fixed task is not sufficiently explained.

    The authors propose a flexible attention mechanism to build a unified architecture. However, for a single task, such as non-causal global reconstruction of an entire sequence, why does ViGeo perform better than DA3, VGGT, and similar methods?

Questions for authors
  1. Could newer related methods, such as GemDepth and VGGT-Omega, be included for a more comprehensive evaluation?

  2. Could the architecture choices and training procedure of the depth-completion teacher be described in more detail, to better assess how these architectural and strategy choices affect the final model?

  3. In Figure 6, ViGeo's point clouds have noticeably fewer outliers than Pi3 and VGGT. Does this result from post-processing or an advantage of the model itself?

  4. Please provide a clearer and more complete explanation or evidence for the source of ViGeo's advantage on VGGT-style tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.