acceptodds
Withdrawn from ICLR 2027

From Visibility to Geometry: Efficient Training for Feed-Forward 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Feed-forward reconstruction models, such as VGGT- and , have demonstrated impressive reconstruction performance, often leveraging powerful pretrained visual features such as DINOv2 or DINOv3. In this work, we show that restructuring the training process around geometric relationships can substantially improve the efficiency of learning multi-view geometry. Our key insight is that predicting covisibility for multiple pairs of images provides a strong, unified learning signal for geometric relationships across views. Instead of directly predicting camera poses and depth maps, we first pretrain the network to predict covisibility maps derived from the ground-truth geometry, and then fine-tune it to regress camera poses and depth maps. This two-stage training strategy substantially accelerates convergence, yielding significantly better reconstruction performance under a fixed compute budget. Remarkably, our approach achieves competitive reconstruction quality when trained entirely from scratch, without relying on any pretrained visual features. Code and models will be made available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Pre-training, geometry, multi-view, covisibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CQ3iOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a new pre-training method for Feed-Forward 3D reconstruction relying on dense covisibility class prediction. The main idea is to predict for each pixel in the source image, whether the corresponding 3D point is visible in another image, occluded, or outside of the field of view in the second image. The authors claim that in this way the network learns relationships between views through a lightweight head that is discarded in the second stage and replaced by depth and camera heads that are fine-tuned.

Critical strengths

Overall, I remain unconvinced by the paper and do not identify a sufficiently compelling strength to support acceptance. Nevertheless, the paper has several positive aspects worth acknowledging:

  1. Using covisibility, which can be readily derived from available geometry, as a supervision signal is a promising direction. Although the idea builds on prior work such as Gekko and Alligat0R, Table 1 provides convincing evidence of its effectiveness within the proposed training framework, particularly when used in the two-stage curriculum, however these results concern mainly ScanNet. The gap on other datasets (Table 3) such as KITTI depth and CO3Dv2 pose estimation, performance on RE10K remains significative compared to VGGT-Ω and π³.

  2. The paper proposes ablations and analyses of the proposed pipeline evaluating the role of pretraining, showing the complementarity of covisibility and masked modeling, robustness to noisy labels, and assessing the quality of frozen representations across several benchmark including patch correspondences.

Critical weaknesses

Main concerns:

  • The paper’s positioning is unclear.
  • The evaluation does not sufficiently establish the proposed method as a reusable geometric pre-training strategy.
  • The lack of controlled end-to-end comparisons with alternative pre-training objectives and evaluation of the 3D reconstruction.

Weaknesses in details:

  1. The paper’s positioning should be clarified. The paper proposing a pre-training step for 3D reconstruction, but it seems that the model is only fine-tuned for and evaluated on depth and camera prediction. While in Figure 1 the paper illustrates a few 3D examples obtained with their model (supposedly from depth and pose), a direct evaluation of the reconstructed 3D geometry, including accuracy, completeness, or cross-view consistency would better support the papers claim of “pre-training for 3D reconstruction”.

  2. Pre-training typically aims to learn reusable representations that transfer across downstream tasks and datasets. Here, the pre-training is only used as a first stage in a two-stage curriculum learning for depth and camera prediction. Furthermore, the first stage uses geometric annotations from the reconstruction training data and is primarily evaluated as preparation for depth and camera estimation. Although the frozen-feature and correspondence evaluations provide some evidence of reusability, broader transfer remains insufficiently explored. The paper would benefit from experiments demonstrating transfer to additional downstream tasks and datasets unseen during pre-training, supporting reusability of the learned geometric representations.

  3. The main contribution appears to be a multi-view geometric pre-training objective and a two-stage training curriculum. Although the paper includes useful controlled and frozen-encoder comparisons, its evaluation does not systematically compare alternative pre-training strategies using the same reconstruction architecture and end-to-end fine-tuning protocol. In particular, the masking-only baseline is evaluated only with a frozen backbone in Table 6 but it is absent from the main fine-tuning comparison.

  4. I am also concerned by the motivation of replacing dense geometric targets with three-class covisibility supervision. It might simplify or reduce the cost of optimization and indeed some of the labels might be robust to noise, still I think such discretization discards potentially useful information about 3D structure. The paper should, for example, provide comparison with alternative cross-view geometric objectives such as dense pointmaps or correspondence supervision trained on the same.

  5. Furthermore, unlike self-supervised masking and cross-view completion, the proposed objective requires depth and calibrated cameras to generate its labels. This substantial annotation requirement limits its applicability to unannotated image collections and should be more explicitly considered when discussing its efficiency and scalability. Pseudo-labels offer a possible alternative, but then the associated cost should also be considered in comparison.

  6. The main depth-and-camera evaluation omits relevant feed-forward geometry models, particularly Depth Anything 3 and MapAnything. Although DA3 appears in the feature-correspondence comparison, the authors should evaluate and provide comparisons for depth and camera predictions, together with MapAnything in an images-only setting. Additional monocular depth baselines could also help establish the benefit of multi-view processing beyond the comparison with MoGe-2.

Questions for authors
  1. Since the covisibility prediction head operates on image pairs and this head is discarded before reconstruction fine-tuning, it would be useful to compare the proposed multi-view pre-training with a pairwise covisibility pre-training baseline. This, on the one hand, would enable to evaluate/compare e.g. with CroCo-like two-view pretraining, and on the other hand, initializing the multi-view architecture with these weights before refinement could evaluate the benefits of directly pre-training the multi-view architecture.

  2. The training and evaluation splits could be described more explicitly. In particular, lines 350–351 do not specify which ScanNet scans are used to train the supervised ViT-S and generate its covisibility pseudo-labels, or whether pseudo-label generation is restricted to the training split. Although the paper specifies 120 held-out ScanNet scans, it should clearly confirm that these scenes are excluded from all training and pseudo-labeling stages. The relationship between ScanNet-1500 and the other ScanNet splits also needs clarification: since ScanNet-1500 is a benchmark derived from ScanNet, the statement that its scenes lie outside the training data should be supported by an explicit account of scene overlap for each evaluated encoder.

2 comments
AuthorsOpenReview ↗

Reply to CQ3i (Part 1)

We thank the reviewer for a thorough review. W1 to W6 follow the order of the detailed weaknesses. The general response describes how covisibility is learned since submission and the scaled ("post-submission") models.

W1. Positioning and direct 3D evaluation. Our claim concerns how to train a feed-forward reconstruction model: we propose a geometric first stage, in the role that CroCo plays for DUSt3R or DINO for VGGT, rather than a general-purpose representation, and the introduction will say so. We now report point-cloud accuracy and completeness on every benchmark with depth, using π3\pi^3's protocol on 7-Scenes and Neural-RGBD (our implementation reproduces π3\pi^3's published numbers within 0.002). Each cloud fuses all the views of a clip, so these errors also reflect cross-view consistency. Accuracy / completeness in meters, for the post-submission ViT-B:

training (GPU-h, DINO + multi-view)7-ScenesNeural-RGBDScanNet
VGGT22k + 13.8k A1000.023 / 0.0270.015 / 0.0150.018 / 0.021
VGGT-Ω\Omega61k + ~34k H1000.015 / 0.0170.010 / 0.0070.017 / 0.022
π3\pi^322k + ≥13.8k A1000.016 / 0.0220.013 / 0.0140.008 / 0.011
ours1.2k H1000.020 / 0.0250.029 / 0.0250.013 / 0.016

In domain (ScanNet), our model is ahead of VGGT and VGGT-Ω\Omega and behind π3\pi^3. On Neural-RGBD, its errors are about twice those of VGGT and π3\pi^3, which we report as a limitation.

W2. Use beyond depth and cameras. Within geometry, we add three evaluations:

  • With frozen features (Table 4), the submitted encoder already led every released encoder on ScanNet-1500 (0.757 of queries within 16 pixels, against 0.522 for DA3-Large) but trailed on NAVI, which we never train on (0.649 against 0.797). The post-submission ViT-B reaches 0.968 and 0.917.
  • On two-view pose (ScanNet-1500), the post-submission ViT-L has the best AUC@5 (50.0, against 46.9 for π3\pi^3 and 42.3 for VGGT-Ω\Omega), although its mean rotation error is larger than π3\pi^3's (2.42° against 1.38°).
  • On dynamic-scene pose (TUM-dynamics, never trained on), both our models are close to π3\pi^3 and VGGT (ATE 0.016 and 0.015, against 0.014 and 0.012), though behind VGGT-Ω\Omega (0.006).

Transfer to non-geometric tasks is outside our scope, and the revision will say so.

W3. Comparison with other pre-training objectives. We compare against multi-view masked modeling with dropped patches (MuM's objective), with the same architecture, data, budget and fine-tuning. The first row is Table 1's setting. Every row trains for 120k steps in total, so its ViT-B is not the longer-trained model of W6. ScanNet AbsRel, fine-tuned / frozen:

settingscratchmasked modelingcovisibility
ViT-S, 23 datasets, since submission0.12250.0710 / 0.12430.0607 / 0.0739
ViT-B, 23 datasets, since submission0.07770.0332 / 0.08390.0324 / 0.0555

With the objective as learned since submission, covisibility is better than masked modeling.

AuthorsOpenReview ↗

Reply to CQ3i (Part 2)

W4. Why a discrete target, point maps, and correspondences.

  • A first stage that regresses ground-truth depth, cameras and point maps (VGGT-Ω\Omega's loss), followed by the same fine-tuning, does no better than training directly (ViT-S, 23 datasets: 0.1222 against 0.1225), while the submitted covisibility objective reaches 0.0801.
  • Used as a regression target, VGGT-Ω\Omega's predicted depth and cameras give 0.0822. Turned into covisibility labels, the same predictions give 0.0405 (Table 7).
  • We agree that the discretization discards information. In particular, three per-view classes do not say where a pixel is visible. The finer classification we now use keeps that information and still separates occluded from outside (0.0607 in the table of W3 and the ablation is in our reply to Yhg9, W1).

W5. Annotation requirements. This is a fair point, and the revision will make it explicit. Feed-forward reconstruction models such as DUSt3R, VGGT and π3\pi^3 are trained on depth and calibrated cameras, and our first stage reuses the annotations that the second stage needs anyway. The labels also need not be ground truth: those from the weak ViT-S labeller of Table 2 still give 0.0474, against 0.0863 without pre-training (same split). Labels from a model can also be denser than ground truth: in our training data, ground truth gives a label to only 0.8% of DL3DV pixels (from sparse COLMAP reconstructions), 6% of CO3D's and 14-21% of the lidar driving data, whereas our ViT-B's predicted geometry labels every pixel. A ViT-L pre-trained on these dense labels, with the same recipe as our ground-truth ViT-L, ends 10% better on 7-Scenes depth and 3.5% better on KITTI, 1.7% worse on Argoverse 2, and level elsewhere. The labeller is our ViT-B before its 448x592 stage, itself trained with ground truth. Its training took about 450 H100-hours, and running it adds about 13% to each pre-training step.

W6. Missing baselines. We scored DA3 and MapAnything on the same clips as ours, without confidence masks and at the input resolution of their official demos. MapAnything receives images only. Our rows are the post-submission models:

training (GPU-h, DINO + multi-view)ScanNet AbsRel7-Scenes rot.KITTI AbsRelScanNet-1500 AUC@5
DA3-Largenot reported*0.03522.41°0.050139.2
DA3-Giantnot reported*0.03262.33°0.045244.6
MapAnything22k A100 + 15.4k H2000.03872.43°0.047830.4
ours, ViT-B1.2k H1000.02152.48°0.063847.2
ours, ViT-L2.3k H1000.01752.41°0.059750.0

* DA3 reports 128 H100 GPUs for 200k steps, but not the duration.

Our models lead in domain, and DA3-Giant leads out of domain. As a monocular baseline, we scored Depth Anything V2 under its own protocol, which our implementation reproduces on NYU-v2 and KITTI. Under that protocol it is clearly ahead of our models on single-image depth (KITTI AbsRel 0.071, against 0.171 for our ViT-L), and so is π3\pi^3 on NYU-v2 (0.054, against 0.13 for both our models). Single-image depth is therefore a weakness of our models. They are never trained on a single image, since every training clip has between 2 and 24 views, which may explain part of this gap. All these models will go into Table 3.

Q1. Pairwise pre-training. We pre-trained the same architecture (ViT-S, submitted objective, ground-truth labels) on two-view and on 24-view clips, with the same number of images and supervised pairs per step, and fine-tuned both with 24 views. Multi-view pre-training lowers the error on ScanNet depth by 12% and rotation by 19%, on 7-Scenes depth by 7% and on KITTI depth by 8%, all significantly, and is level on DL3DV rotation.

Q2. Splits. Pseudo-labels are only ever generated on the training split of the model they supervise: the labeller of Table 2 is trained on ScanNet training scans and labels training clips only. The 120 held-out ScanNet scans are excluded from all training and pseudo-labelling stages of our models. ScanNet-1500 is derived from ScanNet's test set, so it is not in our training data either.

Reviewer_Yhg9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper pre-trains feed-forward reconstruction models through multi-view covisibility prediction, then fine-tunes them for depth and camera estimation. Under the same optimizer-step budget, pre-training reduces ViT-S’s ScanNet depth AbsRel from 0.0863 to 0.0385. Larger models achieve competitive ScanNet depth results but remain weaker on several pose and transfer benchmarks.

Critical strengths

1. Substantial gains over training from scratch. Table 1 shows clear improvements in depth and camera estimation. Frozen-encoder results further support the usefulness of the learned representations.

2. Relevant controls and transparent limitations. The paper includes joint-training baselines, predicted-label experiments, scene-level bootstrap comparisons, and limited multi-seed evaluation. It also acknowledges weaker transfer and dependence on depth coverage.

3. Evidence beyond reconstruction accuracy. Strong frozen-feature matching and successful pre-training with noisy predicted labels provide additional evidence for the practical value of covisibility supervision.

Critical weaknesses

1. Covisibility provides limited constraints on precise geometry. The three-class labels specify neither correspondence locations nor disparity, and different geometries can produce identical labels. Although Table 4 demonstrates useful features, the conditions under which this supervision supports precise geometry remain unclear. Please evaluate performance across overlap ratios, baselines, and disparities, and assess potential shortcuts based on image position or coarse overlap (§3.1, Table 4, Appendix D.4).

2. Applicability to dynamic scene reconstruction remains unclear. Equation (1) transforms source points using camera poses without accounting for object motion. Consequently, Equation (2) may label motion-induced inconsistencies as occlusion. Please clarify whether this supervision is suitable for dynamic scene reconstruction and evaluate its impact on reconstruction accuracy in moving regions (Eqs. 1–2).

3. Robustness to label noise requires further validation. Hard thresholds absorb only perturbations that do not cross classification boundaries. The stability analysis in Appendix C assumes reprojection onto the same target pixel, leaving reprojection shifts unaddressed. Please independently perturb depth, rotation, and translation, and report threshold sensitivity and label-flip rates to establish the limits of robustness (Tables 2 and 7, Appendix C).

4. Competitive performance is concentrated in a limited range of settings. ViT-L remains substantially behind released models on KITTI depth and CO3Dv2 pose estimation, while its NAVI matching score is lower than all listed released encoders. Moreover, only 7-Scenes is out of domain in the scaled depth and camera evaluation described in §4.1. Broader comparisons under matched compute budgets are needed to establish generalization and the accuracy–efficiency trade-off (Tables 3–4).

5. Small-scale ablations do not establish the benefits at larger scales. The main ablations use ViT-S on ScanNet, whereas the final models use larger architectures and broader training data. The frozen-feature comparisons in Appendix D.2 do not replace a full-training comparison. Please compare direct regression and two-stage training on at least ViT-B using the same data and compute budget to verify that the benefits persist at scale.

Questions for authors

All my concerns have been detailed in the weaknesses section.

2 comments
AuthorsOpenReview ↗

Reply to Yhg9 (Part 1)

We thank the reviewer for the careful reading and the concrete requests. The general response describes the objective as learned since submission and the scaled models.

W1. Three classes constrain geometry weakly, and possible shortcuts. We agree that a per-view class does not say where in the other view a pixel is visible.

  • We find no positional shortcut. On 7-Scenes, never used for training, the submitted head labels 91.9% of pixels correctly. A predictor that sees only pixel position reaches 59.4%, and 70.0% when also given each pair's true overlap. Replacing the target view with one from another ScanNet room lowers the predicted covisible fraction from 0.44 to 0.13.

  • The gain over random initialization holds on hard pairs. We split the ScanNet and 7-Scenes test clips into thirds by ground-truth overlap, by frame spacing and by parallax (baseline over depth, which sets disparity). At ViT-B with AdamW, the submitted objective beats random initialization in every third of every split, and DINOv3 initialization on 7-Scenes depth in every third, but it trails DINOv3 on ScanNet throughout. On the 7-Scenes third with the widest frame spacing, rotation error is 6.13° with the submitted objective, against 8.50° from DINOv3 and 12.09° from random initialization. The objective as learned since submission reaches 3.05° at the same budget with Muon, against 5.21° from random initialization with Muon.

  • Our finer classification scores covisibility per target patch, which addresses the point directly (ViT-S, same data and budget, fine-tuned AbsRel):

    covisible scoreScanNetKITTI
    per target view, dedicated head (submitted)0.08010.0998
    per target patch, occluded and outside merged0.06130.0952
    per target patch, occluded and outside separate0.06070.0819

    Scoring per target patch gives most of the ScanNet gain (removing the head alone gives 0.0766), and keeping occluded and outside apart gives a further 14% on KITTI. The revision will add this ablation.

W2. Dynamic scenes. The reviewer is right that Eq. 1 assumes a static scene, so moving pixels might be labelled occluded. Only our pre-training labels make this assumption. The fine-tuning losses on depth and cameras do not. On BEDLAM, which shows moving people, 16% of labels are "occluded", and this share falls only 1.7 times when the depth tolerance τr\tau_r is widened 12 times, far more slowly than for real occlusion. Our post-submission training data therefore excludes BEDLAM and Virtual KITTI 2, so it holds much less dynamic data than released models (comparison in our reply to 3vKq, W2 and Q2). On π3\pi^3's dynamic benchmarks, our camera pose is competitive with theirs at a small fraction of their training compute, while our depth is worse. We have not isolated moving regions, but our depth error concentrates on close-range pixels, as on static scenes. We will remove the dynamic-scene claim from Figure 1 and list the static-scene assumption of pre-training as a limitation. We could extend the method with a fourth class for moving pixels, labelled from scene flow or motion masks where a dataset provides them, but keep that for future work.

AuthorsOpenReview ↗

Reply to Yhg9 (Part 2)

W3. Label noise. We will state that the bound in Appendix C only covers reprojections that stay on the same target pixel. On 60 ScanNet clips, we perturbed depth, rotation and translation independently. The table gives the share of labels that flip at two thresholds, next to the error the same noise causes in a point-map target:

noisepoint-map errorflips, τr\tau_r 0.05flips, τr\tau_r 0.15
depth, 5% per pixel3.7%3.1%0.8%
depth scale, 5% per view3.7%13.6%1.9%
rotation, 2°3.4%7.1%4.2%
translation, 10% of baseline2.7%5.0%2.6%

At τr\tau_r 0.15, only rotation noise flips more labels than it moves the point map. At τr\tau_r 0.05, only per-pixel depth noise flips fewer. Rotation error and tight thresholds are therefore the limits of the robustness claim, and the revision will say so. That said and as Alligat0R pointed out, this pre-training objective is quite robust to noise in the covisibility labels and can even increase downstream performance, acting as some form of regularization.

W4. Generalization and efficiency. For the post-submission models, 7-Scenes, KITTI and RealEstate10K are out of domain.

  • The general response gives all of Table 3's benchmarks for these models, with training compute. The ViT-L's camera pose is level with VGGT and π3\pi^3 on 7-Scenes and between them on RealEstate10K, behind VGGT-Ω\Omega on both. On KITTI depth it is clearly behind. On CO3Dv2, it is level with π3\pi^3 (89.2 against 89.1 AUC@30).
  • On NAVI matching (Table 4's protocol, never trained on), 0.927 of queries fall within 25 pixels for the post-submission ViT-L and 0.917 for the ViT-B, against 0.649 for the submitted ViT-B and 0.797 for the best released encoder (DA3-Large).
  • Counting all training compute, the ViT-B reaches 0.0279 ScanNet AbsRel after about 240 H100-hours, already ahead of VGGT-Ω\Omega (0.0303) and VGGT (0.0314).

W5. Larger scale. With the submitted objective on 23 datasets (ViT-S, AdamW), pre-training beats direct training on all nine metrics (ScanNet AbsRel 0.0801 against 0.1225). At ViT-B, with the same data and compute and AdamW throughout (general response), the submitted objective beats random initialization everywhere and DINOv3 initialization out of domain, but not in domain. The AdamW run of the objective as learned since submission is running. We will share its results soon.

Reviewer_3vKqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to train 3D feed-forward reconstruction models directly from scratch, without using pretrained DINO features. Instead of directly regressing camera poses and depth, it first pretrains the network to predict dense covisibility maps from many sampled view pairs, and then fine-tunes it to regress camera poses and depth. The prediction is compared with covisibility maps computed from ground-truth camera poses and depth. Under matched compute on ScanNet, the two-stage training framework reduces depth and pose errors, compared with optimizing all losses jointly. The recipe also works when the labels come from a weaker model’s noisy predictions. The scaled ViT-B and ViT-L models match or exceed the strongest open-source models on ScanNet depth with much less training compute, but they still perform worse on out-of-domain benchmarks.

Critical strengths
  1. The pretext task is well-motivated and simple. Covisibility maps are dense and cheap to compute on the fly from depth and cameras. The paper demonstrates that covisibility can be a practical alternative to DINO initialization for feed-forward 3D models.
  2. The training is robust. Covisibility labels from a weaker model’s predictions still yield a model that surpasses the label provider. Covisibility labels from VGGT-Ω transfer far better than regressing its geometry directly. This suggests a path to training on unannotated video.
Critical weaknesses
  1. The scaled models clearly trail released models on out-of-domain benchmarks: 7-Scenes rotation error is about twice that of VGGT-Ω (3.09 vs. 1.58 for ViT-B). This suggests the gains may come more from in-domain fitting than from general multi-view geometry, which weakens the claim of competitive reconstruction quality.
  2. The caption of Fig. 1 claims that the models can reconstruct dynamic scenes, but the paper provides no quantitative evaluation on dynamic scenes. The train scene in Fig. 1 seems poorly reconstructed.
Questions for authors
  1. Could the authors analyze what causes the gap on out-of-domain benchmarks? Does sampling pairs with wider baselines and less visual overlap work?
  2. Could the authors provide quantitative results on a dynamic benchmark to support the claim in Fig. 1?
1 comment
AuthorsOpenReview ↗

Reply to 3vKq

We thank the reviewer for the positive assessment and the two precise questions.

W1 and Q1 (first part). The out-of-domain gap. We agree that this is the main limitation. Released models are now scored on our clips at the input resolution of their official demos, so VGGT-Ω\Omega's 7-Scenes rotation error is 1.45° here, against 1.58° in the paper.

  • The post-submission models narrow the gap (table in the general response, where 7-Scenes, KITTI and RealEstate10K are out of domain). 7-Scenes rotation error goes from 3.09° (submitted ViT-B) to 2.48° (new ViT-B) and 2.41° (ViT-L), level with VGGT and π3\pi^3 for the ViT-L but behind VGGT-Ω\Omega (1.45°), and KITTI depth remains clearly behind.
  • No single change closes the gap on its own. A further 80k steps at 448x592 take the ViT-B from 2.56° to 2.48°, moving to the ViT-L gives 2.41°, and initializing the ViT-B from DINOv3 before covisibility pre-training gives 2.70° (trained with AdamW, as in VGGT-style models, whereas ours uses Muon). VGGT-Ω\Omega, which remains ahead, spent about 15 times our ViT-L's total compute on its multi-view training alone.

Q1 (second part). Wider baselines and less overlap. We tested wider baselines with the submitted objective (ViT-S, same budget for both models). Some of our pre-training clips revisit a place from a distant part of the trajectory, so their views see the same surfaces from directions 15 to 60° apart. Removing them makes the frozen encoder worse on all nine metrics, significantly on six and by up to 12.6%. After fine-tuning, the effect is mixed: KITTI is 30% better with these clips and ScanNet 5% better without them. Wide-baseline pairs help the representation, but they do not close the gap alone. As for less overlap, spacing frames further apart mostly produces pairs that share no surface at all, which is why we widen the viewing angle instead.

W2 and Q2. Dynamic scenes. The reviewer is right that we had not evaluated dynamic scenes: we will remove the dynamic-scene claim and the panel with the train scene from Figure 1. Only our pre-training labels assume a static scene (Eq. 1). The fine-tuning losses on depth and cameras do not. Our data is also thin on motion: our 23 datasets contain no dedicated dynamic-scene data (we excluded BEDLAM and Virtual KITTI 2, whose motion our static-scene labels would mislabel), and motion appears only incidentally, in driving and egocentric sequences. We now report π3\pi^3's dynamic benchmarks, on which our implementation reproduces π3\pi^3's published numbers (depth AbsRel, pose ATE), with each model's dynamic training data and compute:

dynamic training datatraining (GPU-h, DINO + multi-view)Sintel depthBonn depthSintel poseTUM pose
VGGTKubric, PointOdyssey, Virtual KITTI22k + 13.8k A1000.3010.0590.1720.012
π3\pi^3an internal dynamic dataset22k + ≥13.8k A1000.2290.0500.0750.014
VGGT-Ω\OmegaDynamic Replica, ~200k dynamic video scenes61k + ~34k H1000.0970.0490.0350.006
ours, ViT-Bnone dedicated1.2k H1000.4820.1770.1340.016
ours, ViT-Lnone dedicated2.3k H1000.6530.3240.1260.015

With little dynamic data and a small fraction of the compute, our camera pose is competitive: close to VGGT and π3\pi^3 on TUM, and better than VGGT on Sintel. Depth is clearly worse. Besides data coverage, our error concentrates on close-range pixels: on Bonn, our error on pixels closer than 2m is 2-15 times that of π3\pi^3, while the gap is much smaller beyond 2m, and the same pattern appears on NYU-v2, which is static. NYU-v2 is also scored on single images, whereas our models are always trained on 2 to 24 views, which may add to the gap there. We will add the static-scene assumption of pre-training to the limitations. We could extend the method with a fourth class for pixels that move between views, labelled from scene flow or motion masks where a dataset provides them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.