- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose a new pre-training method for Feed-Forward 3D reconstruction relying on dense covisibility class prediction. The main idea is to predict for each pixel in the source image, whether the corresponding 3D point is visible in another image, occluded, or outside of the field of view in the second image. The authors claim that in this way the network learns relationships between views through a lightweight head that is discarded in the second stage and replaced by depth and camera heads that are fine-tuned.
Critical strengths
Overall, I remain unconvinced by the paper and do not identify a sufficiently compelling strength to support acceptance. Nevertheless, the paper has several positive aspects worth acknowledging:
-
Using covisibility, which can be readily derived from available geometry, as a supervision signal is a promising direction. Although the idea builds on prior work such as Gekko and Alligat0R, Table 1 provides convincing evidence of its effectiveness within the proposed training framework, particularly when used in the two-stage curriculum, however these results concern mainly ScanNet. The gap on other datasets (Table 3) such as KITTI depth and CO3Dv2 pose estimation, performance on RE10K remains significative compared to VGGT-Ω and π³.
-
The paper proposes ablations and analyses of the proposed pipeline evaluating the role of pretraining, showing the complementarity of covisibility and masked modeling, robustness to noisy labels, and assessing the quality of frozen representations across several benchmark including patch correspondences.
Critical weaknesses
Main concerns:
- The paper’s positioning is unclear.
- The evaluation does not sufficiently establish the proposed method as a reusable geometric pre-training strategy.
- The lack of controlled end-to-end comparisons with alternative pre-training objectives and evaluation of the 3D reconstruction.
Weaknesses in details:
-
The paper’s positioning should be clarified. The paper proposing a pre-training step for 3D reconstruction, but it seems that the model is only fine-tuned for and evaluated on depth and camera prediction. While in Figure 1 the paper illustrates a few 3D examples obtained with their model (supposedly from depth and pose), a direct evaluation of the reconstructed 3D geometry, including accuracy, completeness, or cross-view consistency would better support the papers claim of “pre-training for 3D reconstruction”.
-
Pre-training typically aims to learn reusable representations that transfer across downstream tasks and datasets. Here, the pre-training is only used as a first stage in a two-stage curriculum learning for depth and camera prediction. Furthermore, the first stage uses geometric annotations from the reconstruction training data and is primarily evaluated as preparation for depth and camera estimation. Although the frozen-feature and correspondence evaluations provide some evidence of reusability, broader transfer remains insufficiently explored. The paper would benefit from experiments demonstrating transfer to additional downstream tasks and datasets unseen during pre-training, supporting reusability of the learned geometric representations.
-
The main contribution appears to be a multi-view geometric pre-training objective and a two-stage training curriculum. Although the paper includes useful controlled and frozen-encoder comparisons, its evaluation does not systematically compare alternative pre-training strategies using the same reconstruction architecture and end-to-end fine-tuning protocol. In particular, the masking-only baseline is evaluated only with a frozen backbone in Table 6 but it is absent from the main fine-tuning comparison.
-
I am also concerned by the motivation of replacing dense geometric targets with three-class covisibility supervision. It might simplify or reduce the cost of optimization and indeed some of the labels might be robust to noise, still I think such discretization discards potentially useful information about 3D structure. The paper should, for example, provide comparison with alternative cross-view geometric objectives such as dense pointmaps or correspondence supervision trained on the same.
-
Furthermore, unlike self-supervised masking and cross-view completion, the proposed objective requires depth and calibrated cameras to generate its labels. This substantial annotation requirement limits its applicability to unannotated image collections and should be more explicitly considered when discussing its efficiency and scalability. Pseudo-labels offer a possible alternative, but then the associated cost should also be considered in comparison.
-
The main depth-and-camera evaluation omits relevant feed-forward geometry models, particularly Depth Anything 3 and MapAnything. Although DA3 appears in the feature-correspondence comparison, the authors should evaluate and provide comparisons for depth and camera predictions, together with MapAnything in an images-only setting. Additional monocular depth baselines could also help establish the benefit of multi-view processing beyond the comparison with MoGe-2.
Questions for authors
-
Since the covisibility prediction head operates on image pairs and this head is discarded before reconstruction fine-tuning, it would be useful to compare the proposed multi-view pre-training with a pairwise covisibility pre-training baseline. This, on the one hand, would enable to evaluate/compare e.g. with CroCo-like two-view pretraining, and on the other hand, initializing the multi-view architecture with these weights before refinement could evaluate the benefits of directly pre-training the multi-view architecture.
-
The training and evaluation splits could be described more explicitly. In particular, lines 350–351 do not specify which ScanNet scans are used to train the supervised ViT-S and generate its covisibility pseudo-labels, or whether pseudo-label generation is restricted to the training split. Although the paper specifies 120 held-out ScanNet scans, it should clearly confirm that these scenes are excluded from all training and pseudo-labeling stages. The relationship between ScanNet-1500 and the other ScanNet splits also needs clarification: since ScanNet-1500 is a benchmark derived from ScanNet, the statement that its scenes lie outside the training data should be supported by an explicit account of scene overlap for each evaluated encoder.
2 comments
Reply to CQ3i (Part 2)
W4. Why a discrete target, point maps, and correspondences.
- A first stage that regresses ground-truth depth, cameras and point maps (VGGT-'s loss), followed by the same fine-tuning, does no better than training directly (ViT-S, 23 datasets: 0.1222 against 0.1225), while the submitted covisibility objective reaches 0.0801.
- Used as a regression target, VGGT-'s predicted depth and cameras give 0.0822. Turned into covisibility labels, the same predictions give 0.0405 (Table 7).
- We agree that the discretization discards information. In particular, three per-view classes do not say where a pixel is visible. The finer classification we now use keeps that information and still separates occluded from outside (0.0607 in the table of W3 and the ablation is in our reply to Yhg9, W1).
W5. Annotation requirements. This is a fair point, and the revision will make it explicit. Feed-forward reconstruction models such as DUSt3R, VGGT and are trained on depth and calibrated cameras, and our first stage reuses the annotations that the second stage needs anyway. The labels also need not be ground truth: those from the weak ViT-S labeller of Table 2 still give 0.0474, against 0.0863 without pre-training (same split). Labels from a model can also be denser than ground truth: in our training data, ground truth gives a label to only 0.8% of DL3DV pixels (from sparse COLMAP reconstructions), 6% of CO3D's and 14-21% of the lidar driving data, whereas our ViT-B's predicted geometry labels every pixel. A ViT-L pre-trained on these dense labels, with the same recipe as our ground-truth ViT-L, ends 10% better on 7-Scenes depth and 3.5% better on KITTI, 1.7% worse on Argoverse 2, and level elsewhere. The labeller is our ViT-B before its 448x592 stage, itself trained with ground truth. Its training took about 450 H100-hours, and running it adds about 13% to each pre-training step.
W6. Missing baselines. We scored DA3 and MapAnything on the same clips as ours, without confidence masks and at the input resolution of their official demos. MapAnything receives images only. Our rows are the post-submission models:
| training (GPU-h, DINO + multi-view) | ScanNet AbsRel | 7-Scenes rot. | KITTI AbsRel | ScanNet-1500 AUC@5 | |
|---|---|---|---|---|---|
| DA3-Large | not reported* | 0.0352 | 2.41° | 0.0501 | 39.2 |
| DA3-Giant | not reported* | 0.0326 | 2.33° | 0.0452 | 44.6 |
| MapAnything | 22k A100 + 15.4k H200 | 0.0387 | 2.43° | 0.0478 | 30.4 |
| ours, ViT-B | 1.2k H100 | 0.0215 | 2.48° | 0.0638 | 47.2 |
| ours, ViT-L | 2.3k H100 | 0.0175 | 2.41° | 0.0597 | 50.0 |
* DA3 reports 128 H100 GPUs for 200k steps, but not the duration.
Our models lead in domain, and DA3-Giant leads out of domain. As a monocular baseline, we scored Depth Anything V2 under its own protocol, which our implementation reproduces on NYU-v2 and KITTI. Under that protocol it is clearly ahead of our models on single-image depth (KITTI AbsRel 0.071, against 0.171 for our ViT-L), and so is on NYU-v2 (0.054, against 0.13 for both our models). Single-image depth is therefore a weakness of our models. They are never trained on a single image, since every training clip has between 2 and 24 views, which may explain part of this gap. All these models will go into Table 3.
Q1. Pairwise pre-training. We pre-trained the same architecture (ViT-S, submitted objective, ground-truth labels) on two-view and on 24-view clips, with the same number of images and supervised pairs per step, and fine-tuned both with 24 views. Multi-view pre-training lowers the error on ScanNet depth by 12% and rotation by 19%, on 7-Scenes depth by 7% and on KITTI depth by 8%, all significantly, and is level on DL3DV rotation.
Q2. Splits. Pseudo-labels are only ever generated on the training split of the model they supervise: the labeller of Table 2 is trained on ScanNet training scans and labels training clips only. The 120 held-out ScanNet scans are excluded from all training and pseudo-labelling stages of our models. ScanNet-1500 is derived from ScanNet's test set, so it is not in our training data either.
Reply to CQ3i (Part 1)
We thank the reviewer for a thorough review. W1 to W6 follow the order of the detailed weaknesses. The general response describes how covisibility is learned since submission and the scaled ("post-submission") models.
W1. Positioning and direct 3D evaluation. Our claim concerns how to train a feed-forward reconstruction model: we propose a geometric first stage, in the role that CroCo plays for DUSt3R or DINO for VGGT, rather than a general-purpose representation, and the introduction will say so. We now report point-cloud accuracy and completeness on every benchmark with depth, using π3's protocol on 7-Scenes and Neural-RGBD (our implementation reproduces π3's published numbers within 0.002). Each cloud fuses all the views of a clip, so these errors also reflect cross-view consistency. Accuracy / completeness in meters, for the post-submission ViT-B:
In domain (ScanNet), our model is ahead of VGGT and VGGT-Ω and behind π3. On Neural-RGBD, its errors are about twice those of VGGT and π3, which we report as a limitation.
W2. Use beyond depth and cameras. Within geometry, we add three evaluations:
Transfer to non-geometric tasks is outside our scope, and the revision will say so.
W3. Comparison with other pre-training objectives. We compare against multi-view masked modeling with dropped patches (MuM's objective), with the same architecture, data, budget and fine-tuning. The first row is Table 1's setting. Every row trains for 120k steps in total, so its ViT-B is not the longer-trained model of W6. ScanNet AbsRel, fine-tuned / frozen:
With the objective as learned since submission, covisibility is better than masked modeling.