acceptodds
Under review as a conference paper at ICLR 2027

Pretraining registers with covisibility for long-sequence 3D reconstruction

Abstract

Feed-forward reconstruction transformers rely on all-to-all patch attention across views (global attention), whose quadratic cost in views and patches prevents scaling to a large number of views. Prior attempts to reduce this cost by routing cross-view information through a compact set of per-frame tokens while keeping patch attention intra-frame (register attention) report a significant accuracy drop. We show this gap is largely an initialization issue, not a capacity limitation of register attention. By pretraining the registers on a multi-view covisibility pretext task extended from a two-view, static-scene setting to long, dynamic sequences with a new view-conditioned, query-based segmentation head before finetuning on 3D reconstruction, our model recovers most of the global attention performance. On long sequences, where global attention’s quadratic cost becomes prohibitive, our approach scales efficiently and enables training on long sequences. This curriculum makes register attention a scalable alternative that outperforms chunked global attention on hundreds of views.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-sequence 3D reconstruction, Geometric Pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZJAGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors show that using only register-attention in their multi-view transformer actually works, in contrast to the results of VGGT-Omega, and do so by proposing a novel objective of multi-view covisibility estimation. This leads to a model that scales significantly better to longer sequences, as the number of tokens in the global attention is significantly reduced. While pairwise covisibility scales quadratically, they propose to instead use a sampling strategy. They further propose to deal with dynamic scenes by using epipolar geometry.

The paper provides clear experimental evidence supporting their hypothesis, both in controlled experiments and when comparing to the state-of-the-art.

Critical strengths
  • The manuscript is well-written and clearly structured, detailing a strong story-line that is easy to follow and well-motivated.
  • I think the experiments are really good. I like the very detailed controlled experiments and iso-flop comparisons. The paper follows this with a scaled-up state-of-the-art run, which is appreciated.
  • The experiments support the central claims well and the finding is quite surprising and novel: you don't need global attention if done right.
  • Multi-view covisibility estimation is a natural, but interesting, extension to the two-view Alligat0r paper and I feel the authors accurately position themselves in relation to the field.
Critical weaknesses
  • I am missing an ablation on the specific effect of registers compared to global attention. It seems to currently be intertwined with the change in objective from training from scratch versus using covisibility (if I am understanding the experiments correctly, please correct me otherwise).
  • There is no real clear motivation for why register attention would work. I can tell that it does, which is nice, but would be interesting with some more qualitative / discussion about why it did not work in VGGT-Omega but does here (just saying covisibility is good does not feel like it separates the two things enough). I do see some discussion about this, but might be interesting to hear more / backed up by qualitative or quantitative experiments.

Consistent with both of my weaknesses is the feeling that covisibility and using register-attention are quite intertwined in the experiments and it is not obvious to me why using covisibility suddenly saves the register-attention method.

The authors say in the abstract:

On long sequences, where global attention’s quadratic cost becomes prohibitive, our approach scales efficiently and enables training on long sequences. From my understanding, your method still scales quadratically with the sequence length (however, it is more efficient as you essentially run at a lower resolution thanks to only using register attention). Please consider revising this sentence of the abstract, the current framing makes it appear as if your scaling is sub-quadratic with sequence length.

Questions for authors
  • For your final model, could you provide the results on some more standard benchmarks for relative pose estimation and point cloud estimation? Depending on what is convenient, I suggest you try 7Scenes, ETH3D, or RE10K for multi-view relative pose (only running on 8 or 10 views) and e.g. DTU for point cloud estimation. You can find the code for conducting such an experiment in the Pi3 repo (under the evaluation branch). I feel it is good to provide some further transparency on this and put it in the supplementary, though, I understand that it is not the main point of the paper.
  • How is VRAM affected during training for different sequence lengths when using full register attention, as you do, compared to the 25% that is used by VGGT-Omega?
  • Did you try varying the proportion of your layers that are register-attention (currently 100%)? It is not clear to me if this paper indicates that register attention is as good as global attention when pretrained with covisibility or if it means that just the covisibility objective is a lot better and can save the bad register attention (what would happen if you had e.g. no register attention but still your proposed covisibility training)?
4 comments

We thank the reviewer for the feedback. Below we address their concerns. New experiments are integrated in the revision.

W1 and Q3. Register attention vs. global attention

The reviewer is right: in the submission, the change of objective (scratch → covisibility pretraining) was only applied to register attention. We ran the missing controlled ablation, crossing register and global cross-view attention with training from scratch and covisibility pretraining, with the same ViT-S model, data, finetuning objective and total budget of 300k steps (Table R8; same setting as in our answer to Reviewer EZkY). Covisibility pretraining is a good objective for both attention types, in line with the two-view results of Alligat0r [Loiseau et al., NeurIPS 2025]: it improves global attention by +22.4 / +15.3 and register attention by +12.6 / +12.7, which answers what happens with covisibility pretraining but no register attention. Register attention is harder to train from scratch (6.6 / 8.0 points behind global attention, which reproduces the VGGT-Omega observation), but covisibility pretraining puts it above global attention trained from scratch. At equal supervision, global attention keeps an advantage (+16.4 / +10.6): it can match every patch across views, whereas our registers compress each frame into 17 tokens (Fig. 5 of the revision). Register attention is, however, much cheaper: global attention is 2.5× slower per training step at 24 views, 3.1× slower at 48 views and more than 10× slower beyond 360 views (Q2). This is what makes long sequences affordable, where our largest gains appear (Table R7 in our answer to Reviewer rmL4: +15.5 AUC@30 on ScanNet at 512 frames, +24.7 on KITTI at 256 frames).

Table R8. Pose AUC@20 on ScanNet++ / Kubric (24 views).

Cross-view attentionFrom scratch (300k)Covisibility PT + FT (200k + 100k)
Global attention40.8 / 63.463.2 / 78.7
Register attention (ours)34.2 / 55.446.8 / 68.1
Global − register+6.6 / +8.0+16.4 / +10.6

W2. Why register attention works, and why covisibility saves it

Our intuition is that covisibility makes the cross-view supervision dense. When trained from scratch, the only supervision that requires cross-view information is the relative camera pose, a low-dimensional representation per frame: pointmap prediction is dense but expressed in frame 0, and depth can largely be predicted from a single frame. Consequently, only a few numbers per frame (a 4×4 camera pose matrix) force the registers to carry geometry across views. Covisibility, in contrast, gives a dense, per-pixel, cross-view signal: whether a pixel is visible, occluded or out of view in another frame depends on relative pose, depth and occlusion, and in our head it can only be predicted through the other frame's registers. By sampling informative pairs, we also constrain the whole pose graph, beyond the relation to frame 0. Our objective ablations support this: only dense, per-pixel covisibility helps (Table R9, Tables 1 and A2 of the paper).

Table R9. ScanNet++ camera pose AUC@10 after finetuning, for different pretraining objectives (ViT-S, 12 views).

Pretraining objectiveCross-view signalScanNet++ AUC@10
None (scratch)none29.8
Covisibility matrixone number per frame pair30.1
Relative camera poseone pose per frame pair30.8
Two-view dense covisibilityper pixel38.9
Multi-view covisibility through registers (ours)per pixel, through registers42.9

We have further visualized the cross-view attention of the models of Table R8 in Fig. 5 of the revision, and measured it on 634 ScanNet++ sequences (128 for global attention) and 100 Kubric sequences of 24 views. Covisibility pretraining makes the cross-view attention follow frame overlap, for both attention types.

From scratch, register attention weakly follows frame overlap and falls back on frame 0, in which all cameras are expressed (20% of its cross-view attention, against 4% for uniform attention). Covisibility pretraining fixes this: the attention follows the ground-truth overlap, without the frame-0 sink (4%). After finetuning (ours), the overlap structure remains and frame 0 is attended again (21%), as an anchor for the reference coordinate system. Global attention trained from scratch partly follows the overlap in its patch-to-patch attention, since every patch can directly attend to the patches of other frames, and covisibility pretraining makes this structure much clearer. After finetuning, both models keep this structure in their cross-view channel: the registers for register attention, which are its only channel across views, and the patches for global attention.

This explains why register attention lags behind global attention when trained from scratch, as observed in VGGT-Omega, and overtakes global attention trained from scratch once pretrained with covisibility: from scratch, registers exchange only 17 tokens per frame and are supervised only through a few camera parameters, whereas covisibility pretraining directly supervises which frames, and which pixels, overlap. Global attention, which can also match patches directly, benefits even more (Table R8).

Complexity (abstract)

We agree that our method is not subquadratic. We clarified this in the revised version of the abstract with the following sentence “Our approach is an order of magnitude faster than global attention as only a few tokens per frame are exchanged across views, which enables training on long sequences”. We also formalized the complexity in the method section. Register attention remains quadratic in the number of views S, but with a very small constant: only the 17 registers of each frame attend across views, against 785 tokens per frame inside each frame, so the measured latency grows quasi-linearly up to 1,024 views (Fig. 4b of the paper).

Q1. Standard benchmarks with few views

We report RealEstate10K and 7-Scenes results with the Pi3 protocol in Tables R5 and R6 (see our answer to Reviewer rmL4). Our model is on par with VGGT-Omega on RealEstate10K camera pose (88.4 vs 88.8 AUC@30) and above Pi3 (85.9 reported, 87.1 with our code), and it is the best on every metric of dense 7-Scenes (13 to 25 video frames) among models we evaluated. On sparse 7-Scenes (3 to 5 widely spaced views), VGGT-Omega, which keeps global attention in most of its cross-view blocks, is better (Chamfer distance 0.047 vs 0.069), consistent with the advantage of global attention at equal supervision (Table R8), and we still outperform our reproduction of Pi3 on completion, Chamfer distance and normal consistency. On video frames, our target setting, our register-only model is thus on par on few views and better on long sequences (Table R7 in our answer to Reviewer rmL4: +15.5 AUC@30 at 512 frames on ScanNet).

Q2. VRAM vs. sequence length

Register attention reduces both the training memory and the step time, and the savings grow with the sequence length (Tables R10 and R11). We measure our ViT-L model (1.1B parameters, 24 frame + 24 cross-view blocks, 16 registers) on one H100 (80 GB) with batch size 1, bf16, FlashAttention enabled, and 384×512 images. VGGT-Omega uses global attention in 19 of the 24 cross-view blocks.

Table R10. Peak training memory without rematerialization (GiB).

Cross-view attentionS = 12S = 24S = 48
Global attention51.290.1 (OOM)168.0 (OOM)
VGGT-Omega49.085.6 (OOM)159.1 (OOM)
Register attention (ours)40.969.1125.9 (OOM)

Table R11. Step time with the lightest rematerialization that fits (s/step).

Cross-view attentionS = 12S = 24S = 48
Global attention0.290.721.87
VGGT-Omega0.280.601.66
Register attention (ours)0.180.290.60
  • Memory: the saving grows with S (10, 21 and 42 GiB vs global attention), because global layers store activations for S·785 tokens and register layers only for S·17. At S = 24, ours is the only variant that trains without any rematerialization. The longest sequence that fits without rematerialization is 27 views for ours vs 20 for global attention, and 48 vs 30 with encoder rematerialization only.
  • Speed: the speed-up grows with S. With full rematerialization, all variants fit up to ~370 views, but at 360 views ours trains at 4.5 s/step vs 53.6 s/step for VGGT-Omega (11.9×).

Q3. Proportion of register layers

We varied it in early experiments (ViT-S, 12 views, ScanNet++, trained from scratch with the VGGT-Omega objective). Up to 75%, the differences are about 1 point and non-monotonic on camera pose, so intermediate ratios bring no accuracy benefit over global attention; only 100% register layers clearly drops performance, as also found in VGGT-Omega. Intermediate ratios also lose most of the efficiency, since a single global layer brings back the quadratic all-patch attention: at S = 24, the VGGT-Omega layout saves 5% memory and 17% step time compared with global attention, against 23% and 60% for 100% register layers (Q2). We therefore focus on 100% register layers, the only layout that allows training on long sequences, and close its accuracy gap with better supervision. If the reviewer thinks it is important to add them to the paper, we can re-run these preliminary experiments on our current setup.

"Is register attention as good as global attention, or does covisibility save the bad register attention?" Table R8 answers this directly, since it contains both 0% and 100% register layers, each with and without covisibility pretraining, and the new Fig. 5 of the revision visualizes the mechanism. Covisibility is a good pretraining objective in itself, which also improves global attention, and it is what makes register attention competitive. We therefore claim that, with dense cross-view supervision, a 100%-register model matches or exceeds global attention trained from scratch at a fraction of the cost, while global attention remains more accurate at equal supervision. Covisibility pretraining applies to any ratio of global and register layers, and we will study at which ratio the attention collapses on frame 0.

Reviewer_ZJAGOpenReview ↗

Response

Thank you for the detailed experiments and addressing my concerns. I think it will improve the paper.

I remain optimistic about the paper and will keep my score of 3 (weak acceptance). Which means I favor the paper for acceptance but won't champion it more than that.

I consider the core findings interesting and the paper to be quite timely.

Reviewer_rmL4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new method to reduce computation costs and errors of long-sequence 3D reconstruction for the register-only model. To mitigate the negative influence on register-only cross-view attention setting, this paper introduce a pretraining stage where the dense covisibility head is introduced to strength the representational capacity of register tokens. After the pretraining, this paper processes the finetuning to accomplish the whole feed-forward 3D reconstruction model. Experimental results demonstrate that the register-only design reduces computation costs, while the proposed approach improves camera pose accuracy on long-sequence 3d reconstruction tasks.

Critical strengths
  1. This paper is well-written, the core improvements are clearly presents.

  2. The proposed method is technically sound, by introducing an extra pretraining stage, the bottleneck in the register-only model has been mitigated, thus the computation costs on long-sequence 3d reconstruction is reduced significantly.

Critical weaknesses
  1. The novelty of this paper is somewhat limited. The main contribution seems to be adapting existing covisibility pretraining to register-only models for long-sequence 3D reconstruction. Besides, the experiments in this paper are limited, only a limited number of datasets and algorithms are used for comparison

  2. The conclusions are also somewhat weak. For example, the improved performance on long-sequence tasks may benefit from training with more frames rather than solely from the two-stage training setting.

Questions for authors
  1. Could the authors provide more comprehensive experiments and visualizations, similar to those presented in π3\pi^3 (wang et al.)

  2. Could the authors provide more analysis and additional ablation studies to explain the improvement in long-sequence metrics, e.g., by keeping the number of frames per training sequence consistent across the compared methods?

4 comments

We thank the reviewer for the feedback. Below we address their concerns. New experiments are integrated in the revision, and their setup is described at the end of this response.

W1. Novelty

Our main contribution is an in-depth analysis of a surprising result (the fact that register attention performs well if pretrained with covisibility), along with design choices that enable effective pretraining on long sequences (new covisibility head architecture, incorporation of dynamic regions, pair sampling…). In the following, we reiterate the contributions in detail.

  • A finding. VGGT-Omega [Wang et al., CVPR 2026] concluded that register-only cross-view attention loses accuracy. We show that most of this loss is a supervision problem: in a new controlled 2×2 ablation (Table R8 in our answer to Reviewer ZJAG), covisibility pretraining raises register attention from 34.2 to 46.8 pose AUC@20 on ScanNet++ (55.4 → 68.1 on Kubric), above global attention trained from scratch (40.8 / 63.4), with training steps 2.5× cheaper at 24 views and over 10× cheaper beyond 360 views. Global attention benefits even more from our objective (63.2 / 78.7), at a much higher cost.
  • A new objective and a new covisibility head. Alligat0r [Loiseau et al., NeurIPS 2025] predicts covisibility for two static views with a dense decoder over the pair. We predict per-pixel covisibility between many views of a possibly dynamic sequence, through the registers of the target view (view-conditioned query head), with pair sampling and dynamic-region masking, so we avoid dense decoding of all S × S view pairs. On ScanNet++ (AUC@10), a two-view dense covisibility objective gives 38.9, ours gives 42.9 (scratch 29.8; covisibility matrix 30.1; relative pose 30.8). To our knowledge, no prior work initializes ‘object’ queries from other tokens to segment geometric quantities.
  • An explanation of why it works (new, see our answer to Reviewer ZJAG, W2). After covisibility pretraining, the attention pattern follows frame overlap instead of relying too much on the first frame. Global attention already partly matches frames through its patches from scratch, and covisibility pretraining improves it, too.
  • A practical consequence. Only a few tokens per frame are exchanged across views, and this is the key mechanism that makes training on longer sequences feasible.

W1 and Q1. Limited evaluation datasets and baselines

Besides the three datasets used in the submission, we have added three additional benchmarks on RealEstate10K, 7-Scenes (sparse set) and 7-Scenes (dense set) from the Pi3 official evaluation [Wang et al., ICLR 2026] (Q1), as well as evaluating sequences of 24 to 1,024 frames on ScanNet and KITTI (Table R7). We compare to Pi3 and VGGT-Omega [Wang et al., CVPR 2026] on all of them. On RealEstate10K, our model trained on at most 24 frames is on par with VGGT-Omega (88.4 vs 88.8 AUC@30) and above Pi3 (85.9 reported, 87.1 with our code). On dense 7-Scenes (13 to 25 video frames), it is the best on every metric among models re-evaluated by us. On sparse 7-Scenes (3 to 5 widely spaced views), VGGT-Omega, which keeps global attention in most of its cross-view blocks, is better (Chamfer distance 0.047 vs 0.069), consistent with the advantage of global attention at equal supervision (Table R8 in our answer to Reviewer ZJAG), and we still outperform our reproduction of Pi3 on completion, Chamfer distance and normal consistency. On video frames, our target setting, our register-only model is thus on par with global attention on few views and better on long sequences (Table R7).

Table R5. Camera pose on RealEstate10K (1,719 clips of 10 frames, all frame pairs). AUC@30 ↑, ATE (m) ↓.

MethodAUC@30ATE
Pi3 [Wang et al., ICLR 2026]85.9n/a
Pi3 (our reproduction)87.10.0171
VGGT-Omega88.80.0149
Ours88.40.0160

Table R6. Point maps on 7-Scenes (m). Mean accuracy ↓ / completion ↓ / Chamfer distance ↓ / normal consistency ↑. Sparse: 3 to 5 views. Dense: 13 to 25 views.

MethodSparseDense
Pi3 [Wang et al., ICLR 2026]0.047 / 0.075 / n/a / 0.7420.016 / 0.022 / n/a / 0.689
Pi3 (our reproduction)0.062 / 0.095 / 0.079 / 0.7080.021 / 0.032 / 0.026 / 0.675
VGGT-Omega (our reproduction)0.042 / 0.052 / 0.047 / 0.7260.029 / 0.037 / 0.033 / 0.657
Ours0.062 / 0.076 / 0.069 / 0.7200.018 / 0.028 / 0.023 / 0.676

Bold: best among models evaluated with our code.

We evaluate our ViT-L model trained on clips of at most 24 frames (Sec. A.4.2 of the revision). None of the evaluation datasets is used for training. Frame selection, alignment and metrics follow Pi3; the "Pi3 [Wang et al., ICLR 2026]" rows are the numbers of the Pi3 paper, and the other models are evaluated with our code at their native resolution (ours at 384×512). Pi3 predicts point maps with a dedicated head, whereas we unproject our depth with the predicted intrinsics and poses.

Visualizations. We added qualitative reconstructions in the paper (Fig. A5 in Section A.4.5) in a similar spirit to Pi3. We can provide predicted covisibility maps or more bird’s-eye-view trajectories if requested.

W2 and Q2. Is the long-sequence gain due to training on more frames?

A model trained only on short clips already shows the long-sequence gain. We evaluate the model of Q1, trained only on clips of 2 to 24 frames like the baselines, on 24 to 1,024 frames (Table R7): it outperforms all baselines at every length on both datasets. It is already better at 24 and 48 frames, where all models process the whole sequence at once (+2.3 / +1.5 AUC@30 on ScanNet, +1.1 / +0.4 on KITTI), and the gap grows with the sequence length (up to +15.5 on ScanNet at 512 frames and +24.7 on KITTI at 256 frames), as we use a single 512-frame window instead of 64-frame windows stitched with Sim(3) alignment, which accumulates drift. The gain therefore comes from covisibility pretraining, which makes register attention competitive (Table R8 in our answer to Reviewer ZJAG), and from the larger windows that register attention makes affordable; global attention cannot be trained on such numbers of frames at a reasonable cost (Tables R10 and R11 in our answer to Reviewer ZJAG).

Table R7. Camera pose AUC@30 vs. number of input frames. All models are trained on at most 24 frames per sample. VGGT-Omega uses its more recent 416 checkpoint (Table 3 of the paper uses the 512 checkpoint).

Method / frames2448961282565121,024
ScanNet
VGGT-Omega56.667.373.573.069.863.053.6
Pi356.167.873.671.263.253.238.9
Ours, short clips only58.969.374.976.378.578.558.0
KITTI
VGGT-Omega88.485.470.460.538.324.112.6
Pi390.287.281.167.048.633.918.5
Ours, short clips only91.387.684.584.173.343.122.9

All rows use the same evaluation code, each model at its native resolution, with 512-frame windows for our model and 64-frame windows for the global-attention baselines (Sec. A.4.3 of the revision). Since this model also differs from the submitted one in its training data and schedule (see below), we see longer training sequences as a complementary benefit, affordable only with register attention (69 vs 90 GiB peak training memory at 24 views and 3.1× faster training steps at 48 views, Tables R10 and R11 in our answer to Reviewer ZJAG).

Setup of the experiments. We evaluate a ViT-L model pretrained with covisibility for 330k steps and finetuned for 300k steps on clips of at most 24 frames, without our long-sequence finetuning. This model uses a slightly different training data mixture (adding Waymo [Sun et al., CVPR 2020]), a longer schedule than the models in the submission and 64 registers (+5% more compute in total when training on 24 views).

Thanks a lot for the authors' responses, which have addressed most of my concerns. I believe the paper has been significantly improved, and I would like to raise my rating from 2 to 3.

Reviewer_EZkYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the accuracy degradation observed when cross-view communication in feed-forward 3D reconstruction models is restricted to register tokens. Inspired by Alligat0r, the authors propose covisibility pretraining using a view-conditioned segmentation head, sampled view pairs, and dynamic-region masking. The pretrained model is subsequently finetuned for 3D reconstruction with progressively longer sequences. The approach substantially improves register-attention models and demonstrates favorable efficiency and long-sequence camera pose accuracy.

Critical strengths

1. Well-motivated problem and effective training approach. Understanding how to retain reconstruction accuracy while reducing cross-view attention costs is important. The paper demonstrates that the performance degradation of register attention can be substantially mitigated through their proposed pretraining on covisbility-region segmentation.

2. Controlled evidence of effectiveness. Table 1 and Fig. 3(a) show clear improvements under a matched total training-data budget, showing that stage 1 pretraining not only improves register attention modeling but also outperforms global attention modeling.

Critical weaknesses

W1. Reliance on heuristic supervision in Stage 1. Covisibility labels rely on heuristic thresholds and geometric rules for masking dynamic regions. Their sensitivity to these choices and robustness to noisy depth or camera poses are not evaluated. Additionally, the optimal pretraining setup may vary across datasets.

W2. Concern about framing Stage 1 as “pretraining.” In the pretraining–finetuning paradigm, pretraining typically aims to learn broadly transferable knowledge that can be adapted to downstream tasks. However, Stage 1 appears tailored to reconstruction and relies on heuristic supervision, leaving it unclear whether it learns general geometric representations.

Questions for authors

Q1. Ablation on the role of Stage 1 “pretraining”: Does Stage 1 serve as pretraining, or is it just a suitable initialization? Under the same total budget of 40 epochs, would allocating fewer epochs to Stage 1 than the current 25/15 pretraining–finetuning split achieve comparable performance? If so, this would suggest that Stage 1 primarily provides a suitable initialization rather than serving as broader pretraining.

3 comments

We thank the reviewer for the feedback. Below we address their concerns. New experiments are integrated in the revision, and their common setup is described at the end of this response.

W1. Reliance on heuristic supervision

(a) Sensitivity to the heuristic thresholds. The covisible/occluded depth tolerance τ follows prior work [Loiseau et al., NeurIPS 2025]. We relabel 200 training clips per dataset (4,800 view pairs each for ScanNet++ and Kubric) with τ ∈ {0.01, 0.03, 0.05 (ours), 0.10, 0.15, 0.20} and observe that the label statistics remain stable (Table R1). Over a 7× range (τ ∈ [0.03, 0.20]), the occluded share changes by at most 0.6 point of all pixels (2.8% → 2.2% on ScanNet++, 1.5% → 1.4% on Kubric), because τ only moves pixels at depth discontinuities between covisible and occluded, while out-of-view and missing-depth pixels do not depend on τ. The dynamic-region rule only removes pixels from the loss (0.2–1.2% of the pixels on static real scenes, and the moving objects on dynamic synthetic scenes), so it can reduce the supervision but cannot corrupt the labels. It uses the same tolerance τ, so a larger τ flags fewer pixels as moving and returns them to the loss, mostly as covisible, which is why the covisible share of Kubric grows by 1.6 points between τ = 0.03 and τ = 0.20 (Table R1).

Table R1. Label statistics as the depth tolerance τ varies, on 200 training clips per dataset. "Covisible / occluded" is the % of all pixels. Out-of-view pixels (ScanNet++ 58.7%, Kubric 10.8%) and pixels with invalid depth do not depend on τ.

Datasetτ = 0.01τ = 0.03τ = 0.05 (ours)τ = 0.10τ = 0.15τ = 0.20
ScanNet++Covisible / occluded (%)31.1 / 3.332.2 / 2.832.4 / 2.732.7 / 2.532.9 / 2.333.0 / 2.2
KubricCovisible / occluded (%)52.5 / 2.054.0 / 1.554.3 / 1.454.8 / 1.455.2 / 1.455.6 / 1.4

(b) Robustness to noisy depth and camera poses. Pretraining on real-only data tests this directly: real labels are computed from noisy sensor depth and SfM poses, so if the heuristic labels were too noisy, Stage 1 on real data alone would not help. We run Stage 1 with the same setup and thresholds on real data only (ScanNet++, CO3D and others, without any synthetic data), then finetune it for 100k steps on point tracking with synthetic tracks, a task that requires precise pixel correspondences (Table R2). Pretrained only on noisy real geometry, the model doubles the tracking accuracy on real wide-baseline pairs from held-out scenes (ScanNet1500, the 1,500 test pairs of SuperGlue [Sarlin et al., CVPR 2020], also used in Alligat0r): AJ goes from 19.2 to 40.7. On synthetic Kubric, which this Stage 1 never sees, it stays below the baseline (93.5 vs 96.0), which trains on Kubric for 300k steps instead of 100k. The same setup and thresholds thus work across very different sensors and datasets, and the benefit grows with the coverage of the Stage-1 data: our full mixture further improves ScanNet1500 (47.9 AJ) and ScanNet++ (+5.3 AJ over no Stage 1), which is why our recipe uses a large, diverse mixture.

Table R2. Point tracking (AJ ↑) after Stage 1 on real data only.

Stage 1 dataScanNet1500 pairs (real, wide baseline, held-out scenes)ScanNet++ (real)Kubric (synthetic, in FT data)
No Stage 1 (300k)19.257.896.0
Real data only (200k) → FT (100k)40.760.593.5
Full mixture (200k) → FT (100k), from Table R347.963.195.5

W2. Is Stage 1 "pretraining"? Does it learn transferable geometric representations?

While Stage 1 indeed provides a good initialization for reconstruction, we test whether it also learns transferable geometric representations, as requested by the reviewer. We finetune a single Stage-1 checkpoint (full mixture) on three downstream tasks individually, each with a new task head: camera pose only (VGGT-Omega [Wang et al., CVPR 2026] camera head), reconstruction (camera and depth heads), and point tracking as a new task (a TAPIR-style correlation head [Doersch et al., ICCV 2023] finetuned on synthetic tracks only, evaluated on query points and tracks derived from the depth and camera poses of the validation splits). We compare each against the same model trained from scratch for 300k steps on that task only. The same Stage-1 representation improves all three tasks at a matched budget, with heads it has never seen (Table R3); the only exception is Kubric tracking, where both models are close to saturation (95.5 vs 96.0). Stage 1 therefore learns a multi-view geometric representation that transfers beyond reconstruction. This matches what covisibility encodes: which pixels observe the same 3D surface across views, a signal shared by pose, depth and correspondence.

Table R3. Transfer of one Stage-1 checkpoint to three tasks, at a matched total budget.

Task (new head)MetricNo Stage 1 (300k)Stage 1 (200k) → FT (100k)
Camera pose onlyScanNet++ AUC@20 ↑38.048.1
Kubric AUC@20 ↑57.171.6
ReconstructionScanNet++ AUC@20 ↑34.246.8
Kubric AUC@20 ↑55.468.1
ScanNet++ depth AbsRel ↓0.0580.057
Point tracking (new task)ScanNet++ AJ ↑57.863.1
Kubric AJ ↑96.095.5

Q1. Pretraining or "just a suitable initialization"? Impact of the pretraining/finetuning schedule

We follow the protocol suggested by the reviewer: a fixed total budget of 300k steps with different Stage 1 / Stage 2 splits, each an independent run with its own schedule. In Table R4, we see that allocating fewer steps to Stage 1 does not reach comparable performance: a short Stage 1 (50k steps) brings no gain over training from scratch, and performance then grows steadily with the Stage-1 budget, up to +12.6 / +12.7 AUC@20 (ScanNet++ / Kubric) for the full Stage 1.

Table R4. Stage 1 / Stage 2 split at a fixed total budget of 300k steps (reconstruction).

Stage 1 / Stage 20 / 300k (scratch)50k / 250k100k / 200k200k / 100k (ours)
ScanNet++ AUC@20 ↑34.232.741.246.8
Kubric AUC@20 ↑55.454.163.568.1

Setup of all new experiments. We use a ViT-S/16 encoder (DINOv3 initialization) with our register aggregator (12 layers, 16 registers per frame, register-only cross-view attention) and no other architectural change, trained on the same 2–24-view data mixture as our large-scale model. Stage 1 runs for 200k steps and Stage 2 for 100k; every baseline is the same model trained for 300k steps without Stage 1, i.e., a matched total budget, and every run has its own optimized cosine schedule. As in the paper's ablations, we evaluate on the ScanNet++ and Kubric validation splits of our small-scale experiments.

We added these experiments to the paper (Appendix A.1.3, A.2.4 and A.3.6).

References

  • [Doersch et al., ICCV 2023] C. Doersch, Y. Yang, M. Večerík et al. TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement. ICCV 2023.
  • [Sarlin et al., CVPR 2020] P.-E. Sarlin, D. DeTone, T. Malisiewicz, A. Rabinovich. SuperGlue: Learning Feature Matching with Graph Neural Networks. CVPR 2020.

Thank you for the detailed response and additional experiments.

Concerns addressed. Table R3 provides convincing evidence that Stage 1 transfers beyond reconstruction, addressing my concern about the term “pretraining.” Table R4 also shows that a substantial Stage 1 budget is important.

Remaining limitation. The method still relies on heuristic label construction, and Table R1 measures label stability rather than the effect of threshold choices on downstream performance.

Although the method relies on some heuristics, its results offer a useful clue for designing more effective supervision for register-based cross-view modeling. I find this insight meaningful and have raised my rating to 3.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback. Below we summarize the changes in the revised PDF (marked in blue).

In the main paper, Table 2 and Figure 5 are new. In the appendix, Tables A1 and A4–A9 and Figure A5 are new. The numbering of some of the original tables in the submitted version has therefore changed. All the tables in the rebuttal (marked with the prefix R) are integrated in the revision. Our rebuttal tables appear in the revision as Table R1 → Table A1, R2 → A5, R3 → A4, R4 → A6, R5–R6 → A8, R7 → A9, R8 → Table 2, R10–R11 → A7.

  • Main paper: the role of cross-view supervision in the accuracy drop of register attention and the revised efficiency claim (Abstract, Sections 1, 2 and 5, ZJAG W1/Q3 and complexity), the complexity of register attention (Section 3.1, ZJAG complexity), register vs. global attention with and without covisibility pretraining (Section 4.1, Table 2, ZJAG W1/Q3, rmL4 W1), the cross-view attention analysis (Section 4.3, Figure 5, ZJAG W2), and pointers to the new appendix results (Sections 3.2, 4.1 and 4.2).
  • Appendix A.1.3: sensitivity of the labels to the depth tolerance (Table A1, EZkY W1).
  • Appendices A.2.3–A.2.4: attention statistics, transfer of pretraining to three different tasks, Stage I on real data only (noisy depth and cameras), and Stage I / Stage II split (Tables A4–A6, ZJAG W2, EZkY W1/W2/Q1).
  • Appendices A.3.1 and A.3.6: training data of the new models and the extended ablation setting.
  • Appendices A.4.2–A.4.5: few-view benchmarks (Table A8, ZJAG Q1, rmL4 W1/Q1), long sequences with training on short clips only (Table A9, rmL4 W2/Q2), training memory and step time (Table A7, ZJAG Q2), and qualitative reconstructions (Figure A5, rmL4 Q1).

We thank the Area Chair and reviewers for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.