- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors show that using only register-attention in their multi-view transformer actually works, in contrast to the results of VGGT-Omega, and do so by proposing a novel objective of multi-view covisibility estimation. This leads to a model that scales significantly better to longer sequences, as the number of tokens in the global attention is significantly reduced. While pairwise covisibility scales quadratically, they propose to instead use a sampling strategy. They further propose to deal with dynamic scenes by using epipolar geometry.
The paper provides clear experimental evidence supporting their hypothesis, both in controlled experiments and when comparing to the state-of-the-art.
Critical strengths
- The manuscript is well-written and clearly structured, detailing a strong story-line that is easy to follow and well-motivated.
- I think the experiments are really good. I like the very detailed controlled experiments and iso-flop comparisons. The paper follows this with a scaled-up state-of-the-art run, which is appreciated.
- The experiments support the central claims well and the finding is quite surprising and novel: you don't need global attention if done right.
- Multi-view covisibility estimation is a natural, but interesting, extension to the two-view Alligat0r paper and I feel the authors accurately position themselves in relation to the field.
Critical weaknesses
- I am missing an ablation on the specific effect of registers compared to global attention. It seems to currently be intertwined with the change in objective from training from scratch versus using covisibility (if I am understanding the experiments correctly, please correct me otherwise).
- There is no real clear motivation for why register attention would work. I can tell that it does, which is nice, but would be interesting with some more qualitative / discussion about why it did not work in VGGT-Omega but does here (just saying covisibility is good does not feel like it separates the two things enough). I do see some discussion about this, but might be interesting to hear more / backed up by qualitative or quantitative experiments.
Consistent with both of my weaknesses is the feeling that covisibility and using register-attention are quite intertwined in the experiments and it is not obvious to me why using covisibility suddenly saves the register-attention method.
The authors say in the abstract:
On long sequences, where global attention’s quadratic cost becomes prohibitive, our approach scales efficiently and enables training on long sequences. From my understanding, your method still scales quadratically with the sequence length (however, it is more efficient as you essentially run at a lower resolution thanks to only using register attention). Please consider revising this sentence of the abstract, the current framing makes it appear as if your scaling is sub-quadratic with sequence length.
Questions for authors
- For your final model, could you provide the results on some more standard benchmarks for relative pose estimation and point cloud estimation? Depending on what is convenient, I suggest you try 7Scenes, ETH3D, or RE10K for multi-view relative pose (only running on 8 or 10 views) and e.g. DTU for point cloud estimation. You can find the code for conducting such an experiment in the Pi3 repo (under the evaluation branch). I feel it is good to provide some further transparency on this and put it in the supplementary, though, I understand that it is not the main point of the paper.
- How is VRAM affected during training for different sequence lengths when using full register attention, as you do, compared to the 25% that is used by VGGT-Omega?
- Did you try varying the proportion of your layers that are register-attention (currently 100%)? It is not clear to me if this paper indicates that register attention is as good as global attention when pretrained with covisibility or if it means that just the covisibility objective is a lot better and can save the bad register attention (what would happen if you had e.g. no register attention but still your proposed covisibility training)?
4 comments
W2. Why register attention works, and why covisibility saves it
Our intuition is that covisibility makes the cross-view supervision dense. When trained from scratch, the only supervision that requires cross-view information is the relative camera pose, a low-dimensional representation per frame: pointmap prediction is dense but expressed in frame 0, and depth can largely be predicted from a single frame. Consequently, only a few numbers per frame (a 4×4 camera pose matrix) force the registers to carry geometry across views. Covisibility, in contrast, gives a dense, per-pixel, cross-view signal: whether a pixel is visible, occluded or out of view in another frame depends on relative pose, depth and occlusion, and in our head it can only be predicted through the other frame's registers. By sampling informative pairs, we also constrain the whole pose graph, beyond the relation to frame 0. Our objective ablations support this: only dense, per-pixel covisibility helps (Table R9, Tables 1 and A2 of the paper).
Table R9. ScanNet++ camera pose AUC@10 after finetuning, for different pretraining objectives (ViT-S, 12 views).
| Pretraining objective | Cross-view signal | ScanNet++ AUC@10 |
|---|---|---|
| None (scratch) | none | 29.8 |
| Covisibility matrix | one number per frame pair | 30.1 |
| Relative camera pose | one pose per frame pair | 30.8 |
| Two-view dense covisibility | per pixel | 38.9 |
| Multi-view covisibility through registers (ours) | per pixel, through registers | 42.9 |
We have further visualized the cross-view attention of the models of Table R8 in Fig. 5 of the revision, and measured it on 634 ScanNet++ sequences (128 for global attention) and 100 Kubric sequences of 24 views. Covisibility pretraining makes the cross-view attention follow frame overlap, for both attention types.
From scratch, register attention weakly follows frame overlap and falls back on frame 0, in which all cameras are expressed (20% of its cross-view attention, against 4% for uniform attention). Covisibility pretraining fixes this: the attention follows the ground-truth overlap, without the frame-0 sink (4%). After finetuning (ours), the overlap structure remains and frame 0 is attended again (21%), as an anchor for the reference coordinate system. Global attention trained from scratch partly follows the overlap in its patch-to-patch attention, since every patch can directly attend to the patches of other frames, and covisibility pretraining makes this structure much clearer. After finetuning, both models keep this structure in their cross-view channel: the registers for register attention, which are its only channel across views, and the patches for global attention.
This explains why register attention lags behind global attention when trained from scratch, as observed in VGGT-Omega, and overtakes global attention trained from scratch once pretrained with covisibility: from scratch, registers exchange only 17 tokens per frame and are supervised only through a few camera parameters, whereas covisibility pretraining directly supervises which frames, and which pixels, overlap. Global attention, which can also match patches directly, benefits even more (Table R8).
Complexity (abstract)
We agree that our method is not subquadratic. We clarified this in the revised version of the abstract with the following sentence “Our approach is an order of magnitude faster than global attention as only a few tokens per frame are exchanged across views, which enables training on long sequences”. We also formalized the complexity in the method section. Register attention remains quadratic in the number of views S, but with a very small constant: only the 17 registers of each frame attend across views, against 785 tokens per frame inside each frame, so the measured latency grows quasi-linearly up to 1,024 views (Fig. 4b of the paper).
Q1. Standard benchmarks with few views
We report RealEstate10K and 7-Scenes results with the Pi3 protocol in Tables R5 and R6 (see our answer to Reviewer rmL4). Our model is on par with VGGT-Omega on RealEstate10K camera pose (88.4 vs 88.8 AUC@30) and above Pi3 (85.9 reported, 87.1 with our code), and it is the best on every metric of dense 7-Scenes (13 to 25 video frames) among models we evaluated. On sparse 7-Scenes (3 to 5 widely spaced views), VGGT-Omega, which keeps global attention in most of its cross-view blocks, is better (Chamfer distance 0.047 vs 0.069), consistent with the advantage of global attention at equal supervision (Table R8), and we still outperform our reproduction of Pi3 on completion, Chamfer distance and normal consistency. On video frames, our target setting, our register-only model is thus on par on few views and better on long sequences (Table R7 in our answer to Reviewer rmL4: +15.5 AUC@30 at 512 frames on ScanNet).
Q2. VRAM vs. sequence length
Register attention reduces both the training memory and the step time, and the savings grow with the sequence length (Tables R10 and R11). We measure our ViT-L model (1.1B parameters, 24 frame + 24 cross-view blocks, 16 registers) on one H100 (80 GB) with batch size 1, bf16, FlashAttention enabled, and 384×512 images. VGGT-Omega uses global attention in 19 of the 24 cross-view blocks.
Table R10. Peak training memory without rematerialization (GiB).
| Cross-view attention | S = 12 | S = 24 | S = 48 |
|---|---|---|---|
| Global attention | 51.2 | 90.1 (OOM) | 168.0 (OOM) |
| VGGT-Omega | 49.0 | 85.6 (OOM) | 159.1 (OOM) |
| Register attention (ours) | 40.9 | 69.1 | 125.9 (OOM) |
Table R11. Step time with the lightest rematerialization that fits (s/step).
| Cross-view attention | S = 12 | S = 24 | S = 48 |
|---|---|---|---|
| Global attention | 0.29 | 0.72 | 1.87 |
| VGGT-Omega | 0.28 | 0.60 | 1.66 |
| Register attention (ours) | 0.18 | 0.29 | 0.60 |
- Memory: the saving grows with S (10, 21 and 42 GiB vs global attention), because global layers store activations for S·785 tokens and register layers only for S·17. At S = 24, ours is the only variant that trains without any rematerialization. The longest sequence that fits without rematerialization is 27 views for ours vs 20 for global attention, and 48 vs 30 with encoder rematerialization only.
- Speed: the speed-up grows with S. With full rematerialization, all variants fit up to ~370 views, but at 360 views ours trains at 4.5 s/step vs 53.6 s/step for VGGT-Omega (11.9×).
Q3. Proportion of register layers
We varied it in early experiments (ViT-S, 12 views, ScanNet++, trained from scratch with the VGGT-Omega objective). Up to 75%, the differences are about 1 point and non-monotonic on camera pose, so intermediate ratios bring no accuracy benefit over global attention; only 100% register layers clearly drops performance, as also found in VGGT-Omega. Intermediate ratios also lose most of the efficiency, since a single global layer brings back the quadratic all-patch attention: at S = 24, the VGGT-Omega layout saves 5% memory and 17% step time compared with global attention, against 23% and 60% for 100% register layers (Q2). We therefore focus on 100% register layers, the only layout that allows training on long sequences, and close its accuracy gap with better supervision. If the reviewer thinks it is important to add them to the paper, we can re-run these preliminary experiments on our current setup.
"Is register attention as good as global attention, or does covisibility save the bad register attention?" Table R8 answers this directly, since it contains both 0% and 100% register layers, each with and without covisibility pretraining, and the new Fig. 5 of the revision visualizes the mechanism. Covisibility is a good pretraining objective in itself, which also improves global attention, and it is what makes register attention competitive. We therefore claim that, with dense cross-view supervision, a 100%-register model matches or exceeds global attention trained from scratch at a fraction of the cost, while global attention remains more accurate at equal supervision. Covisibility pretraining applies to any ratio of global and register layers, and we will study at which ratio the attention collapses on frame 0.
Response
Thank you for the detailed experiments and addressing my concerns. I think it will improve the paper.
I remain optimistic about the paper and will keep my score of 3 (weak acceptance). Which means I favor the paper for acceptance but won't champion it more than that.
I consider the core findings interesting and the paper to be quite timely.
We thank the reviewer for the feedback. Below we address their concerns. New experiments are integrated in the revision.
W1 and Q3. Register attention vs. global attention
The reviewer is right: in the submission, the change of objective (scratch → covisibility pretraining) was only applied to register attention. We ran the missing controlled ablation, crossing register and global cross-view attention with training from scratch and covisibility pretraining, with the same ViT-S model, data, finetuning objective and total budget of 300k steps (Table R8; same setting as in our answer to Reviewer
EZkY). Covisibility pretraining is a good objective for both attention types, in line with the two-view results of Alligat0r [Loiseau et al., NeurIPS 2025]: it improves global attention by +22.4 / +15.3 and register attention by +12.6 / +12.7, which answers what happens with covisibility pretraining but no register attention. Register attention is harder to train from scratch (6.6 / 8.0 points behind global attention, which reproduces the VGGT-Omega observation), but covisibility pretraining puts it above global attention trained from scratch. At equal supervision, global attention keeps an advantage (+16.4 / +10.6): it can match every patch across views, whereas our registers compress each frame into 17 tokens (Fig. 5 of the revision). Register attention is, however, much cheaper: global attention is 2.5× slower per training step at 24 views, 3.1× slower at 48 views and more than 10× slower beyond 360 views (Q2). This is what makes long sequences affordable, where our largest gains appear (Table R7 in our answer to ReviewerrmL4: +15.5 AUC@30 on ScanNet at 512 frames, +24.7 on KITTI at 256 frames).Table R8. Pose AUC@20 on ScanNet++ / Kubric (24 views).