- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates the layer redundancy in multi-view geometry models like VGGT, pi3 and VGGT-omega. It removes different intervals of layers in the alternating-attention aggregator and measure the pose and point map accuracy drops on 4 datasets. It finds that the early and late regions result in relatively low degradation. It further finds that the degradation of removing two intervals can be approximated by summing the degradations of removing each interval separately. This finding helps them to reduce the O(L^4) two-interval search to O(L^2) because only single intervals need to be evaluated. They also finds that CKA offers a cheaper way to perform pruning search, and that a linear calibration helps recover accuracy after pruning.
Critical strengths
Compare to existing works, this method is simple to implement and deploy, and it seems to preserve the performance quite well with recovery. I also think the finding of the approximated additivity of the degradation function is clever which yields practical benefits of shrinking the search space.
Critical weaknesses
Overall I think it's a nice paper with good quantitative result. However, the goal of the paper is a bit unclear to me, whether it is reducing the parameters count, improving efficiency (speed or memory) or providing analysis/understanding of the VGGT family. So let's discuss them one by one.
- Parameter: I think it would be nice to also report the pruning ratios with the image encoder included so the percentages make more sense.
- If increasing the efficiency is the goal, I believe a comparison with prior methods in long-sequence scenarios (200, 500 or even 1k frames) should be included, where the speed and memory becomes major bottlenecks. Currently, table 2 only compares the proposed method against the intact model at up to 200 frames. I think at least a comparison with FastVGGT should be included. And it would be great to compare with FastVGGT on longer sequences.
- For analysis part, I think AVGGT[1] and Sparse-VGGT[2] already pointed out that early and late (global) attention layers contributes little to performance. Moreover, Sparse-VGGT includes a layer-skip experiment with findings similar to Finding 1 of this paper. I think it would be great if the authors could justify their contribution from this angle given the prior works.
Reference: [1] Sun et al., AVGGT: Rethinking Global Attention for Accelerating VGGT. [2] Wang et al., Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers
Questions for authors
- The redundancy pattern of VGGT family is quite different from, say, language models. For language models the early regions seem to be more important, while for VGGT family only the middle region seem to be critical. Do you have any insights why the VGGT family behaves this way? Or like what's the role of the middle layers?
- Currently each pruned interval contains both frame-wise and global attention layers. Have you considered analyzing them separately? For example, do the two layer types contribute similarly to the redundancy pattern, or does one dominate? It would also be interesting to see whether they matter differently for different tasks, for example global layers for camera pose and frame-wise layers for dense geometry.
5 comments
Response to Reviewer xf9x (Part 2/3)
(Part 2/3, continued from Part 1/3.)
Analysis: contribution relative to AVGGT and SparseVGGT (continued)
All three find that early global attention contributes little and that middle global attention carries cross-view alignment, and our results agree where they overlap. But they analyze global attention layer by layer, and when they remove or convert layers they use only a few fixed ranges: SparseVGGT's layer-skip experiment skips only global attention layers, and only cumulatively in four fixed patterns (from the front, the back, alternating, or the middle) [3]; AVGGT's layer-level ablations only convert the first or the last few global layers to frame attention [4]. None of them evaluates arbitrary contiguous intervals or touches frame blocks. An account based on global attention alone therefore cannot explain why an early frame block, which never looks across views, is expendable. The explanation below fills this gap.
Our explanation. What a block can contribute depends on the information available at its depth, not on its type. The aggregator's input is the output of a DINO encoder, so each token already describes its own image well. The one thing missing is information from the other views. Existing attention analyses place meaningful cross-view interaction in the middle: AVGGT places the boundary between early and middle global layers before the 10th global layer of VGGT and the 11th of π³ [4], i.e., around blocks 19–20 and 21–22 in our indexing, close to where our early region ends. Until meaningful cross-view interaction forms, blocks of either type change the representation only slightly (see “Further evidence” below); we believe the reason is that they are still refining per-image information that DINO has already processed. Once cross-view information enters the representation, later blocks work on it, and blocks of both types become harder to remove.
A direct test. If this is right, removing only frame blocks and removing only global blocks should give similar pictures. This is exactly the experiment you suggest in your second question, and we ran it in π³ on ScanNet during the rebuttal (Figure 21, Appendix K.1 of the revised PDF; details in the last answer below). The two pictures closely agree: Spearman 0.92 for trajectory and 0.93 for dense geometry. Early frame blocks are about as removable as early global blocks, and middle frame blocks are sensitive as well.
Further evidence. Two further observations fit the same picture. In the paper, CKA between the two ends of early intervals stays high (Finding 3) and a single linear map recovers long early deletions (Finding 4), so early blocks change the representation in modest, largely linear ways. The same proxy shows where this changes: 1 − CKA between the input and output of an interval stays low throughout the early region and rises sharply once intervals reach the middle, tracking measured degradation (Pearson 0.87, Spearman 0.92 on VGGT; Fig. 11).
How this relates to prior work. The global-attention findings in [3–5] are consistent with our explanation when it is restricted to global blocks: early global attention has no meaningful cross-view information to exchange yet, and middle global attention does. Our explanation extends this to frame blocks and MLPs, and explains why the deletion pattern depends little on block type. The late region follows the same logic. Late global attention contributes mainly minor refinements [4], so some blocks become removable, but these refinements accumulate and removing many consecutive layers causes a clear loss, which keeps that region narrow. Prior findings on late global attention vary across models and tasks [4,5].
What we do not claim. We present this as an explanation consistent with our measurements and with prior findings, not as a proof. We take the depth at which cross-view integration begins from existing attention analyses rather than deriving it. To study how the pattern arises, one could track interval-deletion sensitivity across training checkpoints, which we leave to future work.
Contribution relative to prior work. In short, the paper goes further in three respects.
- Scope. As noted above, prior work intervenes on global attention layers only over a few fixed ranges. We delete complete blocks, attention and MLP, of both types over all contiguous intervals, across three backbones (VGGT, π³ and VGGT-Ω), four datasets plus CO3D (Appendix K.3), and seven metrics covering both pose and dense geometry (3,018 configurations). Evaluating all intervals reveals what fixed patterns cannot: the length and boundary of each region, the high cost of intervals that span the middle, and the approximate additivity of the two regions' degradation.
(Continued in Part 3/3 with the second and third points.)
Response to Reviewer xf9x (Part 3/3)
(Part 3/3, continued from Part 2/3: points two and three.)
-
An explanation. The frame/global separation (Figure 21) and the account above indicate that the redundancy pattern is set by depth position rather than by the attention operator: early frame blocks and their MLPs are about as removable as early global blocks, which accounts based on global attention do not explain. This account is consistent with the prior global-attention findings while extending them to frame blocks and MLPs.
-
Practical consequences. Approximate additivity and the CKA proxy cut the pruning search from an estimated 5,920 GPU hours to under one, and closed-form recovery allows removing up to 44% of aggregator layers without retraining, which yields models with genuinely fewer parameters (see “The goal of the paper” above).
Why the VGGT family differs from language models, and the role of the middle layers
Reviewer: The redundancy pattern of VGGT family is quite different from, say, language models. […] Do you have any insights why the VGGT family behaves this way? Or like what's the role of the middle layers?
Why language models differ. LLM analyses [1,2] find that the middle-to-late layers are usually the most expendable, and attribute this to deeper blocks changing the representation little once it has stabilized. The same reasoning applies to the aggregator. What differs is where the representation is already largely in place. An LLM starts from token embeddings and has to build contextual features in its early layers, so those layers do essential work. The aggregator starts from the output of a complete pretrained encoder, so its early layers are in a position much like the late layers of an LLM: there is little left to add until information from the other views arrives.
The role of the middle layers. As described in the previous answer, the middle layers are where cross-view information enters the representation and is integrated. This, rather than anything about attention itself, is what makes them hard to remove: a frame block in the middle never looks across views itself, but the tokens it processes already carry information from the other views, so it is sensitive as well (Figure 21). This is also why deletions there hurt both pose and dense geometry.
Frame vs. global blocks, and pose vs. dense geometry
Reviewer: Currently each pruned interval contains both frame-wise and global attention layers. Have you considered analyzing them separately? For example, do the two layer types contribute similarly to the redundancy pattern, or does one dominate? It would also be interesting to see whether they matter differently for different tasks […].
Thank you for suggesting this. We ran the experiment during the rebuttal (Figure 21, Appendix K.1 of the revised PDF), and it is the most direct test of the explanation above. In π³ on ScanNet, we removed consecutive blocks within the frame-only or the global-only sequence, each with its attention and MLP, while keeping every block of the other type; deleting the first two frame blocks, for example, keeps the global block between them. All results are without recovery, with fixed per-metric normalization and a shared color scale.
Does one type dominate? The short answer is that neither type dominates the pattern. The two deletion landscapes are strongly correlated:
| Frame-only vs. global-only landscape | Trajectory | Dense geometry |
|---|---|---|
| Spearman correlation | 0.92 | 0.93 |
| MAE of normalized degradation | 0.09 | 0.11 |
Where the two types differ. Both types have a broad early region that can be removed with little degradation, slightly smaller for global blocks, and up to about ten removed blocks the two are almost interchangeable. They separate only at aggressive budgets, where removing 11–12 global blocks costs roughly twice as much as removing the same number of frame blocks.
Is there a task split? We also looked for the task split you suggest, global blocks for pose and frame blocks for dense geometry, and did not find it: global-only deletion is the more damaging of the two for both task groups. On this model and dataset, then, the two block types follow a similar removal pattern, which is what the explanation above predicts.
References
[1] Xin Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.
[2] Andrey Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.
[3] Chung-Shien Brian Wang et al. Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers. CVPR, 2026.
[4] Xianbing Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR, 2026.
[5] Jinhao You, Shuo Lyu, et al. RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer. NeurIPS, 2026.
Response to Reviewer xf9x (Part 1/3)
We really appreciate your careful review. Your suggestion to analyze frame and global blocks separately is especially useful (Figure 21, Appendix K.1 of the revised PDF). We respond point by point below.
The goal of the paper
What the paper is for. That is a fair question, and the original framing did not answer it clearly. As the title asks (“Does the VGGT Family Need All Its Layers?”), the paper's primary aim is to understand the redundancy pattern, and above all how general it is: whether the same layers turn out to be dispensable across backbones, datasets, scene types and metrics. That is why the study is built around exhaustive interval deletion over three models, four datasets (now five, with CO3D; see Appendix K.3) and seven metrics, rather than around a single compression target.
Why compression is part of the story. The compression results serve a specific purpose within that aim. A pattern that holds this consistently is only useful if one can act on it without losing accuracy, so we show that it is compatible with compression: removing the layers it identifies, with closed-form recovery and no retraining, keeps accuracy comparable for VGGT-Ω and π³ while removing up to 44% of aggregator layers. Because the pattern is shared, practitioners can treat it as a reliable prior rather than rediscovering it for each new model or dataset, and approximate additivity and the CKA proxy reduce what search remains from an estimated 5,920 GPU hours to under one. The parameter and efficiency results below should be read in that light: as evidence that the observation can be used safely to save compute, not as a claim that layer removal is the fastest way to run VGGT on long sequences.
We have rewritten the introduction and the contribution list to say this explicitly (Sec. 1), and we take your three angles in turn below.
Parameters and efficiency
These results serve the purpose described above: they show that the pattern can be acted on with little loss of accuracy, and that doing so remains compatible with other acceleration methods. Tables 1 and 2 now report parameter counts both for the aggregator alone and with the image encoder included. Table 1 already includes FastVGGT. Layer removal and token merging reduce different costs, and combining them lowers latency further (by 29.5%, 40.9% and 30.9% for VGGT-Ω, π³ and VGGT, against 14.0–21.3% for FastVGGT alone), while our method alone gives the best accuracy in 18 of the 21 backbone–metric comparisons. Table 2 shows that the compressed models keep their accuracy up to 200 frames; we leave comparisons on longer sequences to future work, where we will also include a comparison with FastVGGT.
Analysis: contribution relative to AVGGT and SparseVGGT
We agree that AVGGT [4] and SparseVGGT [3] already report that early and late global attention contributes little, and that SparseVGGT's layer-skip experiment shows a pattern similar to our Finding 1. We now discuss both explicitly in Sec. 1, Appendix A.1 and Appendix L of the revised paper. Below we first set out what prior work explains and what it leaves open, then give our explanation and the experiments that support it, and finally list what the paper adds relative to these works.
What prior work explains. Prior analyses of the VGGT family focus on global attention. The table shows what each covers.
(Continued in Part 2/3.)