acceptodds
Under review as a conference paper at ICLR 2027

Does the VGGT Family Need All Its Layers?

20 views

Abstract

Which layers of a feed-forward geometry model are needed to preserve both camera poses and dense 3D structure? We study layer redundancy in VGGT, , and VGGT-: 3018 pruned configurations, scored on seven camera-pose and dense-geometry metrics across four indoor and outdoor datasets. Four findings follow: (i) Removable layers cluster in two redundancy regions: a dominant early region and a narrower late one, while deletions spanning the intervening layers are consistently more disruptive. This recurring pattern holds across models, datasets, and metrics, and contrasts with the middle-to-late redundancy commonly reported in the literature. (ii) Within these regions, we observe that the joint degradation from deleting two intervals is approximately the sum of their individual degradations, reducing the number of model evaluations for pruning search from to , where is the aggregator depth. (iii) We find that CKA provides a cheaper representation-based proxy for interval degradation, offering a practical trade-off between pruning quality and calibration cost. (iv) Closed-form linear calibration recovers accuracy after pruning without end-to-end retraining. A least-squares analysis shows that using a shared map for special and patch tokens generally incurs excess reconstruction loss, motivating token-aware recovery. Recovery maps fitted on just 100 calibration scenes generalize to held-out scenes and unseen datasets. The resulting models reduce aggregator parameters by up to 44% while maintaining accuracy comparable to their intact counterparts. All code is provided in the supplementary materials and will be released publicly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Layer Pruning, 3D Visual Geometry Transformers, VGGT

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xf9xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the layer redundancy in multi-view geometry models like VGGT, pi3 and VGGT-omega. It removes different intervals of layers in the alternating-attention aggregator and measure the pose and point map accuracy drops on 4 datasets. It finds that the early and late regions result in relatively low degradation. It further finds that the degradation of removing two intervals can be approximated by summing the degradations of removing each interval separately. This finding helps them to reduce the O(L^4) two-interval search to O(L^2) because only single intervals need to be evaluated. They also finds that CKA offers a cheaper way to perform pruning search, and that a linear calibration helps recover accuracy after pruning.

Critical strengths

Compare to existing works, this method is simple to implement and deploy, and it seems to preserve the performance quite well with recovery. I also think the finding of the approximated additivity of the degradation function is clever which yields practical benefits of shrinking the search space.

Critical weaknesses

Overall I think it's a nice paper with good quantitative result. However, the goal of the paper is a bit unclear to me, whether it is reducing the parameters count, improving efficiency (speed or memory) or providing analysis/understanding of the VGGT family. So let's discuss them one by one.

  1. Parameter: I think it would be nice to also report the pruning ratios with the image encoder included so the percentages make more sense.
  2. If increasing the efficiency is the goal, I believe a comparison with prior methods in long-sequence scenarios (200, 500 or even 1k frames) should be included, where the speed and memory becomes major bottlenecks. Currently, table 2 only compares the proposed method against the intact model at up to 200 frames. I think at least a comparison with FastVGGT should be included. And it would be great to compare with FastVGGT on longer sequences.
  3. For analysis part, I think AVGGT[1] and Sparse-VGGT[2] already pointed out that early and late (global) attention layers contributes little to performance. Moreover, Sparse-VGGT includes a layer-skip experiment with findings similar to Finding 1 of this paper. I think it would be great if the authors could justify their contribution from this angle given the prior works.

Reference: [1] Sun et al., AVGGT: Rethinking Global Attention for Accelerating VGGT. [2] Wang et al., Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers

Questions for authors
  1. The redundancy pattern of VGGT family is quite different from, say, language models. For language models the early regions seem to be more important, while for VGGT family only the middle region seem to be critical. Do you have any insights why the VGGT family behaves this way? Or like what's the role of the middle layers?
  2. Currently each pruned interval contains both frame-wise and global attention layers. Have you considered analyzing them separately? For example, do the two layer types contribute similarly to the redundancy pattern, or does one dominate? It would also be interesting to see whether they matter differently for different tasks, for example global layers for camera pose and frame-wise layers for dense geometry.
5 comments
AuthorsOpenReview ↗

Response to Reviewer xf9x (Part 1/3)

We really appreciate your careful review. Your suggestion to analyze frame and global blocks separately is especially useful (Figure 21, Appendix K.1 of the revised PDF). We respond point by point below.

The goal of the paper

Reviewer: The goal of the paper is a bit unclear to me, whether it is reducing the parameters count, improving efficiency (speed or memory) or providing analysis/understanding of the VGGT family.

What the paper is for. That is a fair question, and the original framing did not answer it clearly. As the title asks (“Does the VGGT Family Need All Its Layers?”), the paper's primary aim is to understand the redundancy pattern, and above all how general it is: whether the same layers turn out to be dispensable across backbones, datasets, scene types and metrics. That is why the study is built around exhaustive interval deletion over three models, four datasets (now five, with CO3D; see Appendix K.3) and seven metrics, rather than around a single compression target.

Why compression is part of the story. The compression results serve a specific purpose within that aim. A pattern that holds this consistently is only useful if one can act on it without losing accuracy, so we show that it is compatible with compression: removing the layers it identifies, with closed-form recovery and no retraining, keeps accuracy comparable for VGGT-Ω and π³ while removing up to 44% of aggregator layers. Because the pattern is shared, practitioners can treat it as a reliable prior rather than rediscovering it for each new model or dataset, and approximate additivity and the CKA proxy reduce what search remains from an estimated 5,920 GPU hours to under one. The parameter and efficiency results below should be read in that light: as evidence that the observation can be used safely to save compute, not as a claim that layer removal is the fastest way to run VGGT on long sequences.

We have rewritten the introduction and the contribution list to say this explicitly (Sec. 1), and we take your three angles in turn below.

Parameters and efficiency

Reviewer: Parameter: […] report the pruning ratios with the image encoder included […] If increasing the efficiency is the goal, […] a comparison with prior methods in long-sequence scenarios (200, 500 or even 1k frames) should be included […] at least a comparison with FastVGGT should be included. And it would be great to compare with FastVGGT on longer sequences.

These results serve the purpose described above: they show that the pattern can be acted on with little loss of accuracy, and that doing so remains compatible with other acceleration methods. Tables 1 and 2 now report parameter counts both for the aggregator alone and with the image encoder included. Table 1 already includes FastVGGT. Layer removal and token merging reduce different costs, and combining them lowers latency further (by 29.5%, 40.9% and 30.9% for VGGT-Ω, π³ and VGGT, against 14.0–21.3% for FastVGGT alone), while our method alone gives the best accuracy in 18 of the 21 backbone–metric comparisons. Table 2 shows that the compressed models keep their accuracy up to 200 frames; we leave comparisons on longer sequences to future work, where we will also include a comparison with FastVGGT.

Analysis: contribution relative to AVGGT and SparseVGGT

Reviewer: For analysis part, I think AVGGT[1] and Sparse-VGGT[2] already pointed out that early and late (global) attention layers contributes little to performance. Moreover, Sparse-VGGT includes a layer-skip experiment with findings similar to Finding 1 of this paper. I think it would be great if the authors could justify their contribution from this angle given the prior works.

We agree that AVGGT [4] and SparseVGGT [3] already report that early and late global attention contributes little, and that SparseVGGT's layer-skip experiment shows a pattern similar to our Finding 1. We now discuss both explicitly in Sec. 1, Appendix A.1 and Appendix L of the revised paper. Below we first set out what prior work explains and what it leaves open, then give our explanation and the experiments that support it, and finally list what the paper adds relative to these works.

What prior work explains. Prior analyses of the VGGT family focus on global attention. The table shows what each covers.

ModelsGlobal attentionFrame blocksAll contiguous intervals
SparseVGGT [3]VGGT (+ MapAnything)✓✗✗
AVGGT [4]VGGT, π³✓✗✗
RegimeVGGT [5]VGGT✓✗✗
OursVGGT, π³, VGGT-Ω✓✓✓

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Response to Reviewer xf9x (Part 2/3)

(Part 2/3, continued from Part 1/3.)

Analysis: contribution relative to AVGGT and SparseVGGT (continued)

All three find that early global attention contributes little and that middle global attention carries cross-view alignment, and our results agree where they overlap. But they analyze global attention layer by layer, and when they remove or convert layers they use only a few fixed ranges: SparseVGGT's layer-skip experiment skips only global attention layers, and only cumulatively in four fixed patterns (from the front, the back, alternating, or the middle) [3]; AVGGT's layer-level ablations only convert the first or the last few global layers to frame attention [4]. None of them evaluates arbitrary contiguous intervals or touches frame blocks. An account based on global attention alone therefore cannot explain why an early frame block, which never looks across views, is expendable. The explanation below fills this gap.

Our explanation. What a block can contribute depends on the information available at its depth, not on its type. The aggregator's input is the output of a DINO encoder, so each token already describes its own image well. The one thing missing is information from the other views. Existing attention analyses place meaningful cross-view interaction in the middle: AVGGT places the boundary between early and middle global layers before the 10th global layer of VGGT and the 11th of π³ [4], i.e., around blocks 19–20 and 21–22 in our indexing, close to where our early region ends. Until meaningful cross-view interaction forms, blocks of either type change the representation only slightly (see “Further evidence” below); we believe the reason is that they are still refining per-image information that DINO has already processed. Once cross-view information enters the representation, later blocks work on it, and blocks of both types become harder to remove.

A direct test. If this is right, removing only frame blocks and removing only global blocks should give similar pictures. This is exactly the experiment you suggest in your second question, and we ran it in π³ on ScanNet during the rebuttal (Figure 21, Appendix K.1 of the revised PDF; details in the last answer below). The two pictures closely agree: Spearman 0.92 for trajectory and 0.93 for dense geometry. Early frame blocks are about as removable as early global blocks, and middle frame blocks are sensitive as well.

Further evidence. Two further observations fit the same picture. In the paper, CKA between the two ends of early intervals stays high (Finding 3) and a single linear map recovers long early deletions (Finding 4), so early blocks change the representation in modest, largely linear ways. The same proxy shows where this changes: 1 − CKA between the input and output of an interval stays low throughout the early region and rises sharply once intervals reach the middle, tracking measured degradation (Pearson 0.87, Spearman 0.92 on VGGT; Fig. 11).

How this relates to prior work. The global-attention findings in [3–5] are consistent with our explanation when it is restricted to global blocks: early global attention has no meaningful cross-view information to exchange yet, and middle global attention does. Our explanation extends this to frame blocks and MLPs, and explains why the deletion pattern depends little on block type. The late region follows the same logic. Late global attention contributes mainly minor refinements [4], so some blocks become removable, but these refinements accumulate and removing many consecutive layers causes a clear loss, which keeps that region narrow. Prior findings on late global attention vary across models and tasks [4,5].

What we do not claim. We present this as an explanation consistent with our measurements and with prior findings, not as a proof. We take the depth at which cross-view integration begins from existing attention analyses rather than deriving it. To study how the pattern arises, one could track interval-deletion sensitivity across training checkpoints, which we leave to future work.

Contribution relative to prior work. In short, the paper goes further in three respects.

  • Scope. As noted above, prior work intervenes on global attention layers only over a few fixed ranges. We delete complete blocks, attention and MLP, of both types over all contiguous intervals, across three backbones (VGGT, π³ and VGGT-Ω), four datasets plus CO3D (Appendix K.3), and seven metrics covering both pose and dense geometry (3,018 configurations). Evaluating all intervals reveals what fixed patterns cannot: the length and boundary of each region, the high cost of intervals that span the middle, and the approximate additivity of the two regions' degradation.

(Continued in Part 3/3 with the second and third points.)

AuthorsOpenReview ↗

Response to Reviewer xf9x (Part 3/3)

(Part 3/3, continued from Part 2/3: points two and three.)

  • An explanation. The frame/global separation (Figure 21) and the account above indicate that the redundancy pattern is set by depth position rather than by the attention operator: early frame blocks and their MLPs are about as removable as early global blocks, which accounts based on global attention do not explain. This account is consistent with the prior global-attention findings while extending them to frame blocks and MLPs.

  • Practical consequences. Approximate additivity and the CKA proxy cut the pruning search from an estimated 5,920 GPU hours to under one, and closed-form recovery allows removing up to 44% of aggregator layers without retraining, which yields models with genuinely fewer parameters (see “The goal of the paper” above).

Why the VGGT family differs from language models, and the role of the middle layers

Reviewer: The redundancy pattern of VGGT family is quite different from, say, language models. […] Do you have any insights why the VGGT family behaves this way? Or like what's the role of the middle layers?

Why language models differ. LLM analyses [1,2] find that the middle-to-late layers are usually the most expendable, and attribute this to deeper blocks changing the representation little once it has stabilized. The same reasoning applies to the aggregator. What differs is where the representation is already largely in place. An LLM starts from token embeddings and has to build contextual features in its early layers, so those layers do essential work. The aggregator starts from the output of a complete pretrained encoder, so its early layers are in a position much like the late layers of an LLM: there is little left to add until information from the other views arrives.

The role of the middle layers. As described in the previous answer, the middle layers are where cross-view information enters the representation and is integrated. This, rather than anything about attention itself, is what makes them hard to remove: a frame block in the middle never looks across views itself, but the tokens it processes already carry information from the other views, so it is sensitive as well (Figure 21). This is also why deletions there hurt both pose and dense geometry.

Frame vs. global blocks, and pose vs. dense geometry

Reviewer: Currently each pruned interval contains both frame-wise and global attention layers. Have you considered analyzing them separately? For example, do the two layer types contribute similarly to the redundancy pattern, or does one dominate? It would also be interesting to see whether they matter differently for different tasks […].

Thank you for suggesting this. We ran the experiment during the rebuttal (Figure 21, Appendix K.1 of the revised PDF), and it is the most direct test of the explanation above. In π³ on ScanNet, we removed consecutive blocks within the frame-only or the global-only sequence, each with its attention and MLP, while keeping every block of the other type; deleting the first two frame blocks, for example, keeps the global block between them. All results are without recovery, with fixed per-metric normalization and a shared color scale.

Does one type dominate? The short answer is that neither type dominates the pattern. The two deletion landscapes are strongly correlated:

Frame-only vs. global-only landscapeTrajectoryDense geometry
Spearman correlation0.920.93
MAE of normalized degradation0.090.11

Where the two types differ. Both types have a broad early region that can be removed with little degradation, slightly smaller for global blocks, and up to about ten removed blocks the two are almost interchangeable. They separate only at aggressive budgets, where removing 11–12 global blocks costs roughly twice as much as removing the same number of frame blocks.

Is there a task split? We also looked for the task split you suggest, global blocks for pose and frame blocks for dense geometry, and did not find it: global-only deletion is the more damaging of the two for both task groups. On this model and dataset, then, the two block types follow a similar removal pattern, which is what the explanation above predicts.

References

[1] Xin Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.

[2] Andrey Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.

[3] Chung-Shien Brian Wang et al. Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers. CVPR, 2026.

[4] Xianbing Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR, 2026.

[5] Jinhao You, Shuo Lyu, et al. RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer. NeurIPS, 2026.

Reviewer_EZGtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work analyzes the usefulness of different layers of VGGT (and other variants of the family) on the different downstream tasks. The author analyzed this primarily through the removal of layers (single, or a continuous block) at different stages of the model (early, middle, late). The empirical finding is that removal of the early and later layers are relatively safe (with less degradation to the downstream task) while removing the middle layers have more meaningful negative impact on the downstream task. The author also analyzed the composability of the removal such as the effect combination when multiple layers are removed, proposed the CKA-based method for faster proxy and discussed linear proxy for adjustment after the removal instead of end to end finetuning. Overall, the paper has made concrete some empirical findings that were not directly available from existing works.

Critical strengths

Strength

The work performed substantial and detailed empirical study. Across different VGGT models, different dataset, different downstream tasks. The removal of the layer study is exhaustive and I would say makes sense at a high level (more on this later). The strong empirical study is consequential for my assessment as it is a highlight of the paper and it provides a strong basis/information point that layer works can rely on.

The proposed CKA and proxy linear mapping are reasonable methods to speed up this study or the adjustment of the model with the layer removed. The composability by itself can be informative for future works. This is a relatively weaker point for my assesement. The proposed methods are relatively simple and the empirical finding on the composability (mostly linear, may not be always) makes sense but is relatively simple.

Critical weaknesses

The work largely feels like an additional empirical data point among a large plethora of existing works, and in close relation to many potential other works from adjacent fields of transformer study.

I would expect the study to extend beyond simply whether the CKA remains good proxy or composability of the degradation. It is important to at least analyze some aspects of why the empirical pattern is happening (early and late layer are much more safe to remove than middle layers) and/or what are the different layers are doing. Simply showing empirically that some layers are more safe to remove than others makes the contribution too limited.

Questions for authors

The authors have very thorough empirical studies within their scope, I don’t have particular new questions on that. However, I do find some broader questions and would like to hear the author’s thought on.

Note: By no means I meant the authors need to compare with these works, but I think these works study issues that are very close to what the authors are analyzing, and a new empirical study at this stage should move a bit beyond simply telling which layer is safe to remove empirically. Do the authors have thoughts or findings beyond what is presented in existing work? Or could have a more general explanation that captures the existing works' findings?

AVGGT: Rethinking Global Attention for Accelerating VGGT: what is the author’s opinion on whether or not could the empirical finding on middle layer substantially matters relate to aspects of the global attention components? RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer: could the layer importance be explained by attention pattern emirical finding in this work, or there are additional important factors beyond what is implied in these existing works

And many other language transformer analysis works that can also be relevant for extrapolation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer EZGt (Part 1/2)

Thank you for these questions; they go to the heart of the paper. The criticism that the original submission described the redundancy pattern without explaining it is fair. Much of our rebuttal effort went into that question, and the revised paper now contains an explanation that, we believe, also accounts for the findings of AVGGT and RegimeVGGT (Sec. 7 and Appendix L).

Why early and late layers are easier to remove than middle ones

Reviewer: […] It is important to at least analyze some aspects of why the empirical pattern is happening (early and late layer are much more safe to remove than middle layers) and/or what are the different layers are doing. Simply showing empirically that some layers are more safe to remove than others makes the contribution too limited. Do the authors have thoughts or findings beyond what is presented in existing work? Or could have a more general explanation that captures the existing works' findings?

What prior work explains. Prior analyses of the VGGT family focus on global attention. The table shows what each covers.

ModelsGlobal attentionFrame blocksAll contiguous intervals
SparseVGGT [3]VGGT (+ MapAnything)✓✗✗
AVGGT [4]VGGT, π³✓✗✗
RegimeVGGT [5]VGGT✓✗✗
OursVGGT, π³, VGGT-Ω✓✓✓

All three find that early global attention contributes little and that middle global attention carries cross-view alignment, and our results agree where they overlap. They analyze global attention layer by layer (attention patterns, probes or per-layer perturbations), and when they remove or convert layers they use only a few fixed ranges (the first layers, the last layers or a middle block); none of them evaluates arbitrary contiguous intervals or studies how the effects of two intervals combine. In addition, an account based on global attention alone cannot explain why an early frame block, which never looks across views, is expendable. The explanation below fills this gap.

Our explanation. What a block can contribute depends on the information available at its depth, not on its type. The aggregator's input is the output of a DINO encoder, so each token already describes its own image well. The one thing missing is information from the other views. Existing attention analyses place meaningful cross-view interaction in the middle: AVGGT places the boundary between early and middle global layers before the 10th global layer of VGGT and the 11th of π³ [4], i.e., around blocks 19–20 and 21–22 in our indexing, close to where our early region ends. Until meaningful cross-view interaction forms, blocks of either type change the representation only slightly (see “Further evidence” below); we believe the reason is that they are still refining per-image information that DINO has already processed. Once cross-view information enters the representation, later blocks work on it, and blocks of both types become harder to remove.

A direct test. If this is right, removing only frame blocks and removing only global blocks should give similar pictures. We ran this test in π³ on ScanNet during the rebuttal, removing intervals of one block type at a time, each with its MLP and without recovery (Figure 21, Appendix K.1 of the revised PDF). The two pictures closely agree: Spearman 0.92 for trajectory and 0.93 for dense geometry, with MAEs of 0.09 and 0.11 in normalized degradation. Early frame blocks are about as removable as early global blocks, and middle frame blocks are sensitive as well.

Further evidence. Two further observations fit the same picture. In the paper, CKA between the two ends of early intervals stays high (Finding 3) and a single linear map recovers long early deletions (Finding 4), so early blocks change the representation in modest, largely linear ways. The same proxy shows where this changes: 1 − CKA between the input and output of an interval stays low throughout the early region and rises sharply once intervals reach the middle, tracking measured degradation (Pearson 0.87, Spearman 0.92 on VGGT; Fig. 11). This location is broadly consistent with the AVGGT boundary mentioned above.

How this relates to prior work. The global-attention findings in [3–5] are consistent with our explanation when it is restricted to global blocks: early global attention has no meaningful cross-view information to exchange yet, and middle global attention does. Our explanation extends this to frame blocks and MLPs, and explains why the deletion pattern depends little on block type. The late region follows the same logic. Late global attention contributes mainly minor refinements [4], so some blocks become removable, but these refinements accumulate and removing many consecutive layers causes a clear loss, which keeps that region narrow. Prior findings on late global attention vary across models and tasks [4,5].

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer EZGt (Part 2/2)

(Part 2/2, continued from Part 1/2.)

Why early and late layers are easier to remove than middle ones (continued)

What we do not claim. We present this as an explanation consistent with our measurements and with prior findings, not as a proof. We take the depth at which cross-view integration begins from existing attention analyses rather than deriving it. To study how the pattern arises, one could track interval-deletion sensitivity across training checkpoints, which we leave to future work.

AVGGT and RegimeVGGT: can global attention or attention patterns explain layer importance?

Reviewer: AVGGT […]: what is the author's opinion on whether or not could the empirical finding on middle layer substantially matters relate to aspects of the global attention components? RegimeVGGT […]: could the layer importance be explained by attention pattern emirical finding in this work, or there are additional important factors beyond what is implied in these existing works

The table above places both works relative to ours; we answer the two questions directly here.

On AVGGT. Middle global attention clearly matters, as AVGGT reports, but middle frame blocks also matter (Figure 21), so the importance of the middle reflects the cross-view information being processed there rather than the global-attention operator itself. Block type does make a difference at aggressive budgets: removing 11–12 global blocks costs roughly twice as much as removing the same number of frame blocks, which fits global attention being the only channel through which views exchange information.

On RegimeVGGT. We do not think attention patterns can explain layer importance on their own, since early MLPs and complete frame blocks are removable as well. Our reading is that the patterns RegimeVGGT reports are one visible sign of a broader transition, from per-image to cross-view processing, and that the same transition also governs the frame blocks and the MLPs.

Language-model analyses

Reviewer: And many other language transformer analysis works that can also be relevant for extrapolation.

Why the location differs from LLMs. LLM analyses [1,2] find that the middle-to-late layers are usually the most expendable, and attribute this to deeper blocks changing the representation little once it has stabilized. The same reasoning applies to the aggregator. What differs is where the representation is already largely in place. An LLM starts from token embeddings and has to build contextual features in its early layers, so those layers do essential work. The aggregator starts from the output of a complete pretrained encoder, so its early layers are in a position much like the late layers of an LLM: there is little left to add until information from the other views arrives.

Contribution beyond existing work

Reviewer: The work largely feels like an additional empirical data point among a large plethora of existing works, and in close relation to many potential other works from adjacent fields of transformer study.

We understand the concern, and we hope the answers above show that the paper now offers more than another data point. In brief, it adds three things.

  • Scope. Existing VGGT analyses study global attention. We delete complete blocks of both types over all contiguous intervals, across three backbones (VGGT, π³ and VGGT-Ω), four datasets plus CO3D (Appendix K.3), and seven metrics (3,018 pruned configurations).

  • An explanation. The frame/global separation (Figure 21) and the account above indicate that the redundancy pattern is set by depth position rather than by the attention operator, and they are consistent with the prior global-attention findings while extending them to frame blocks and MLPs.

  • Practical consequences. Approximate additivity and the CKA proxy cut the pruning search from an estimated 5,920 GPU hours to under one, and closed-form recovery allows removing up to 44% of aggregator layers without retraining.

References

[1] Xin Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.

[2] Andrey Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.

[3] Chung-Shien Brian Wang et al. Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers. CVPR, 2026.

[4] Xianbing Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR, 2026.

[5] Jinhao You, Shuo Lyu, et al. RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer. NeurIPS, 2026.

AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Thank you again for your thoughtful review. Your central question, why early and late layers are easier to remove than middle ones, was the one we spent the most effort on during the rebuttal, and we believe the paper is better for it. Our answer is in the new Sec. 7 and Appendix L, supported by a new experiment that separates frame and global blocks (Figure 21).

As the discussion period closes in about a day, we would sincerely appreciate hearing whether this addresses your concerns. If anything remains unconvincing, please let us know and we will do our best to respond in the remaining time. If your concerns have been satisfactorily addressed, we would be grateful if you would consider updating your score accordingly.

Thank you very much for your time!

Reviewer_pFhaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies efficient empirical recipes for removing redundant layers from feed-forward geometric VGGT family of models. It first examines degradation caused by removing consecutive blocks of layers and establishes that there are two separate regions of low degradation: a broad early region (layers closer to input) and a narrower late region (layers closer to output).

The paper then empirically observes that the degradation caused by removing one interval from each region is approximately the sum of degradations caused by removing them individually. This observation could be used to guide the search of intervals to remove, reducing search complexity from O(L^4) to O(L^2).

Finally, the paper establishes that one can use Centered Kernel Alignment (CKA) measure between two layers to approximate degradation caused by removing the block between them, allowing estimating the degradation with model activations using single model forward pass instead of O(L2) model evaluation.

Through these incremental steps, the paper successfully reduce the search time from 5920 GPU hours to less than 1 GPU hour. Additionally, the paper proposes learning a token type-dependent closed-form linear calibration between the input and the output of the removed blocks to recover accuracy without retraining.

The paper shows that the resulting models maintain accuracy comparable to their intact counterparts while removing up to 44% of aggregator layers.

Critical strengths
  • The paper presents a strong empirical experimental design. It starts with an exhaustive interval deletion experiment to find redundancy regions in the models, then establish the additivity of interval-wise degradation before studying different approximation for performance degradation using only model activations. This systematic design leads to an excellent efficiency gain from 5290 GPU hours to less than 1 GPU hour.

  • The paper systematically validates different claims and hypothesis it presents. The results in most case are convincing. The final results look good for VGGT-\omega and \pi^3. The paper shows that pruned models improve results while being less computationally expensive.

  • The paper is in general well organised, well written and easy to follow.

Critical weaknesses
  • The paper is experimentally heavy and at times aims only to present results and observations without providing insights into these observations. For example, it would be interesting to understand why there are two low degradation regions in the geometric models while the observation is different from LLMs and ViTs. Also, why is this structured shared across geometric models, datasets and metrics?

  • The additivity of interval-wise degradation is interesting but it is empirically established only for models before the final recovery. Some study into this would be valuable.

  • The paper only considers indoor and driving data. It is not clear if the observations in the paper still hold on other domains of data.

  • The paper claims that pruned models perform comparably to intact models but there are significant drops with VGGT in certain benchmarks. The claim should be more nuanced.

Questions for authors

Suggestions:

  • It is not clear the equality in L.211 is correct because normalising factor alpha changes in the conditional case versus the unconditional case. The paper seems to say that alpha_m is unchanged but this should be clarified to avoid misunderstanding.

  • The conclusion of section 5.1 — that the shared-mapping penalty is strictly positive whenever T_s ≠ T_p, and that balanced weights do not remove it — follows in one line from strict convexity of each L_r (Σ_r ≻ 0) and uniqueness of T_r, none of the math machinery is required for it.

  • The word calibration appears multiple times and seemingly carries different meanings. Please clarify what it means and considers using other words when suitable.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pFha (Part 1/3)

We thank the reviewer for a careful and constructive review. The questions below sharpened the paper, and in particular pushed us to say why the redundancy pattern looks the way it does, not only that it exists. We respond point by point.

W1 — Why two low-degradation regions, and why the pattern is shared

Reviewer: The paper is experimentally heavy and at times aims only to present results and observations without providing insights into these observations. For example, it would be interesting to understand why there are two low degradation regions in the geometric models while the observation is different from LLMs and ViTs. Also, why is this structured shared across geometric models, datasets and metrics?

Thank you for this valuable question. All three reviewers raised it, and most of our rebuttal work went into it. Below we share our own thinking on it, which the revised paper now develops in Sec. 7 and Appendix L. Our short answer is that whether a block can be removed depends on what information is available at its depth, not on what kind of block it is.

To see why, it helps to follow a token through the aggregator, from input to output.

  • Why the early region is broad. Start with what the aggregator receives. Its input is the output of a DINO encoder, so every token already carries a rich description of its own image. What the tokens do not yet carry is anything from the other views. Prior work on global attention [3–5] finds that meaningful cross-view interaction is weak in early layers and only forms in the middle. Until it does, blocks of either type change the representation only slightly and in a nearly linear way: CKA between the two ends of early intervals stays high (Finding 3), and a single linear map recovers long early deletions (Finding 4). We believe the reason is that these blocks are still working on per-image information that DINO has already processed, and they have comparatively little to add. This is consistent with the broad early region.

  • Why the middle is sensitive. This is where the views start to interact. Prior analyses place meaningful cross-view alignment in the middle of the aggregator; AVGGT places the boundary between early and middle global layers before the 10th global layer of VGGT and the 11th of π³ (in its 0-based indexing, t_early = 9 and 10) [4], i.e., around blocks 19–20 and 21–22 in our indexing, close to where our early region ends. Once that information enters the representation, the blocks that follow have genuinely new work to do: reconciling what the different views say about the same scene into one consistent geometry. That is why deletions in or across this region degrade both camera pose and dense geometry, and why this holds for frame blocks as well as global ones. A frame block in the middle never looks across views itself, but the tokens it processes already carry information from the other views, so it takes part in the integration.

  • Why the late region is narrow. Toward the end, global attention contributes mainly minor refinements [4], so some blocks become dispensable again. But these refinements accumulate: short intervals can be removed, while removing many consecutive layers causes a clear loss, so the region forms only a narrow band along the diagonal. Prior findings on late global attention vary across models and tasks [4,5].

Two results support this reading.

  • Why we think block type is not the cause. The cleanest test of this reading is a new experiment we ran during the rebuttal (Figure 21, Appendix K.1 of the revised PDF). In π³ on ScanNet, we removed intervals of frame blocks only or of global blocks only, each with its MLP and without recovery. Prior work attributes early redundancy to global attention itself: AVGGT argues that features at this stage still lack sufficient 3D information [4], and RegimeVGGT that cross-view content is not yet operative in shallow layers [5]. AVGGT and SparseVGGT further assign frame attention the role of first forming per-view structure and adapting features for matching in later global layers [3,4]. If that were the whole story, early frame blocks should still matter. They do not: the two deletion landscapes are highly similar (Spearman 0.92 for trajectory, 0.93 for dense geometry), even though frame blocks never look across views.

  • What the similarity proxies show. The similarity proxies in the paper point the same way. 1 − CKA between the input and output of each interval stays low throughout the early region and rises sharply once intervals reach the middle, tracking measured degradation (Pearson 0.87, Spearman 0.92 on VGGT; Fig. 11).

(Continued in Part 2/3. References are listed at the end of Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer pFha (Part 2/3)

(Part 2/3, continued from Part 1/3.)

W1 — Why two low-degradation regions, and why the pattern is shared (continued)

Why this differs from LLMs and ViTs. The contrast with LLMs and ViTs follows from the same reasoning. Those models start from token or patch embeddings, so their early layers have to build the features everything else depends on; in LLMs it is the middle-to-late layers that turn out to be expendable [1,2]. The aggregator starts after a complete pretrained encoder, so we believe that work has largely been done already.

Why the pattern is shared across models, datasets and metrics. The same argument accounts for the consistency you point out. Nothing in it depends on scene content or on the metric. It depends on the input features and the architecture. All three backbones encode each image with a DINO-initialized ViT (DINOv2 for VGGT and π³, DINOv3 for VGGT-Ω) and then alternate per-frame and cross-view attention in an aggregator whose output feeds both pose and dense geometry. The pattern also reappears on object-centric CO3D (see W3 and Appendix K.3). We note that part of the similarity between π³ and VGGT is probably inherited: π³ is initialized from VGGT and reuses VGGT's fine-tuned image encoder unchanged. VGGT-Ω is the more independent check: it is trained from a DINOv3 initialization rather than from VGGT weights, uses a different patch size, and replaces a quarter of its global-attention layers with register attention, yet it shows the same pattern.

What we do not claim. We offer this as an explanation consistent with our measurements and with prior work, not as a proof. In particular, we take the depth at which cross-view integration begins from existing attention analyses rather than deriving it. To study how the pattern arises, one could track interval-deletion sensitivity across training checkpoints, which we leave to future work.

W2 — Additivity after recovery

Reviewer: The additivity of interval-wise degradation is interesting but it is empirically established only for models before the final recovery. Some study into this would be valuable.

Agreed, and we have now checked it. In the paper, additivity was established on degradation measured before recovery; the open question is whether it survives once each interval has its own fitted recovery map.

What we tested. We took 24 interval pairs (three backbones, budgets b = 10, 12, …, 24), chosen exactly as in the paper: same ScanNet + nuScenes calibration data, objective and constraints, and no use of post-recovery or test results. Where the original search had picked a single interval, we used the best two-interval alternative. Recovery maps were fitted on the calibration set and evaluated on held-out scenes. Figure 22 (Appendix K.2 of the revised PDF) plots the sum of the separately recovered costs, DMrec(A)+DMrec(B)D_{\mathcal{M}}^{\mathrm{rec}}(A) + D_{\mathcal{M}}^{\mathrm{rec}}(B), against the cost of recovering both intervals together, DMrec(A∪B)D_{\mathcal{M}}^{\mathrm{rec}}(A \cup B), all with the paper's fixed normalization.

What we found. For most pairs the relation holds approximately. VGGT stays close to the diagonal throughout. The larger departures come at the most aggressive budgets (b = 22, 24) for π³ and VGGT-Ω, where the sum underestimates the joint cost for π³ and overestimates it for VGGT-Ω. At the budgets of our final models (b = 18 for VGGT-Ω, 16 for π³, 10 for VGGT), the points sit near the diagonal.

What this means for the method. Our search does not rely on post-recovery additivity, so these results do not affect the reported models. They do suggest that searching directly on post-recovery costs may be feasible at moderate budgets, which we see as a natural next step that needs testing on more interval pairs.

W3 — Data beyond indoor and driving scenes

Reviewer: The paper only considers indoor and driving data. It is not clear if the observations in the paper still hold on other domains of data.

A fair concern. Our original four datasets are indoor (ScanNet, 7Scenes) and driving (nuScenes, Waymo), so during the rebuttal we repeated the whole analysis on object-centric CO3D with π³ (Appendix K.3 of the revised PDF), following the original protocol: 30 frames per scene, and disjoint calibration and evaluation splits of 50 scenes each.

(Continued in Part 3/3. References are listed at the end of Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer pFha (Part 3/3)

(Part 3/3, continued from Part 2/3.)

W3 — Data beyond indoor and driving scenes (continued)

Do the findings hold on CO3D? The findings carry over. The deletion heatmap shows the same two regions, and the marginal-degradation map agrees closely with it (MAE 0.06, Pearson 0.95, Spearman 0.95; Figure 23), so approximate additivity holds here too.

Does CKA still work as a proxy? CKA again tracks measured degradation (Spearman 0.90 for 1 − CKA; Figure 24), and choosing intervals by CKA gives degradation curves similar to those from measured-degradation selection before recovery, and nearly identical ones after recovery (Figure 25). Recovery maps fitted on the calibration split transfer to the test split.

The pruned model. The resulting model keeps 20 of π³'s 36 aggregator layers and matches or improves all seven CO3D metrics at the reported precision (Table 3, Appendix K.3 of the revised PDF):

MethodLayersAPE ↓ARE ↓ (°)RPE-t ↓RPE-r ↓ (°)Acc. ↓Comp. ↓CD ↓
π³360.0764.3620.0591.8580.0230.0250.024
Ours200.0512.3780.0550.9770.0230.0240.023

W4 — The claim for VGGT

Reviewer: The paper claims that pruned models perform comparably to intact models but there are significant drops with VGGT in certain benchmarks. The claim should be more nuanced.

We agree and have narrowed the claim. Comparable accuracy now refers only to VGGT-Ω and π³ (abstract, Sec. 1 and the conclusion), and VGGT's larger drop is reported explicitly (Sec. 6).

S1 — Normalization in the conditional-degradation identity

Reviewer: It is not clear the equality in L.211 is correct because normalising factor alpha changes in the conditional case versus the unconditional case. The paper seems to say that alpha_m is unchanged but this should be clarified to avoid misunderstanding.

Thank you. The equality does hold, but we agree the text made the reader work too hard to see why. αm\alpha_m is computed once per model and dataset (Sec. 2.3, lines 131–133 of the revised PDF) and held fixed across budgets and evaluation splits, so both sides of the identity use the same αm\alpha_m. We now say this right next to the equation (Sec. 4.1, lines 232–235).

S2 — Simplifying the shared-mapping argument

Reviewer: The conclusion of section 5.1 — that the shared-mapping penalty is strictly positive whenever T_s ≠ T_p, and that balanced weights do not remove it — follows in one line from strict convexity of each L_r (Σ_r ≻ 0) and uniqueness of T_r, none of the math machinery is required for it.

This is a nice simplification, and we have adopted it. With Σr≻0\Sigma_r \succ 0 and positive weights on both token types, a single shared map cannot reach both minima when Ts≠TpT_s \neq T_p, and the main text now makes the argument in one line (Sec. 5.1). We have kept the closed form in Appendix F because it says something the one-line argument does not: how large the penalty is, as a function of how much the two maps disagree, the token-type weights and the feature second moments. In particular, it shows how a small pooled penalty can hide a large penalty on the special tokens when the weights are imbalanced.

S3 — The term “calibration”

Reviewer: The word calibration appears multiple times and seemingly carries different meanings. Please clarify what it means and considers using other words when suitable.

Agreed; the word was doing two jobs. We now reserve “calibration” for recovery, that is, fitting the linear maps on a small set of scenes without retraining (Sec. 5). The same scenes are also used to select which intervals to prune, and we now say so explicitly (Sec. 2.2), so that “calibration set” has a single, consistent meaning throughout.

References

[1] Xin Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.

[2] Andrey Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.

[3] Chung-Shien Brian Wang et al. Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers. CVPR, 2026.

[4] Xianbing Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR, 2026.

[5] Jinhao You, Shuo Lyu, et al. RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer. NeurIPS, 2026 (arXiv:2606.18439).

AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Thank you again for your careful and constructive review. Your questions led directly to the main additions in the revision: the explanation of why the two redundancy regions arise (new Sec. 7 and Appendix L), and the new experiments on additivity after recovery and on CO3D (Appendix K).

As the discussion period closes in about a day, we would sincerely appreciate hearing whether our response addresses your concerns. If anything remains unclear, please let us know and we will do our best to clarify it in the remaining time. If you feel the revision has strengthened the paper, we would be grateful if you would consider reflecting that in your final assessment.

Thank you very much for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.