acceptodds
Under review as a conference paper at ICLR 2027

PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers

Abstract

Multi-view geometry transformers are feed-forward 3D foundation models that jointly predict depth maps, point maps, and camera poses for N images in a single forward pass. Most of them build on an alternating-attention (AA) stack whose cross-view attention runs over the patch tokens of all N views in every block, so its cost grows quadratically in N: on 300-frame ScanNetv2 clips, VGGT peaks at 72.3 GiB, and MapAnything does not fit on a 93 GiB card. Existing methods reduce tokens or attention inside the AA stack, so the full patch grid still enters it, and the pre-AA interface, where the encoder hands its tokens to the first AA block, has been overlooked. We propose PrePARE, which prunes patch tokens once at the pre-AA interface and restores the dense grid once after the last AA block. Every block of the frozen AA stack then runs on the reduced tokens, while the prediction heads receive the full grid. A Token Scorer, supervised by how the unpruned AA stack uses each token, and a Feature-guided Restoration module are the only trained parts. The Token Routing module is rule-based. On ScanNetv2 at N = 300, PrePARE reduces the peak memory of VGGT by 80%, from 72.3 to 14.4 GiB, below every in-AA method we ran, and runs 5.4 times faster, while its per-scene Chamfer distance is lower than the unpruned model's on 34 of 50 scenes (sign test p = 0.015). On π³ and Depth Anything 3 it saves 15 and 20% of peak memory, where the in-AA methods save at most 9%, and at 1000 frames it is the only method we ran that fits Depth Anything 3 and MapAnything on one GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
efficient AI, token pruning, multi-view geometry transformers, 3D foundation models, feed-forward 3D reconstruction, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AP1aOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PrePARE, which prunes patch tokens once before the alternating-attention stack of a frozen multi-view geometry transformer and restores the dense grid once after it, so that every attention block operates on a reduced token set. A lightweight scorer distilled from the cross-view attention of the unpruned model selects the tokens to keep, and a feature-guided restoration module rebuilds the full grid for the dense prediction heads. Experiments on four backbones show substantial memory and latency savings with reconstruction and pose accuracy close to or better than the unpruned models, although the gains are most pronounced on VGGT and rely partly on in-domain supervised training.

Critical strengths

The design is simple and broadly applicable: by intervening only at the interfaces before and after the alternating-attention stack, PrePARE leaves the backbone frozen, adds only a small number of trainable parameters, and transfers to four different backbones by specifying their interfaces and anchor tokens, while making long input sequences feasible on a single GPU.

The token selection criterion is well motivated: instead of relying on encoder feature similarity, the scorer is distilled from how the unpruned model actually uses each token in cross-view attention, and the ablations clearly show that this criterion and the learned restoration outperform similarity-based and interpolation-based alternatives.

Critical weaknesses

**Major Weakness **

Train-test length mismatch. Both training stages use very short clips (2 to 8 frames on VGGT and 2 to 4 frames on the other backbones, Appendix A and Table A1), whereas evaluation is performed on sequences of 100 to 1000 frames. Since the attention distribution of a transformer depends strongly on the number of tokens it attends over, a difference of two to three orders of magnitude can lead to substantially different attention patterns. This affects the method in two ways: the AA-usage target that the scorer learns to predict is computed only on short clips and may not reflect which tokens matter in long sequences, and the AA outputs that the restoration module reads at inference differ from those it was trained on. The paper provides no analysis of why the trained modules still work under this shift. The authors should explain this and support it with evidence, for example by reporting how the restoration error with respect to the unpruned features varies with sequence length (which is feasible for VGGT up to 300 frames), or by comparing against modules trained on longer clips.

Unjustified loss weighting. The downstream loss (line 737) sums four terms, with all weight 1.0. These terms differ considerably in form. They may not own comparable magnitude, and equal weights do not imply equal contributions to training. The authors should report the value ranges and gradient norms of each term during training, so that readers can assess which terms actually drive the results. If their ranges indeed vary a lot, the authors should explain why “all weight 1.0” works.

**Minor Weakness **

Unequal training conditions in comparisons. FastVGGT is a training-free method, and LiteVGGT is fine-tuned on a general mixture of datasets that does not include ScanNetv2, whereas the trainable modules of PrePARE are trained on the ScanNetv2 training split with ground-truth depth supervision and then evaluated on the ScanNetv2 test split. The main comparisons in Tables 2 and 3 therefore pit an in-domain fine-tuned method against baselines that have not seen the target distribution, which favors PrePARE and makes it difficult to attribute its advantage to the pre-AA pruning design alone. The zero-shot results in Table 4 partially address this concern, but the advantage over the unpruned model is less consistent there. The authors should discuss whether this comparison is fair.

Inconsistency with previously reported baseline results. The paper states that its evaluation follows the FastVGGT protocol on ScanNetv2, yet the reported LiteVGGT results differ notably from those in the original LiteVGGT paper, which uses the same protocol. In the LiteVGGT paper (Table 1), LiteVGGT achieves a clearly lower Chamfer distance than VGGT at around 300 frames (0.365 versus 0.417), whereas in Table 2 of this submission it is worse than the unpruned VGGT (0.4589 versus 0.4512). The authors should explain this difference.

Typos and notation errors. The notation around Equation (2) is inconsistent: the equation writes the attention weight with the anchor token a or the token i as the query, while the following sentence defines it with a query index u that does not appear in the equation. In addition, the manuscript contains a number of spelling and grammatical errors, for example "computaional" and "lerger" in Section 1, "Motivatied" in Section 3.2, "eveluate" and "protocal" in Section 4.4, "PatchEnbed" in Figures 1 and 2, "apply the our framework" in Section 1, and "at theirs released head-sparsity profiles" in Section 4.1. A careful proofreading pass is recommended.

Questions for authors

Please check Critical Weaknesses and address these concerns.

6 comments

1. W1: Does training on short clips generalize to long sequences?

Direct answer. Training on short clips and evaluating on substantially longer sequences is an established practice in multi-view geometry transformers, including VGGT, Fast3R, LiteVGGT, and MapAnything (Wang et al., 2025;Yang et al., 2025;Shu et al., 2026;Keetha et al., 2026), as summarized below. Specifically, the original VGGT (Wang et al., 2025) itself trained on N=2-24, and evaluate on N=100-1000.

Res-Table 8. Comparison between training clip length of different methods.

MethodTraining length (frames)Evaluation length (frames)
FastVGGT2–24 (VGGT)100 – 1000
Fast3R201,000+
LiteVGGT4–4896 – 1,000
MapAnything≤4 (Stage 1), 2–24 (Stage 2)- 100
PrePARE (Ours)2–8 (VGGT), 2–4 (others)100–1,000

2. W2: Why are equal loss weights used?

Direct answer. We use unit weights for the main loss components because their magnitudes are comparable during training. We report their loss magnitudes and gradient norms from stage-2 training (664 optimization steps) in Res-Table 9. The results support using unit weights without additional loss-scale normalization.

Res-Table 9. Per-term loss magnitudes and gradient norms during VGGT stage-2 training.

TermWeightSteps 1–100Steps 250–350Grad norm (scorer)Grad norm (restoration)
Prediction consistency10.240 [0.116–0.552]0.179 [0.105–0.342]0.3272.213
GT depth, scale-invariant10.152 [0.140–0.169]0.144 [0.132–0.164]0.0240.067
GT depth, tail10.287 [0.207–0.323]0.224 [0.138–0.275]0.1050.800
GT depth, range hinge10.145 [0.012–0.198]0.095 [0.051–0.124]0.1650.142
L_downstream (sum)—0.8240.642——

Loss values are reported as mean [min–max]; gradient norms are measured at the start of stage-2 training.

The four unit-weight downstream losses have comparable mean magnitudes (0.095–0.287). As the gradient norms shows that equal weights are not intended to enforce equal gradient contributions, as the terms supervise different aspects (scoring, feature restoration, prediction consistency, and depth supervision) of geometric prediction.

The revision. We add the per-term loss magnitudes and gradient norms to Appendix H.6 (Table A13) and explain the rationale for our loss weighting strategy.

3. W3: Fair Comparison with FastVGGT and LiteVGGT?

Direct answer. PrePARE trains its lightweight modules using ScanNetv2 ground-truth depth, whereas FastVGGT is training-free and LiteVGGT finetunes on a broader dataset mixture. However, the efficiency gains of PrePARE primarily arise from reducing the token workload throughout the entire AA stack.

Training protocol. The ScanNetv2 training and evaluation scenes are building-disjoint. Furthermore, VGGT includes ScanNet in its original pretraining, so FastVGGT and LiteVGGT inherits this domain exposure, both of them have seen the target distribution.

Zero-shot generalization. Our original Table 4 evaluates PrePARE on 7-Scenes and NRGBD without retraining. PrePARE outperforms both Co-Me and LiteVGGT on all 12 reported metrics (Acc, Comp, and NC across two datasets and two sampling strides), demonstrating that its advantages over these acceleration baselines extend beyond the training distribution.

Compared with unpruned VGGT, the results are mixed: PrePARE improves Acc and Comp in 3/4 settings each, but NC in only 1/4. As discussed in our response to Reviewer WXqi (Q2), this reflects the trade-off between removing redundant tokens and preserving distinctive geometric information. We therefore distinguish the in-domain reconstruction improvements from the less consistent zero-shot improvements over the unpruned model.

4. W4: Why do the LiteVGGT results differ from its original paper?

Direct answer. The result's differences comes from scene selection, frame sampling, or preprocessing contribute to the observed discrepancy. First, LiteVGGT does not specify a fixed 50-scene manifest or the exact frame indices underlying its reported results on their official repository. Their evaluation script selects scenes from the locally processed ScanNetv2 directory and samples frames according to the available images and valid poses. Second, LiteVGGT reports their results at 296 frames in their original paper, while our evaluation reports LiteVGGT at 300 frames, following VGGT, FastVGGT and Co-Me's settings for a fair comparison. Notably, VGGT baselines also vary across independent evaluations: FastVGGT reports 0.416 at 300 frames, LiteVGGT reports 0.417 at 296 frames, RegimeVGGT reports 0.454 at 300 frames, and our evaluation obtains 0.4512 at 300 frames. Third, Table 5 in paper gives LiteVGGT against unpruned VGGT at N=300 as better on 22 of 50 scenes, p = 0.48. The mean gap is 0.0077, under our protocol we do not find LiteVGGT to be worse than the unpruned VGGT. Our experiments evaluate all baselines under a unified evaluation protocol, providing internally consistent comparisons under the same settings.

5. W5: Notation inconsistencies and typos

We thank the reviewer for identifying these issues. We correct the inconsistent attention notation around Eq. (2), explicitly define the query and key indices, and carefully proofread the manuscript to resolve the listed spelling, grammar, and figure-label errors.

Reviewer_AP1aOpenReview ↗

Response to W1 is not persuasive

I thank the authors for their response. My main concern still lies in W1, and I do not find the current response convincing.

It has been documented in prior work that VGGT suffers from error accumulation and degraded accuracy on long sequences, and methods such as FastVGGT and LiteVGGT were developed specifically with this in mind. FastVGGT, for example, is motivated by the token collapse phenomenon it observes in the global attention of VGGT on long sequences. Its merging decisions are made at inference time on the full sequence in every global-attention layer, and the tokens of the reference frame are always preserved. This targeted treatment of tokens is what allows it to remain robust on long sequences.

First, the paper states in Section 4.2 that "The efficiency and performance gain of PrePARE improves as the clip grows compared to other methods." The results in Table 2 do not provide strong support for the performance part of this claim.

Second, I would like the authors to discuss in more depth where and how PrePARE is designed to preserve accuracy on long clips. All the design choices and claims presented so far, such as pruning once before the AA stack, are motivated by reducing computational cost. The trained modules operate per frame, are trained on clips of at most 8 frames, without observing the sequence length or the content of other frames, so their ability to adapt their token selection to long sequences is not proven. The argument in Section 4.2 that pruned tokens are more likely to have kept counterparts in other views on longer clips is a post-hoc explanation that is not verified experimentally. I would appreciate concrete evidence AND interpretation that support your claim about the good accuracy on long clips.

Reviewer_GYdJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrePARE, a token-pruning framework for frozen multi-view geometry transformers. It prunes tokens once before the alternating-attention stack using a learned scorer and keep/merge/prune routing, then restores dense features for the prediction heads. Experiments on four backbones show substantial inference-time and memory reductions, particularly for long sequences.

Critical strengths
  1. Pruning once before the attention stack is an appealing design: subsequent blocks benefit from fewer tokens without repeated selection overhead.
  2. Evaluation across four backbones demonstrates broad applicability, and the ablations examine the framework's main components.
Critical weaknesses
  1. Some downstream accuracy losses are substantial. At the reported r=0.4 operating point, zero-shot depth AbsRel on NRGBD increases from 0.013 to 0.052, while point-tracking AJ drops from 0.183 to 0.130 on the 26 TAP-Vid DAVIS clips completed by both models. These losses limit the scope of accuracy-preservation claims. Although statistical tests and a keep-ratio sweep are provided, a nonsignificant difference does not establish equivalence; paired intervals and an acceptable-loss margin would help characterize the trade-off.
  2. The large peak-memory difference between learned restoration and ToMe-style unmerging at the same keep ratio remains insufficiently explained. The reconstruction rules and common measurement protocol are specified, but an allocation breakdown would clarify the source of this gap.
Questions for authors
  1. What causes the depth and tracking degradation, and what do the existing keep-ratio and restoration results indicate about the achievable accuracy-efficiency trade-off?
  2. Could the authors report paired uncertainty estimates and clarify whether the operating points in the keep-ratio sweep were selected on a separate validation set?
  3. What intermediate allocations and tensor lifetimes account for the restoration variants' peak-memory gap at the same keep ratio?
  4. Could the authors discuss the differences from AVGGT [1] and the cited HeSS [2] and FlashVGGT [3], and provide feasible comparisons at matched latency or memory budgets on compatible tasks?

References: [1] Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR 2026. [2] Kim et al. HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT. CVPR 2026. [3] Wang and Xu. FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention. CVPR 2026.

4 comments

1. W1 + Q1: What causes depth and tracking degradation, and what is the accuracy–efficiency trade-off?

Direct answer. As shown in Figure 1(b), substantial token redundancy exists both within and across frames. Removing redundant tokens can reduce noisy interactions in global attention, potentially improving reconstruction quality. However, depth and tracking rely more heavily on fine-grained spatial information, which may be lost during pruning and cannot always be fully recovered by restoration. At r = 0.4, 60% of patch positions require restoration, making dense predictions particularly sensitive to information loss. We provide an accuracy–efficiency trade-off analysis on VGGT across different keep ratios.

Res-Table 4. Accuracy–efficiency trade-off of PrePARE on VGGT.

Keep ratioCD ↓ATE ↓Depth AbsRel ↓Tracking AJ ↑Time (s) ↓Mem (GiB) ↓
1.0 (Unpruned)0.45120.0960.01310.183048.972.3
0.60.41860.1090.02480.159415.817.8
0.4 (default)0.41870.0920.05200.13039.114.4
0.20.43790.1010.08010.09218.010.5

At r = 0.4, PrePARE achieves a 7.2% reduction in CD, an 80% reduction in peak memory, and a 5.4× speedup, providing the balance between accuracy and efficiency. These improvements suggest that removing redundant tokens can benefit global geometric reconstruction. However, depth AbsRel increases from 0.0131 to 0.0520, while tracking AJ decreases from 0.1830 to 0.1303. This indicates that dense predictions are more sensitive to the fine-grained information discarded during pruning. Further reducing the keep ratio to r = 0.2 also worsens CD and ATE compared with r = 0.4.

The revision. We include the expanded accuracy–efficiency analysis and clarify how token redundancy and fine-grained information loss affect different downstream tasks in Appendix.H.3 and Table A9.

2. Q2: Paired uncertainty estimates and operating-point selection

Direct answer. The default keep ratio (r = 0.4) is selected based on the accuracy–efficiency trade-off measured on a held-out validation subset drawn from the training data. The additional ratios (r = 0.6 and r = 0.2) are reported to characterize the trade-off.

To quantify the uncertainty of accuracy differences, we additionally report 95% paired-bootstrap confidence intervals using 10,000 scene-level resamples over the same 50 ScanNetv2 scenes evaluated in the original submission. We compute paired differences relative to unpruned VGGT (N = 300), where negative values indicate improvement.

Res-Table 5. Paired accuracy differences across keep ratios.

Keep ratioΔCD [95% CI] ↓ΔATE [95% CI] ↓Better scenes (CD)
0.6−0.0326 [−0.074, +0.011]+0.013 [−0.004, +0.034]26/50
0.4 (default)−0.0325 [−0.055, −0.011]−0.004 [−0.014, +0.001]34/50
0.2−0.0133 [−0.031, +0.003]+0.005 [−0.012, +0.025]31/50

At r = 0.4, PrePARE improves CD on 34/50 scenes (paired sign test, p = 0.015), with the entire 95% CI below zero, supporting a statistically significant reconstruction improvement. For ATE, the interval includes zero, but its upper bound is only +0.001 m (1.0% relative to unpruned VGGT). The intervals at r = 0.6 and r = 0.2 include zero for both metrics. Rather than interpreting nonsignificant differences as equivalence, we report these bounds to quantify potential accuracy degradation without introducing an arbitrary non-inferiority margin.

The revision. We add the paired confidence intervals and validation-based operating-point selection protocol to Appendix H.3 (Table A10).

3. W2 + Q3: What explains the peak-memory gap between learned restoration and ToMe-style unmerging?

Direct answer. The peak-memory difference mainly arises from restoration design choices. Our feature-guided restoration is designed for chunk-wise processing, whereas the evaluated ToMe-style baseline performs unchunked unmerging by their design, causing the different intermediate feature allocations and tensor lifetimes. We provide a stage-wise GPU memory breakdown for VGGT (N = 300, r = 0.4).

Res-Table 6. Stage-wise GPU memory profiling (GiB).

StageLearned restoration (Peak / Increment)ToMe-style unmerging (Peak / Increment)
Encoder features14.4 / 9.118.5 / 9.0
Scorer & routing9.0 / 2.412.1 / 1.4
AA stack14.1 / 7.036.9 / 25.8
AA features retained after stack3.823.1
Restoration & heads14.4 / 3.563.6 / 31.4
Overall peak14.463.6

The main difference comes from intermediate AA feature retention (3.8 vs. 23.1 GiB) and restoration-stage allocations (3.5 vs. 31.4 GiB). Our chunk-wise restoration limits the lifetime of dense intermediate features, whereas the evaluated unchunked ToMe-style implementation retains more AA features and materializes larger dense tensors during unmerging. This explains the substantially higher peak memory of the ToMe-style baseline under the evaluated implementations.

The revision. We include the stage-wise memory breakdown and clarify the restoration implementation choices and their impact on peak memory in Appendix.H.4 and Table A11.

4. Q4: How does PrePARE differ from AVGGT, HeSS, and FlashVGGT?

Direct answer. PrePARE reduces the number of tokens processed throughout the entire alternating-attention stack by merging or pruning. Thus our original evaluation only include token-pruning and token-merging methods for fair comparison. In contrast, AVGGT, HeSS, and FlashVGGT primarily optimize global attention computation while preserving the full patch-token stream. Specifically, AVGGT converts early global-attention layers into frame attention and subsamples keys and values in subsequent global-attention layers. HeSS adaptively redistribute sparsity across attention heads, allocating more computation to sensitive heads. FlashVGGT replacing dense global self-attention with cross-attention to compact per-frame descriptors.

We provide an additional comparison on the same reconstruction and pose-estimation benchmark on the same experiment machine. We do not include AVGGT because no implementation is publicly available.

Res-Table 7. Additional comparison on ScanNetv2 (N = 100).

MethodCD ↓ATE ↓ARE ↓RPEr ↓RPEt ↓Time (s) ↓Mem (GiB) ↓
Unpruned VGGT0.44060.0923.921.160.0555.427.2
FlashVGGT0.43830.0984.071.370.0675.111.0
HeSS0.43790.1034.221.280.0774.612.1
PrePARE0.41850.0953.821.280.0611.66.5

At N = 100, PrePARE achieves the lowest CD, inference time, and peak memory among the evaluated methods. Due to the resource and time limitation, we will provide the full comparison on N=300-1000 on future revision.

The revision. We expand Related Work to distinguish token reduction from attention-level acceleration and include the additional comparison in the Appendix.H.5 and Table A12.

Dear reviewer,

Thank you again for your constructive review. We would like to follow up on whether our response and the new results address your concerns on the cause of the depth and tracking degradation and the accuracy–efficiency trade-off (W1, Q1), paired uncertainty and operating-point selection (Q2), the peak-memory gap between learned restoration and ToMe-style unmerging (W2, Q3), and the comparison with AVGGT, HeSS and FlashVGGT (Q4).

If any question remains, we would be glad to clarify it. Thank you for your time.

Reviewer_GYdJOpenReview ↗

thanks for the response

I thank the authors for the detailed response and the additional experiments. The rebuttal clarifies several points: the keep-ratio sweep (Res-Table 4) and the paired bootstrap intervals (Res-Table 5) characterize the accuracy-efficiency trade-off and confirm that the operating point was selected on a held-out validation subset, the stage-wise profile (Res-Table 6) identifies where the peak-memory gap arises, and the new comparison with FlashVGGT and HeSS (Res-Table 7) is helpful. However, my main concerns remain. First, the sweep shows that the depth and tracking losses are not specific to the default operating point: both metrics are already clearly worse than the unpruned model at the largest keep ratio tested and degrade further as the ratio decreases. The response attributes this to lost fine-grained information but does not show how it could be reduced, and the paired intervals cover only CD and ATE, not the tasks that degrade. Second, the memory gap to ToMe-style unmerging is explained by chunk-wise versus unchunked processing, which suggests that a large part of the gap comes from implementation choices rather than from learned restoration itself; a chunked baseline would make this comparison more convincing. Third, the comparison with attention-level methods is reported only at N = 100 and not at matched latency or memory budgets and the longer-sequence results are deferred to a future revision. My suggestion is that the authors limit the accuracy-preservation claims to reconstruction and pose, and present the depth and tracking degradation clearly as a limitation.

Reviewer_WXqiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed PrePARE, a method for acceleration of transformer based VGGT style 3D foundation models. The core idea is to stop doing repeated token pruning/merging/expansion, but rather decide a set of tokens that are suitable for pruning and merging before the transformer inference stack. To do so, the authors proposed a few techniques, using the anchor (camera token) and token to token attention to decide which ones should be pruned or merged, a distilled student for learning such decision, which and how the tokens are restored via cross-attention.

Critical strengths

The work achieved relatively strong empirical results, showing reduction in speed and memory while improving the reconstruction quality in some cases. The evaluation is comprehensive, covering different models and different datasets, and the model to model differences and variations in treatment are also discussed.

The proposed methods form a coherent set, the pruning and decision to prune is distilled to a relatively small student network so during inference time the performance gain is real rather than having to run the full transformer again. The restoration considered the need for the learned restoration rather than simply copy pasting or interpolation, which through the ablation study demonstrated more accurate prediction.

Critical weaknesses

The distilled student for predicting the which tokens are suitable for pruning has a rather small 7x7 local attention window. It is counter-intuitive that this level of information suffices for the decision of which tokens are important or not. The ablations demonstrated such student choice is reasonable compared to not having it, but the impact of why it is sufficient is not properly demonstrated.

Studies regarding which tokens are important or not important, i.e. a qualitative or quantitative evaluation of the decision process is missing. Making the mechanism of the rather impressive empirical gain a bit black box, especially how does it improve the performance in some cases.

Questions for authors

Why did the authors decide to go with a distilled student that is rather local, what is the key characteristic that determines if a token is important or not?

The results when compared to the original model, especially under cross scene/data transfer scenarios, could be better or worse than the original. Do the authors have explanations and investigations on why this is the case?

3 comments

1. W1: Why is a 7×7 local attention window sufficient for token scoring?

Direct answer. We chose the 7×7 window to balance scoring accuracy and computational efficiency. Our additional ablation with 3×3, 7×7, and 14×14 windows reveals a clear diminishing-return pattern: increasing the window from 3×3 to 7×7 substantially improves accuracy with minimal overhead, whereas expanding it further to 14×14 yields only marginal gains at a much higher computational cost.

Evidence. We compare three attention window sizes under identical evaluation settings.

Res-Table 1. Comparison between different attention window sizes.

Attention windowCD ↓ATE ↓Time (s) ↓Mem (GiB) ↓
3×30.43630.1038.914.0
7×7 (default)0.41870.0929.114.4
14×140.41830.0919.915.2

Increasing the window from 3×3 to 7×7 reduces CD by 4.0% and ATE by 10.7%, with only a 2.2% increase in inference time and 2.9% in memory. In contrast, expanding from 7×7 to 14×14 improves CD by only 0.10% and ATE by 1.1%, while increasing inference time by 8.8% and memory by 5.6%. The results suggest that 7×7 attention window size is sufficient to capture the local contextual information. Further expanding the window size gets no significant improvement in accuracy but with increasing computational cost. Importantly, our scorer operates on pretrained encoder features and is supervised by the full cross-view attention of the frozen AA stack, allowing it to learn globally informed importance scores without performing expensive global attention during inference.

The revision. We add the complete window-size ablation to the Appendix.H.1, Table A6 and clarify the motivation for the 7×7 local attention design in Section 3.3.

2. W2 + Q1: What characteristics determine token importance?

Direct answer. In PrePARE, the matching scores calculated from in-AA tokens determining the importance. Specifically, our scorer learns token importance from both frozen model's attention behavior and local appearance. In Res-Table 2, we analyze the characteristics of kept, merged, and pruned tokens on ScanNetv2.

Res-Table 2. Characteristics of tokens selected by our learned scorer.

Token characteristicKeptMergedPruned
Mean image gradient (×0.01)4.563.402.08
Depth discontinuities rate within class (%)18.112.96.1
Wall / floor / ceiling rate within class (%)38.542.252.2

For example, kept tokens exhibit 2.2× higher image gradients and are 3.0× more likely to contain depth discontinuities than pruned tokens. Conversely, homogeneous surfaces (walls, floors, and ceilings) account for 52.2% of pruned tokens, compared with 38.5% of kept tokens. These results indicate that our scorer distinguishes geometrically informative regions from redundant ones to decide between keep/merge/prune; Figure 3(a) also shows a qualitative example.

The revision. We include these token-distribution statistics in the Appendix.H.2 and Table A7 to better explain the scorer's selection behavior.

3. Q2: Why does PrePARE sometimes outperform the unpruned model?

Direct answer. In some cases, PrePARE can improve reconstruction quality by selectively removing redundant tokens while preserving geometrically informative ones. Such redundancy can introduce unnecessary or noisy interactions during global attention. As observed in Figure 1(b), patch tokens exhibit substantial redundancy both within and across frames. On ScanNetv2 (N = 300), PrePARE achieves a 7.2% reduction in CD compared with unpruned VGGT, outperforming it on 34/50 scenes. In other cases, the spatial information may be lost during pruning and cannot always be fully recovered by restoration, leading to accuracy degradation. Additionally, the result of Table 4 implies that, on transferred dataset, the influence of redundancy or information loss could be magnified because the trained module has not seen the data distribution, e.g., PrePARE performs better than unpruned VGGT on 7-Scenes Stride 10 by 0.0016, but worse on NRGBD Stride 10 by 0.005.

Evidence. We quantitatively evaluate the token-selection decisions and their impact on reconstruction accuracy.

Res-Table 3. Token-selection effectiveness on VGGT (ScanNetv2, N = 300, r = 0.4).

Token-selection strategyOverlap with AA target ↑CD ↓
Unpruned VGGT—0.4512
Encoder cosine similarity (40% chance level)56%0.5193
Learned scorer (Ours)73%0.4187

Our learned scorer achieves 73% overlap with the attention-derived importance target while improving the CD compared to unpruned VGGT. To prove the learned scorer is critical to reconstruction quality, we additionally compare to naive encoder-similarity, the selection simply determine by cosine similarity will prune useful information with the degraded overlap of 55%, also leading to a performance drop of CD from 0.5193 to 0.4187. Furthermore, our token-distribution analysis (Res-Table 2) shows that learned scorer preferentially preserves distinctive geometric information while removing redundant content.

The revision. We add the quantitative token-selection analysis and clarified how the learned pruning criterion can contribute to improved reconstruction quality in Appendix.H.2 and Table A8.

I thank the additional experiments and they have addressed my concerns. I will maintain my acceptance rating.

Comments on the paper

1 comment

Thank you for your thoughtful feedback. We have submitted our responses and would appreciate your consideration of whether they address your concerns. If our clarifications have changed your assessment, we would be grateful if you could update your scores accordingly.

We would be more than happy to engage in further discussion and provide any additional clarification.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.