- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PrePARE, which prunes patch tokens once before the alternating-attention stack of a frozen multi-view geometry transformer and restores the dense grid once after it, so that every attention block operates on a reduced token set. A lightweight scorer distilled from the cross-view attention of the unpruned model selects the tokens to keep, and a feature-guided restoration module rebuilds the full grid for the dense prediction heads. Experiments on four backbones show substantial memory and latency savings with reconstruction and pose accuracy close to or better than the unpruned models, although the gains are most pronounced on VGGT and rely partly on in-domain supervised training.
Critical strengths
The design is simple and broadly applicable: by intervening only at the interfaces before and after the alternating-attention stack, PrePARE leaves the backbone frozen, adds only a small number of trainable parameters, and transfers to four different backbones by specifying their interfaces and anchor tokens, while making long input sequences feasible on a single GPU.
The token selection criterion is well motivated: instead of relying on encoder feature similarity, the scorer is distilled from how the unpruned model actually uses each token in cross-view attention, and the ablations clearly show that this criterion and the learned restoration outperform similarity-based and interpolation-based alternatives.
Critical weaknesses
**Major Weakness **
Train-test length mismatch. Both training stages use very short clips (2 to 8 frames on VGGT and 2 to 4 frames on the other backbones, Appendix A and Table A1), whereas evaluation is performed on sequences of 100 to 1000 frames. Since the attention distribution of a transformer depends strongly on the number of tokens it attends over, a difference of two to three orders of magnitude can lead to substantially different attention patterns. This affects the method in two ways: the AA-usage target that the scorer learns to predict is computed only on short clips and may not reflect which tokens matter in long sequences, and the AA outputs that the restoration module reads at inference differ from those it was trained on. The paper provides no analysis of why the trained modules still work under this shift. The authors should explain this and support it with evidence, for example by reporting how the restoration error with respect to the unpruned features varies with sequence length (which is feasible for VGGT up to 300 frames), or by comparing against modules trained on longer clips.
Unjustified loss weighting. The downstream loss (line 737) sums four terms, with all weight 1.0. These terms differ considerably in form. They may not own comparable magnitude, and equal weights do not imply equal contributions to training. The authors should report the value ranges and gradient norms of each term during training, so that readers can assess which terms actually drive the results. If their ranges indeed vary a lot, the authors should explain why “all weight 1.0” works.
**Minor Weakness **
Unequal training conditions in comparisons. FastVGGT is a training-free method, and LiteVGGT is fine-tuned on a general mixture of datasets that does not include ScanNetv2, whereas the trainable modules of PrePARE are trained on the ScanNetv2 training split with ground-truth depth supervision and then evaluated on the ScanNetv2 test split. The main comparisons in Tables 2 and 3 therefore pit an in-domain fine-tuned method against baselines that have not seen the target distribution, which favors PrePARE and makes it difficult to attribute its advantage to the pre-AA pruning design alone. The zero-shot results in Table 4 partially address this concern, but the advantage over the unpruned model is less consistent there. The authors should discuss whether this comparison is fair.
Inconsistency with previously reported baseline results. The paper states that its evaluation follows the FastVGGT protocol on ScanNetv2, yet the reported LiteVGGT results differ notably from those in the original LiteVGGT paper, which uses the same protocol. In the LiteVGGT paper (Table 1), LiteVGGT achieves a clearly lower Chamfer distance than VGGT at around 300 frames (0.365 versus 0.417), whereas in Table 2 of this submission it is worse than the unpruned VGGT (0.4589 versus 0.4512). The authors should explain this difference.
Typos and notation errors. The notation around Equation (2) is inconsistent: the equation writes the attention weight with the anchor token a or the token i as the query, while the following sentence defines it with a query index u that does not appear in the equation. In addition, the manuscript contains a number of spelling and grammatical errors, for example "computaional" and "lerger" in Section 1, "Motivatied" in Section 3.2, "eveluate" and "protocal" in Section 4.4, "PatchEnbed" in Figures 1 and 2, "apply the our framework" in Section 1, and "at theirs released head-sparsity profiles" in Section 4.1. A careful proofreading pass is recommended.
Questions for authors
Please check Critical Weaknesses and address these concerns.
6 comments
4. W4: Why do the LiteVGGT results differ from its original paper?
Direct answer. The result's differences comes from scene selection, frame sampling, or preprocessing contribute to the observed discrepancy. First, LiteVGGT does not specify a fixed 50-scene manifest or the exact frame indices underlying its reported results on their official repository. Their evaluation script selects scenes from the locally processed ScanNetv2 directory and samples frames according to the available images and valid poses. Second, LiteVGGT reports their results at 296 frames in their original paper, while our evaluation reports LiteVGGT at 300 frames, following VGGT, FastVGGT and Co-Me's settings for a fair comparison. Notably, VGGT baselines also vary across independent evaluations: FastVGGT reports 0.416 at 300 frames, LiteVGGT reports 0.417 at 296 frames, RegimeVGGT reports 0.454 at 300 frames, and our evaluation obtains 0.4512 at 300 frames. Third, Table 5 in paper gives LiteVGGT against unpruned VGGT at N=300 as better on 22 of 50 scenes, p = 0.48. The mean gap is 0.0077, under our protocol we do not find LiteVGGT to be worse than the unpruned VGGT. Our experiments evaluate all baselines under a unified evaluation protocol, providing internally consistent comparisons under the same settings.
5. W5: Notation inconsistencies and typos
We thank the reviewer for identifying these issues. We correct the inconsistent attention notation around Eq. (2), explicitly define the query and key indices, and carefully proofread the manuscript to resolve the listed spelling, grammar, and figure-label errors.
Response to W1 is not persuasive
I thank the authors for their response. My main concern still lies in W1, and I do not find the current response convincing.
It has been documented in prior work that VGGT suffers from error accumulation and degraded accuracy on long sequences, and methods such as FastVGGT and LiteVGGT were developed specifically with this in mind. FastVGGT, for example, is motivated by the token collapse phenomenon it observes in the global attention of VGGT on long sequences. Its merging decisions are made at inference time on the full sequence in every global-attention layer, and the tokens of the reference frame are always preserved. This targeted treatment of tokens is what allows it to remain robust on long sequences.
First, the paper states in Section 4.2 that "The efficiency and performance gain of PrePARE improves as the clip grows compared to other methods." The results in Table 2 do not provide strong support for the performance part of this claim.
Second, I would like the authors to discuss in more depth where and how PrePARE is designed to preserve accuracy on long clips. All the design choices and claims presented so far, such as pruning once before the AA stack, are motivated by reducing computational cost. The trained modules operate per frame, are trained on clips of at most 8 frames, without observing the sequence length or the content of other frames, so their ability to adapt their token selection to long sequences is not proven. The argument in Section 4.2 that pruned tokens are more likely to have kept counterparts in other views on longer clips is a post-hoc explanation that is not verified experimentally. I would appreciate concrete evidence AND interpretation that support your claim about the good accuracy on long clips.
1. W1: Does training on short clips generalize to long sequences?
Direct answer. Training on short clips and evaluating on substantially longer sequences is an established practice in multi-view geometry transformers, including VGGT, Fast3R, LiteVGGT, and MapAnything (Wang et al., 2025;Yang et al., 2025;Shu et al., 2026;Keetha et al., 2026), as summarized below. Specifically, the original VGGT (Wang et al., 2025) itself trained on N=2-24, and evaluate on N=100-1000.
Res-Table 8. Comparison between training clip length of different methods.
2. W2: Why are equal loss weights used?
Direct answer. We use unit weights for the main loss components because their magnitudes are comparable during training. We report their loss magnitudes and gradient norms from stage-2 training (664 optimization steps) in Res-Table 9. The results support using unit weights without additional loss-scale normalization.
Res-Table 9. Per-term loss magnitudes and gradient norms during VGGT stage-2 training.
Loss values are reported as mean [min–max]; gradient norms are measured at the start of stage-2 training.
The four unit-weight downstream losses have comparable mean magnitudes (0.095–0.287). As the gradient norms shows that equal weights are not intended to enforce equal gradient contributions, as the terms supervise different aspects (scoring, feature restoration, prediction consistency, and depth supervision) of geometric prediction.
The revision. We add the per-term loss magnitudes and gradient norms to Appendix H.6 (Table A13) and explain the rationale for our loss weighting strategy.
3. W3: Fair Comparison with FastVGGT and LiteVGGT?
Direct answer. PrePARE trains its lightweight modules using ScanNetv2 ground-truth depth, whereas FastVGGT is training-free and LiteVGGT finetunes on a broader dataset mixture. However, the efficiency gains of PrePARE primarily arise from reducing the token workload throughout the entire AA stack.
Training protocol. The ScanNetv2 training and evaluation scenes are building-disjoint. Furthermore, VGGT includes ScanNet in its original pretraining, so FastVGGT and LiteVGGT inherits this domain exposure, both of them have seen the target distribution.
Zero-shot generalization. Our original Table 4 evaluates PrePARE on 7-Scenes and NRGBD without retraining. PrePARE outperforms both Co-Me and LiteVGGT on all 12 reported metrics (Acc, Comp, and NC across two datasets and two sampling strides), demonstrating that its advantages over these acceleration baselines extend beyond the training distribution.
Compared with unpruned VGGT, the results are mixed: PrePARE improves Acc and Comp in 3/4 settings each, but NC in only 1/4. As discussed in our response to Reviewer WXqi (Q2), this reflects the trade-off between removing redundant tokens and preserving distinctive geometric information. We therefore distinguish the in-domain reconstruction improvements from the less consistent zero-shot improvements over the unpruned model.