acceptodds
Under review as a conference paper at ICLR 2027

QuadVGGT: Quotient Attention for Accelerating Dense-Token Inference in Visual Geometry Grounded Transformers

Abstract

Feed-forward geometric foundation models such as VGGT have significantly ad- vanced multi-view 3D reconstruction by directly predicting camera parameters and dense scene geometry from uncalibrated images. However, their global at- tention jointly processes all image tokens, causing computational and memory costs to grow rapidly with the number of input views and limiting long-sequence inference. Existing acceleration methods primarily reduce, merge, or compress tokens, potentially altering token-specific interactions within attention. We intro- duce QuadVGGT, a training-free framework based on Quotient Attention. Our key observation is that representational redundancy and computational redun- dancy are distinct: tokens that retain distinct representations for dense geomet- ric prediction can nonetheless induce similar attention interactions, allowing their computations to be shared without merging their representations. QuadVGGT uses sparse operator-induced profiles to select original token representatives for independent key–value and query computation sharing while preserving the orig- inal dense token stream. To further support long-sequence inference, we combine Quotient Attention with scene-adaptive anchor construction and hierarchical fea- ture offloading. QuadVGGT can be directly applied to pretrained VGGT without additional training or modification of its prediction heads. Extensive experiments across dense 3D reconstruction, depth estimation, and camera pose estimation demonstrate substantial efficiency gains while maintaining comparable geometric accuracy. On 1,000-view reconstruction, QuadVGGT achieves a 14.8× speedup and reduces peak GPU memory from 72.4 to 55.9 GB.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-view 3D reconstruction, visual geometry grounded transformers, efficiency, training-free acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_avEyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QuadVGGT accelerates long-sequence VGGT-based 3D reconstruction and reduces memory usage by sharing query and key-value attention computations through anchor-guided representatives while preserving dense token representations and geometric accuracy.

Critical strengths

I think the proposed acceleration method is thoughtfully designed by jointly considering the underlying computational mechanics of attention and the unique characteristics of 3D reconstruction. The method is simple, well-motivated, and effective. The authors also provide extensive experiments to substantiate their claims, and I have carefully examined the appendix as well.

Critical weaknesses
  1. The 500- and 1,000-frame experiments are primarily conducted on densely sampled sequences with small inter-frame motion and substantial visual overlap. However, the method is not evaluated on genuinely large-scale, long-range capture sequences, such as those commonly used to benchmark SLAM systems. Consequently, the current experiments mainly demonstrate that the proposed acceleration is effective for highly redundant inputs, but do not establish its robustness in long-sequence settings involving broad spatial coverage, large viewpoint changes, or limited frame overlap.
  2. Table 5 shows that the memory reduction mainly comes from HFO rather than Quotient Attention. Since this technique could also be applied to other methods, the memory comparison may not be fully fair. Applying the same offloading strategy to the baselines would provide a more controlled comparison.
Questions for authors

Please check the weaknesses and answer my questions.

4 comments
AuthorsOpenReview ↗

Author Response 1/N

Question 1 : How robust is QuadVGGT on large-scale, long-range sequences with substantial viewpoint changes and limited frame overlap?

We thank the reviewer for highlighting the need to evaluate QuadVGGT on long-range sequences with limited frame overlap and substantial viewpoint changes. This insightful suggestion prompted us to conduct an additional evaluation on KITTI Odometry 00–10, through which we uncovered an important aspect of QuadVGGT's adaptive behavior under challenging viewing conditions. KITTI is a standard outdoor SLAM benchmark covering kilometre-scale driving routes with limited overlap between consecutive views. We uniformly sample each sequence over its full length to at most 2,500 frames, further increasing the gap between consecutive inputs (Table R8).

On KITTI, QuadVGGT runs 11.3× faster than dense VGGT and is also faster than FlashVGGT, while using 27% less peak memory than both. It also achieves lower ATE, ARE, and RPE-R than FlashVGGT, with comparable RPE-T. Across all sequences, its worst-case ATE remains within 1.74× of dense VGGT, whereas FlashVGGT degrades by 13× on seq04. On ETH3D, QuadVGGT achieves camera-pose accuracy comparable to dense VGGT (Table 12(b)). Further ablations show that operator-induced assignments preserve near-dense accuracy, whereas simpler anchor and representative-selection strategies exhibit noticeable degradation (Table 14).

Importantly, these additional experiments reveal the adaptive nature of Delta-Walking. Unlike fixed-ratio methods such as FlashVGGT and FastVGGT, Delta-Walking adjusts the anchor budget according to scene redundancy. Specifically, a frame is selected as a new anchor when 1 − ⟨g_s, g_{a_last}⟩ > Δ* (Eq. (2)), where g_s is the normalized frame descriptor and a_last denotes the latest anchor. Using the same threshold (Δ* = 0.002), Delta-Walking retains approximately 2.5× more anchors on KITTI than on 7-Scenes, while anchors still account for only about 5% of the frames. Non-anchor tokens are assigned to representative tokens in nearby views observing the same structure, while preserving the original token representations of all frames for per-frame predictions. Thus, QuadVGGT adapts to limited overlap without sacrificing most of the attention reduction.

Although absolute errors remain high for all feed-forward models on kilometre-scale routes without loop closure, including dense VGGT, QuadVGGT closely preserves the dense model's behavior as frame overlap decreases. We will incorporate the KITTI evaluation and the corresponding discussion into the revised manuscript.

We sincerely appreciate the reviewer's insightful suggestion. It not only helped us strengthen the experimental evaluation but also led us to uncover and better understand an important characteristic of QuadVGGT's adaptive design. We are truly grateful for this valuable feedback.

Table R8. KITTI 00–10, 2,500 frames (mean of 11 sequences). Speed-up over VGGT in parentheses. Best values are shown in bold.

MethodATE↓ARE↓RPE-T↓RPE-R↓Total time↓Peak mem↓
VGGT (Wang et al., 2025a)188.5249.8716.408.093,900 s (×1.0)71.7 GB
FastVGGT (Shen et al., 2026)185.4242.9516.627.721,642 s (×2.4)71.9 GB
LiteVGGT (Shu et al., 2026)202.7947.8615.977.68703 s (×5.5)50.3 GB
FlashVGGT (Wang & Xu, 2026)207.7870.6220.477.57393 s (×9.9)71.5 GB
SparseVGGT (Wang et al., 2026a)OOMOOMOOMOOMOOMOOM
SAF3R (Deng et al., 2026)OOMOOMOOMOOMOOMOOM
QuadVGGT (Ours)178.8754.1120.896.20344 s (×11.3)52.2 GB
AuthorsOpenReview ↗

Author Response 2/N

Question 2 : Is the memory comparison fair without applying the same HFO strategy to the baselines?

We thank the reviewer for this suggestion and agree that a controlled memory comparison is necessary. Quotient Attention and HFO address independent bottlenecks: the former reduces attention computation, while the latter offloads intermediate features retained by VGGT-style prediction heads. Since these heads consume four intermediate layers only after the final block, we applied the same HFO strategy to dense VGGT and all five baselines without modifying their attention. Only feature storage and restoration were changed, and outputs remain bit-identical (Table R9).

The memory savings confirm that HFO addresses a common architectural bottleneck: approximately 24 GB for dense VGGT and FlashVGGT (fp32), and 11.9 GB for FastVGGT and LiteVGGT (bf16). HFO also enables SparseVGGT and SAF3R to complete all 7-Scenes sequences at 1,000 frames, where they previously failed.

With HFO applied equally, QuadVGGT's peak memory falls within the baseline range, separating HFO's memory contribution from that of Quotient Attention. Importantly, QuadVGGT remains the fastest method, achieving 1.55× speedup over FlashVGGT, 2.26× over LiteVGGT, 2.02× over FastVGGT, and 14.55× over dense VGGT. Thus, the controlled comparison confirms that Quotient Attention provides the speed-up, while HFO provides architecture-independent memory savings. We will add Table R9 and clarify this attribution in the revision.

Table R9. Controlled comparison with HFO applied to every method (7-scene, N-BGRD, ScanNet 1000 frames)

MethodHFORel↓τ↑Acc↓Comp↓CD↓NC↑APE↓ARE↓RPE-T↓RPE-R↓Time (s)↓Mem (GB)↓
VGGT✗.0313.9670.0119.0088.0104.6979.0548.0985.0083.0143678.2772.40
✓.0313.9670.0119.0088.0104.6979.0548.0985.0083.0143666.4348.5
FastVGGT✗.0383.9712.0134.0079.0101.6924.0459.0621.0058.013994.8372.7
✓.0383.9712.0134.0079.0101.6924.0459.0621.0058.013992.7460.8
LiteVGGT✗.0402.9690.0175.0089.0132.6504.0711.1313.0104.0184108.7751.1
✓.0402.9690.0175.0089.0132.6504.0711.1313.0104.0184103.5339.2
FlashVGGT✗.0301.9745.0115.0079.0097.6701.0403.0639.0053.007172.2372.2
✓.0301.9745.0115.0079.0097.6701.0403.0639.0053.007170.7748.4
SparseVGGT✗OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
✓.0452.9238.0110.0097.0104.6182.0483.0680.0061.0085532.7669.6
SAF3R✗OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
✓.0440.9238.0108.0094.0101.6244.0434.0652.0051.0069183.5869.7
QuadVGGT (Ours)✗.0274.9670.0087.0083.0085.7229.0362.0577.0058.007558.7276.5
✓.0274.9670.0087.0083.0085.7229.0362.0577.0058.007545.8055.9

Thanks for the response. The newly added experiments are useful for addressing my concerns.

We sincerely thank the reviewer for their positive feedback. We are glad that the additional experiments helped address your concerns. We greatly appreciate your constructive comments, which have helped us strengthen our work.

Reviewer_TgXYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces QuadVGGT, a training-free method for increasing the inference speed of VGGT and similar geometric foundation models. Its central contribution is Quotient Attention, which independently reduces key-value computation and shares query outputs while preserving the original dense residual-token stream.

The framework also incorporates Delta-Walking for scene-adaptive anchor selection and Hierarchical Feature Offloading for GPU memory reduction.

At 1000 views, the method reports a 14.8x speedup over the original VGGT, with peak GPU memory decreasing from 72.4 GB to 55.9 GB, while the reconstruction accuracy remains competitive.

Critical strengths
  • The method achieves up to 14.8x faster inference than VGGT in the 1000 frames setting while maintaining competitive accuracy. Surprisingly, sometimes it even outperforms the original model.
  • The method is training-free and applicable to not only VGGT but also other geometric foundation models such as DA3, π3\pi^3 and VGGT-Ω\Omega.
  • The paper evaluates multiple reconstruction tasks and provides comprehensive comparisons with recent baselines and ablation studies.
Critical weaknesses
  • The technical novelty is less than the presentation suggests. The Delta-Walking Anchor Construction is essentially threshold-based keyframe selection. And I believe comparing a feature distance to the last keyframe against a fixed threshold is a standard heuristic in computer vision, which offers limited novelty and is better presented as an implementation choice than as part of the contribution.
  • Also, the Hierarchical Feature Offloading is more like an implementation detail, hardly to be justified as a contribution.
  • Beyond the two components discussed above, the main novelty appears to lie in the proposed Quotient Attention and Sparse Operator-Induced Profiles for representative selection. However, the overall approach closely resembles existing token merging and clustering methods, such as FastVGGT[1] and Fast Transformers with Clustered Attention[2]. The proposed profiles essentially provide an alternative similarity criterion for selecting representative tokens. While the empirical improvements are encouraging, I am not convinced that it constitutes a sufficient methodological contribution over prior work.
  • Lastly, presentation clarity is poor. Several concepts, terminologies and mathematical notations introduced without definition or explanation. Readers often have to infer their meaning from context. This makes the methodology unnecessarily difficult to follow.

Reference: [1] Shen et al., Fastvggt: Fast visual geometry transformer. [2] Vyas et al. Fast transformers with clustered attention.


Rebuttal Update (Nov 18): The authors' clarifications have addressed most of my concerns. I have therefore updated my rating from 2 (Weak Reject) to 3 (Weak Accept).

Questions for authors
  1. Could the authors further compare the key differences between Quotient Attention and existing token merging or clustering approaches, clarifying which components are shared with prior work and which are novel contributions of this paper?
  2. Could the authors provide a detailed inference-time breakdown for different input sequence lengths, including anchor selection, operator-induced profile computation, representative matching, attention computation, and hierarchical feature offloading? This would help clarify the computational overhead.
5 comments
AuthorsOpenReview ↗

Author Response 1/N

Weakness 1 : (i) Limited technical novelty over threshold-based keyframe selection; (ii) limitations of the standard heuristic

(i) Delta-Walking draws inspiration from threshold-based keyframe selection but serves a different purpose in Quotient Attention. Rather than selecting which input frames to retain, it identifies anchor frames that define the candidate pool for representative-token selection before global attention. This candidate restriction is necessary for our design; Delta-Walking is one practical strategy for constructing this pool, and we also evaluate uniformly spaced anchors as an alternative (Appendix, Tables 13 and 14). Unlike conventional keyframe selection, it does not discard any input frames, dense token states, or per-frame predictions; anchors only restrict the pool from which representatives are selected. It requires only one feature inner product per frame using patch-embedding descriptors, without requiring camera poses or additional training.

(ii) Although thresholding itself is a standard heuristic, Table 8 in the Appendix A.1 presents a threshold sensitivity analysis demonstrating the robustness of Delta-Walking to threshold variations. Halving the threshold from Δ* = 0.002 to Δ* = 0.001 roughly doubles the retained KV ratio on 7-Scenes, yet changes every metric by at most 2%, while both ETH3D AUC metrics change only slightly. This indicates that Delta-Walking does not require precise threshold tuning within the tested range. Thus, Delta-Walking provides an efficient and robust way to construct a scene-adaptive candidate set, while the main accuracy gains come from Sparse Operator-Induced Profiles for representative selection, rather than from the anchor heuristic itself.


Weakness 2 : Hierarchical Feature Offloading is more like an implementation detail, hardly to be justified as a contribution

We respectfully disagree that HFO is merely an implementation detail. While CPU offloading itself is established, to our knowledge, HFO is the first inference-time CPU offloading strategy specifically tailored to the hierarchical prediction-head architecture of VGGT-style models. Unlike conventional activation offloading, which primarily targets tensors retained for backward propagation, HFO addresses the delayed consumption of intermediate features during inference. It offloads each feature upon production and restores only the frame slabs required by the prediction head, rather than transferring and rematerializing the entire feature set.

This is a distinct methodological contribution because it addresses a memory bottleneck that attention acceleration alone cannot resolve. As shown in Table 5, HFO reduces dense VGGT's peak GPU memory from 72.4 to 55.9 GB without affecting reconstruction accuracy. Moreover, even with HFO applied equally to the baseline, QuadVGGT remains 15.4× faster (648.6 vs. 42.2 s). We therefore consider HFO an important methodological contribution of this work, complementary to Quotient Attention, rather than a generic implementation optimization.


Weakness 4 : Presentation: concepts, terminology and notation used without definition

We thank the reviewer for this feedback. We agree that some concepts and notations are introduced too compactly. We will improve clarity by adding a notation explanation, explicitly defining key terms and assignment mappings (Eqs. 3–8), and clarifying the three error terms in Eq. 9. We will also refine the explanation of how Delta-Walking, Quotient Attention, and HFO interact, making the methodology easier to follow.

AuthorsOpenReview ↗

Author Response 2/N

Weakness 3 & Question 1: How does Quotient Attention differ from existing token-merging and clustering methods, and what are its novel contributions?

We thank the reviewers for raising this important question. Quotient Attention differs from FastVGGT [1] and Clustered Attention [2] in three respects: what determines computation sharing, how groups are represented, and which sides of attention are reduced. Table R6 summarizes these key differences. In multi-view geometry transformers, tokens observing overlapping regions may induce similar attention interactions while requiring distinct features for dense predictions. We therefore target redundancy in attention computation rather than token representations.

Table R6. Comparison of Quotient Attention with FastVGGT and Clustered Attention across three key design aspects: sharing criterion, group representation, and attention reduction.

Comparison CriterionQuotient AttentionFastVGGTClustered Attention
Sharing CriterionAttention operator equivalenceFeature Cosine SimilarityEuclidean Query Distance
Group RepresentationOriginal representative tokensMerged tokensCluster-based representations
Reduction TargetIndependent Query and KV reductionShared merging of Query and KVQuery-only reduction

First, we decide which tokens can share computation based on their attention behavior, rather than simply how similar their features look. Two queries produce the same attention distribution when their scores (logits) against the retained keys differ only by a constant (Prop. B.5), since softmax ignores such constant shifts. Our Query Profile estimates this condition using a small set of probe keys (Eq. 4), while the KV Profile compares incoming logits using probe queries (Eq. 3). This is effective because VGGT's query-key interactions have a low effective rank, allowing a small number of probes to capture their main structure (Fig. 3(a–c)). The resulting assignments also show greater 3D consistency than query cosine similarity (Fig. 7(d)). In comparison, Clustered Attention groups queries based on Euclidean distance, while FastVGGT uses feature cosine similarity.

Second, we use existing tokens as representatives instead of averaging tokens into new ones. Each non-anchor patch is assigned to an original representative selected from patches in its two temporally nearest anchor frames. This avoids iterative clustering and centroid updates while preserving the representative's original keys and values. The approximation is exact when the grouped tokens have identical attention distributions and values.

Third, we reduce queries and KV separately rather than using the same reduction for both. Quotient Attention uses different criteria to select representatives for queries and KV. FastVGGT instead uses one shared set of merged tokens for both, while Clustered Attention reduces queries but keeps all keys and values. Our ablations show that reducing KV provides most of the acceleration, whereas reducing queries alone leaves the KV computation unchanged.

Taken together, Quotient Attention goes beyond similarity-based token selection by deriving sparse Query and KV profiles from the attention operator's equivalence conditions. This enables independent computation sharing on both sides of attention while preserving token-specific representations. Our theoretical analysis establishes when this sharing is exact and bounds its approximation error, while ablations validate the contributions of Query and KV reduction.

AuthorsOpenReview ↗

Author Response 3/N

Question 2 : Inference-time breakdown across sequence lengths

Table R7 breaks down one forward pass from 100 to 1,000 frames. The non-overlapping computation phases sum to the reported total, while HFO transfer time is reported separately to avoid double counting overlapped operations. The phases our method adds are cheap, and their share falls as the sequence grows. Anchor selection takes under 0.01 s. Profile computation uses only m = 16 probes per layer. Representative matching compares each token only with the patches of its two candidate anchor frames. Together with output gathering, these phases fall from 11% of our runtime at 100 frames to 6% at 1,000 frames. The reduced attention takes 5–29% of our runtime, whereas the dense attention it replaces takes 76–97% of VGGT’s. At 1,000 frames, 64% of our time is spent in unchanged VGGT components, and the speed-up over dense VGGT grows from 2.7× at 100 frames to 15.2× at 1,000. HFO copies overlap with computation, and at most 0.34 s of them is exposed. The pinned host buffers are allocated once per process (12.6 s at 1,000 frames); this one-time cost is excluded from the table.

Table R7. Inference time per phase in seconds for one forward pass (7-Scenes, N-RGBD, ScanNet; A100 80GB, bf16, steady state). All phases are timed with CUDA events and synchronized only at the end. The non-overlapping computation phases sum to the reported total, while the exposed HFO transfer time is reported separately to avoid double counting overlapping operations.

FramesAnchor selectionProfile computationRepresentative matchingOutput gatheringReduced attentionHFO copies (exposed)Unchanged VGGT componentsTotal (ours)Dense VGGT
100<0.010.090.250.060.170.033.023.599.53
300<0.010.210.590.190.970.109.1811.1468.84
5000.010.340.980.323.200.1715.3520.18184.86
7000.010.461.260.447.450.2421.5931.21364.07
1,0000.010.691.800.6314.460.3431.5449.12744.75
Reviewer_ZKa5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QuadVGGT, a training-free framework for accelerating visual geometry transformers. Its Quotient Attention independently shares KV and query computations without merging token representations. Together with adaptive anchor selection and feature offloading, it substantially reduces inference cost while retaining competitive geometric accuracy.

Critical strengths
  1. Jointly reducing KV and query computation is an interesting design, supported by component-wise ablations.
  2. Preserving dense token representations offers a useful alternative to token merging, and the approximation analysis helps explain the design.
Critical weaknesses
  1. Delta-Walking has a substantial efficiency limitation on shuffled inputs, although the paper explicitly targets ordered long-video sequences. Delta-cover mitigates this limitation with an accuracy trade-off on ordered inputs; the applicability of each strategy could be clearer.
  2. Some approximation and implementation details need further validation. Although Fig. 7 measures mass/query distortion and Table 10 studies value averaging, the attention-mass-weighted value-substitution term in Eq. 9 is not directly quantified. The supplementary code also reuses KV partitions across global layers; its relationship to the reported results and the end-to-end cost of offloading could be clearer.
Questions for authors
  1. Could the authors quantify the value-substitution term across layers and scenes, and discuss value-aware representative selection, as distinct from the value averaging already studied?
  2. Could the authors clarify the accuracy-efficiency trade-off between Delta-Walking and Delta-cover, and recommend which to use for ordered versus shuffled inputs?
  3. Were the main results obtained with cross-layer KV-partition reuse? Please clarify its accuracy/runtime trade-off, peak host memory, and whether latency includes transfers and initial setup.
  4. Could the authors discuss the differences from AVGGT [1], HeSS [2], and Co-Me [3], and provide feasible comparisons under matched settings, accounting for training or calibration costs?

References: [1] Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR 2026. [2] Kim et al. HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT. CVPR 2026. [3] Chen et al. Co-Me: Confidence Guided Token Merging for Visual Geometric Transformers. CVPR 2026.

4 comments
AuthorsOpenReview ↗

Author Response 1/N

Weakness 1 & Question 2: When should Delta-Walking or Delta-cover be preferred, considering their accuracy–efficiency trade-offs on ordered and shuffled inputs?

Table 9 compares the two strategies under ordered and shuffled inputs. Rather than exhibiting a direct accuracy–efficiency trade-off under the same input ordering, their relative advantages depend on the input order.

On ordered input, Delta-Walking achieves better accuracy and slightly lower runtime than Delta-cover (20.3 vs. 20.9 s), with comparable anchor counts (29–44 vs. 25–36). Delta-cover occasionally selects visually similar but geometrically distinct anchors, resulting in higher RPE-Trans and RPE-Rot (31% and 18%, respectively).

On shuffled input, Delta-Walking loses its compression efficiency, retaining 440–467 anchors and taking 152.1 s (1.2× speedup). In contrast, Delta-cover retains only 19–30 anchors and runs in 20.1 s (8.9× speedup), while also achieving better results on some accuracy metrics. Its accuracy remains within approximately 3.5% of ordered Delta-Walking.

Therefore, we recommend Delta-Walking when capture order is known (e.g., video or sequential capture), and Delta-cover when input order is unknown or unreliable (e.g., unordered photo collections). We will clarify this order-dependent strategy selection in the revised manuscript.


Weakness 2 & Question 3: How do (i) cross-layer KV-partition reuse and (ii) offloading overhead affect accuracy, runtime, peak host memory usage, and end-to-end efficiency?

We thank the reviewer for asking us to clarify the effects of cross-layer KV-partition reuse and HFO overhead. We provide controlled comparisons to quantify their respective impacts on accuracy, runtime, and memory usage.

(i) Cross-layer KV-partition reuse. We support two modes: building the KV partition once at the first global layer and reusing it, or rebuilding it at every layer. All reported results use the first mode. Each layer still uses its own keys and values, while only the representative assignments determined by the initial partition are reused. Since Eq. 9 holds for any assignment, it also bounds the error under reuse. Table R1 shows that rebuilding provides no consistent accuracy gain, while reuse achieves better NC and most pose metrics at lower runtime (21.29 vs. 21.82 s). Thus, reuse is a design choice rather than merely a speed shortcut.

Table R1. KV partition reuse vs. rebuilding at every layer (7-Scenes, 500 frames; identical settings).

KV partitionAbs Rel↓τ↑Acc↓Comp↓CD↓NC↑APE↓ARE↓RPE-T↓RPE-R↓Time (s)↓
Built once, reused.0430.9225.0110.0114.0112.6335.0372.0624.0050.006921.29
Rebuilt at every layer.0429.9222.0110.0108.0109.6274.0378.0613.0057.007521.82

(ii) HFO memory and latency. All reported latencies are CUDA-event measurements of the complete forward pass, including Delta-Walking, profiling, partitioning, bidirectional host transfers, and prediction heads. Excludes data loading and pinned-buffer allocation (12.6 s).

At 1,000 frames, HFO increases peak host memory from 35.0 to 67.0 GB (including 32.0 GB of pinned buffers), while reducing peak GPU memory from 76.5 to 52.7 GB (Table R2). Transfers overlap with computation, leaving only 0.34 s exposed. On 7-Scenes, HFO reduces runtime from 67.10 to 49.12 s under high GPU memory pressure. Accuracy remains identical at the reported precision.

Table R2. HFO overhead on 7-Scenes. Time , GPU memory, host memory and accuracy: all scenes, 1,000 frames. Without HFO, the all-scene run requires PyTorch's expandable-segments allocator.

Without HFOWith HFO
Time (s)67.1049.12
Exposed transfer time (s)–0.34
Peak GPU memory (GB)76.552.7
Peak host memory (GB)35.067.0
Pinned HFO buffers (GB)0.032.0
Depth: Abs Rel / δ<1.25.0428 / .9235.0428 / .9235
Point: Acc / Comp / CD / NC.0108 / .0109 / .0109 / .6414.0108 / .0109 / .0109 / .6414
Camera: APE / ARE / RPE-T / RPE-R.0418 / .0671 / .0050 / .0068.0418 / .0671 / .0050 / .0068
AuthorsOpenReview ↗

Author Response 2/N

Question 1: (i) Quantify how the value-substitution error varies across layers and scenes, and (ii) discuss value-aware representative selection.

Our profiles primarily target the two attention-routing terms in Eq. 9: KV mass distortion and query sharing. While the KV profile also captures the logit deviations involved in the value-substitution bound (Corollary B.4), it does not explicitly minimize within-group value variation. We therefore quantify the previously unreported value-substitution term across all 24 global layers (Table R3) and 27 scenes from 7-Scenes, N-RGBD, and ETH3D (Table R4).

(i) Value substitution. The value-substitution term in Eq. 9 measures how well a representative's value approximates those of its group members, weighted by the attention mass assigned to each group. Although this term contributes to the upper bound on KV approximation error, it is conservative in practice: the measured KV-only output error is approximately 8.6× smaller than this term and remains below 10% of the output norm across every evaluated layer and scene. The term is larger on highly redundant video sequences, where fewer representatives are retained (0.57 on 7-Scenes and N-RGBD), than on ETH3D (0.30).

Table R3. Value-substitution term of Eq. 9 and actual KV output error per global layer, both relative to the norm of the exact attention output. Mean over 27 scenes (7-Scenes 7, N-RGBD 9, ETH3D 11), with datasets weighted equally.

LayerValueKV err.LayerValueKV err.LayerValueKV err.
L0.12.014L8.97.066L16.70.097
L1.61.038L9.74.062L17.67.054
L2.49.036L10.56.055L18.39.057
L3.46.041L11.36.066L19.34.082
L4.47.041L12.33.065L20.49.061
L5.52.048L13.36.072L21.35.059
L6.64.056L14.15.044L22.52.061
L7.69.060L15.36.076L23.24.028

(ii) Value-aware representative selection. We evaluate value-aware representative selection to examine its effect on value substitution and overall KV approximation. Selecting representatives based on token-feature similarity yields representatives with closer values and reduces the value-substitution term across all 27 scenes. However, this reduction does not translate into lower overall KV error: the actual KV error increases in all 27 scenes, and pose accuracy also degrades. This occurs because a representative's key also determines how much attention its group receives; selecting a representative based on value similarity can therefore distort the group's attention mass, increasing KV mass distortion. Adding value similarity to our operator-induced profile leaves accuracy essentially unchanged, with all 7-Scenes metrics remaining within 0.5% of the original results and ETH3D AUC@30 changing from 0.8242 to 0.8260.

Table R4. Dataset-level means of the value-substitution term in Eq. 9 and actual KV output error (averaged over 24 global layers) for our profile and feature-similarity selection. Each dataset mean covers all scenes; the final row reports the equally weighted mean across the three datasets.

DatasetRetainedOurs: ValueOurs: KV err.Feature: ValueFeature: KV err.
7-Scenes (7 scenes)11–24%.57.057.47.066
N-RGBD (9 scenes)6–11%.57.069.46.082
ETH3D (11 scenes)53–89%.30.041.26.047
Mean (three datasets, weighted equally)—.48.056.40.065

These results suggest that value substitution is not the dominant factor limiting geometric accuracy under our current setting. Instead, preserving attention mass and controlling query sharing remain more consequential. This supports our design choice of prioritizing the two attention-routing terms in Eq. 9 rather than explicitly optimizing value similarity. We will incorporate this analysis and discuss value-aware selection as a potential extension.

AuthorsOpenReview ↗

Author Response 3/N

Question 4 : Could the authors discuss the differences from AVGGT [1], HeSS [2], and Co-Me [3], and provide feasible comparisons under matched settings, accounting for training or calibration costs?

We thank the reviewer for this suggestion. AVGGT, HeSS, and Co-Me are concurrent methods that accelerate VGGT's global attention. They differ from QuadVGGT in their reduction criteria, token representations, and computational-budget allocation.

AVGGT replaces early global-attention layers with frame attention, then subsamples KV tokens on a fixed spatial grid while retaining all queries. It uses diagonal preservation and mean-fill to compensate for removed KV entries. Although training-free, its subsampling ratio is fixed.

HeSS redistributes a fixed block-sparse attention budget across heads using Fisher sensitivity scores derived from camera-pose and point-cloud losses on labeled calibration data. It requires offline supervision and backpropagation; the resulting scores are fixed at inference, although selected blocks can vary by input.

Co-Me distills a confidence predictor from a frozen VGGT backbone to merge low-confidence 3×3 token groups at a predefined ratio. Selected groups vary by input, but the overall budget remains fixed. Merged tokens are processed as single tokens, and spatial resolution is restored through replication-based splitting, without preserving individual representations.

QuadVGGT differs in three respects.

(i) Training and calibration. QuadVGGT requires neither. It determines computation sharing at inference time using attention-operator equivalence.

(ii) Token representation and computation sharing. All original token features are preserved through the network and prediction heads; only attention computation is shared. Representatives are original tokens, not averages, and sharing is exact when group members duplicate their representative (Prop. B.1). Although Co-Me's log n bias and our log M correction both follow ToMe's proportional attention, ours operates on original representatives rather than merged averages.

(iii) Adaptive computational budget. Delta-Walking determines the anchor budget from scene redundancy rather than a preset ratio. It retains fewer anchors for redundant video and more for low-overlap inputs. Delta-cover (Appendix A.2) extends this adaptivity to unordered inputs. In contrast, AVGGT, HeSS, and Co-Me use predetermined subsampling, sparsity, and merging budgets, respectively.

Matched comparison. We evaluated all three methods using the same frames, resolution, metrics, precision, and GPU as Table 3. HeSS and Co-Me use official code and released checkpoints. Since AVGGT has no public code, we reimplemented it and verified that disabling subsampling reproduces dense VGGT. Table R5 reports 1,000-frame averages over 7-Scenes, N-RGBD, and ScanNet-50. QuadVGGT outperforms AVGGT on 8/10 accuracy metrics and Co-Me and HeSS on 9/10. It is also 1.7× faster than AVGGT and approximately 5× faster than Co-Me and HeSS, without training or calibration.

Table R5. Comparison with AVGGT, Co-Me, and HeSS under matched evaluation settings (1,000 frames; averages over 7-Scenes, N-RGBD, and ScanNet-50).

MethodRel↓τ↑Acc↓Comp↓CD↓NC↑APE↓ARE↓RPE-T↓RPE-R↓Time (s)↓Mem (GB)↓
VGGT.0313.9670.0119.0088.0104.6979.0548.0985.0083.0143678.2772.40
FastVGGT.0383.9712.0134.0079.0101.6924.0459.0621.0058.013994.8372.60
LiteVGGT.0402.9690.0175.0089.0132.6504.0711.1313.0104.0184108.7751.10
FlashVGGT.0301.9745.0115.0079.0097.6701.0403.0639.0053.007172.2372.40
SparseVGGTOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
SAF3ROOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
AVGGT†.0331.9671.0110.0071.0091.6652.0450.0668.0077.011079.0472.38
Co-Me.0414.9490.0153.0082.0117.6356.0549.0775.0145.0206242.1768.12
HeSS.0516.9504.0223.0080.0152.6214.0932.2103.0170.0349219.1574.90
QuadVGGT (Ours).0274.9670.0087.0083.0085.7229.0362.0577.0058.007545.8055.90

† AVGGT is reimplemented from the paper because official code is unavailable.

References

[1] Sun et al. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR 2026.

[2] Kim et al. HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT. CVPR 2026.

[3] Chen et al. Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers. CVPR 2026.

Reviewer_ZKa5OpenReview ↗

thanks for the response

Thank you for the detailed response. The rebuttal addresses my main questions: (1) Table 9 clarifies that the two anchor strategies are complementary that Delta-Walking is preferred for ordered inputs, while Delta-cover restores the speedup on shuffled inputs with accuracy within about 3.5% of ordered Delta-Walking, although it is less accurate on ordered inputs; (2) the authors confirm that all reported results use cross-layer KV-partition reuse, show that rebuilding at every layer brings no consistent gain, and report the end-to-end cost of HFO, including the increase in peak host memory and the excluded one-time buffer allocation; (3) the value-substitution term is now quantified across 24 layers and 27 scenes, and the value-aware selection experiment supports the choice of prioritizing the attention-routing terms; (4) the matched comparison with AVGGT, HeSS, and Co-Me is helpful. My remaining reservation is that the anchor strategy must still be chosen based on prior knowledge of the input order, and I suggest the authors to improve it in future version. Overall, the response confirms my initial assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all reviewers for their thoughtful comments and constructive feedback. We have addressed the individual concerns in our responses and revised the manuscript accordingly. The key updates are summarized below (revised portions are highlighted in blue in the manuscript):

  • Figure 2 and Sections 3.1–3.3 have been revised to improve presentation clarity, with more explicit notation definitions, explanations of representative selection, and clarification of the roles of Delta-Walking, Quotient Attention, and HFO.

  • Appendix A.1 (Table 8) provides a threshold sensitivity analysis of Delta-Walking, demonstrating its robustness to variations in the anchor-selection threshold.

  • Appendix A.2 (Table 9) analyzes ordered and shuffled inputs, introduces Delta-cover as an order-robust alternative, and clarifies when each anchor-construction strategy should be preferred.

  • Appendix A.3 (Tables 10–11) quantifies the value-substitution term across 24 global-attention layers and 27 scenes, and compares operator-induced representative selection with feature-similarity selection.

  • Appendix A.5 (Tables 13–14) provides additional comparisons with simple anchor baselines, including uniformly spaced anchors and alternative representative-assignment strategies.

  • Appendix A.7 (Table 18) adds long-range evaluation on KITTI Odometry 00–10, covering kilometre-scale trajectories with limited frame overlap and substantial viewpoint changes.

  • Appendix A.8 (Tables 19–21) provides an end-to-end inference-time breakdown, HFO-controlled comparisons across baselines, and detailed GPU/host memory and transfer-overhead measurements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.