acceptodds
Withdrawn from ICLR 2027

GeoCircuit: Training-Free Task-Aware Head-Circuit Extraction for Visual Geometry Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Feed-forward 3D reconstruction models such as the Visual Geometry Grounded Transformer (VGGT) use a single network to predict camera pose, depth, and point maps. However, their deployment has been constrained by the billion-scale parameter and quadratic complexity. Existing VGGT acceleration methods mainly reduce tokens or sparsify attention entries, while the redundancy and task-dependent roles of complete attention heads remain underexplored. We instead ask whether a frozen visual geometry transformer contains smaller, structurally executable head circuits specialized for different geometric task sets. To investigate this question, we present GeoCircuit, which instruments both frame and global attention with complete-head gates and treats the pretrained model as a frozen parent network of candidate geometric subnetworks. Using dense-model predictions as label-free task teachers, GeoCircuit estimates task-aware gate Gauss-Newton matrices and applies shrinkage to preserve useful cross-head interactions while suppressing estimation noise. Given a target head retention ratio, GeoCircuit directly selects a fixed-budget circuit through a normalized multi-task objective, producing subnets of different sizes without fine-tuning. With 50% of heads retained, GeoCircuit outperforms state-of-the-art method at the corresponding retention setting by 5.93 percentage points on camera pose estimation and improves the point map estimation results by 43.5%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Geometry Grounded Transformer, 3D reconstruction, Circuit extraction, Efficienct AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DDfXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose GeoCircuit, a training-free method, to build task-specific subnetworks of 3D foundation models by removing attention heads from both frame and global attention. It uses the dense model’s own predictions as label-free targets and then a second-order approximation that accounts for interactions between heads estimates how much removing a group of heads would hurt each task. Given a head budget, the method selects which heads to keep for a requested task or set of tasks and packs them into a smaller network without fine-tuning. The paper reports better pose and point-map accuracy with superior efficiency over some baselines.

Critical strengths
  • Generally, I think the paper is well-written and is easy to follow.
  • The proposed GeoCircuit idea is cheap and easy to adapt. From the presented results, it generalizes to several different datasets and backbone networks.
  • The task to address is important. How to more efficiently perform 3D reconstruction could be important to a lot of down-streamed applications.
Critical weaknesses
  • Some baselines are missing. FastVGGT and AVGGT are cited but not compared while Co-Me (CVPR 2026) and LiteVGGT (CVPR 2026) are not cited at all. The closest head-pruning method, Kwon et al. (2022), is also missing. Since head removal is orthogonal to token reduction, both matched-latency comparisons and combined results are needed. More advanced variants (e.g. GoToHunt) are also welcome to compare with.
  • Comparison is not compute-matched. HeSS, the only accuracy baseline, counts kept attention blocks rather than kept heads. So equal “retention” does not mean equal compute and Appendix I concedes this. 
  • The evaluation testbeds are limited. Currently, only Co3Dv2 and DTU are used, omitting standard protocols such as RealEstate10K, ScanNet-50, NRGBD and ETH3D. Depth and the multi-task objective (Eq. 12) are never evaluated. Accuracy is measured only on short inputs, while latency is reported at 100 frames.
  • The experimental gains are limited. At 80-90% retention, GeoCircuit is near parity with HeSS. And on Pi-3, it trails HeSS on DTU at 70-90%. At 50% it still loses about 7 AUC@30 and ~40% in CD relative to dense for only a 1.6-1.8× speedup. The benefit of modeling head interactions over the diagonal-only variant is also inconsistent across retention ratios.
  • The conceptual novelty is somewhat limited. Second-order head-mask pruning, random-projection curvature estimation and Hessian damping are established. And HeSS already uses Hessian-based head sensitivity on VGGT. The “circuit” framing adds little mechanistic analysis beyond pruning heads.
  • Sec. 3.3 describes choosing \alpha on a validation set but the experiments fix \alpha = 0.5 based on Table 1, which uses test results. The same 50% setting also differs between Tables 1 and 2 (86.94 vs. 87.15 AUC@30), and no variance is reported.
Questions for authors

Please see the weakness part.

1 comment
Reviewer_DDfXOpenReview ↗

Official Comment by Reviewer DDfX

Thank you for the detailed response. The RFP comparison and new results on RealEstate10K, ETH3D, and KITTI address some of my concerns. However, I remain concerned about how GeoCircuit compares with other efficient VGGT methods.

  1. Speed and accuracy. At 50% head retention, GeoCircuit achieves a 1.63–1.80× speedup at 100 frames, but loses about 7 AUC@30 points on Co3Dv2 and increases DTU CD by 26–51% relative to dense VGGT. FastVGGT and AVGGT report comparable speedups of approximately 1.7× and 2×, respectively, with little accuracy loss. LiteVGGT and Co-Me report larger speedups of approximately 4.7× at 96 frames and 5.4× at 128 frames. Could you evaluate VGGT, GeoCircuit (80% and 50% head retention), FastVGGT, AVGGT, Co-Me, LiteVGGT, and GoToHunt on the same GPU and input sequences at 100 and 500 frames, reporting latency, speedup, AUC@30, and CD?
  2. Combined methods. FastVGGT+Ours and LiteVGGT+Ours are evaluated only at 10 frames, where token merging offers limited speedup, and their latency is not reported. At 50% head retention, adding GeoCircuit to LiteVGGT reduces AUC@30 by 6.2 points and AUC@5 by 27.1 points, while increasing CD by 26%. What additional speedup does this combination achieve at 100 frames and longer sequences?
  3. FastVGGT configuration. FastVGGT alone achieves 78.50 AUC@30, compared with 94.22 for dense VGGT, although its paper reports near parity at 100 frames. Which merge ratio was used in your evaluation? What explains the increase from 78.50 to 81.57 AUC@30 when 30% of the heads are removed?
Reviewer_riVHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the concept of circuit extraction into feed-forward 3D visual geometry foundation models (specifically VGGT). It proposes GeoCircuit, a training-free and label-free structural pruning framework that physically removes complete attention heads across both frame and global attention layers, yielding compact subnetworks tailored to specific geometric tasks (e.g., camera pose or point-map estimation) while preserving the majority of the original dense model's capabilities.

The core technical innovations of this work are two-fold:

  1. Modeling cross-head Interactions via second-order Hessian/GGN. Using the frozen dense model's predictions as label-free targets, the authors formulate a local quadratic fidelity predictor based on the gate-space Generalized Gauss-Newton (GGN) matrix (which equals the exact loss Hessian at the dense model). By incorporating off-diagonal pairwise interactions alongside shrinkage regularization and greedy/swap search, the method effectively accounts for head redundancy and synergy, substantially improving pruning accuracy over independent head scoring.
  2. Tractable estimation via random projections. To overcome the prohibitive computational cost of explicitly constructing the full Jacobian of high-dimensional geometric outputs with respect to all head gates, the paper introduces an unbiased and engineering-friendly estimator using Rademacher random projections, making second-order interaction estimation fast and scalable on a small calibration set.

Overall, the experimental validation is comprehensive and convincing, demonstrating clear accuracy improvements over state-of-the-art baselines (such as HeSS), strong task-specificity, generalization to the π3\pi^3 architecture, and practical end-to-end wall-clock inference speedups through structural weight packing.

Critical strengths
  1. The derivation of the Jacobian-based quadratic fidelity predictor is mathematically rigorous and sound. The authors cleverly leverage the zero-residual property at the dense model (δ=0\delta=0) to prove that the gate-space Gauss-Newton matrix (Jt⊤Jt)\left(J_{t}^{\top} J_{t}\right) recovers the exact local Hessian , providing a solid theoretical basis for capturing cross-head interactions without computing second-order network derivatives.
  2. Addressing the engineering bottleneck of differentiating high-dimensional 3D geometric outputs (such as dense point maps), the proposed Rademacher random projection estimator is both well-motivated and effective. It makes estimating the full head-interaction matrix computationally tractable while maintaining unbiasedness and positive semidefiniteness.
  3. The empirical evaluation is extensive. It covers accuracy comparisons across multiple retention ratios, cross-task specificity, ablation studies on the shrinkage weight, compatibility with block-sparse attention (SparseVGGT) with actual wall-clock latency measurements, and generalization to another 3D architecture (π3\pi^3).
Critical weaknesses

The paper lacks a thorough empirical study on how the choice, size, and domain distribution of the unlabeled calibration set $ D_{\text{cal}} $ affect circuit extraction.

For a 3D visual geometry foundation model like VGGT, multiple attention heads are designed not only to support multiple geometric tasks, but also to handle diverse visual domains, scene scales, and camera motion patterns (e.g., indoor vs. outdoor scenes, object-centric vs. scene-level captures). However, according to Appendix D, the calibration set DcalD_{\text{cal}} consists of only 30 images organized into just N=5N=5 six-view inputs, all drawn exclusively from 7-Scenes. This setup raises several critical yet unexplored questions:

  • Mask Consistency Across Domains: Would calibrating on outdoor scenes (e.g., ETH3D, Tanks & Temples), object-centric videos, or synthetic datasets yield substantially different head-selection masks compared to calibrating on 7-Scenes? Are certain attention heads specialized for specific scene scales or domains rather than being universally redundant?
  • Cross-Domain Generalization Degradation: When a circuit is extracted using only indoor sequences (7-Scenes), does its performance degrade when evaluated on challenging outdoor or wide-baseline benchmarks?
  • Sample Size Stability: Given that H^t∈R768×768\hat{H}_t \in \mathbb{R}^{768 \times 768}, estimating pairwise interactions from merely N=5N=5 scene samples seems highly susceptible to sampling variance. An ablation study on the number of calibration scenes NN and the variance across different random subsets of calibration images is missing.

In summary, this is a solid and well-rounded paper. The theoretical formulation is sound, and the empirical results effectively support the main claims.

Questions for authors
  1. Referring to the Weakness section, could the authors provide further discussion and supplementary experiments on the choice of calibration images? Since GeoCircuit is training-free, running these experiments should be feasible during the rebuttal period.

  2. In Eq. 11, α=0\alpha = 0 completely ignores cross-head interactions while α=1\alpha = 1 fully considers them. According to the theoretical derivation in Proposition 3.1 (qt(δ)=12δ⊤Htδq_t(\delta) = \frac{1}{2}\delta^\top H_t \delta), α=1\alpha = 1 accounts for head interactions most thoroughly and should theoretically be optimal. Why does α=0.5\alpha = 0.5 empirically outperform α=1\alpha = 1 in Table 1? Could the authors explain the reason behind this discrepancy between theory and practice?

Reviewer_FGUHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes GeoCircuit, a training-free method that extracts task-aware circuits of complete attention heads from a frozen visual geometry transformer such as VGGT. In particular, it estimates a gate-space Gauss-Newton matrix per task from dense-model predictions on unlabeled images. It then shrinks the cross-head interactions and greedily selects heads under a fixed budget. Experiments on Co3Dv2 and DTU report better pose and point-map accuracy than HeSS at 50% retention, with a 1.63× to 1.80× speedup over VGGT.

Critical strengths
  • The idea of treating a frozen VGGT as a parent network of task-specific head circuits is interesting. It can serve as a compression tool and also inspire further analysis of which heads support pose versus geometry.

  • The circuit selection of GeoCircuit is training-free and only requires a few unlabeled images. Since the selected heads are removed, it yields end-to-end speedups and can be combined with block-sparse attention.

Critical weaknesses
  • The main comparison is not conducted at matched compute. The abstract states that GeoCircuit "outperforms state-of-the-art method at the corresponding retention setting by 5.93 percentage points" (L027-028). However, HeSS keeps 50% of the attention blocks inside heads, whereas GeoCircuit keeps 50% of complete heads. App. I also mentions that "These retention ratios use different sparsity units and do not imply matched FLOPs" (L905-906). Yet no latency or FLOPs are reported for HeSS, so I am not convinced by this comparison. This matters since the accuracy of GeoCircuit degrades considerably at 50% retention, e.g., the DTU accuracy error rises from 1.499 to 2.788 mm (Tab. 1). It would be better to plot accuracy against measured latency for all methods, including SparseVGGT alone.

  • Limited novelty over post-training head pruning. Kwon et al. (2022) already remove complete heads without retraining, using Fisher-based mask importance with intra-layer interactions under a latency constraint. I understand that it was developed for language Transformers such as BERT, not for VGGT. However, its mask search only needs a small sample set and a differentiable loss. It seems that it could be adapted to VGGT with a task loss on a few labeled scenes. The fidelity loss itself cannot be used, as its gradient vanishes at the dense model (L281). GeoCircuit mainly extends it with limited modifications, i.e., a label-free curvature estimate, cross-layer interactions and shrinkage. Its diagonal-only setting (α\alpha = 0 in Tab. 1) only partly covers such a baseline, as it lacks their mask rearrangement and mask tuning. I find it difficult to see what new methodological insights GeoCircuit could bring beyond this prior work. It is highly recommended to compare with Kwon et al. adapted to VGGT under the same head budget.

Questions for authors
  1. The dense VGGT accuracy on DTU is 1.499 mm in Tab. 1 but 1.393 mm in Tab. 2. The 50% point-map circuit also reaches a CD of 2.08 mm in Fig. 3 but 2.679 mm in Tab. 2. Which evaluation protocol is used in each case, and which circuit is used for the HeSS comparison in Fig. 2?

  2. Depth is one of the three target tasks, yet no depth benchmark is reported. How well do the depth-conditioned circuits preserve depth accuracy?

  3. Calibration only uses 30 indoor images from 7-Scenes. It would be interesting to see how stable the selected circuits are under a different calibration set, e.g., outdoor scenes.

  4. Sec. 3.3 states that "Validation against the dense model determines the interaction weight" (L300-301). However, α\alpha = 0.5 is fixed (L343) and justified on Co3Dv2 and DTU in Tab. 1. There, "no single value performs best at every retention ratio" (L427). How does GeoCircuit perform when α\alpha is selected on the unlabeled validation set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.