- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the issues of quadratic computational complexity and error propagation caused by dense all-to-all cross-view attention in feed-forward 3D foundation models. Drawing inspiration from view-graph edge pruning in classical Structure-from-Motion (SfM), the authors propose the Masked Geometric Encoder (MGE). During training, MGE randomly masks frame-to-frame attention connections and distills knowledge from a full-attention teacher, forcing the network to learn robust per-frame geometric representations under incomplete cross-view context.
Building upon the robust representations learned by MGE, the authors naturally introduce Anchor-Guided Adaptive (AGA) Token Merging for efficient inference. AGA preserves full-resolution anchor frames to anchor the global geometry while adaptively compressing redundant non-anchor views based on their feature similarity.
Notably, this framework is model-agnostic and can be readily applied to various encoder-decoder 3D foundation models (e.g., and VGGT). Comprehensive experiments across standard benchmarks and stress tests (occlusions and visually ambiguous doppelgangers) validate the effectiveness and robustness of the proposed approach.
In summary, the core claimed contributions are twofold: (1) a model-agnostic MGE training strategy adaptable to various 3D foundation models, and (2) an inference-time AGA strategy that synergizes with MGE to enhance efficiency while preserving geometric accuracy.
Critical strengths
-
Well-Motivated Problem: Addressing cross-view redundancy and error propagation in dense all-to-all attention by drawing inspiration from classical SfM view-graph pruning is intuitive and well-grounded.
-
Effective Synergy of MGE and AGA: The combination of training-time MGE and inference-time AGA works synergistically, improving long-sequence efficiency while reliably preserving reconstruction accuracy.
-
Model-Agnostic Design: The proposed framework is versatile and demonstrated to be effective across different 3D foundation architectures, including both and VGGT.
-
Extensive Empirical Evaluation and clear presentation: The experiments are comprehensive, covering not only standard pose and depth benchmarks, but also targeted stress tests on occlusions and visual ambiguities. And the paper is well-organized, logically structured, and easy to follow.
Critical weaknesses
-
While mitigating cross-view redundancy is well-motivated, using an unconstrained random Bernoulli mask during training may not fully substantiate the claim that the model learns to identify and discard redundant or misleading pairs. Random dropping could inadvertently discard critical covisible pairs or retain uninformative ones. Given that training datasets typically provide explicit ground-truth poses and depths (from which visual overlap and covisibility can be computed), would it be more principled to supervise the network with an explicit, geometry-aware connectivity matrix (or learning a soft view-graph relation) rather than purely relying on random dropouts?
-
On standard benchmarks without extreme noise or token reduction (e.g., Table 1, Table 2, Table 8, Table 9), the performance gains of standalone MGE over the dense baselines ( and VGGT) appear relatively marginal. The pronounced advantages mainly emerge when combined with AGA or under synthetic stress tests (occlusions and doppelgangers). This raises a slight concern regarding whether MGE serves primarily as a regularizer tailored for downstream token reduction, rather than an intrinsically superior geometric representation learner in general scenarios.
-
A core claim of the paper is improving scalability and long-sequence inference efficiency via AGA. However, the manuscript lacks explicit hardware-level benchmarks such as actual inference runtime (FPS/latency), peak GPU memory (VRAM) footprint, and FLOPs comparisons against baselines like FastVGGT. Adding these runtime profiles would provide a more complete and substantiated validation of the efficiency claims.
-
There are a few minor formatting and typographical issues that could be polished in the revision (e.g., line 199, Inconsistent bolding and underlining in Table 3).
Overall, I find the primary weakness of this manuscript to be the conceptual soundness of the random masking strategy in MGE. The marginal quantitative gains observed on standard benchmarks further amplify this concern.
From my perspective, while the paper presents a solid motivation and fairly extensive experiments suitable for ICLR, the central claims around MGE as a standalone geometric representation learner do not feel entirely solid. The authors might consider reframing the narrative: centering the core claims on efficiency and long-sequence scalability, and presenting the MGE + AGA synergy as an effective solution that substantially boosts the practical efficiency of real-time 3D foundation models while preserving accuracy, would likely make the paper much more compelling and convincing.
Questions for authors
-
Clarification and Attention Analysis on the MGE Mechanism: As mentioned in the Weaknesses section, my primary reservation remains the conceptual soundness of the random masking strategy in MGE. Since the model is trained under a teacher-student distillation framework, could the authors provide qualitative visualizations or quantitative analyses comparing the cross-frame attention maps of the Student (MGE) versus the Teacher? Specifically, does the Student naturally learn to assign higher attention weights to geometrically informative/covisible image pairs while attenuating attention on misleading ones? Providing such empirical evidence would greatly help demystify how and why the masked student surpasses the dense teacher, substantiating the core claims behind MGE.
-
Hardware-Level Efficiency Benchmarks for AGA: Could the authors provide concrete runtime benchmarks to substantiate the efficiency claims of the AGA strategy?
3 comments
2. Practical Inference Efficiency
In the original manuscript, we evaluate all token merging methods under the same token budget (around 30% of total tokens), reducing attention computation costs significantly. To further substantiate our efficiency claims, we have added a new table reporting end-to-end inference time and peak GPU memory across different sequence lengths.
Our method consistently achieves lower latency and memory consumption than the feed-forward baselines, particularly for long sequences. At the densest sampling (stride=3), our method reduces inference time from 95.6s to 61.2s and peak memory from 31.7 GB to 22.3 GB compared to FastVGGT.
*Table: Inference efficiency under different keyframe strides. We report inference time (s) and peak GPU memory (GB) on single RTX 5000. *
| Method | stride=200 | stride=100 | stride=40 | stride=25 | stride=10 | stride=3 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Time (s) | Mem (GB) | Time (s) | Mem (GB) | Time (s) | Mem (GB) | Time (s) | Mem (GB) | Time (s) | Mem (GB) | Time (s) | Mem (GB) | |
| TTT3R | 1.1 | 6.4 | 1.9 | 6.4 | 4.4 | 6.4 | 6.9 | 6.4 | 17.0 | 6.4 | 56.1 | 6.4 |
| StreamVGGT | 1.0 | 8.2 | 2.3 | 10.4 | 7.2 | 12.6 | 14.4 | 21.5 | OOM | OOM | OOM | OOM |
| FastVGGT | 0.7 | 8.4 | 1.3 | 8.7 | 2.8 | 9.2 | 4.8 | 9.9 | 15.4 | 14.7 | 95.6 | 31.7 |
| VGGT+AGA | 0.7 | 8.4 | 1.4 | 8.7 | 2.8 | 9.2 | 4.8 | 10.2 | 15.2 | 14.0 | 94.9 | 29.4 |
| Ours | 0.4 | 5.8 | 0.8 | 6.0 | 2.0 | 6.8 | 3.5 | 7.5 | 11.3 | 10.6 | 61.2 | 22.3 |
3. Formatting in the manuscipt
Thanks for pointing out. We have revised the manuscript accordingly.
Dear Reviewer 6R3E,
We kindly invite you to review our rebuttal since the discussion phase is ending soon. We have carefully addressed all your concerns, including:
- MGE mechanism analysis and clarification of our claims
- inference efficiency comparison
- formatting issues
If you have any further questions or would like additional clarifications, we would be happy to continue the discussion.
Addressing questions from Reviewer 6R3E
We thank Reviewer 6R3E for positive feedback and for recognizing the effective synergy between MGE and AGA, the generalizability of our framework, and the extensive experimental evaluation, which are also acknowledged by the other reviewers. We address the questions and provide additional evidence below. We hope that they address the reviewer's concerns and would love to hear the reviewer's feedback.
1. MGE mechanism analysis and core claims
Why use random masking? In MGE, our objective is to reduce the model's reliance on specific cross-view connections and thus can work together with AGA for more effective token merging. While geometry-aware connectivity is a reasonable alternative, high covisibility does not necessarily imply high geometric informativeness: highly overlapping views may contain redundant or misleading information, whereas weakly overlapping views can still provide complementary geometric constraints. Moreover, our training sequences contain at most 24 frames with generally high mutual covisibility, meaning that a covisibility-based view graph remain densely connected, approaching all-to-all attention. Importantly, high covisibility alone does not guarantee reliable geometric associations. We illustrate this with a failure case below, where even highly covisible views lead the dense teacher to incorrect geometric reconstruction.
Attention Analysis. We provide additional analysis comparing the π3 teacher, a student trained with identical distillation but without masking (π3-distilled), and MGE on the Temple of Heaven (ToH) scene. It contains 20 images captured from the same side of a rotationally symmetric building, with pairwise covisibility exceeding 0.95 for all image pairs. As shown in Figure 10 of the revised manuscript, when processing all images jointly, π3 incorrectly places frames 13–19 on the opposite side of the building. π3-distilled reproduces this failure, whereas MGE recovers a geometrically consistent camera trajectory. For attention behaviours shown in Figure 11, the teacher and unmasked student develop highly concentrated attention on particular view groups, accompanied by incorrect correspondences between repetitive structures (Figure 12). In contrast, MGE maintains more balanced frame-level attention and more geometrically consistent patch-level correspondences. At layer 31, MGE achieves 30/30 correct top-attended patch pairs, compared with only 2/30 for both baselines. Furthermore, if we replace the teacher's late-layer frame-level attention with uniform weights, the maximum camera azimuth drops from 75.8° to 3.8°, supporting the role of concentrated attention in this failure case. Need to mention, if we use uniform weights for other examples, the reconstruction will fail. These findings suggest that random masking regularizes cross-view information aggregation rather than explicitly learning an optimal view graph. Random masking training exposes the model to different subsets of view connections across training iterations and reduces its dependence on specific, potentially misleading view groups, allowing more reliable geometric correspondences to emerge despite supervision from the same teacher.
Modest improvement on standard benchmarks. MGE is distilled to preserve the dense teacher's reconstruction capability, so substantial improvements under clean, full-context inference (what we called standard benchmarks in the manuscipt) are not necessarily expected. Its main advantage emerges when cross-view information becomes unreliable or incomplete, as demonstrated by our occlusion and Doppelganger evaluations. We further validate this through inference-time masking on 7-Scenes: while π3 and MGE achieve identical accuracy without masking (0.019), MGE degrades much more gracefully as connectivity is reduced (0.024 vs. 0.051 at ρ=0.5). Importantly, these improvements are observed without AGA, which demonstrates that MGE's robustness is not specific to downstream token merging. AGA further exploits this property to enable more aggressive token reduction while preserving reconstruction quality for long sequences.
Table 15: Reconstruction accuracy error (↓) under different inference-time masking ratios.
We revised the manuscript to clarify that MGE's contribution is robustness and scalability to long sequences.