acceptodds
Withdrawn from ICLR 2027

Less Context, Better Geometry: Masked Geometric Encoder for Robust 3D Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in 3D foundation models has enabled rapid 3D reconstruction and camera calibration by leveraging learned 3D priors from vast amount of spatial data. However, the all-to-all global attention design leads to quadratic complexity and limits long-sequence inference; unconstrained cross-view interactions also can propagate unreliable evidence from occluded or visually similar but geometrically distant views. In this paper, We introduce a Masked Geometric Encoder (MGE), which promotes the learning of robust geometric representations under incomplete cross-view context. During training, MGE strategically drops frame tokens from global attention and distills from a pretrained full-context teacher model. This allows the model to learn an intrinsically richer per-frame representation while providing sufficient intermediate supervision to avoid performance degradation. Through extensive experiments, we show that MGE leads to much stronger performance under occlusion and doppelganger views while retaining high performance on standard benchmarks. Such a richer frame representation also leads to more effective token reduction during inference. To this end, we develop a novel Anchor-Guided Adaptive token merging technique that preserves representative anchor frames while jointly merging redundant tokens from the remaining views. Compared to other efficient inference approaches, we can achieve inference speedup while consistently maintaining higher reconstruction quality, particularly in limited-view settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3d reconstruction, masked encoder

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6R3EOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the issues of quadratic computational complexity and error propagation caused by dense all-to-all cross-view attention in feed-forward 3D foundation models. Drawing inspiration from view-graph edge pruning in classical Structure-from-Motion (SfM), the authors propose the Masked Geometric Encoder (MGE). During training, MGE randomly masks frame-to-frame attention connections and distills knowledge from a full-attention teacher, forcing the network to learn robust per-frame geometric representations under incomplete cross-view context.

Building upon the robust representations learned by MGE, the authors naturally introduce Anchor-Guided Adaptive (AGA) Token Merging for efficient inference. AGA preserves full-resolution anchor frames to anchor the global geometry while adaptively compressing redundant non-anchor views based on their feature similarity.

Notably, this framework is model-agnostic and can be readily applied to various encoder-decoder 3D foundation models (e.g., π3\pi^3 and VGGT). Comprehensive experiments across standard benchmarks and stress tests (occlusions and visually ambiguous doppelgangers) validate the effectiveness and robustness of the proposed approach.

In summary, the core claimed contributions are twofold: (1) a model-agnostic MGE training strategy adaptable to various 3D foundation models, and (2) an inference-time AGA strategy that synergizes with MGE to enhance efficiency while preserving geometric accuracy.

Critical strengths
  1. Well-Motivated Problem: Addressing cross-view redundancy and error propagation in dense all-to-all attention by drawing inspiration from classical SfM view-graph pruning is intuitive and well-grounded.

  2. Effective Synergy of MGE and AGA: The combination of training-time MGE and inference-time AGA works synergistically, improving long-sequence efficiency while reliably preserving reconstruction accuracy.

  3. Model-Agnostic Design: The proposed framework is versatile and demonstrated to be effective across different 3D foundation architectures, including both π3\pi^3 and VGGT.

  4. Extensive Empirical Evaluation and clear presentation: The experiments are comprehensive, covering not only standard pose and depth benchmarks, but also targeted stress tests on occlusions and visual ambiguities. And the paper is well-organized, logically structured, and easy to follow.

Critical weaknesses
  1. While mitigating cross-view redundancy is well-motivated, using an unconstrained random Bernoulli mask AA during training may not fully substantiate the claim that the model learns to identify and discard redundant or misleading pairs. Random dropping could inadvertently discard critical covisible pairs or retain uninformative ones. Given that training datasets typically provide explicit ground-truth poses and depths (from which visual overlap and covisibility can be computed), would it be more principled to supervise the network with an explicit, geometry-aware connectivity matrix (or learning a soft view-graph relation) rather than purely relying on random dropouts?

  2. On standard benchmarks without extreme noise or token reduction (e.g., Table 1, Table 2, Table 8, Table 9), the performance gains of standalone MGE over the dense baselines (π3\pi^3 and VGGT) appear relatively marginal. The pronounced advantages mainly emerge when combined with AGA or under synthetic stress tests (occlusions and doppelgangers). This raises a slight concern regarding whether MGE serves primarily as a regularizer tailored for downstream token reduction, rather than an intrinsically superior geometric representation learner in general scenarios.

  3. A core claim of the paper is improving scalability and long-sequence inference efficiency via AGA. However, the manuscript lacks explicit hardware-level benchmarks such as actual inference runtime (FPS/latency), peak GPU memory (VRAM) footprint, and FLOPs comparisons against baselines like FastVGGT. Adding these runtime profiles would provide a more complete and substantiated validation of the efficiency claims.

  4. There are a few minor formatting and typographical issues that could be polished in the revision (e.g., line 199, Inconsistent bolding and underlining in Table 3).

Overall, I find the primary weakness of this manuscript to be the conceptual soundness of the random masking strategy in MGE. The marginal quantitative gains observed on standard benchmarks further amplify this concern.

From my perspective, while the paper presents a solid motivation and fairly extensive experiments suitable for ICLR, the central claims around MGE as a standalone geometric representation learner do not feel entirely solid. The authors might consider reframing the narrative: centering the core claims on efficiency and long-sequence scalability, and presenting the MGE + AGA synergy as an effective solution that substantially boosts the practical efficiency of real-time 3D foundation models while preserving accuracy, would likely make the paper much more compelling and convincing.

Questions for authors
  1. Clarification and Attention Analysis on the MGE Mechanism: As mentioned in the Weaknesses section, my primary reservation remains the conceptual soundness of the random masking strategy in MGE. Since the model is trained under a teacher-student distillation framework, could the authors provide qualitative visualizations or quantitative analyses comparing the cross-frame attention maps of the Student (MGE) versus the Teacher? Specifically, does the Student naturally learn to assign higher attention weights to geometrically informative/covisible image pairs while attenuating attention on misleading ones? Providing such empirical evidence would greatly help demystify how and why the masked student surpasses the dense teacher, substantiating the core claims behind MGE.

  2. Hardware-Level Efficiency Benchmarks for AGA: Could the authors provide concrete runtime benchmarks to substantiate the efficiency claims of the AGA strategy?

3 comments
AuthorsOpenReview ↗

Addressing questions from Reviewer 6R3E

We thank Reviewer 6R3E for positive feedback and for recognizing the effective synergy between MGE and AGA, the generalizability of our framework, and the extensive experimental evaluation, which are also acknowledged by the other reviewers. We address the questions and provide additional evidence below. We hope that they address the reviewer's concerns and would love to hear the reviewer's feedback.

1. MGE mechanism analysis and core claims

Why use random masking? In MGE, our objective is to reduce the model's reliance on specific cross-view connections and thus can work together with AGA for more effective token merging. While geometry-aware connectivity is a reasonable alternative, high covisibility does not necessarily imply high geometric informativeness: highly overlapping views may contain redundant or misleading information, whereas weakly overlapping views can still provide complementary geometric constraints. Moreover, our training sequences contain at most 24 frames with generally high mutual covisibility, meaning that a covisibility-based view graph remain densely connected, approaching all-to-all attention. Importantly, high covisibility alone does not guarantee reliable geometric associations. We illustrate this with a failure case below, where even highly covisible views lead the dense teacher to incorrect geometric reconstruction.

Attention Analysis. We provide additional analysis comparing the π3\pi^3 teacher, a student trained with identical distillation but without masking (π3\pi^3-distilled), and MGE on the Temple of Heaven (ToH) scene. It contains 20 images captured from the same side of a rotationally symmetric building, with pairwise covisibility exceeding 0.95 for all image pairs. As shown in Figure 10 of the revised manuscript, when processing all images jointly, π3\pi^3 incorrectly places frames 13–19 on the opposite side of the building. π3\pi^3-distilled reproduces this failure, whereas MGE recovers a geometrically consistent camera trajectory. For attention behaviours shown in Figure 11, the teacher and unmasked student develop highly concentrated attention on particular view groups, accompanied by incorrect correspondences between repetitive structures (Figure 12). In contrast, MGE maintains more balanced frame-level attention and more geometrically consistent patch-level correspondences. At layer 31, MGE achieves 30/30 correct top-attended patch pairs, compared with only 2/30 for both baselines. Furthermore, if we replace the teacher's late-layer frame-level attention with uniform weights, the maximum camera azimuth drops from 75.8° to 3.8°, supporting the role of concentrated attention in this failure case. Need to mention, if we use uniform weights for other examples, the reconstruction will fail. These findings suggest that random masking regularizes cross-view information aggregation rather than explicitly learning an optimal view graph. Random masking training exposes the model to different subsets of view connections across training iterations and reduces its dependence on specific, potentially misleading view groups, allowing more reliable geometric correspondences to emerge despite supervision from the same teacher.

Modest improvement on standard benchmarks. MGE is distilled to preserve the dense teacher's reconstruction capability, so substantial improvements under clean, full-context inference (what we called standard benchmarks in the manuscipt) are not necessarily expected. Its main advantage emerges when cross-view information becomes unreliable or incomplete, as demonstrated by our occlusion and Doppelganger evaluations. We further validate this through inference-time masking on 7-Scenes: while π3\pi^3 and MGE achieve identical accuracy without masking (0.019), MGE degrades much more gracefully as connectivity is reduced (0.024 vs. 0.051 at ρ=0.5\rho=0.5). Importantly, these improvements are observed without AGA, which demonstrates that MGE's robustness is not specific to downstream token merging. AGA further exploits this property to enable more aggressive token reduction while preserving reconstruction quality for long sequences.

Table 15: Reconstruction accuracy error (↓) under different inference-time masking ratios.

Methodρ=0\rho=0ρ=0.1\rho=0.1ρ=0.3\rho=0.3ρ=0.5\rho=0.5ρ=0.7\rho=0.7
π3\pi^30.0190.0210.0440.0510.091
MGE0.0190.0200.0230.0240.044

We revised the manuscript to clarify that MGE's contribution is robustness and scalability to long sequences.

2. Practical Inference Efficiency

In the original manuscript, we evaluate all token merging methods under the same token budget (around 30% of total tokens), reducing attention computation costs significantly. To further substantiate our efficiency claims, we have added a new table reporting end-to-end inference time and peak GPU memory across different sequence lengths.

Our method consistently achieves lower latency and memory consumption than the feed-forward baselines, particularly for long sequences. At the densest sampling (stride=3), our method reduces inference time from 95.6s to 61.2s and peak memory from 31.7 GB to 22.3 GB compared to FastVGGT.

*Table: Inference efficiency under different keyframe strides. We report inference time (s) and peak GPU memory (GB) on single RTX 5000. *

Methodstride=200stride=100stride=40stride=25stride=10stride=3
Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)
TTT3R1.16.41.96.44.46.46.96.417.06.456.16.4
StreamVGGT1.08.22.310.47.212.614.421.5OOMOOMOOMOOM
FastVGGT0.78.41.38.72.89.24.89.915.414.795.631.7
VGGT+AGA0.78.41.48.72.89.24.810.215.214.094.929.4
Ours0.45.80.86.02.06.83.57.511.310.661.222.3

3. Formatting in the manuscipt

Thanks for pointing out. We have revised the manuscript accordingly.

Dear Reviewer 6R3E,

We kindly invite you to review our rebuttal since the discussion phase is ending soon. We have carefully addressed all your concerns, including:

  1. MGE mechanism analysis and clarification of our claims
  2. inference efficiency comparison
  3. formatting issues

If you have any further questions or would like additional clarifications, we would be happy to continue the discussion.

Reviewer_vey2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a Masked Geometric Encoder (MGE), which randomly masks cross-view attention connections during training while distilling geometric predictions and features from a full-context teacher. At inference, Anchor-Guided Adaptive (AGA) token merging preserves full-resolution anchor frames and allocates merging ratios to other frames according to their feature similarity to the anchors. The method improves robustness to occlusion and visual ambiguity while largely maintaining performance on standard benchmarks.

Critical strengths

1. A well-motivated approach to regularizing cross-view dependencies. Restricting cross-view connections encourages robustness to incomplete context. The approach integrates into existing architectures and is evaluated on both π3 and VGGT.

2. Substantial improvements under challenging inputs. Under large occlusion on ETH3D, MGE reduces reconstruction accuracy error from 0.278 to 0.171 and relative rotation error from 7.969 to 2.043 compared with π3, demonstrating meaningful robustness gains (Table 2).

3. Relevant ablations of training and compression strategies. The paper examines masking ratios, masking patterns, supervision strategies, and token-merging designs. Comparisons at the same overall token retention ratio help assess the contributions of the merging components (§4.7, Appendix A.8).

Critical weaknesses

1. Unclear Robustness Mechanism of Teacher-Student Distillation MGE aims to overcome incorrect geometric associations in full-context models, yet relies on their predictions and features for distillation. It remains unclear how the student corrects rather than inherits teacher errors. The authors should analyze teacher failure cases with and without feature distillation. Additionally, the inferior performance of ground-truth supervision deserves further investigation. 2. Insufficient Evidence for Practical Inference Efficiency The experiments mainly compare reconstruction quality at similar token retention ratios, without reporting actual inference costs. AGA introduces additional overhead from similarity computation, token matching, and recovery. The authors should provide end-to-end latency, peak GPU memory, and quality–efficiency comparisons across different sequence lengths. 3. Potential Sensitivity of Anchor Selection to Input Ordering AGA uniformly selects anchor frames, which may not ensure sufficient geometric coverage or viewpoint diversity. Different input permutations could lead to inconsistent anchor selections and reconstruction results. The authors should evaluate permutation sensitivity and compare against diversity-based anchor selection strategies. 4. Limited Analysis of Mask Ratio and Generalization The mask ratio ablation is limited to two sampling configurations on NRGBD. Additional evaluations on real-world datasets, such as On-the-Fly NeRF, would strengthen the conclusions. The authors should also compare π³ and MGE under different inference-time masking ratios to clarify the benefits of masked training. 5. Limited Justification for Random Masking over Sliding-window Attention Random masking shows relatively modest improvements over sliding-window attention in the reported experiments. The authors should provide broader evaluations and further explain why random connectivity is more effective for learning robust geometric representations.

Questions for authors

All questions are detailed in the weaknesses above.

6 comments
AuthorsOpenReview ↗

Addressing questions from Reviewer vey2

We thank Reviewer vey2 for his/her constructive comments and for recognizing our well-motivated approach, promising robustness improvements, and comprehensive ablation studies, which are also appreciated by the other reviewers. The reviewer raised a series of questions, which we will provide answers and clarifications below. We hope that they address the reviewer's concerns and would love to hear the reviewer's feedback.

1. Robustness Mechanism

To isolate the effect of masked training, we introduce a baseline termed π3\pi^3-distilled, which follows the same teacher-student distillation framework as MGE, including identical supervision and training settings, but without cross-view attention masking. We then analyze a teacher failure case on the Temple of Heaven (ToH), containing 20 images captured from the same side of a rotationally symmetric building. As shown in Figure 10 of the revised manuscript, when processing all images jointly, π3\pi^3 incorrectly places frames 13–19 on the opposite side of the building. π3\pi^3-distilled reproduces this failure, whereas MGE recovers a geometrically consistent camera trajectory.

In this example, the teacher's failure is amplified by additional views. When processing image pairs independently, π3\pi^3 correctly estimates 75% of relative poses, but this drops to 52% when all 20 images are jointly processed. To understand this behavior, we further analyze the evolution of frame-level attention distributions across layers (Figure 11) and visualize the top-30 attended query-key token pairs to assess their geometric correspondency (Figure 12) in the revised manuscript. In later layers (27–33), both π3\pi^3 and π3\pi^3-distilled develop highly concentrated frame-level attention patterns (see Figure 11), accompanied by incorrect patch-level associations between repetitive structures (see Figure 12). In contrast, MGE maintains more balanced cross-view attention and substantially improves correspondence consistency. At layer 31, all 30 top-attended patch pairs are correct for MGE, compared with only 2/30 for both baselines.

To further validate the role of attention concentration through an inference-time intervention: replacing π3\pi^3's late-layer frame-level attention with uniform weights reduces the maximum camera azimuth jump from 75.8° to 3.8° (π3\pi^3-intervened in Figure 10), which provides direct evidence that the concentrated attention contributes to the misregistration.

These observations help explain why MGE can mitigate teacher errors despite distillation. Without masking, the distilled student inherits its failure. In contrast, random cross-view masking exposes MGE to varying connectivity patterns during training, reducing its reliance on fixed groups of mutually reinforcing but potentially misleading views. Consequently, MGE can learn more reliable cross-view associations rather than simply reproducing the teacher's erroneous full-context predictions.

As for the inferior performance of ground-truth supervision, it can be explained by two factors. First, incomplete GT annotations. As illustrated in Figure 5 of our appendix, MegaDepth ground-truth depth maps contain substantial missing and invalid regions, whereas the pretrained teacher produces spatially dense predictions that preserve the principal scene structure, object boundaries, and relative depth ordering. Such dense pseudo-labels provide more comprehensive geometric supervision, including regions where GT annotations are unavailable. Second, training convergence and computational cost. Learning accurate geometric predictions directly from GT requires substantial optimization. For reference, the original π3\pi^3 model underwent two main training stages, each consisting of 80 epochs with 800 iterations per epoch, using 16 and 64 A100 GPUs, respectively, followed by separate confidence-head training. Distillation leverages the geometric knowledge already acquired by a pretrained teacher, enabling faster convergence under a limited training budget, as shown in Figure 4.

2. Practical Inference Efficiency

We have added a new table reporting end-to-end inference time and peak GPU memory across different sequence lengths.

Table: Inference efficiency under different keyframe strides. We report inference time (s) and peak GPU memory (GB) on single RTX 5000.

Methodstride=200stride=100stride=40stride=25stride=10stride=3
Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)
TTT3R1.16.41.96.44.46.46.96.417.06.456.16.4
StreamVGGT1.08.22.310.47.212.614.421.5OOMOOMOOMOOM
FastVGGT0.78.41.38.72.89.24.89.915.414.795.631.7
VGGT+AGA0.78.41.48.72.89.24.810.215.214.094.929.4
Ours0.45.80.86.02.06.83.57.511.310.661.222.3

Our method consistently achieves lower latency and memory consumption than the feed-forward baselines, particularly for long sequences. At the densest sampling (stride=3), our method reduces inference time from 95.6s to 61.2s and peak memory from 31.7 GB to 22.3 GB compared to FastVGGT.

To isolate AGA's computational overhead, we additionally compare VGGT+AGA with FastVGGT under identical token budgets, ensuring equal attention FLOPs. The overall computation consists of frame-level similarity estimation, token merging, and attention computation. First, AGA introduces a lightweight frame-level similarity computation to determine adaptive merging ratios using pooled frame descriptors. This is performed only once per forward pass and takes merely 0.33 ms for 100-frame sequences. Second, both methods perform token-level merging by matching source tokens to destination tokens. AGA uses fewer destinations (25,700 vs. 26,484 at 100 frames) and avoids duplicating protected tokens, reducing memory usage and making merge construction slightly faster (70.3 vs. 70.9 ms per layer), despite the additional frame-level overhead. Third, attention computation has identical FLOPs because both methods retain the same number of tokens. Consequently, at stride=3 on NRGBD, VGGT+AGA achieves slightly lower end-to-end latency (94.9 vs. 95.6 s) and peak memory (29.4 vs. 31.7 GB) than FastVGGT.

Together with our reconstruction quality comparisons (Figure 3 and Table 8) in the manuscript, these results demonstrate the favorable quality–efficiency trade-off of our MGE+AGA while confirming that AGA introduces negligible additional overhead.

3. Sensitivity of Anchor Selection to Input Ordering

"The authors should evaluate permutation sensitivity and compare against diversity-based anchor selection strategies."

We conduct additional experiments on ETH3D to evaluate input-order sensitivity by randomly permuting the input images with 10 different seeds and reporting the mean and standard deviation.

Table 13: Sensitivity to input image ordering on ETH3D. We report reconstruction accuracy (Acc), completeness (Comp), and normal consistency (NC) as mean ± standard deviation over 10 random input permutations.

MethodAcc ↓Comp ↓NC ↑
VGGT+AGA0.3938 ± 0.04790.4631 ± 0.06570.7979 ± 0.0128
π3\pi^3+AGA0.2513 ± 0.01530.3145 ± 0.04730.8294 ± 0.0105
MGE+AGA0.2307 ± 0.01010.2516 ± 0.00940.8414 ± 0.0038

Compared with VGGT, π3\pi^3 exhibits lower sensitivity to input ordering, consistent with its reference-free design and pairwise geometric supervision. MGE further reduces the variance across permutations, which can be attributed to masked cross-view training mitigating reliance on specific frames and thus improving robustness to changes in input ordering.

We also compare uniform anchor selection with diversity-based selection on ETH3D under the same anchor budget, denoted as MGE+Replace with diverse frames in Table 5. We follow the work Diversity-aware VGGT[1] to select the anchor sets.

However, diversity-based selection yields slightly worse reconstruction quality compared to uniform anchors. That's because visually distinct frames have limited geometric overlap with other views, making them less effective as anchors for cross-view reconstruction and redundancy estimation. In contrast, uniform sampling provides a simple and stable strategy without additional selection overhead. Moreover, our long-sequence benchmarks primarily consist of temporally ordered videos, where uniformly spaced anchors naturally provide coverage across the sequence.

Table 5: Ablation of merging designs on ETH3D. All variants retain the same overall token ratio for a fair comparison.

MethodAcc ↓Comp ↓NC ↑
MGE+FastVGGT-style merging0.3080.3110.815
MGE+Replace with anchor frames0.2300.2560.838
MGE+Replace with diverse frames0.2410.3120.836
VGGT+AGA0.3050.3110.821
MGE+AGA0.2160.2430.842

We have included these experiments and analysis to the revised manuscript.

[1] Park, J., Choi, D., Seo, A., Cho, M., & Son, J. (2026, September). Diversity-Aware View Partitioning for Scalable VGGT. In European Conference on Computer Vision (pp. 547-564). Cham: Springer Nature Switzerland.

Reviewer_gfMJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to improve multi-view 3D reconstruction quality while reducing attention computation at the same time. More specifically, during training, in the "inter-frame" attention layers, a random attention drop strategy is applied and distill from the teacher with full context, and then during inference, a heuristic frame-selection-and-token-merging strategy (uniformly sample anchor frames first then decide the token merging ratio of other frames based on dinov2 similarity) is applied. The experiments show that the model has better performance than the baseline VGGT and Pi-3 model.

Critical strengths
  • The main idea of this is easy to follow -- a method to reduce the attention computation with "somewhat a sparsification" method.
  • Regarding the experimental result, the shown robustness seems promising. Compared to the base model, MGE shows better robustness in the designed experiments with visual ambiguity and occulusion.
  • The token merging ablation study is also helpful. Compared to FastVGGT-style merging, preserving anchor frames shows better performance, which is further improved by adaptive token allocation.
  • The main result is based on pi-3 while the same method is applied to VGGT too showing the idea is not limited to just one model.
Critical weaknesses
  • Table 4 compares masking ratios of 0.3, 0.5 and 0.7, but lacks a no-masking baseline trained under the same settings. This comparison is particularly important to validate the effectiveness of masked training itself.
  • The paper claims a improved efficiency, but does not report the inference time (and peak GPU memory). Figure 3 and Table 7 only report reconstruction quality across input lengths.

The two weaknesses above already influence the claimed contribution regarding the validity of the method. For the rebuttal: I would appreciate (1) a controlled no-masking baseline and (2) latency and memory measurements against the base model.

Overall, the results are encouraging, but the two points I present above affect the main claims. I would consider changing my rating to weak accept if the rebuttal presents a clear benefit from masking (beyond post-training) and a competitive efficiency–quality trade-off.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Addressing questions from Reviewer gfMJ

We thank Reviewer gfMJ for the thoughtful feedback and for recognizing the effectiveness of our robustness improvements, token merging design, and generalizability across different 3D foundation models, which are also acknowledged by the other reviewers. The reviewer raised some questions, which we will provide answers and clarifications below. We hope that they address the reviewer's concerns and would love to hear the reviewer's feedback.

1. No-masking baseline

We thank the reviewer for this suggestion. We have added a no-masking baseline (ρ=0\rho=0) trained under identical settings in the updated Table 4.

Table 4: Effect of mask ratio (ρ\rho). We compare different masking ratios under two keyframe sampling strides.

Settingstride=100stride=10
CD ↓NC ↑CD ↓NC ↑
ρ=0\rho=00.01750.90460.01360.7588
ρ=0.3\rho=0.30.01500.91840.01340.7607
ρ=0.5\rho=0.50.01590.91040.01260.7660
ρ=0.7\rho=0.70.01820.89830.01560.7440

The results show that masked training consistently improves reconstruction quality with token merging under both sampling strides. In particular, at stride=100, ρ=0.3\rho=0.3 reduces CD from 0.0175 to 0.0150 and improves NC from 0.9046 to 0.9184. At stride=10, ρ=0.5\rho=0.5 achieves the best performance. These results demonstrate that masked training learns representations that are more robust to token reduction, enabling more effective token merging while preserving reconstruction quality.

2. Inference time and memory usage

In the original manuscript, we evaluate all token merging methods under the same token budget (around 30% of total tokens), reducing attention computation costs significantly. To further substantiate our efficiency claims, we have added a new table reporting end-to-end inference time and peak GPU memory across different sequence lengths.

Table: Inference efficiency under different keyframe strides. We report inference time (s) and peak GPU memory (GB) on single RTX 5000.

Methodstride=200stride=100stride=40stride=25stride=10stride=3
Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)Time (s)Mem (GB)
TTT3R1.16.41.96.44.46.46.96.417.06.456.16.4
StreamVGGT1.08.22.310.47.212.614.421.5OOMOOMOOMOOM
FastVGGT0.78.41.38.72.89.24.89.915.414.795.631.7
VGGT+AGA0.78.41.48.72.89.24.810.215.214.094.929.4
Ours0.45.80.86.02.06.83.57.511.310.661.222.3

Our method consistently achieves lower latency and memory consumption than the feed-forward baselines, particularly for long sequences. At the densest sampling (stride=3), our method reduces inference time from 95.6s to 61.2s and peak memory from 31.7 GB to 22.3 GB compared to FastVGGT.

Dear Reviewer gfMJ,

We kindly invite you to review our rebuttal since the discussion phase is ending soon. We have carefully addressed all your concerns, including no-masking baseline and inference efficiency comparison. If you have any further questions or would like additional clarifications, we would be happy to continue the discussion.

Reviewer_gfMJOpenReview ↗

Comment

Thank you authors for the additional experiments. These addressed my concerns.

Thank you for your positive feedback. We are glad that our additional experiments have addressed your concerns.

As you previously mentioned that you would consider revising your score if these concerns were resolved, we would greatly appreciate it if you could reconsider your rating based on our responses and additional results.

Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.