acceptodds
Withdrawn from ICLR 2027

MosaicAttention: Dual-Pooled Sparse Attention for Accelerating Multi-View 3D Geometry Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward multi-view geometry Transformers recover cameras and scene geometry from images in a single forward pass by aggregating cross-view evidence through global attention. However, the global attention scales quadratically with the number of input views, making long-sequence inference extremely expensive. Block-sparse attention here provides an intuitive way to accelerate global attention. Existing methods typically treat visual tokens as a flattened 1D sequence and represent each block through only average statistics with uniform sparsity across all layers, which overlooks the 2D spatial layout, salient variations within each block, and layer-wise varying sensitivity to block removal, leading to unreliable block-importance estimation and potential performance loss. These limitations motivate us to introduce MosaicAttention, a sparse attention framework that jointly exploits both 2D average- and maximum-pooled views for more reliable block routing. Our theoretical analysis shows that Avg preserves mean regional logits while Max captures additional within-region variation, motivating dual pooling over 2D visual regions as coarse-grained proxies of token interactions. We then calibrate these dual-pooled proxies against dense value-weighted output contributions to obtain reliable block priorities, and allocate computation according to both block importance and layer-wise sparsification sensitivity. We further bound regional logit approximation and sparse-attention output errors and characterize when calibration preserves block selection. Experiments on multiple feed-forward geometry Transformers and 3D prediction tasks show that MosaicAttention achieves a favorable accuracy-efficiency trade-off over previous methods. Especially, on inputs with 128 views, it achieves up to 2.89 end-to-end inference speedup while maintaining competitive performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Computer vision, Efficient AI, Sparse attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GD7KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MosaicAttention, a block-sparse attention framework for accelerating feed-forward multi-view 3D geometry Transformers. The method constructs lightweight attention proxies using spatially structured average and maximum pooling, and employs an offline ridge-regression calibrator to predict block priorities based on dense attention output contributions. It further introduces sensitivity-aware layer-wise budget allocation and mixed-precision sparse attention execution.

Experiments on VGGT, π³, and Depth Anything 3 demonstrate promising accuracy-efficiency trade-offs across camera pose estimation and dense geometry reconstruction. The method achieves up to 2.89× end-to-end speedup on 128-view inputs.

Critical strengths
  1. Compared with prior block-sparse attention methods relying on average-pooled Q/K features, the proposed max pooling captures salient information that averaging may miss. In my experience, this is beneficial for 3D geometry tasks.
  2. The contribution-based calibration strategy is an interesting approach to aligning lightweight block importance estimates with actual attention output contributions.
Critical weaknesses

The calibration target considers value-weighted attention contributions, yet the calibrator only uses Q/K-derived features without incorporating V, which seems inconsistent with the motivation of estimating actual block contributions.

Questions for authors

Same as Weakness 1. Since V is already computed during inference, shouldn't incorporating V-related features improve block importance estimation? Why do the authors exclude V, especially when doing so would introduce only negligible computational overhead?

1 comment
AuthorsOpenReview ↗

Rebuttal for Weakness 1 and Question1:

W1 / Q1. Why Use Value-Aware Calibration with Q/K-Only Features?

We thank the reviewer for this insightful suggestion. We clarify the distinction between value-aware supervision and online feature extraction, and provide a controlled V-feature ablation.

(1) Value-aware supervision versus Q/K-only prediction.

Our calibration target is value-aware, but the online predictor need not explicitly access V. Specifically, the dense value-weighted output contribution (Definition 3.4) provides the offline supervision target, while the lightweight Q/K-derived dual-pooled features predict its relative block importance without computing dense attention. This is an approximation rather than an assumption that Q/K fully determines the contribution.

(2) Does incorporating V improve block importance estimation?

Following the reviewer's suggestion, we independently fit three calibrators using the same dense teacher and calibration split, adding either block-wise V RMS or the norm of mean V to the original Q/K features.

Table R1. Value-feature ablation. VGGT, 50% sparsity, held-out validation.

Calibrator featuresOutput Rel. ℓ2{\ell}_2 ↓Retained utility ↑AUC@30 ↑Latency (s) ↓
Q/K only0.076380.940288.570.520
Q/K + V RMS0.076340.940688.560.529
Q/K + mean-V norm0.076420.940288.570.521

Adding V RMS slightly improves output approximation and retained utility, while the mean-V-norm variant provides no improvement in output distortion. These results suggest that the tested V statistics provide only marginal additional information beyond our Q/K-derived features.

Reviewer_eq6COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a way to speed up feed-forward 3D models (VGGT, π³, Depth Anything 3) with block-sparse attention: in global-attention layers, only the important query–key blocks are computed, and the rest are skipped. To choose blocks, tokens in each frame are grouped into 2D regions, then they are summarized by average and max pooling. A small linear scorer, fitted offline on calibrated data, turns features from these maps into block importance scores. Each layer gets its own budget based on how sensitive it is to sparsity. The kept blocks run in 8-bit or 4-bit. On CO3Dv2 pose and on 7 Scenes, NRGBD and ETH3D geometry, the method is more accurate than SparseVGGT at the same sparsity. It gives up to 2.89× end-to-end speedup at 128 frames.

Critical strengths
  1. Works on three different model families with real speedups. The method is applied to VGGT, π³ and DA3 without retraining, and the speedup gains are better than other methods.
  2. Better accuracy than SparseVGGT at the same sparsity At matched sparsity and 8-bit precision, it beats SparseVGGT on pose for almost every model and input length and on most geometry metrics (Tab. 2). The ablations show the effect of each part: 2D regions, avg+max fusion, calibration, layer budgets and precision.
Critical weaknesses
  1. The speed comparisons are not clean.

    • The paper does not explain why it is faster than SparseVGGT at the same sparsity and the same 8-bit precision. The same sparsity should mean similar attention compute.
    • The method runs in 8-bit (4/8-bit in Fig. 5), but FastVGGT runs in FP16 and SAF3R in BF16 (l363–366), so part of the gain is just quantization. There is no dense FP16 baseline to isolate the effect of sparsity.
    • The low-precision execution gives very little speedup. In Table 5, FP8 is only 5% faster than FP16 (0.505 vs. 0.533 s), and mixed FP4/FP8 only 3% faster than FP8 (0.491 s).
    • On the geometry benchmarks (Tab. 2), which are not used for calibration, the method clearly beats only SparseVGGT. Against SAF3R and FastVGGT it does not outperform.
  2. Incremental novelty. Block-sparse attention already exists (SparseVGGT, SpargeAttention), and using channel-wise key maxima to estimate block importance is already proposed in Quest. The new parts are 2D regions, avg+max fusion and a 69-feature linear scorer. The gain from the scorer is moderate (87.79 → 88.66 AUC in Tab. 3). The ablations also do not support the 2D-region design: in Table 3, 1D naive fusion mostly beats 2D naive fusion, and there is no "1D + calibration" variant to show that 2D regions matter in the final method.

  3. Key evaluations are missing.

    • The geometry benchmarks are heavily subsampled (App. B, l901–905: every 20th frame on 7 Scenes, every 40th on NRGBD, 14–76 views on ETH3D), and the number of input views per scene is not reported for 7 Scenes and NRGBD. All ablations use VGGT with 10 frames, where there is almost no speedup.
    • The scorer and layer budgets are fitted on a CO3Dv2 training subset, and pose is evaluated on CO3Dv2, which is the same dataset. All pose ablations (Tab. 3–5) are on CO3Dv2 too. There is no cross-dataset pose evaluation, e.g., on RealEstate10K, which VGGT uses, or on 7 Scenes / ETH3D, which already have GT poses and are used in Tab. 2. So the pose gains, which are the main results, may partly reflect in-domain calibration. It is also unclear how well this transfers to real in-the-wild videos (outdoor, dynamic).
Questions for authors
  1. Why is MosaicAttention faster than SparseVGGT at the same sparsity and precision (up to ~40% at 100 frames), but slower at 10 frames?

  2. Could authors report accuracy (pose and geometry) at 128 frames and above, ideally several hundred frames, against dense and all baselines?

  3. How should the calibration be done for a new setting? Does one scorer work for all input lengths and sparsity levels? How many calibration videos are needed, how long does calibration take?

5 comments
AuthorsOpenReview ↗

Rebuttal for Weakness 1: Fairness of Efficiency Comparisons

We thank the reviewer for raising these important concerns. We provide additional controlled profiling and FP16-only evaluations to clarify the sources of acceleration.

(1) Why can MosaicAttention and SparseVGGT differ in latency at the same sparsity?

Equal nominal sparsity does not necessarily imply equal end-to-end latency. Beyond sparse attention computation, the two implementations differ in pooling, mask construction, operand preparation, and kernel execution. We additionally conduct paired-input profiling with identical VGGT inputs at 50% sparsity across 10 and 128 views on an RTX 4090 with 518×518 resolution, with full latency breakdowns reported in Table R1.

Table R1. Paired-input end-to-end latency.

ViewsSparsityMethod# bitPool/draftScore/calib.Mask/routingQuant/packKernelOtherE2EAUC@30 ↑
1050%SparseVGGT8-bit4.254–2.3401.75437.547418.880464.77586.12
1050%Ours8-bit0.6689.00534.3659.37925.176416.439495.03288.66
12850%SparseVGGT8-bit70.061–208.08827.6195531.8175638.70111476.28686.83
12850%Ours8-bit5.428138.215479.438111.3123850.2595599.22510183.87789.84

These results show that the relative efficiency depends on sequence length and sparsity. At 128 views and 50% sparsity, our native implementation is faster, whereas at 75% sparsity the two methods have nearly identical latency. We will clarify that the original dataset-average speedups do not imply universal same-input superiority.

(2) Separating sparsification from low-precision execution.

To isolate sparsification benefits, we additionally evaluate MosaicAttention using FP16 sparse attention, without low-precision kernels.

To isolate sparsification benefits, we additionally evaluate MosaicAttention with FP16 sparse attention on VGGT at a fixed 518 × 518 resolution using an RTX 5090, reporting end-to-end latency in seconds in Table R2.

Table R2. FP16-only end-to-end latency and speedup.

Method32F64F100F128F256F
Dense FP161.721 (1.00×)5.116 (1.00×)10.998 (1.00×)17.010 (1.00×)61.793 (1.00×)
Ours FP16, 50%1.451 (1.19×)3.849 (1.33×)7.846 (1.40×)11.739 (1.45×)40.241 (1.54×)
Ours FP16, 75%1.227 (1.40×)2.961 (1.73×)5.647 (1.95×)8.197 (2.08×)25.789 (2.40×)

At 256 views, FP16 sparsification alone provides 1.54×/2.40× speedup at 50%/75% sparsity. Thus, the acceleration is not solely attributable to low-precision execution. We will explicitly distinguish native mixed-precision comparisons from FP16-controlled results.

(3) Why is the precision-related speedup small in Table 5?

Table 5 evaluates only 10-view inference, where global attention constitutes a relatively small fraction of end-to-end latency. Consequently, accelerating sparse attention has a limited effect on the full model. We further report execution-precision ablation results for VGGT and π3\pi^3 at 128 views on an RTX 5090, with latency measured in seconds, in Table R3.

Table R3. Execution-precision ablation on VGGT and π3\pi^3.

VGGTπ3\pi^3
SparsityFP16 latency (s)8-bit latency (s)Mixed 4/8-bit latency (s)FP16 latency (s)8-bit latency (s)Mixed 4/8-bit latency (s)
0% (Dense FP16)17.010——15.745——
50%11.7398.0337.5619.8446.7766.517
75%8.1976.4936.2946.8225.6485.122

For VGGT at 128 views, 8-bit execution provides an additional 1.46×/1.26× speedup over sparse FP16 at 50%/75% sparsity for VGGT, respectively. This explains why the precision benefit becomes more visible as global attention increasingly dominates long-sequence inference.

(4) Performance on geometry benchmarks.

We sincerely thank the reviewer for this constructive comment, which prompted us to make Table 2 fully transparent by adding the dense reference, latency, and all baselines. We respectfully invite the reviewer to inspect the table directly. We agree that MosaicAttention does not outperform every baseline on every geometry metric, and we do not claim uniform metric-wise superiority. Its contribution is a controllable accuracy–efficiency trade-off: at matched sparsity and 8-bit execution, contribution-calibrated routing preserves geometry-relevant interactions better than average-pooled routing on most metrics, stays close to dense at 50% sparsity, and degrades gracefully at 75%.

AuthorsOpenReview ↗

Rebuttal for Weakness 2: Methodological Novelty and Effectiveness of Spatial Regions

We thank the reviewer for the insightful comments. We clarify our methodological contributions and provide the missing 1D + calibration control to directly validate our design.

(1) Novelty beyond existing block-sparse attention.

We acknowledge that block-sparse attention and channel-wise max statistics have been explored in SparseVGGT, SpargeAttention, and Quest. Our novelty lies not in these individual operations, but in their joint formulation for contribution-aware routing: spatially structured dual-pooled proxies preserve complementary regional statistics, while a lightweight calibrator maps them to dense value-weighted output contribution ranks for input-adaptive block selection. This directly connects spatial representation to the actual sparse-attention objective.

(2) Role of contribution calibration.

The 69-feature linear calibrator is deliberately lightweight rather than a new prediction architecture. Its role is to translate inexpensive spatial Avg/Max evidence into more reliable block priorities, instead of directly selecting blocks from approximate attention scores. As established in Section 3.3, the calibration target is the percentile rank of dense value-weighted output contributions. Theorem 3.5 further characterizes when sufficiently accurate predictions preserve the contribution-ranked Top-K support.

(3) Missing 1D + calibration factorial control.

Following the reviewer's suggestion, we complete the missing factorial ablation across 1D/2D layouts and uncalibrated/calibrated routing for VGGT on multiple datasets under 50% sparsity with FP16 precision, as summarized in Table R5.

Table R5. Layout and calibration ablation.

LayoutCalibrationAUC@30 ↑7Scenes Acc.↓NRGBD Acc.↓ETH3D Acc.↓Output Rel. ℓ2{\ell}_2 ↓
1D✗88.240.01920.01730.16610.1165
1D✓88.180.02120.01900.18170.1247
Ragged-2D✗88.290.01920.01790.16990.0936
Ragged-2D✓88.760.01870.01580.14650.0785

These results demonstrate that the benefits arise from the interaction between spatial representation and contribution calibration, rather than either component alone. We will incorporate this complete ablation study into the revised main paper.

AuthorsOpenReview ↗

Rebuttal for Weakness 3: Evaluation Coverage and Cross-Domain Generalization

We really thank the reviewer for highlighting these evaluation gaps. We provide additional long-sequence and cross-domain experiments, with frozen CO3Dv2-calibrated parameters throughout.

(1) Geometry sampling and long-sequence evaluation.

We now report the long-sequence accuracy-efficiency curve under the same precision that addresses the 10-view ablation limitation in Table R8. The evaluation uses 18 7Scenes sequences (850 views; 25–50 per sequence), 9 NRGBD sequences (278 views; 10–42), and 11 ETH3D scenes (377 views; 14–76).

(2) Cross-dataset camera-pose generalization.

To evaluate generalization beyond the calibration dataset, we additionally test on a RealEstate10K (Re10K) [1] dataset with 10 frames, without refitting the CO3Dv2-calibrated router.

Table R6. Cross-dataset camera-pose estimation on Re10K (VGGT, AUC@30 ↑).

Method# bitAUC@30 ↑
DenseFP1675.02
Ours, 50%FP1673.49
Ours, 50%8-bit73.33
Ours, 75%FP1669.55
Ours, 75%8-bit69.34
SparseVGGT, 50%8-bit69.52
SparseVGGT, 75%8-bit64.49
SAF3R, 85%BF1672.58
FastVGGT, merge=0.9FP1670.94

At 50% sparsity, MosaicAttention retains AUC@30 = 73.33 with 8-bit execution, compared with 75.02 for Dense. These results provide evidence of cross-dataset pose transfer without recalibration, while also revealing non-negligible accuracy degradation at higher sparsity.

(3) Generalization to dynamic videos.

We further evaluate on eight 90-view sequences from TUM-dynamics [2], covering indoor dynamic scenarios.

Table R7. TUM-dynamics camera-pose estimation.

BackboneMethodSparsity/Parameter# bitATE RMSE m ↓RPE trans m ↓RPE rot ° ↓Eval mean s ↓
VGGTDense0%FP160.011770.009980.30918.205
VGGTOurs50%8-bit0.012180.009990.30834.273
VGGTOurs75%8-bit0.014510.010740.31503.762
VGGTSparseVGGT50%8-bit0.014780.010560.32144.551
VGGTSparseVGGT75%8-bit0.017860.012320.35393.901
VGGTSAF3R85%BF160.012290.010210.30984.518
VGGTFastVGGTmerge=0.9FP160.012890.011440.32025.059
π3\pi^3Dense0%FP160.014080.008740.30707.876
π3\pi^3Ours50%8-bit0.013700.008810.30774.160
π3\pi^3Ours75%8-bit0.016270.010900.32493.601
π3\pi^3SparseVGGT50%FP160.015050.009560.31304.998
π3\pi^3SparseVGGT75%FP160.017080.010750.32624.009
π3\pi^3SAF3R85%BF160.014760.008840.30863.689
π3\pi^3FastVGGTmerge=0.9FP160.014870.009240.31484.756
DA3Dense0%FP160.015460.012110.31984.117
DA3Ours50%8-bit0.015880.012240.32193.003
DA3Ours75%8-bit0.018430.013350.33482.700
DA3SparseVGGT50%FP160.017090.012730.32533.253
DA3SparseVGGT75%FP160.020310.013750.33942.884
DA3FastVGGTmerge=0.9FP160.018500.013270.32503.309

At 50% sparsity, MosaicAttention achieves ATE close to Dense across all three backbones, demonstrating encouraging transfer to real indoor dynamic sequences. Higher sparsity leads to larger errors.


[1] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018.

[2] Ju ̈rgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of RGB-D SLAM systems. In IROS, pages 573–580. IEEE, 2012.

Reviewer_DW5nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the quadratic cost of global attention in feed-forward multi-view geometry Transformers with block-sparse attention. Tokens are grouped into compact 2D regions within each frame, and each region's queries and keys are summarized by average and maximum pooling to form low-resolution draft attention maps. Features derived from these maps are passed to a linear (ridge) calibrator, fitted offline on CO3Dv2, which predicts each block's contribution to the dense attention output. Blocks are then selected by a single top-K shared across all heads of a layer, under per-layer budgets set by an offline sensitivity analysis. The paper reports up to 2.89× end-to-end speedup at 128 views with competitive accuracy.

Critical strengths
  • A relevant problem and a practical setting. Global attention dominates inference time as the number of views grows, and the method keeps the pretrained backbone frozen. Only a small calibrator and a per-layer budget schedule are fitted offline, so it can be applied to existing models without retraining.
  • A sensible design for estimating block importance. Pooling over spatially compact 2D regions, rather than contiguous 1D token segments, respects the image layout. Targeting the value-weighted output contribution, rather than raw attention scores, is also reasonably motivated. Table 3 shows that calibration lowers the attention-output error compared with using the pooled scores directly.
Critical weaknesses
  • The advantage over existing efficient methods is not clearly established. The speedup is small at 10–30 views (at most about 1.06× at 10 views), so the benefit is limited to long sequences. At 100 views, however, the method is not clearly better than SAF3R. On π³, MosaicAttention at 50% sparsity and SAF3R reach nearly the same AUC@30 at nearly the same latency (Table 1), even though MosaicAttention uses an 8-bit kernel while SAF3R runs in BF16. Table 2 also reports neither the dense model nor latency, so the accuracy loss and the speed gained cannot be judged. Accuracy–latency curves for each method at long sequences, together with the dense reference, would make the comparison much clearer.

  • The headline speedup combines sparsity with low precision, and its accuracy is not reported. The 2.89× speedup in the abstract is measured with π³ at 75% sparsity and mixed 4/8-bit execution at 128 views (Fig. 5), but no accuracy is reported for this setting. At 100 views, the same configuration loses more than 6 AUC@30 points relative to the dense model (Table 7), which makes the claim of “maintaining competitive performance” difficult to support. Moreover, all reported speedups include low-precision kernels, while the only precision ablation (Table 5) is at 10 views, so it is unclear how much of the speedup comes from sparsification itself. The dependence of FP4 execution on recent GPUs should also be stated as a limitation.

  • A closely related training-free method, AVGGT, is not discussed or compared. AVGGT (Sun et al., CVPR 2026) analyzes the role of each global-attention layer in VGGT and π³, converts early global layers into frame attention, and subsamples keys and values in the remaining layers. Without calibration or low-precision kernels, it reports about 2× speedup at 100 views and larger gains at 300–800 views, with little loss in accuracy. Its analysis also reaches the observation in Fig. 3 that global-attention layers differ substantially in their role and in their tolerance to sparsification. The layer-wise budget allocation is therefore an application of a known observation rather than a new insight. A direct comparison with AVGGT, ideally at the same precision, is needed to show the benefit of input-adaptive block routing.

References

  • Sun, X., Zhu, Z., Lou, Z., Yang, B., Tang, J., Zhang, L., Wang, H., and Zhang, J. AVGGT: Rethinking Global Attention for Accelerating VGGT. CVPR 2026.
Questions for authors
  • Can you provide accuracy–latency curves at 100–128 views that include the dense model, SAF3R and AVGGT [1], with all methods at the same precision?
  • How sensitive are the calibrator and the layer budgets to the calibration data, for example when fitted on a different dataset or on longer sequences (64–128 frames)?
5 comments
AuthorsOpenReview ↗

Rebuttal for Weakness 1: Advantage over Existing Efficient Methods (Part 1)

Many thanks for the detailed and constructive assessment. These suggestions helped us present the scaling behavior and the relation to AVGGT more clearly.

(1) Limited speedup at short sequences.

MosaicAttention targets long-sequence inference, where global attention dominates. At short lengths, routing overhead is a larger fraction of runtime, so the gain is smaller. Table R1 isolates this scaling with FP16 sparse attention on VGGT (RTX 5090, 518×518).

Table R1. Fixed-shape end-to-end latency and speedup scaling.

Method32F64F100F128F256F
Dense FP161.721 (1.00×)5.116 (1.00×)10.998 (1.00×)17.010 (1.00×)61.793 (1.00×)
Ours FP16, 50%1.451 (1.19×)3.849 (1.33×)7.846 (1.40×)11.739 (1.45×)40.241 (1.54×)
Ours FP16, 75%1.227 (1.40×)2.961 (1.73×)5.647 (1.95×)8.197 (2.08×)25.789 (2.40×)

At 256 views, FP16 sparsification alone gives 1.54×/2.40× speedup at 50%/75%, independently of low-precision execution.

(2) Long-sequence accuracy–latency comparison with existing efficient methods.

Table R2a adds 100/128-view pose with dense and SAF3R; Table R2b adds 256-view dense geometry with dense, SparseVGGT, FastVGGT, and SAF3R.

Table R2a. Camera-pose accuracy and latency — CO3Dv2

100F128F
Method# bitAUC@30 ↑Latency (s) ↓AUC@30 ↑Latency (s) ↓
VGGTDenseFP1690.6712.21090.6318.624
Ours, 50%FP1689.638.61789.9213.301
Ours, 75%FP1687.906.44688.079.341
Ours, 50%8-bit89.605.61989.848.175
Ours, 75%8-bit87.914.93788.096.753
SAF3R, 85%BF1689.528.23889.6111.121
π3\pi^3DenseFP1691.1012.10291.1618.635
Ours, 50%FP1690.597.06090.6610.435
Ours, 75%FP1684.985.52385.277.692
Ours, 50%8-bit90.595.10690.857.620
Ours, 75%8-bit84.954.32785.126.185
SAF3R, 85%BF1690.386.45990.458.719

At 50% sparsity with 8-bit precision and 128 views, MosaicAttention outperforms SAF3R in both accuracy and inference speed. Our method delivers compelling accuracy–efficiency trade-offs across all evaluated sparsity and precision regimes, and establishes clear performance advantages over competing approaches in representative settings.

Table R2b. 256-view dense geometry on 7Scenes

BackboneMethodPrecisionSparsity/Parameterδ. ↑Acc. ↓Comp. ↓Latency (s) ↓
VGGTDenseFP160%95.690%0.015710.0206160.266
Ours8bi t-QK/FP8-PV50%95.891%0.015590.0215723.779
SparseVGGT8-bit50%95.679%0.017130.0219028.919
SparseVGGTFP1650%95.688%0.016970.0225849.076
Ours8-bit75%96.176%0.016320.0210520.384
SparseVGGT8-bit75%95.938%0.018170.0232325.236
SparseVGGTFP1675%95.989%0.018090.0228342.814
FastVGGTFP16merge=0.995.953%0.015640.0235121.991
SAF3RBF1685%95.851%0.016100.0225220.592
π3\pi^3DenseFP160%98.805%0.009010.0116651.300
Ours8-bit50%98.786%0.009730.0107917.895
SparseVGGT8-bit50%98.785%0.014190.0116219.813
SparseVGGTFP1650%98.786%0.014240.0111925.476
Ours8-bit75%98.749%0.019770.0116513.291
SparseVGGT8-bit75%98.712%0.023870.0124714.902
SparseVGGTFP1675%98.728%0.022990.0122317.874
FastVGGTFP16merge=0.998.456%0.012830.0117213.952
SAF3RBF1685%98.739%0.010730.0109111.986
DA3DenseFP160%97.662%0.015490.0159521.036
Ours8-bit50%97.746%0.015640.0150111.332
SparseVGGT8-bit50%97.680%0.015850.0153312.746
SparseVGGTFP1650%97.685%0.015790.0153914.445
Ours8-bit75%97.749%0.017370.0151110.074
SparseVGGT8-bit75%97.702%0.017540.0158311.365
SparseVGGTFP1675%97.701%0.017870.0156213.902
FastVGGT-styleFP16merge=0.997.576%0.016420.0157312.428

In the 256-view geometry comparison, 8-bit MosaicAttention beats SparseVGGT across all sparsity settings on three backbones with higher accuracy, lower latency, and near-dense accuracy, and outperforms SAF3R/FastVGGT in efficiency and performance for nearly all configurations.

AuthorsOpenReview ↗

Rebuttal for Weakness 1: Advantage over Existing Efficient Methods (Part 2)

(3) Separating routing efficiency from execution precision.

We agree that comparing native 8-bit execution against BF16 alone cannot isolate the advantage of block routing. We therefore additionally evaluate our method with FP16 sparse attention and report the precision-dependent latency separately on RTX5090.

Table R3. Precision ablation on VGGT and π3\pi^3 with 128 inputs.

VGGTπ3\pi^3
SparsityFP16 latency (s)8-bit latency (s)Mixed 4/8-bit latency (s)FP16 latency (s)8-bit latency (s)Mixed 4/8-bit latency (s)
0% (Dense FP16)17.010——15.745——
50%11.7398.0337.5619.8446.7766.517
75%8.1976.4936.2946.8225.6485.122

The FP16 results establish that sparsification itself provides substantial acceleration at long sequence lengths, while low-precision execution offers an additional, complementary gain. We will distinguish these effects and explicitly label the execution precision of each baseline in the revision.

(4) Missing dense references and latency in Table 2.

We add dense geometry metrics and dataset-level evaluation latency in Table 2. The corresponding sampling comprises 18 7Scenes sequences (850 views; 25–50 each), 9 NRGBD sequences (278 views; 10–42), and 11 ETH3D scenes (377 views; 14–76).

AuthorsOpenReview ↗

Rebuttal for Weakness 2: Headline Speedup, Accuracy Preservation, and Execution Precision

We sincerely thank the reviewer for pointing out the need to explicitly associate the reported speedup with its accuracy and to disentangle sparsification from low-precision execution. We address these concerns separately below.

(1) Accuracy–efficiency trade-off at 128 views.

We additionally evaluate the 128-view π3\pi^3 setting, including the previously missing accuracy of the mixed 4/8-bit configuration.

Table R5. Camera-pose accuracy at 128 views on CO3Dv2. (All rows except the mixed 4/8-bit row are measured on RTX 4090; only the mixed 4/8-bit row is measured on RTX 5090.)

Sparsity# bitAUC@30 ↑Latency (s) ↓Speedup ↑
0%Dense FP1691.1618.6351.00×
50%Sparse FP1690.6610.4351.79×
50%8-bit90.857.6202.45×
50%Mixed 4/8-bit90.065.361—
75%Sparse FP1685.277.6922.42×
75%8-bit85.126.1853.01×
75%Mixed 4/8-bit85.044.545—

At 50% sparsity, the mixed-precision configuration remains close to dense AUC@30. At 75%, it offers more aggressive acceleration at a noticeable accuracy cost. We will explicitly associate our headline efficiency claim with its corresponding accuracy.

(2) Separating sparsification from low-precision acceleration.

To isolate their respective contributions, we extend the original 10-view precision ablation to long-sequence inference and report fixed-shape end-to-end latency under π3\pi^3 on an RTX 5090 with 518 × 518 inputs.

Table R6. Fixed-shape end-to-end latency and speedup by execution precision.

Configuration32F64F100F128F256F
Dense FP161.464 (1.00×)4.247 (1.00×)9.956 (1.00×)15.745 (1.00×)47.758 (1.00×)
Sparse FP16, 50%1.298 (1.13×)3.335 (1.27×)6.623 (1.50×)9.844 (1.60×)32.117 (1.49×)
8-bit, 50%1.102 (1.33×)2.565 (1.66×)4.448 (2.24×)6.776 (2.32×)19.498 (2.45×)
Mixed 4/8-bit, 50%1.091 (1.34×)2.507 (1.69×)4.289 (2.32×)6.517 (2.42×)19.140 (2.50×)
Sparse FP16, 75%1.104 (1.33×)2.589 (1.64×)4.789 (2.08×)6.822 (2.31×)20.668 (2.31×)
8-bit, 75%1.042 (1.40×)2.297 (1.85×)3.964 (2.51×)5.648 (2.79×)14.466 (3.30×)
Mixed 4/8-bit, 75%1.036 (1.41×)2.271 (1.87×)3.998 (2.49×)5.122 (3.07×)14.009 (3.41×)

At 256 frames, FP16 sparsification alone achieves 1.49×/2.31× speedup at 50%/75% sparsity, respectively. Mixed 4/8-bit execution further improves these to 2.50×/3.41× on the same hardware. This confirms that sparsification provides substantial acceleration independently of quantization, while low-precision execution provides additional benefits.

(3) Hardware requirements of FP4 execution.

Native FP4 sparse execution requires supported GPU hardware. As FP4 support becomes increasingly prevalent in future GPU architectures, FP4-based sparse attention holds significant potential for further acceleration. Importantly, MosaicAttention's block routing is independent of the execution precision and supports FP16 and 8-bit alternatives, as demonstrated above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.