- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MosaicAttention, a block-sparse attention framework for accelerating feed-forward multi-view 3D geometry Transformers. The method constructs lightweight attention proxies using spatially structured average and maximum pooling, and employs an offline ridge-regression calibrator to predict block priorities based on dense attention output contributions. It further introduces sensitivity-aware layer-wise budget allocation and mixed-precision sparse attention execution.
Experiments on VGGT, π³, and Depth Anything 3 demonstrate promising accuracy-efficiency trade-offs across camera pose estimation and dense geometry reconstruction. The method achieves up to 2.89× end-to-end speedup on 128-view inputs.
Critical strengths
- Compared with prior block-sparse attention methods relying on average-pooled Q/K features, the proposed max pooling captures salient information that averaging may miss. In my experience, this is beneficial for 3D geometry tasks.
- The contribution-based calibration strategy is an interesting approach to aligning lightweight block importance estimates with actual attention output contributions.
Critical weaknesses
The calibration target considers value-weighted attention contributions, yet the calibrator only uses Q/K-derived features without incorporating V, which seems inconsistent with the motivation of estimating actual block contributions.
Questions for authors
Same as Weakness 1. Since V is already computed during inference, shouldn't incorporating V-related features improve block importance estimation? Why do the authors exclude V, especially when doing so would introduce only negligible computational overhead?
Rebuttal for Weakness 1 and Question1:
W1 / Q1. Why Use Value-Aware Calibration with Q/K-Only Features?
We thank the reviewer for this insightful suggestion. We clarify the distinction between value-aware supervision and online feature extraction, and provide a controlled V-feature ablation.
(1) Value-aware supervision versus Q/K-only prediction.
Our calibration target is value-aware, but the online predictor need not explicitly access V. Specifically, the dense value-weighted output contribution (Definition 3.4) provides the offline supervision target, while the lightweight Q/K-derived dual-pooled features predict its relative block importance without computing dense attention. This is an approximation rather than an assumption that Q/K fully determines the contribution.
(2) Does incorporating V improve block importance estimation?
Following the reviewer's suggestion, we independently fit three calibrators using the same dense teacher and calibration split, adding either block-wise V RMS or the norm of mean V to the original Q/K features.
Table R1. Value-feature ablation. VGGT, 50% sparsity, held-out validation.
Adding V RMS slightly improves output approximation and retained utility, while the mean-V-norm variant provides no improvement in output distortion. These results suggest that the tested V statistics provide only marginal additional information beyond our Q/K-derived features.