acceptodds
Withdrawn from ICLR 2027

VASC: Value-Aware Sparse Attention with Cross-layer Memory for Efficient 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward 3D vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass. However, their quadratic global attention makes long image sequences expensive, while existing sparse methods may favor highly attended yet value-redundant regions. To address these limitations, we introduce VASC, a training-free sparse attention method combining value-aware block selection and execution-aware cross-layer memory. Our value-aware block selection integrates pooled query–key relevance with neighboring value contrast, reducing redundancy while preserving query-relevant and distinctive content. Cross-layer memory tracks unserved demand across layers and updates this state according to actual execution, enabling previously underserved blocks to compete under a fixed computation budget. Experiments on 7Scenes and NeuralRGB-D with VGGT and demonstrate improved pose estimation and reconstruction quality compared with FasterVGGT, together with up to faster inference than dense VGGT. Code and evaluation scripts will be made available.

Primary area:
generative models
Keywords:
sparse attention, 3D reconstruction, attention optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kNpUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VASC, a training-free sparse-attention method for feed-forward 3D reconstruction. The authors demonstrate on commonly used datasets in this area, including 7-Scenes and NRGBD, that the proposed method achieves an improved trade-off between computational efficiency and accuracy preservation.

Critical strengths

In addition to the main comparisons on reconstruction metrics, the authors provide experimental observations and analysis in Section 3, which constitute a valuable contribution.

Critical weaknesses
  1. The manuscript is unnecessarily verbose and insufficiently polished. Section 4 mainly enumerates implementation details and absolute numerical changes, with little meaningful analysis of the observed results and their causes. The repetitive and formulaic writing gives the impression of no adequate human revision.
  2. The proposed method mainly combines existing techniques from the sparse-attention area. The paper's overall methodological novelty therefore appears limited.
  3. Although the authors evaluate frame strides of 5 and 10, both still correspond to relatively small input intervals with substantial view overlap. The method should also be evaluated on the same datasets using large frame intervals. Otherwise, it remains unclear whether the reported improvements generalize to input settings with limited view overlap.
Questions for authors
  1. Please check the weaknesses.
  2. Please provide more substantive analysis of the experimental results.
  3. Please provide reconstruction comparison at larger frame intervals (20, 50, 100).
Reviewer_BQR7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VASC, a training-free method to speed up the global attention of feed-forward 3D reconstruction models such as VGGT and Pi3. It keeps a fixed block-sparse budget and selects which query–key block interactions to execute. Selection combines pooled query–key relevance with the contrast between adjacent value blocks. Experiments on 7Scenes, Neural-RGB-D, and ScanNet-50 with VGGT and Pi3 show better pose and reconstruction metrics than FasterVGGT at the same sparsity, and about two times speedup over dense VGGT.

Critical strengths
  1. The method is training-free and plug-in. The components are simple and effective: pooled relevance, a value-contrast score, and one scalar per block pair for memory.

  2. The gains over the sparse baseline are consistent on two backbones and two datasets at high sparsity, and also transfer to ScanNet.

  3. The speedup over dense VGGT, about two times, comes with only a small overhead over the sparse baseline.

Critical weaknesses
  1. Baselines and efficiency. The main tables compare only with FasterVGGT. FastVGGT is only in an appendix table, and AVGGT and QuantVGGT are not compared. It would be better to compare these methods.

  2. Cross-layer memory. There is no direct evidence that the memory actually serves underserved blocks. It would be better to report how often high-deficit blocks are selected later, how much the memory changes the selected set, and how much it overlaps with dense selection. The ablation already suggests most of the gain comes from value contrast.

  3. Design choices. The pairing scheme, memory weight, block size, and choice of sparse layers are all fixed without ablation. It would be better to provide a detailed ablation of these design choices.

Questions for authors

Please see above weaknesses.

Reviewer_nU1LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces VASC, a training-free framework for accelerating Transformer-based geometry backbones such as VGGT and π3\pi^3 for feed-forward 3D reconstruction. The method combines value-aware block selection and execution-aware cross-layer memory to improve sparse attention allocation. Experimental results demonstrate improvements in reconstruction quality and camera pose estimation compared with FasterVGGT, while maintaining comparable computational efficiency.

Critical strengths
  1. Well-motivated value-aware selection. The motivation for introducing value-aware selection is sound and supported by illustrative experiments. The ablation studies further demonstrate that value-aware selection contributes to the overall performance.

  2. Simple and effective framework. The proposed method introduces relatively few hyperparameters and thresholds that require tuning. The normalization strategy also makes the approach adaptable to different inputs. Experiments demonstrate improvements over FasterVGGT while largely maintaining computational efficiency.

Critical weaknesses
  1. Limited experimental evaluation. The main experiments focus on two indoor datasets, 7Scenes and NeuralRGB-D, which contain substantial redundancy across images. Although the paper additionally evaluates on a subset of ScanNet, all three datasets consist of indoor scenes. More extensive experiments on diverse datasets, particularly outdoor scenes such as ETH3D, would better demonstrate the generalizability of the method. In addition, the paper does not report GPU memory consumption or overhead, which is an important aspect of efficiency for long-sequence 3D reconstruction.

  2. Insufficient justification for cross-layer memory. Since QKV representations differ across Transformer layers, it is unclear whether demand scores are meaningfully comparable across layers or why previously underserved blocks should receive higher priority. Moreover, the demand–service difference does not directly measure information loss or reconstruction quality. Further justification or empirical analysis would strengthen the motivation for this mechanism. The ablation studies also show that adding memory does not consistently improve performance over value-aware selection alone.

Additional concerns:

  • Qualitative results. It is difficult to visually distinguish the reconstructions produced by VASC and FasterVGGT in Figure 5, making it challenging to assess the qualitative improvements claimed by the paper. More detailed visual comparisons or error visualizations would help demonstrate the benefits.
  • Presentation clarity (minor). The paper's writing could be clearer in several places. For example, the notation (j_{p,1}) and (j_{p,2}) is not explicitly introduced when first used in Section 4.1, and the term "connection" is used without a clear initial definition. Some sentences are also difficult to follow, making the method more challenging to understand.
Questions for authors
  1. Can the authors provide additional evaluations on more diverse datasets, particularly outdoor scenes, to better demonstrate the generalizability of the method? It would also be helpful to report peak GPU memory usage alongside the inference latency comparisons.

  2. Can the authors provide further analysis of the role of cross-layer memory, ideally with supporting statistics or empirical justification? In particular, are demand scores comparable across layers, and does the accumulated demand–service difference provide a meaningful signal for future attention selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.