- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces VECA (Visual Elastic-Core Attention), a vision transformer featuring core-periphery structured attention mediated by a small set of learned cores, which reduces the computational complexity of attention from O(N²) to O(N). The authors thoroughly explore several key design aspects and conduct extensive evaluations across multiple visual tasks (e.g., image classification, semantic segmentation, and depth estimation). These results demonstrate that effective visual representations can be learned without direct all-to-all patch interaction.
Critical strengths
- The core design philosophy of this architecture is well-conceived. Unlike latent-compression architectures, VECA limits global communication through a set of compact persistent cores while preserving dense patch representations. This represents a novel direction for high-resolution visual representation learning.
- Controlled comparative experiments are equally valuable. The authors evaluated various alternative models with linear complexity under comparable parameter budgets and a unified teacher distillation scheme. The key results demonstrate that, for both global and dense prediction tasks, VECA significantly narrows the performance gap relative to ViTs employing full attention mechanisms.
- Experiments conducted on the elastic core budget and the emergent semantic organization provide further insights into the proposed architecture. In particular, ablation experiments comparing the "persistent core" with the "hierarchical local core" validate the effectiveness of a key component of the design.
Critical weaknesses
W1. The paper's core claim is insufficiently supported due to its reliance on a teacher model that employs a full-attention mechanism. This paper aims to investigate whether Vision Transformers necessarily rely on all-to-all interactions between patches to achieve effective representation learning. However, all VECA models are trained via feature distillation using DINOv3, which itself relies on a full-attention mechanism between patches. Consequently, the current experimental results merely demonstrate that a core-mediated architecture can effectively approximate the representations learned by a full-attention teacher model; they fail to prove that an all-to-all attention mechanism is unnecessary for learning such representations. Given the paper's central research question, this distinction is crucial and warrants a clearer explanation.
W2. Insufficient end-to-end efficiency evaluation. While computational efficiency is a primary design consideration for VECA, the efficiency analysis in Appendix A.10 focuses solely on the attention module rather than the complete backbone network. Although Table 2 lists partial FLOPs data for VECA variants during the full forward pass, a comprehensive end-to-end comparison with full-attention models is lacking. Given that VECA retains all dense patch tokens and continues to apply projection and feed-forward operations to them, the reduction in attention computational complexity does not necessarily translate into a proportional improvement in overall inference efficiency. Without measurements regarding metrics such as latency, throughput, and memory footprint for the complete model, its actual computational advantages cannot be fully substantiated.
W3. Comparisons with recent methods are incomplete, and the analysis of positional encoding is limited. Comparisons with recent efficient vision architectures are not comprehensive; notably, ViT³ (CVPR 2026 Oral) [1] is omitted. Furthermore, while the "learned core coordinates" constitute an interesting component, existing ablation studies fail to sufficiently demonstrate their necessity. The full-attention baseline employs a different positional encoding scheme than VECA, making it difficult to fully isolate and evaluate the contribution of the proposed attention structure itself.
Reference:
[1] Han, Dongchen, et al. "ViT : Unlocking Test-Time Training in Vision." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
Questions for authors
See Weaknesses.
5 comments
The key changes in our revision are summarized here:
Page 3:we add a discussion and citation for ViT³. (foPV)Page 6:We clarify the role of core embeddings through quantitative comparisons and add additional results on Page 47 of the appendix. (TDwS, foPV)Page 34:We provide additional evaluation results for DINOv3-B and VECA-B at resolutions of 1024px and 1120px. (NkPS)Page 44:We present end-to-end inference efficiency comparisons across resolutions ranging from 256px to 1120px. (TDwS, foPV)Page 46:We add evaluation results of VECA distilled from TIPSv2 on image classification, semantic segmentation, and depth estimation.(TDwS, NkPS, foPV)Page 47:We add ablation results on the core design and analyze the contribution of each component. (TDwS, foPV)Page 48:We provide controlled comparisons against CLS tokens and Registers under VECA routing. (NkPS)
We thank you again for your valuable feedback! We believe that your suggestions have helped us strengthen our paper.
Official Comment by Reviewer foPV
Your response addressed the vast majority of my concerns. We will maintain the score. There is one more point we don’t quite understand: why do some linear-attention models (e.g., PolaFormer, ViT$^3$) exhibit even higher latency than DINOv3, which uses a standard attention mechanism? Could you provide some further classifications? Thank you very much.
Thank you for your detailed and constructive review! We appreciate your suggestions and have updated the paper according to your feedback.
Here, we conduct two additional experiments: (1) distilling VECA-B from Google's latest TIPSv2 model instead of DINOv3.
(2) Training a vanilla ViT and VECA from scratch on ImageNet-1K classification supervision for 150 epochs, following the same training recipe with a global batch size of 1536. Surprisingly VECA achieves better performance on dense prediction tasks. We believe this may be due to our extensive use of cores, which are functionally similar to registers and have been demonstrated to improve dense representations.
We benchmark the complete student forward pass compared to 3 parameter-matched baselines (85.73M) under the same setting on a single RTX PRO 6000 GPU. We used a batch size of 8, torch.inference_mode(), and BF16 torch.autocast. Latency was measured with CUDA events after 50 warm-up iterations, using 200 timed iterations in each of 3 rounds. We explicitly called
torch.cuda.synchronize()after warm-up and before reading the timing events in each round to ensure that asynchronous CUDA execution was completed. We use VECA with 64 active cores and a packed-QKV inference implementation. Median batch latency is reported over 600 measured forward passes. PolaFormer is our most competitive (accuracy) alternative linear attention model and ViT³ is the most recent linear-time baseline.We thank the reviewer for pointing out the missing reference. We have carefully reviewed ViT³ and will cite and discuss it in the next version. This work provides important prior contributions to linear attention and test-time training, and we will revise our related-work section to acknowledge these connections and clarify the distinctions between ViT³ and VECA.
To isolate the contribution of VECA's core positional encoding, we conduct additional controlled ablations. Because core tokens are not initially associated with image patches, they lack an inherent spatial reference. Our design assigns each core a learnable 2D coordinate used by RoPE. These coordinates are initialized using farthest-point sampling, are learnable, and evolve throughout the network layers as the cores specialize.
We conduct the following ablations:
We appreciate your helpful suggestions! Following your feedback, we have conducted additional experiments on different training strategies and on end-to-end efficiency. We also compared against additional baselines, measured the impact of coordinate-based positional encoding, and cited the related work ViT³ in the revised manuscript.