acceptodds
Under review as a conference paper at ICLR 2027

Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores

Abstract

Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design implicitly assumes that direct pairwise patch interactions are necessary for effective representation learning. In this work, we challenge this assumption and show that representations supporting both global recognition and dense prediction can be learned without direct patch-to-patch interaction. We propose **VECA** (Visual Elastic-Core Attention), a vision transformer with core-periphery structured attention mediated by a small set of learned cores. Patch tokens exchange global information exclusively through these cores, while the full set of dense patches are preserved and iteratively updated across layers. This reduces attention complexity from to , linear in the number of patches for a fixed core budget . Unlike prior latent-token cross-attention architectures, VECA facilitates sparse global communication without compressing the spatial representation itself. Nested training along the core axis further enables a single model to elastically trade off computation and accuracy at inference time without retraining. Across image classification and dense prediction tasks, VECA remains competitive with full-attention backbones and outperforms the evaluated linear-complexity alternatives on most benchmarks. Moreover, without explicit supervision, these cores develop semantically organized structures that support object-label transfer across views. These results show that effective visual representations can be learned without direct all-to-all patch interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual representation learning, vision transformers, efficient attention, dense prediction, elastic inference, object-centric representations

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_foPVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces VECA (Visual Elastic-Core Attention), a vision transformer featuring core-periphery structured attention mediated by a small set of learned cores, which reduces the computational complexity of attention from O(N²) to O(N). The authors thoroughly explore several key design aspects and conduct extensive evaluations across multiple visual tasks (e.g., image classification, semantic segmentation, and depth estimation). These results demonstrate that effective visual representations can be learned without direct all-to-all patch interaction.

Critical strengths
  1. The core design philosophy of this architecture is well-conceived. Unlike latent-compression architectures, VECA limits global communication through a set of compact persistent cores while preserving dense patch representations. This represents a novel direction for high-resolution visual representation learning.
  2. Controlled comparative experiments are equally valuable. The authors evaluated various alternative models with linear complexity under comparable parameter budgets and a unified teacher distillation scheme. The key results demonstrate that, for both global and dense prediction tasks, VECA significantly narrows the performance gap relative to ViTs employing full attention mechanisms.
  3. Experiments conducted on the elastic core budget and the emergent semantic organization provide further insights into the proposed architecture. In particular, ablation experiments comparing the "persistent core" with the "hierarchical local core" validate the effectiveness of a key component of the design.
Critical weaknesses

W1. The paper's core claim is insufficiently supported due to its reliance on a teacher model that employs a full-attention mechanism. This paper aims to investigate whether Vision Transformers necessarily rely on all-to-all interactions between patches to achieve effective representation learning. However, all VECA models are trained via feature distillation using DINOv3, which itself relies on a full-attention mechanism between patches. Consequently, the current experimental results merely demonstrate that a core-mediated architecture can effectively approximate the representations learned by a full-attention teacher model; they fail to prove that an all-to-all attention mechanism is unnecessary for learning such representations. Given the paper's central research question, this distinction is crucial and warrants a clearer explanation.

W2. Insufficient end-to-end efficiency evaluation. While computational efficiency is a primary design consideration for VECA, the efficiency analysis in Appendix A.10 focuses solely on the attention module rather than the complete backbone network. Although Table 2 lists partial FLOPs data for VECA variants during the full forward pass, a comprehensive end-to-end comparison with full-attention models is lacking. Given that VECA retains all dense patch tokens and continues to apply projection and feed-forward operations to them, the reduction in attention computational complexity does not necessarily translate into a proportional improvement in overall inference efficiency. Without measurements regarding metrics such as latency, throughput, and memory footprint for the complete model, its actual computational advantages cannot be fully substantiated.

W3. Comparisons with recent methods are incomplete, and the analysis of positional encoding is limited. Comparisons with recent efficient vision architectures are not comprehensive; notably, ViT³ (CVPR 2026 Oral) [1] is omitted. Furthermore, while the "learned core coordinates" constitute an interesting component, existing ablation studies fail to sufficiently demonstrate their necessity. The full-attention baseline employs a different positional encoding scheme than VECA, making it difficult to fully isolate and evaluate the contribution of the proposed attention structure itself.

Reference:

[1] Han, Dongchen, et al. "ViT 3^ 3: Unlocking Test-Time Training in Vision." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors

See Weaknesses.

5 comments

Thank you for your detailed and constructive review! We appreciate your suggestions and have updated the paper according to your feedback.

Q1) Choice of teacher model and training from scratch

Here, we conduct two additional experiments: (1) distilling VECA-B from Google's latest TIPSv2 model instead of DINOv3.

Training StrategyImageNet-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Pretrained TIPSv277.7185.1149.850.3882
VECA distill from TIPSv274.6984.6248.810.3971
Pretrained DINOv383.5687.5051.350.3684
VECA distill from DINOv382.1887.0750.690.3705

(2) Training a vanilla ViT and VECA from scratch on ImageNet-1K classification supervision for 150 epochs, following the same training recipe with a global batch size of 1536. Surprisingly VECA achieves better performance on dense prediction tasks. We believe this may be due to our extensive use of cores, which are functionally similar to registers and have been demonstrated to improve dense representations.

ModelImageNet-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Vanilla ViT-B79.2164.9628.320.6497
VECA-B78.9367.1029.330.6501

Q2) End-to-end efficiency evaluation

We benchmark the complete student forward pass compared to 3 parameter-matched baselines (85.73M) under the same setting on a single RTX PRO 6000 GPU. We used a batch size of 8, torch.inference_mode(), and BF16 torch.autocast. Latency was measured with CUDA events after 50 warm-up iterations, using 200 timed iterations in each of 3 rounds. We explicitly called torch.cuda.synchronize() after warm-up and before reading the timing events in each round to ensure that asynchronous CUDA execution was completed. We use VECA with 64 active cores and a packed-QKV inference implementation. Median batch latency is reported over 600 measured forward passes. PolaFormer is our most competitive (accuracy) alternative linear attention model and ViT³ is the most recent linear-time baseline.

ModelResolutionLatency(ms)Throughput(img/s)
DINOv32564.8591646.36
PolaFormer2568.298964.11
ViT³25610.870735.96
VECA-C642566.1521300.37
DINOv351217.109467.58
PolaFormer51267.341118.80
ViT³51224.603325.16
VECA-C6451213.541590.81
DINOv3102486.15592.86
PolaFormer1024144.56055.34
ViT³1024102.21378.27
VECA-C64102455.693143.65

Q3) Additional related work

We thank the reviewer for pointing out the missing reference. We have carefully reviewed ViT³ and will cite and discuss it in the next version. This work provides important prior contributions to linear attention and test-time training, and we will revise our related-work section to acknowledge these connections and clarify the distinctions between ViT³ and VECA.

Q4) Positional Encoding Ablations

To isolate the contribution of VECA's core positional encoding, we conduct additional controlled ablations. Because core tokens are not initially associated with image patches, they lack an inherent spatial reference. Our design assigns each core a learnable 2D coordinate used by RoPE. These coordinates are initialized using farthest-point sampling, are learnable, and evolve throughout the network layers as the cores specialize.

We conduct the following ablations:

  1. Learnable positional embeddings for both core tokens and dense patch tokens.
  2. Fixed core coords used by RoPE and no farthest-point initialization (fixed at start of training, later layers do not update coordinate).
  3. Fixed core coords used by RoPE with farthest-point initialization (fixed at start of training, later layers do not update coordinate).
  4. Layers can update coordinates with farthest-point initialization (VECA)
Core positional designImageNet-1K Top-1 Acc ↑VOC mIoU ↑ADE20K mIoU ↑NYUv2 RMSE ↓
Learnable position embedding81.4485.8649.730.3842
No learnable RoPE + no farthest-point initialization81.7186.0349.800.3771
No learnable RoPE + farthest-point initialization81.6186.9050.320.3740
Learnable RoPE + farthest-point initialization (VECA)82.1887.0750.690.3705

Conclusion

We appreciate your helpful suggestions! Following your feedback, we have conducted additional experiments on different training strategies and on end-to-end efficiency. We also compared against additional baselines, measured the impact of coordinate-based positional encoding, and cited the related work ViT³ in the revised manuscript.

The key changes in our revision are summarized here:

  1. Page 3: we add a discussion and citation for ViT³. (foPV)
  2. Page 6: We clarify the role of core embeddings through quantitative comparisons and add additional results on Page 47 of the appendix. (TDwS, foPV)
  3. Page 34: We provide additional evaluation results for DINOv3-B and VECA-B at resolutions of 1024px and 1120px. (NkPS)
  4. Page 44: We present end-to-end inference efficiency comparisons across resolutions ranging from 256px to 1120px. (TDwS, foPV)
  5. Page 46: We add evaluation results of VECA distilled from TIPSv2 on image classification, semantic segmentation, and depth estimation.(TDwS, NkPS, foPV)
  6. Page 47: We add ablation results on the core design and analyze the contribution of each component. (TDwS, foPV)
  7. Page 48: We provide controlled comparisons against CLS tokens and Registers under VECA routing. (NkPS)

We thank you again for your valuable feedback! We believe that your suggestions have helped us strengthen our paper.

Reviewer_foPVOpenReview ↗

Official Comment by Reviewer foPV

Your response addressed the vast majority of my concerns. We will maintain the score. There is one more point we don’t quite understand: why do some linear-attention models (e.g., PolaFormer, ViT$^3$) exhibit even higher latency than DINOv3, which uses a standard attention mechanism? Could you provide some further classifications? Thank you very much.

Reviewer_NkPSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed method, VECA, replaces the all-to-all vanilla attention scheme with routing information to core tokens which are supposed to capture global information. With this, a small set of core tokens is appended and all patch tokens attend to them, the core tokens attend to all tokens. Through nested dropout, the core budget can be changed at inference without retraining. The authors perform extensive experiments on DINOv3 distillation.

Critical strengths
  • Using additional tokens and routing the relevant global information is generally useful for representation learning
  • The paper is relevant since efficient attention has a lot of impact across various fields
  • The paper is well written
  • The experiments in the paper are extensive, even if not complete
Critical weaknesses
  • The main principle of the paper is to investigate a general purpose ViT modification. However, the experiments only cover the observable effect when using distillation with ViTs since the main experiments center around a DINOv3 distillation setup. To actually investigate the full breadth of what the title suggest, additional experiments are required (see below)
  • Furthermore, the authors fail to clearly demonstrate the claimed benefit at high resolution. Accuracy is only evaluated up to 768px, which efficiency experiments are performed at 1024px. So they are both never reported on the same resolution. This also requires further experiments
  • The general idea of appending additional tokens that story global information has been proposed in CLS tokens and, in some form, also in register tokens, without the proposed routing. The paper acknowledges the existence of these, but never compares them under similar routing conditions.
Questions for authors
  • How does the proposed method compare to the vanilla ViT when trained from scratch?
  • How does the approach perform wrt accuracy and efficiency at higher resolutions than 1024px?
  • Compare against CLS/register tokens in the same setting: How does a ViT with CLS/Registers perform with the routing? How does your approach perform vs. ViT with CLS/Registers without the routing?
7 comments

Thank you for the concrete suggestions and helpful feedback! We will address specific questions below.

Q1) Training VECA from scratch

To address this, we train both VECA-B and a vanilla ViT-B from scratch on ImageNet-1K for 150 epochs, following the same training recipe with a global batch size of 1536. At least using our training epochs and batch size, surprisingly VECA consistently achieves better performance on dense prediction tasks. We believe this may be due to our extensive use of cores, which are functionally similar to registers and have been demonstrated to improve dense representations.

ModelImageNet-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Vanilla ViT-B79.2164.9628.320.6497
VECA-B78.9367.1029.330.6501

Q2) Performance and Efficiency at higher resolutions

We agree that accuracy and efficiency should be reported at higher resolutions. We therefore evaluate DINOv3-B and VECA-B at 1024px and 1120px under the same linear-probing protocol, where the backbone is frozen and only a linear prediction head is trained. VECA can still demonstrates competitive performance, despite never being trained at higher resolutions.

ModelResolutionImageNet-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
DINOv3768px84.0888.3252.370.3665
VECA768px82.8787.9451.460.3704
DINOv31024px83.6288.6051.950.3725
VECA1024px82.2288.4151.490.3769
DINOv31120px83.4288.4251.920.3728
VECA1120px82.1188.4351.590.3779

We benchmark the complete student forward pass compared to 3 parameter-matched baselines (85.73M) under the same setting on a single RTX PRO 6000 GPU. We used a batch size of 8, torch.inference_mode(), and BF16 torch.autocast. Latency was measured with CUDA events after 50 warm-up iterations, using 200 timed iterations in each of 3 rounds. We explicitly called torch.cuda.synchronize() after warm-up and before reading the timing events in each round to ensure that asynchronous CUDA execution was completed.

We use VECA with 64 active cores and a packed-QKV inference implementation. The median batch latency is reported over 600 measured forward passes. PolaFormer is our most competitive (accuracy) alternative linear attention model and ViT³ is the most recent linear-time baseline.

ModelResolutionLatency(ms)Throughput(img/s)
DINOv376839.207204.05
PolaFormer76889.35289.53
ViT³76854.392147.08
VECA-C6476829.234273.65
DINOv3102486.15592.86
PolaFormer1024144.56055.34
ViT³1024102.21378.27
VECA-C64102455.693143.65
DINOv31120103.73377.12
PolaFormer1120166.59648.02
ViT³1120119.93866.70
VECA-C64112065.227122.65

Q3) Comparison against CLS/Registers with VECA Routing

To address this question, we replace the original attention in pretrained DINOv3 with the core-mediated attention mechanism, using its CLS (or CLS + register) tokens as cores and bidirectional attention routing (like VECA). We then fine-tune the modified model for 50k steps with the same multi-resolution recipe as VECA, using the same DINOv3 teacher, distillation losses, training data, and downstream evaluation protocol.

Directly replacing the original attention leads to a substantial performance collapse. However, after 50k steps of fine-tuning, the modified model recovers significantly, e.g., reaching 88.1% of the original DINOv3's mIoU on VOC. These findings highlight the potential of core-mediated attention as an effective alternative to patch-to-patch attention.

Controlled ComparisonImageNet-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Pretrained DINOv383.5687.5051.350.3684
Pretrained DINOv3 with CLS-to-Patch Routing5.245.032.471.1239
Fine-tuned DINOv3 with CLS-to-Patch Routing72.8465.3627.390.7418
Pretrained DINOv3 with CLS/Reg-to-Patch Routing7.4213.256.260.9590
Fine-tuned DINOv3 with CLS/Reg-to-Patch Routing77.8577.0637.210.5216
VECA (C = 64)82.1887.0750.690.3705

Conclusion

We thank the reviewer for the helpful suggestions. In response, we have conducted additional experiments on training models from scratch, evaluated accuracy and efficiency at matched high resolutions (up to 1120px), and compared core-mediated routing using CLS/register tokens. We will include all of these results in the revision.

We are grateful for your clear and helpful comments. In light of these results and the positive evaluations from other reviewers, we hope you might view our work more favorably. Please feel free to let us know if you have any additional questions!

We summarize the key changes in our revised manuscript below:

  1. Page 3: we add a discussion and citation for ViT³. (foPV)
  2. Page 6: We clarify the role of core embeddings through quantitative comparisons and add additional results on Page 47 of the appendix. (TDwS, foPV)
  3. Page 34: We provide additional evaluation results for DINOv3-B and VECA-B at resolutions of 1024px and 1120px. (NkPS)
  4. Page 44: We present end-to-end inference efficiency comparisons across resolutions ranging from 256px to 1120px. (TDwS, foPV)
  5. Page 46: We add evaluation results of VECA distilled from TIPSv2 on image classification, semantic segmentation, and depth estimation.(TDwS, NkPS, foPV)
  6. Page 47: We add ablation results on the core design and analyze the contribution of each component. (TDwS, foPV)
  7. Page 48: We provide controlled comparisons against CLS tokens and Registers under VECA routing. (NkPS)

We appreciate your insightful suggestions! We believe that your feedback has helped us strengthen our work.

Reviewer_NkPSOpenReview ↗

Response to Rebuttal

Thanks for the detailed rebuttal. The from-scratch results address my concern about the distillation-only setup, and it's good to see VECA hold up without a full-attention teacher. Please add them to the paper, since they're not in the revised PDF yet.

On high resolutions, my concern was about datasets with substantially higher resolution, e.g. 2048px. Going from 1024px to 1120px adds only about 20% more tokens, and the speedup barely changes (1.55x to 1.59x). Also, VOC, ADE20K, NYUv2 and ImageNet are natively around 500px, so evaluating them at 1024px mostly tests upsampling, not high-resolution perception. Could you instead report frozen-backbone results for VECA and DINOv3 on Cityscapes [1] (native 2048x1024) and URUR [2] (resized to 2048x2048), together with end-to-end latency and peak memory? No retraining is needed for this.

On CLS/registers, the new experiment doesn't precisely address my question. It uses 1 or 5 tokens instead of 64 and fine-tunes DINOv3 for only 50k steps, so the gap to VECA mixes token count, initialization and training budget. What would really benefit the paper is an analysis where a ViT-B with 64 registers is trained with your exact recipe, once with masked patch-to-patch attention and once with full attention. That would show what VECA's core design adds beyond simply routing through registers. Table S.18 already hints that most of the gain comes from the routing itself, which is fine, but it should be stated clearly. I understand this needs full training runs, so the final version is fine for this.

Given the positive assessment of the other reviewers, I'm generally inclined to support acceptance, but I'd make this contingent on the two remaining points: the high-res evaluation and the register comparison in the final version.

[1] M. Cordts et al. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR 2016.

[2] D. Ji et al. Ultra-High Resolution Segmentation with Ultra-Rich Context: A Novel Benchmark. CVPR 2023.

Reviewer_TDwSOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VECA (Visual Elastic-Core Attention), a Vision Transformer architecture that replaces dense patch-to-patch self-attention with a core-periphery attention pattern. A small set of learned core tokens mediates global communication, while all patch tokens are preserved and updated throughout the network. This design reduces the attention interaction complexity from quadratic to linear in the number of patches for a fixed core budget. The method further incorporates learned core coordinates with 2D rotary positional embeddings and nested training over different core budgets, enabling inference-time trade-offs between computational cost and representation quality. The authors evaluate VECA on image classification, semantic segmentation, depth estimation, and cross-frame semantic label transfer. Results show competitive representation quality against full-attention vision backbones and improvements over the evaluated linear-complexity alternatives on most benchmarks. Ablation studies support the importance of persistent patch updates, core-patch interactions, and learned core coordinates. High-resolution efficiency measurements also demonstrate reduced attention-layer FLOPs and latency. Overall, the paper presents a promising architecture for efficient visual representation learning without direct all-to-all patch interaction.

Critical strengths
  1. A compelling architectural design that preserves dense spatial representations. VECA addresses an important limitation of latent-bottleneck approaches by retaining and iteratively updating all patch tokens while restricting global communication to a small set of learned cores. The combination of linear attention complexity and competitive dense prediction performance is consequential, particularly for high-resolution vision applications. The ablations provide meaningful evidence that persistent patch updates and core-patch interactions are important to the method's effectiveness.

  2. A useful combination of representation quality, computational efficiency, and elastic inference. The evaluation spans classification, segmentation, depth estimation, multiple resolutions, and different core budgets. The high-resolution latency measurements complement the theoretical complexity analysis, while nested-budget experiments show that one model can adapt its computational cost at inference time without substantial degradation at moderate budget reductions. Together, these results make the approach practically relevant beyond a purely theoretical reduction in attention complexity.

Critical weaknesses

The reported FLOPs and latency comparisons primarily isolate the attention projection-attention-projection path rather than establishing equivalent end-to-end acceleration for the complete model and downstream pipelines.

Moreover, the representation-learning experiments rely on distillation from a strong DINOv3 teacher trained on large-scale data. It is therefore unclear how much of the observed performance depends on the teacher's representations and whether the architectural advantage would persist under alternative training regimes.

These limitations do not negate the reported results, but they qualify the broader claims about VECA as a generally superior replacement for standard self-attention.

Questions for authors
  1. Clarification of architectural novelty: Could the authors more explicitly distinguish VECA from prior latent-mediated attention architectures, particularly in terms of information flow and the preservation of dense patch representations? Which component accounts for the largest performance improvement under matched parameter and computational budgets?

  2. Attribution of the core-mediated design: The ablations show that persistent patch updates and core-patch interactions are important. Can the authors provide further controlled evidence separating the contributions of learned core coordinates, dynamic 2D RoPE, and repeated core-patch communication? This would help establish which design choices are essential rather than complementary implementation details.

  3. End-to-end efficiency: How does VECA compare with full-attention baselines in end-to-end inference latency, peak memory consumption, and throughput at high resolutions, including the complete backbone and downstream prediction heads? Do the attention-layer speedups translate into comparable application-level gains?

  4. Dependence on teacher distillation: How sensitive are the results to the choice and scale of the teacher? Would VECA retain its advantages when distilled from a different pretrained vision model, or when trained with a less expensive supervision strategy? This would clarify whether the findings primarily reflect the attention architecture or its compatibility with DINOv3 representations.

3 comments

Thank you for your detailed suggestions and positive evaluation of our work! We address each question below and include all new results and clarifications in the revision.

Q1) Conceptual differences from prior latent-mediated attention architectures

We attribute our improvements to three primary reasons, starting from the most important:

  1. Bidirectional information passing from dense tokens to core (latent) tokens and from cores to dense tokens, within each layer.
    • The Perceiver family only passes information from dense patches into latent tokens in each cross-attention block.
    • For set transformers, the inducing tokens are unique to every layer and are not updated by the input.
  2. Continual MLP driven updates of both the dense patches and propagated cores across layers.
    • The Perceiver family refines only the latent tokens every layer. Perceiver IO must derive the dense tokens from this compact latent.
    • For set transformers, the inducing tokens are input independent and are not refined across layers.
  3. Learnable coordinates for each core that are updated throughout the network.
    • Neither Perceiver nor set transformers parameterize their latent tokens with explicit spatial coordinates. We show that learnable positions updated throughout the network improve accuracy slightly.

Our paper validated this by using DINOv3 as teacher and different architectures as student, using the same training recipe as VECA (with a best faith effort to optimize hyperparameters for each student).

Q2) Ablations of the Core Design

Because core tokens are not initially associated with image patches, they lack an inherent spatial reference. Our design assigns each core a learnable 2D coordinate used by RoPE. These coordinates are initialized using farthest-point sampling, are learnable, and evolve throughout the network layers as the cores specialize.

We conduct the following ablations:

  1. Learnable positional embeddings for both core and dense patch tokens.
  2. Random core coords used by RoPE without farthest-point initialization (fixed at start of training, later layers do not update coordinate).
  3. Random core coords used by RoPE with farthest-point initialization (fixed at start of training, later layers do not update coordinate).
  4. Layers can update coordinates with farthest-point initialization (FPS)
Core positional designIN-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Learnable position embedding81.4485.8649.730.3842
No learnable RoPE + no FPS81.7186.0349.800.3771
No learnable RoPE + FPS81.6186.9050.320.3740
Learnable RoPE + FPS (VECA)82.1887.0750.690.3705

Q3) Additional Efficiency Comparison

We benchmark the complete student forward pass compared to 3 parameter-matched baselines (85.73M) under the same setting on a RTX PRO 6000 GPU. We used a batch size of 8, torch.inference_mode(), and BF16 torch.autocast. Latency was measured with CUDA events after 50 warm-up iterations, using 200 timed iterations in each of 3 rounds. We explicitly called torch.cuda.synchronize() after warm-up and before reading the timing events in each round to ensure that asynchronous CUDA execution was completed. VECA uses 64 active cores and a packed-QKV inference implementation. The median batch latency is reported over 600 measured forward passes.

ModelResolutionLatency(ms)Throughput(img/s)
DINOv32564.8591646.36
PolaFormer2568.298964.11
ViT³25610.870735.96
VECA-C642566.1521300.37
DINOv351217.109467.58
PolaFormer51267.341118.80
ViT³51224.603325.16
VECA-C6451213.541590.81
DINOv3102486.15592.86
PolaFormer1024144.56055.34
ViT³1024102.21378.27
VECA-C64102455.693143.65

Q4) Choice of teacher model

Here are two additional experiments: (1) distilling VECA-B from a TIPSv2 instead of DINOv3.

Training StrategyIN-1K Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Pretrained TIPSv277.7185.1149.850.3882
Distill from TIPSv274.6984.6248.810.3971
Pretrained DINOv383.5687.5051.350.3684
Distill from DINOv382.1887.0750.690.3705

(2) Training VECA from scratch on ImageNet-1K classification supervision for 150 epochs, following the same training recipe with a global batch size of 1536. Surprisingly VECA achieves better performance on dense prediction tasks. We believe this may be due to our extensive use of cores, which are functionally similar to registers and have been demonstrated to improve dense representations.

ModelIN Top-1 Acc ↑VOC mIoU ↑ADE mIoU ↑NYUv2 RMSE ↓
Vanilla ViT-B79.2164.9628.320.6497
VECA-B78.9367.1029.330.6501

Conclusion

We appreciate the concrete feedback and positive evaluation of our work! All your suggestions will be included in our revision.

Here is a summary of changes in our revision:

  1. Page 3: we add a discussion and citation for ViT³. (foPV)
  2. Page 6: We clarify the role of core embeddings through quantitative comparisons and add additional results on Page 47 of the appendix. (TDwS, foPV)
  3. Page 34: We provide additional evaluation results for DINOv3-B and VECA-B at resolutions of 1024px and 1120px. (NkPS)
  4. Page 44: We present end-to-end inference efficiency comparisons across resolutions ranging from 256px to 1120px. (TDwS, foPV)
  5. Page 46: We add evaluation results of VECA distilled from TIPSv2 on image classification, semantic segmentation, and depth estimation.(TDwS, NkPS, foPV)
  6. Page 47: We add ablation results on the core design and analyze the contribution of each component. (TDwS, foPV)
  7. Page 48: We provide controlled comparisons against CLS tokens and Registers under VECA routing. (NkPS)

We thank you again for your suggestions! And believe our paper has been strengthened with your feedback.

Your reply addresses my concerns. I maintain the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Summary

We thank all reviewers for your detailed and constructive suggestions! We believe the paper has been substantially improved through your feedback.

We are encouraged by the reviewers' positive assessment of our work: A compelling architectural design (TDwS); VECA's useful for representation learning (NkPS); and VECA also represents a novel direction. (foPV).

The suggestions primarily focus on: (1) Adding measurements of VECA's real world computational efficiency. (2) Conducting ablation studies to elucidate the design choices of VECA's core-mediated architecture. (3) Additional experiments using a different teacher or exploring dense/classification accuracy at even higher resolutions.

1) General Summary of our Work

  1. We propose VECA, a visual backbone that replaces quadratic patch-to-patch self-attention with core-periphery structured attention that scales linearly in the number of patches for a fixed core budget.
  2. Unlike prior latent-compression architectures, VECA preserves and continuously updates the complete dense patch stream, while using a resolution-invariant set of cores as an efficient global communication interface.
  3. Through nested core-prefix training, VECA further transforms this communication interface into an elastic computation axis, enabling a single backbone to operate at multiple inference budgets without retraining.
  • We demonstrate that VECA approaches the DINOv3 teacher across classification and dense spatial tasks while providing scalable, elastic inference.
  • We believe VECA establishes core-periphery attention as a promising alternative to dense patch-to-patch self-attention and opens new directions for scalable vision backbones.

2) New Experiments in the Rebuttal and Paper Revision

Additional evaluation on training strategies

We add additional experiments to evaluate VECA's training strategies, including distillation from Google's TIPSv2 foundation model, as well as training from scratch on ImageNet-1K classification supervision. (TDwS, NkPS, foPV)

End-to-End efficiency evaluation

We add results measuring the latency and throughput the complete forward pass of of the VECA model at different resolutions. (TDwS, foPV)

Additional design choice ablations

We provide additional ablations clarifying the roles of learnable core coordinates, spatial initialization, and coordinate-based positional encoding. (TDwS, foPV)

Additional evaluation at higher resoulutions

we add further evalution results on classfication and dense prediciton tasks at higher resolutions than 1024px. (NkPS)

Comparison against CLS/Registers with VECA Routing

We add results on replacing the original attention in DINOv3 with VECA style bidirectional core-mediated attention routing. With metrics from direct pruning, and after fine-tuning using the full attention model. (NkPS)

3) Writing and Clarity

We appreciate the reviewers’ suggestions regarding writing, clarity, and related work. In the revision, we have expanded the discussion of closely related approaches, clarified our conceptual differences more precisely.

Conclusion

We genuinely appreciate the suggestions, and believe our paper will be improved with your feedback! The additional experiments and clarifications will be reflected in the final version as well.

Best,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.