acceptodds
Under review as a conference paper at ICLR 2027

AVQ-Attention-2: Adaptive Vector-Quantized Attention with Exact Local Blocks and Linear-Cost Training

Abstract

Attention costs in the number of tokens. Vector-quantized attention reduces this to by clustering the keys, attending to the codewords instead, and its adaptive form, AVQ-attention, concentrates codebook resolution where a query's attention mass actually falls. We present AVQ-Attention-2, a better design that merges it with exact local attention and attends to the mean of each cluster's keys rather than to a codeword. We show three improvements: consistently better quality at comparable cost; training that now also costs rather than ; and an approximation accurate enough that the layer can be dropped into a pretrained transformer without fine-tuning. Across a variety of sequence lengths the result is competitive not merely with other approximate attention but with exact dense attention, running faster than FlashAttention at little to no loss in quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Efficient attention, Vector Quantization, Attention approximation, GPU kernels

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CzFAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AVQ-Attention-2, an improvement of AVQ-Attention that combines exact local attention with approximate global attention. Specifically, it uses cluster means as representatives rather than codewords, reducing the training complexity. Similar to sliding window attention, it also allows each query to attend exactly to the keys in the same local tile. The two designs are integrated into a shared softmax computation, with corrections to avoid double counting. Experiments on ImageNet-1K, ADE20K, and Diving-48 show its improved efficiency over AVQ-Attention and standard attention.

Critical strengths
  1. Intuitive improvement over AVQ-Attention.
  2. I notice that AVQ-Attention-2 uses two kinds of strategies to group tokens: clusters (similarity-based) and tiles (spatial-based local window), which is uncommon in other efficient attention designs. For example, in MoBA [1] and NSA, there are only blocks. Such combined granularities are more flexible, but incur new problems such as double counting. The authors explicitly notice this point and address it.

[1] MoBA: Mixture of Block Attention for Long-Context LLMs

Critical weaknesses
  1. Insufficient comparisons with related work. The paper mainly compares against VQ-Attention, AVQ-Attention, and Standard Attention (FA2). More recent and similar work should be included. For example, Agent Attention [2] and Mixture-of-top-k attention [3] are also used on visual tasks with similar designs. MoBA and NSA can be excluded due to domain mismatch.

  2. The ablation study is limited. For example, what is the performance if double counting is not dealt with?

I will raise my rating if the weaknesses are addressed.

[2] Agent Attention: On the Integration of Softmax and Linear Attention

[3] Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights

Questions for authors
  1. I notice that Agent Attention can be viewed as a form of VQ-Attention that uses cross-attention to replace k-means. In this way, VQ-Attention seems more burdensome and less flexible. Can the authors further clarify the unique advantage of VQ-Attention?
  2. From my viewpoint, MoBA combines global routing and local attention; NSA combines three branches: global routing, global compression, and local attention; Mixture-of-top-k attention combines global routing and global compression; and AVQ-2 combines global compression and local attention. Can the authors discuss the advantages and disadvantages of their different choices?
2 comments

We thank the reviewer for the careful reading and for the interest in the design of our method. We appreciate the recognition that our specific token grouping, clusters by similarity and tiles by position, is uncommon in efficient attention designs. We have added the requested comparisons, and one extra method, to every experiment, as well as new ablations, and rewrote the related work section to cover these and more methods. Below we address the weaknesses and questions.

Comparisons with related work (Weakness 1). We have added Agent Attention [2], mixture-of-top-k attention [3] and MRA [5] to every experiment: ImageNet and ADE20K in Table 1 (all settings in Appendix C), video at 9,216 tokens in Figure 5 and at 18,432 tokens in Table 3. Each is run in its authors' implementation and trained under our recipe, where Agent Attention, which has new parameters, gets three times the training budget so it can converge. Our method is Pareto-optimal against all three in every experiment.

New ablations (Weakness 2). We ran the ablation the reviewer asks for: Appendix I (Table 10) evaluates every image configuration without the double-counting correction, and accuracy is lower in all of them. A local key that is counted both exactly and through its cluster mean gets too much weight, and the larger normaliser takes weight away from every key outside the tile. We also added an analysis of how a tile chooses which clusters to refine (Appendix J, Table 11), comparing attention mass, key variance and their product against a greedy reference: attention mass alone already reaches most of the achievable error reduction. Together with Table 2, which builds the layer up from AVQ-attention one component at a time, each design choice is now ablated.

VQ-attention and Agent Attention (Question 1). We agree with this view: in both methods, queries attend to a small set of representatives that summarise the keys. However, because in Agent Attention a query does not score the keys directly (it attends to a few agents, each of which has made its own softmax over the keys), it does not in fact approximate dense softmax attention, even in the limit, and is instead a generalized form of linear attention. VQ-attention with cluster means is actually still just softmax attention with each key replaced by an approximation that becomes better as the codebook grows, thus approaching the original dense attention. In fact, we see the original keys still being directly represented, as opposed to mixed into abstract agent contributions, as enabling more flexibility: it is what makes hierarchical refinement in the same softmax possible, as well as the combination with exact local attention, which requires knowing the contribution of the local tile in the first place. Furthermore, as the authors of Agent Attention note, like other linear attention it "suffers from insufficient feature diversity", which they restore with a depthwise convolution on the values, a learned module beside the attention. Finally, while training VQ-attention with a straight-through estimator and its extra hyperparameters is indeed cumbersome, the cluster-mean approach needs neither, as worked out in Appendix H.

Routing, compression and local attention (Question 2). We like this way of categorising the methods, and would add that AVQ-Attention-2 has a form of routing as well, which selects resolution, not membership. In [1], [3] and [4] a key is seen at one of two levels: exact if it is selected, and otherwise compressed, or in [1] not at all. Our layer has three: the keys of the local tile are exact, and for the rest each query tile selects which clusters to refine, while the others stay coarse. The intermediate level is cheaper than exact attention, and because the local keys are left out of the selection, the refinement budget is spent on the rest of the sequence.

A disadvantage is that distant keys are never attended exactly, only through their cluster. Another is that one tile serves two purposes: the queries in a tile share one selection of clusters to refine, and the same tile is the window of keys they attend exactly. For the selection a smaller tile is better, since fewer queries have to share it. For local attention a larger tile is better, since more keys are attended exactly. A local window larger than the query tile would decouple the two, but we have decided against this, as it makes the layer slower: keys have to be read more than once, as in NATTEN.

  • [1] Lu et al., MoBA: Mixture of Block Attention for Long-Context LLMs, 2025.
  • [2] Han et al., Agent Attention, 2024.
  • [3] Wen et al., Mixture-of-Top-k Attention, 2026.
  • [4] Yuan et al., Native Sparse Attention (NSA), 2025.
  • [5] Zeng et al., Multi Resolution Analysis (MRA) for Approximate Self-Attention, 2022.

Thanks for the clarification and experiments. Most of my concerns are resolved. I will raise my rating.

Reviewer_Q6zROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AVQ-Attention-2, which combines adaptive attention over key-cluster means with exact local attention. Using cluster means enables linear-cost training for fixed codebook and refinement budgets, with fused Triton kernels providing an efficient implementation. Experiments on ImageNet, ADE20K, and Diving-48 evaluate drop-in replacement and fine-tuning, showing improved quality–speed trade-offs over AVQ-attention and attention-layer speedups over FlashAttention-2.

Critical strengths
  1. Computing cluster means from the input lets gradients reach the keys directly, removing AVQ-attention’s quadratic straight-through path and enabling linear-cost training for fixed codebook and refinement budgets.

  2. The fused kernel combines exact local attention with cluster-level approximation while correcting for double counting. Excluding local keys from the refinement scores helps direct the adaptive budget toward global context.

  3. Experiments cover image classification, segmentation, and video classification. Ablations show that cluster means and exact local attention help individually and work best together; drop-in results also improve substantially over AVQ-attention.

Critical weaknesses
  1. The paper omits closely related work, particularly SVG-EAR[1], which already combines exact attention with cluster-mean compensation under shared normalization. The novelty of the forward approximation should be distinguished from that of the trainable hierarchical formulation and its implementation.

  2. Refinement uses attention mass, which does not directly measure the error reduction obtained by replacing a parent mean with child means. Comparison with a lightweight error-aware refinement heuristic would help justify how the limited budget is allocated.

  3. The linear-cost training result concerns the attention operator, whereas fine-tuning uses per-layer distillation from a dense teacher whose attention cost remains quadratic. The paper should distinguish operator-level complexity from the cost of the adaptation pipeline; the reported timings do not establish linear-cost end-to-end adaptation.

  4. The quality-preservation claim is too broad. On Diving-48, drop-in replacement loses approximately 3.3–3.4 accuracy points at the two longer sequence lengths, and distillation still leaves a 2.79-point gap at 9,216 tokens. Near-lossless results should be attributed to specific configurations.

  5. Efficiency is evaluated through attention-layer timings on a single RTX 3090, leaving end-to-end throughput gains unverified. Matched-cost comparisons with related methods such as SVG-EAR are also missing, making their relative quality–speed trade-offs unclear.

[1] SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

Questions for authors
  1. How does AVQ-Attention-2 differ from SVG-EAR, which also combines exact attention with cluster-mean compensation under a shared softmax? If feasible, please compare the two in the drop-in V-JEPA 2 setting at matched attention cost.

  2. Would incorporating within-cluster key variance improve refinement over the mass-based score in Eq. (3)? For example, could you evaluate attention mass weighted by cluster variance as a simple heuristic, including the overhead of computing the statistics and routing?

  3. Could you report end-to-end inference speedups, at least for Diving-48 at the longest sequence length, and the total adaptation cost including the dense teacher used for distillation?

3 comments

We thank the reviewer for the careful reading and the concrete suggestions, which we have run as stated. We are glad the reviewer values the breadth of our experiments and ablations, and the improvement in drop-in quality. Below we address the weaknesses and questions; the comparison with SVG-EAR (Question 1) follows in a separate comment. In summary, we have rewritten the related work section, compared against SVG-EAR and three other efficient-attention methods, added the suggested variance-weighted refinement score, and now report whole-encoder speedups.

Other related work (Weakness 1). We agree, and invite the reviewer to read our rewritten related work section, which now covers SVG-EAR and many other methods. Besides the comparison with SVG-EAR (see our separate comment), we have added MRA [2], Agent Attention [3] and mixture-of-top-k attention [4] to every experiment of our submission, each in its authors' implementation and trained under our recipe. None of them matches our accuracy at our speed (Table 1, Appendix C, Figure 5, Table 3).

Table 3, encoder panel (Diving-48 at 18,432 tokens, every attention layer of the encoder replaced):

drop-indistilledms per layer
Dense88.83–15.9
AVQ-Attention-288.7388.220.94
Mixture-of-top-k73.1085.585.85
Agent Attention–82.544.02
MRA79.9086.907.53

Refinement criterion (Weakness 2, Question 2). We analyzed several criteria for refinement in Appendix J, and find that attention mass alone already captures 79-92% of the realizable error reduction over our 3 datasets. The mass × variance heuristic the reviewer suggests indeed improves on this, but only marginally, achieving 82-92% (Table 9). We also derive the form of the error of the cluster-mean approximation, finding that mass is a direct factor but that the role of variance is less straightforward than anticipated, which could clarify why it only gives small improvements. As using the statistic adds up to 7% to the time of the layer, we keep attention mass as the default criterion.

Table 9 (Appendix J), share of the realizable error reduction per criterion:

criterionImageNetADE20KV-JEPA 2
random0.2470.2290.279
key variance0.4860.6130.331
attention mass (ours)0.7890.8810.918
mass × key variance0.8150.9100.921
top-P by each parent's own reduction0.9810.9810.983
sequential reference1.0001.0001.000

Note that the bottom two criteria serve only as references: they need the exact attention output, so they cannot be used in practice.

Training cost (Weakness 3). We agree that the cost of the layer and the cost of distillation should be kept apart, and wish to clarify that our claim concerns only the former: we did not intend to claim that distillation from a dense teacher runs in linear time. Our efficient-training result is about training a model with this layer from scratch. Table 3 shows this case: the probe at 18,432 tokens is trained from scratch, at 11 ms per forward and backward pass against 74 ms for dense. Distillation, by contrast, takes a few epochs on top of a pretrained model whose own training took orders of magnitude more compute, so efficient training matters most where a model is trained from scratch.

Quality claims (Weakness 4). We agree that results should be attributed to their configurations, and now report a sweep of ten at 9,216 tokens (Figure 5, Appendix E). As a drop-in, accuracy depends mainly on codebook size. After distillation the gap the reviewer points to has closed for the larger budgets, and at 18,432 tokens the drop-in layer is level with dense attention. For these runs we use bf16 instead of fp16, turned weight decay off during fine-tuning, and present means over 3 runs.

Diving-48 top-1 and whole-encoder speedup (from Figure 5, Table 3 and Section 5.2):

denseours, drop-inours, distilledspeedup
9,216 tokens (ten configurations)86.582.1–85.785.0–86.41.9–2.3×
18,432 tokens88.888.788.23.5×

End-to-end speed (Weakness 5, Question 3). Over the whole encoder the speedup is 3.5× at 18,432 tokens (Section 5.2) and about 2× at 9,216 tokens, where Figure 5 now reports accuracy against whole-encoder speedup. The full distillation at 18,432 tokens takes about 9 GPU-hours on an RTX 3090 (model compute, without data loading), including the dense teacher.

  • [1] Zhou et al., SVG-EAR, 2026.
  • [2] Zeng et al., Multi Resolution Analysis (MRA) for Approximate Self-Attention, 2022.
  • [3] Han et al., Agent Attention, 2024.
  • [4] Wen et al., Mixture-of-Top-k Attention, 2026.

Relation to SVG-EAR (Question 1). This comment continues our reply above.

We ran SVG-EAR [1] in the V-JEPA 2 encoder, with the authors' code and their Wan 720p settings. The encoder then becomes four to eight times slower than with dense attention, so a comparison at matched attention cost is not possible there. The reason is that SVG-EAR runs k-means on the queries and keys in every call. In video diffusion that cost is amortised, as each denoising step starts from the clusters of the previous one; a model with a single forward pass has nothing to amortise over. The authors note as their main limitation that the method indeed has not been evaluated beyond diffusion transformers.

This is also a key difference between the two methods: in AVQ-Attention-2 the clustering is done during training (or calibration in case of drop-in), and what remains at inference can be viewed as the last step of a k-means whose other steps have already been paid for, which is what makes the layer usable in single-pass models. A second difference is in the scaling: SVG-EAR attends exactly on selected blocks between query clusters and key clusters, which costs O(ρN²), where ρ is the fraction of query–key pairs attended exactly, where ours is exact only on the local tile and costs O(MN) overall. As for the shared softmax: mathematically SVG-EAR indeed combines exact attention and cluster means under one softmax, but the two parts are computed by separate kernels, with the full output and normaliser written to memory in between, where our attention kernel accumulates both in one pass.

Interestingly, SVG-EAR's amortization of k-means inspired us to realize that AVQ-attention can benefit from it on diffusion too: by turning on our codebook training path during the diffusion steps, the codebook is overwritten (fully, with EMA decay 0), effectively performing k-means inside the diffusion steps just like SVG-EAR, but starting from a precomputed codebook. (Updating the codebook during generation is practically free, an estimated 1% overhead, as the cluster means are already computed by our VQ-precompute kernel.)

To test this and compare to SVG-EAR on its own task, we ran both on video generation with Wan2.2 5B, adapting the authors' setting to it as we could not run Wan 14B: 50 prompts from VBench, 50 denoising steps of which the first 10 are dense, as in SVG-EAR's own setup. For SVG-EAR we use the authors' fastest (Turbo) setting, with its cluster counts scaled to the 4.1× fewer tokens of the 5B model; their default setting runs no faster than dense here. Our layer (codebook 256/32/32, tile 128) is used as a drop-in: the model is not trained, the codebook is only calibrated once offline. PSNR, SSIM and LPIPS are against the dense model's videos; speedups are over dense on an RTX 3090.

PSNRSSIMLPIPSVBenchattention speedupgeneration speedup
Dense–––82.231×1×
SVG-EAR21.680.7230.28579.832.6×1.23×
Ours, fixed codebook21.060.6950.31280.413.9×1.32×
Ours, moving codebook21.200.6970.31680.213.9×1.32×

SVG-EAR stays somewhat closer to the dense videos, our layer scores slightly higher on VBench, and it is faster, both per attention layer and for the full generation. The generation speedups are small for both methods because the rest of the Wan network has a fixed cost. The moving codebook brings our videos slightly closer to the dense ones in PSNR.

We have not placed this in the paper, as the experiments section had no space left for an experiment of this size and the appendix does not seem the right place for it. However, we are thankful for the pointer, which could inspire future research whereby AVQ-attention's full potential on diffusion is investigated: currently we have only tested the config in the drop-in setting, but we expect it to benefit a lot from fine-tuning as well.

  • [1] Zhou et al., SVG-EAR, 2026.
Reviewer_Q6zROpenReview ↗

Official Comment by Reviewer Q6zR

Thank you for the rebuttal. I think the authors addressed my concerns well, and I am happy to raise my score to 3.

Reviewer_rCMKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AVQ-Attention-2, combining adaptive vector-quantized global attention with exact local attention for vision and video Transformers. Input-dependent cluster means enable linear-cost forward and backward computation under fixed budgets, while custom kernels efficiently combine local and global contributions. Experiments evaluate direct replacement and brief adaptation of pretrained models on image classification, segmentation, and video classification.

Critical strengths
  1. Promising accuracy–speed tradeoffs. AVQ2 approaches dense-attention accuracy after brief adaptation. At 18,432 video tokens, it achieves a 16.9× attention inference speedup with a 0.91-point accuracy decrease.

  2. Informative ablations and implementation analysis. Component ablations support the benefits of cluster means and exact local attention, while kernel benchmarks explain the efficiency gains from sorted aggregation.

Critical weaknesses
  1. Efficient-attention comparisons remain incomplete. Although the image experiments include VQ/AVQ and local-attention baselines, the long-sequence video experiments compare only against dense attention. This limits the evidence that AVQ2 offers a better accuracy–speed tradeoff than other efficient alternatives.
  2. The main technique components are similar to existing studies that are not discussed. Exact sparse part plus a corrected global approximation is a Scatterbrain decomposition (Chen et al., NeurIPS 2021). Fast Multipole Attention (Kang, Tran & De Sterck) and H-Transformer-1D both compute near interactions exactly and distant ones through coarse key and value summaries. The term exp(q⋅Kˉm)⋅nmexp(q·\bar{K}_m)·n_m equals exp(q⋅Kˉm+log⁡nm)exp(q·\bar{K}_m + \log n_m). This is exactly the "proportional attention" over merged tokens in ToMe (Bolya et al., 2023). The related work section and introduction should position these precedents, rather than presenting the combinations are new.
Questions for authors
  1. Could the authors extend the long-sequence video experiments to include the exact-local-only variant and representative efficient-attention baselines, under comparable adaptation budgets? This would clarify AVQ2’s accuracy–speed advantage beyond dense attention.
  2. How sensitive are the video results to codebook size and refinement budget? Can larger budgets recover the accuracy gap at 9,216 tokens while retaining meaningful speedups?
  3. How do you position the work relative to Scatterbrain, Fast Multipole Attention / H-Transformer-1D, MRA, ToMe's proportional attention, and the Routing Transformer?
4 comments

We thank the reviewer for the careful reading, and in particular for the pointers to related work we had not discussed, which led us to rewrite the related work section and to place our method more precisely in the field. We appreciate the recognition of the accuracy–speed trade-off of our method and the ablations and kernel analysis behind it. We have further added three efficient-attention baselines to every experiment and extended the video experiments. Below we address the weaknesses and questions.

Comparisons with other efficient-attention methods (Weakness 1, Question 1). We have added MRA [1], Agent Attention [2] and mixture-of-top-k attention [3] to every experiment, each run in its authors' implementation and trained under our recipe. None of them matches our accuracy at our speed. ImageNet and ADE20K results are in Table 1, with all settings in Appendix C.

For video at 9,216 tokens, Figure 5 now reports accuracy against whole-encoder speedup for the three baselines and the exact-local-only variant, as a drop-in and after distillation. At 18,432 tokens (Table 3) our drop-in layer is level with dense attention (88.7 against 88.8), where the best baseline reaches 79.9. After distillation the baselines come closer, at 4–8× our cost per attention layer.

Sensitivity to codebook config (Question 2). We replaced the single configuration at 9,216 tokens by a sweep over codebook size, refinement budget and tile size: ten configurations, shown in Figure 5 and listed in Appendix E. For these runs we made some small changes to the code for our video experiments: we use bf16 instead of fp16, we found turning weight decay off during fine-tuning helps across the board, and we present means over 3 runs. All video results were re-run to ensure our method and the baselines share one setup.

The results are mainly sensitive to codebook size under the drop-in setting: accuracy rises from 82.1 with the smallest codebook to 85.7 with the largest, against 86.5 for dense. After distillation the differences get much smaller (all results lie between 85.0 and 86.4), with in particular larger budgets indeed recovering the gap w.r.t. the original model, while still running twice as fast (full encoder throughput).

Positioning against related work (Weakness 2, Question 3). We agree, and invite the reviewer to read our rewritten related work section. We view many of the mentioned works as part of a broader family of efficient attention whereby a query attends to a representative of a group of keys, and acknowledge this is indeed a widely used idea, and so is combining it with exact attention; we did not intend to claim either. (Besides ToMe, the exp⁡(qKˉm) nm\exp(q \bar{K}_m)\, n_m term is indeed also found in [4,5,6,7], and was not intended as a contribution, but just a mathematical necessity.)

We wish to clarify that our contributions are focused on VQ-attention in particular, which we view as a sub-family of this broader family. Our rewritten results section can now also be viewed as covering 3 distinct categories: 1) Exact local attention, comparing our tiling with Swin and NATTEN. 2) The general efficient attention family, comparing to [1,2,3]. 3) The VQ-attention sub-family, comparing to [5,8]. VQ-attention forms its groups from the content of the keys, but does the clustering during training: at inference only an assignment pass remains, which can be tiled like FlashAttention. Of the works the reviewer names, some instead group by position, or pay extra at inference to build and read their representatives. AVQ-attention adds a hierarchical codebook that each query tile refines adaptively, still inside that tiled pass. Our method brings exact local keys into the same pass. For an adaptive method, when implemented properly, this also does more than add exact attention: the local keys no longer need refinement, which frees that budget for the rest of the sequence. AVQ-Attention-2 combines in one layer several ideas from the field: exact local attention, combining coarse and fine representatives, content-based routing, memory-efficient tiling and adaptive inference. Many methods indeed cover some of these; to our knowledge few combine all of them as efficiently.

  • [1] Zeng et al., Multi Resolution Analysis (MRA) for Approximate Self-Attention, 2022.
  • [2] Han et al., Agent Attention, 2024.
  • [3] Wen et al., Mixture-of-Top-k Attention, 2026.
  • [4] Kong et al., GOAT: A Global Transformer on Large-scale Graphs, 2023.
  • [5] Lingle, Transformer-VQ, 2024.
  • [6] Hooper et al., Multipole Attention for Efficient Long Context Reasoning, 2025.
  • [7] Alonso et al., Online Vector Quantized Attention, 2026.
  • [8] van den Dool et al., AVQ-Attention, 2026.
AuthorsOpenReview ↗

The new results referred to in our reply above, copied here from the revised paper for convenience.

Table 1: other efficient-attention methods on ADE20K and ImageNet-1k, as a drop-in and after fine-tuning under the same recipe. ms is the time of one attention layer on an RTX 3090. All settings of each method are in Appendix C.

ADE20K settingmsdrop-in mIoUfine-tuned mIoUImageNet settingmsdrop-in top-1fine-tuned top-1
FlashAttention-2–0.28045.8846.54–0.18085.82–
MRAb=81.56631.5545.88b=81.04183.6485.32
Agent Attention15×150.576–44.647×70.183–77.27
Mixture-of-top-k7×72.16221.3846.177×72.13979.9584.76
AVQ-Attention-232/8/80.17844.1746.7132/8/80.12584.7585.60

Table 3: Diving-48 at 18,432 tokens (V-JEPA 2 ViT-B/16 encoder with an attentive probe). Left: every attention layer of the encoder replaced. Right: the encoder left dense and the probe's attention replaced and trained from scratch.

Encoderdrop-indistilledms per layer
Dense88.83–15.9
AVQ-Attention-288.7388.220.94
Mixture-of-top-k73.1085.585.85
Agent Attention–82.544.02
MRA79.9086.907.53
Probe, trained from scratchtop-1fwd + bwd ms
Dense88.8374.3
AVQ-Attention-290.0511.1
Mixture-of-top-k89.4916.5
Agent Attention89.4446.2
MRA86.7515.1

Figure 5: Diving-48 at 9,216 tokens, the values of the plotted points. Top-1 against the speedup of the whole encoder over the original one (dense: 86.51). Our ten configurations, by codebook M0/P/C and query-tile size:

AVQ-Attention-2tile× densedrop-indistilled
32/8/8642.3282.1085.69
64/8/8642.2783.7785.89
32/8/81282.2582.7885.00
64/16/8642.2283.7885.69
64/8/81282.2183.6385.94
128/8/8642.1884.8285.53
64/16/81282.1483.7185.28
128/8/81282.1284.0486.09
128/16/16642.0385.6285.89
128/16/161281.9085.6586.35

The baselines on the same figure (Table 7 in Appendix E):

Methodsetting× densedrop-indistilled
MRA2 blocks per row1.4460.6180.86
MRA4 blocks per row1.2966.1982.84
MRA8 blocks per row1.0577.1685.18
Agent Attention32 agents1.94–65.03
Agent Attention128 agents (2×8×8)1.82–62.59
Agent Attention128 agents (8×4×4)1.80–70.46
Agent Attention196 agents1.69–73.15
Agent Attention256 agents1.65–71.62
Agent Attention512 agents1.40–77.72
Mixture-of-top-k32 routers1.5420.0055.63
Mixture-of-top-k128 routers1.3151.5280.51
Mixture-of-top-k256 routers0.9975.8983.40

Thank you for your response to my review. I have read it and will take it into account in my final assessment. It seems the paper has not be revised during this period. Could the author provide a revision plan?

Thank you for reading our response and getting back to us. The paper has already been revised: the PDF on this page is the new version. The main changes to look for:

  • Section 2 (related work) is rewritten, and in particular now includes a second paragraph "Attention over group representatitives".
  • Table 1 (sec 5.1) and Table 3 (sec 5.2) are new, comparing against MRA, Agent Attention and mixture-of-top-k attention on images and on video at 18,432 tokens.
  • Figure 5 (sec 5.2) is new: all methods on video at 9,216 tokens, with a sweep of ten of our configurations (details in Appendix E: Figure 6 and Table 7).
  • Appendices C, I and J hold the remaining new results and ablations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.