- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AVQ-Attention-2, an improvement of AVQ-Attention that combines exact local attention with approximate global attention. Specifically, it uses cluster means as representatives rather than codewords, reducing the training complexity. Similar to sliding window attention, it also allows each query to attend exactly to the keys in the same local tile. The two designs are integrated into a shared softmax computation, with corrections to avoid double counting. Experiments on ImageNet-1K, ADE20K, and Diving-48 show its improved efficiency over AVQ-Attention and standard attention.
Critical strengths
- Intuitive improvement over AVQ-Attention.
- I notice that AVQ-Attention-2 uses two kinds of strategies to group tokens: clusters (similarity-based) and tiles (spatial-based local window), which is uncommon in other efficient attention designs. For example, in MoBA [1] and NSA, there are only blocks. Such combined granularities are more flexible, but incur new problems such as double counting. The authors explicitly notice this point and address it.
[1] MoBA: Mixture of Block Attention for Long-Context LLMs
Critical weaknesses
-
Insufficient comparisons with related work. The paper mainly compares against VQ-Attention, AVQ-Attention, and Standard Attention (FA2). More recent and similar work should be included. For example, Agent Attention [2] and Mixture-of-top-k attention [3] are also used on visual tasks with similar designs. MoBA and NSA can be excluded due to domain mismatch.
-
The ablation study is limited. For example, what is the performance if double counting is not dealt with?
I will raise my rating if the weaknesses are addressed.
[2] Agent Attention: On the Integration of Softmax and Linear Attention
[3] Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights
Questions for authors
- I notice that Agent Attention can be viewed as a form of VQ-Attention that uses cross-attention to replace k-means. In this way, VQ-Attention seems more burdensome and less flexible. Can the authors further clarify the unique advantage of VQ-Attention?
- From my viewpoint, MoBA combines global routing and local attention; NSA combines three branches: global routing, global compression, and local attention; Mixture-of-top-k attention combines global routing and global compression; and AVQ-2 combines global compression and local attention. Can the authors discuss the advantages and disadvantages of their different choices?
2 comments
Thanks for the clarification and experiments. Most of my concerns are resolved. I will raise my rating.
We thank the reviewer for the careful reading and for the interest in the design of our method. We appreciate the recognition that our specific token grouping, clusters by similarity and tiles by position, is uncommon in efficient attention designs. We have added the requested comparisons, and one extra method, to every experiment, as well as new ablations, and rewrote the related work section to cover these and more methods. Below we address the weaknesses and questions.
Comparisons with related work (Weakness 1). We have added Agent Attention [2], mixture-of-top-k attention [3] and MRA [5] to every experiment: ImageNet and ADE20K in Table 1 (all settings in Appendix C), video at 9,216 tokens in Figure 5 and at 18,432 tokens in Table 3. Each is run in its authors' implementation and trained under our recipe, where Agent Attention, which has new parameters, gets three times the training budget so it can converge. Our method is Pareto-optimal against all three in every experiment.
New ablations (Weakness 2). We ran the ablation the reviewer asks for: Appendix I (Table 10) evaluates every image configuration without the double-counting correction, and accuracy is lower in all of them. A local key that is counted both exactly and through its cluster mean gets too much weight, and the larger normaliser takes weight away from every key outside the tile. We also added an analysis of how a tile chooses which clusters to refine (Appendix J, Table 11), comparing attention mass, key variance and their product against a greedy reference: attention mass alone already reaches most of the achievable error reduction. Together with Table 2, which builds the layer up from AVQ-attention one component at a time, each design choice is now ablated.
VQ-attention and Agent Attention (Question 1). We agree with this view: in both methods, queries attend to a small set of representatives that summarise the keys. However, because in Agent Attention a query does not score the keys directly (it attends to a few agents, each of which has made its own softmax over the keys), it does not in fact approximate dense softmax attention, even in the limit, and is instead a generalized form of linear attention. VQ-attention with cluster means is actually still just softmax attention with each key replaced by an approximation that becomes better as the codebook grows, thus approaching the original dense attention. In fact, we see the original keys still being directly represented, as opposed to mixed into abstract agent contributions, as enabling more flexibility: it is what makes hierarchical refinement in the same softmax possible, as well as the combination with exact local attention, which requires knowing the contribution of the local tile in the first place. Furthermore, as the authors of Agent Attention note, like other linear attention it "suffers from insufficient feature diversity", which they restore with a depthwise convolution on the values, a learned module beside the attention. Finally, while training VQ-attention with a straight-through estimator and its extra hyperparameters is indeed cumbersome, the cluster-mean approach needs neither, as worked out in Appendix H.
Routing, compression and local attention (Question 2). We like this way of categorising the methods, and would add that AVQ-Attention-2 has a form of routing as well, which selects resolution, not membership. In [1], [3] and [4] a key is seen at one of two levels: exact if it is selected, and otherwise compressed, or in [1] not at all. Our layer has three: the keys of the local tile are exact, and for the rest each query tile selects which clusters to refine, while the others stay coarse. The intermediate level is cheaper than exact attention, and because the local keys are left out of the selection, the refinement budget is spent on the rest of the sequence.
A disadvantage is that distant keys are never attended exactly, only through their cluster. Another is that one tile serves two purposes: the queries in a tile share one selection of clusters to refine, and the same tile is the window of keys they attend exactly. For the selection a smaller tile is better, since fewer queries have to share it. For local attention a larger tile is better, since more keys are attended exactly. A local window larger than the query tile would decouple the two, but we have decided against this, as it makes the layer slower: keys have to be read more than once, as in NATTEN.