- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Direct Attention, an alternative to multiplication-based Softmax attention that replaces coordinatewise dot product with subtraction. The authors derive the formulation from equivalence to coordinatewise minimum, and performed experiments on ViT-Small on ImageNet-1K to show that under lightweight conversion training, the new Direct Attention can maintain most of the pretrained accuracy.
Critical strengths
- The proposed Direct Attention provides a new pathway for designing alternatives to multiplication based softmax attention.
Critical weaknesses
-
Limited experimental evaluation and insufficient presentation of results. The main paper provides a detailed evaluation of only one model on one task (ViT-S on ImageNet-1k) in Table 1. Although the authors mention additional experiments on MiniMind-V and NLLB-200, these results are only briefly discussed in text, without accompanying tables or detailed analysis in the main paper. The limited experimental evidence and insufficient presentation of results make it difficult to comprehensively assess the effectiveness and generalizability of the proposed method.
-
In sufficient ablation studies. Tab.1 is the only experiment present in the main paper, without any further ablation studies on design choice justifying why this particular form of direction attention is chosen. The authors should at least provide a breakdown ablation, as well as comparison with other multiplication-free attention variant methods (e.g. [1-2]) and justify why other potential design choices are inferior than the current form of direct attention.
-
The main methodology section (Section 3) is difficult to follow. In particular, the connection between the mathematical derivations and the motivation underlying the proposed design choices or why it is beneficial is still weak and hard to follow.
[1] You, Haoran, et al. "ShiftAddViT: Mixture of multiplication primitives towards efficient vision transformer." Advances in Neural Information Processing Systems 36 (2023): 33319-33337.
[2] Shu, Han, et al. "Adder attention for vision transformer." Advances in Neural Information Processing Systems 34 (2021): 19899-19909.
Questions for authors
- Does the improved memory for direct atttention translates to improved real-time efficiency gains?
We thank the reviewer for these comments. The comparisons below were completed after the submission deadline, and all of them will be included in the revised paper.
(Ablations and multiplication-free comparisons.) The difference from the closest prior score is a key bias, not a new similarity kernel. Since −max(0,q−k)=min(q,k)−q, the submitted score is row-softmax equivalent to coordinatewise minimum, and modulo query-only constants it equals 21∑rKjr−21∥Qi−Kj∥1. Negative-ℓ1 scoring, as used by Adder Attention and EcoTransformer, gives −∑r∣qir−kjr∣, which modulo the same query-only constant equals 2∑rmin(qir,kjr)−∑rKjr, that is 21∑rKjr−21∥Qi−Kj∥1 at half scale. The two coincide only when the coordinate sum of the key is constant across keys; otherwise Direct carries a key bias that the symmetric form lacks. That bias is the operator-level distinction, and the equal-budget ladder prices it: negative-ℓ1 reaches 81.006% against Direct's 80.114%, so on this protocol the bias does not buy accuracy. ShiftAddViT and EcoTransformer train multiplication-free primitives across a whole network rather than substituting the scalar score of a fixed checkpoint, so a score-level comparison against them would confound the primitive with the training budget and the trainable parameter set; we therefore separate the operator-level argument, which we test, from the cross-design comparison, which we do not claim.
Our score keeps only the positive part of each coordinate difference, Sij=−∑rmax(0,qir−kjr), so it records which of the two vectors is larger. The undirected alternative, −∑r∣qir−kjr∣, is symmetric in q and k and removes that distinction. A post-submission experiment interpolates between the two directions, Sij(λ)=−∑r[(1−λ)max(0,qir−kjr)+λmax(0,kjr−qir)], with λ=0 the submitted score, λ=1 its reverse, and λ=0.5 the symmetric midpoint that equals negative-ℓ1 at half scale. On Stable Diffusion UNet cross-attention at 10,000 COCO prompts, FID reads 194.453 for dot, 207.573 at λ=0, 237.499 at 0.25, 249.378 at 0.50, 233.690 at 0.75 and 228.849 at λ=1. The curve therefore has one peak and two better ends: the directed end is the best rectified setting, the reversed end is next, the symmetric midpoint is worst, and dot sits lowest. Relative to the dot anchor the directed end is 13.1 FID points away and the midpoint 54.9, so under this recipe the directed score is the closer of the two to dot.
(Reading Section 3.) The section is a chain. Proposition 1 shows the score is a representative of the coordinatewise-minimum class, so it is not a new similarity function. Proposition 2 turns both scores into a retrieval term plus a key bias, a coordinate sum for the submitted score and a squared norm for dot product, and the two-coordinate example shows the ranking consequence that follows.
(Memory and real-time efficiency.) Yes at the score unit, no end to end. At 16 bits the Direct score unit closes at 1.39 times the frequency of the dot unit, 92.42 against 66.45 MHz, each driven to its own maximum under the same flow, and it routes with 3,270 against 24,456 segments. It is also 8.67 times smaller in matched synthesis and 8.60 times smaller after routing, 20,958.0 against 180,283.0 μm2. Those runs have no power distribution network and no DRC-clean verdict. Whether that reaches real time is not measured, because it would need a chip-level clock and memory system: under the model the same numbers give a 0.42% total-area reduction and at most 1.46% fewer whole-image cycles, since the score stage is 5.2% of layer cycles. Memory itself is unchanged: the attention matrix and the key-value cache have the same size.
(Evidence breadth.) On the vision-language model the vision-encoder-only arm reads 1.853 evaluation cross-entropy against a 1.841 baseline, the decoder-only arm 2.409, and the whole-model replacement 2.416; Appendix M tabulates those arms with both splits. On the translation model every arm stays above the 1.6976 baseline and the encoder-only arm is the best at 1.9326; Appendix N, Table N.1, carries the complete arm record. Table 1 reports 80.782% against 81.392%, a difference of 0.610 points, and the equal-budget ladder puts Direct at 80.114% against 81.444% for same-data dot.