acceptodds
Under review as a conference paper at ICLR 2027

Attention Score without Multiplication

Abstract

Attention needs a query–key interaction, but that interaction need not be a dot product. We introduce Direct Attention, , which replaces coordinatewise products with subtraction, rectification, and accumulation while retaining softmax and value aggregation. We derive its exact row-softmax equivalence to coordinatewise minimum, its -distance-plus-key-bias form, and the conditions under which simpler separable scores lose query dependence. A budget-bounded ImageNet-1K conversion of a pretrained ViT-Small updates only Q/K weights and keeps every other parameter fixed. The resulting Direct checkpoint reaches 80.782% top-1 accuracy, compared with 81.392% for the frozen dot-product reference, a difference of −0.610 percentage points under identical official preprocessing. This is one adaptive development trajectory with repeated validation use. In a separate matched Sky130 HD synthesis study, the 16-bit Direct score unit occupies 0.115 times the dot-unit area, giving a Dot/Direct ratio of 8.67. The width sweep links this saving to the different arithmetic primitives; the SRAM-inclusive modeled total area falls by 0.42%. Together, the results identify query–key arithmetic as a practical co-design choice: a rectified difference supports accurate checkpoint conversion and a substantially smaller score unit, with system-level value determined by the surrounding memory and computation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
direct attention, multiplication-free attention, query-key scoring, vision transformers, efficient attention

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_bX7aOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Direct Attention, an alternative to multiplication-based Softmax attention that replaces coordinatewise dot product with subtraction. The authors derive the formulation from equivalence to coordinatewise minimum, and performed experiments on ViT-Small on ImageNet-1K to show that under lightweight conversion training, the new Direct Attention can maintain most of the pretrained accuracy.

Critical strengths
  1. The proposed Direct Attention provides a new pathway for designing alternatives to multiplication based softmax attention.
Critical weaknesses
  1. Limited experimental evaluation and insufficient presentation of results. The main paper provides a detailed evaluation of only one model on one task (ViT-S on ImageNet-1k) in Table 1. Although the authors mention additional experiments on MiniMind-V and NLLB-200, these results are only briefly discussed in text, without accompanying tables or detailed analysis in the main paper. The limited experimental evidence and insufficient presentation of results make it difficult to comprehensively assess the effectiveness and generalizability of the proposed method.

  2. In sufficient ablation studies. Tab.1 is the only experiment present in the main paper, without any further ablation studies on design choice justifying why this particular form of direction attention is chosen. The authors should at least provide a breakdown ablation, as well as comparison with other multiplication-free attention variant methods (e.g. [1-2]) and justify why other potential design choices are inferior than the current form of direct attention.

  3. The main methodology section (Section 3) is difficult to follow. In particular, the connection between the mathematical derivations and the motivation underlying the proposed design choices or why it is beneficial is still weak and hard to follow.

[1] You, Haoran, et al. "ShiftAddViT: Mixture of multiplication primitives towards efficient vision transformer." Advances in Neural Information Processing Systems 36 (2023): 33319-33337.

[2] Shu, Han, et al. "Adder attention for vision transformer." Advances in Neural Information Processing Systems 34 (2021): 19899-19909.

Questions for authors
  1. Does the improved memory for direct atttention translates to improved real-time efficiency gains?
1 comment

We thank the reviewer for these comments. The comparisons below were completed after the submission deadline, and all of them will be included in the revised paper.

(Ablations and multiplication-free comparisons.) The difference from the closest prior score is a key bias, not a new similarity kernel. Since −max⁡(0,q−k)=min⁡(q,k)−q-\max(0,q-k)=\min(q,k)-q, the submitted score is row-softmax equivalent to coordinatewise minimum, and modulo query-only constants it equals 12∑rKjr−12∥Qi−Kj∥1\tfrac12\sum_r K_{jr}-\tfrac12\lVert Q_i-K_j\rVert_1. Negative-ℓ1\ell_1 scoring, as used by Adder Attention and EcoTransformer, gives −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|, which modulo the same query-only constant equals 2∑rmin⁡(qir,kjr)−∑rKjr2\sum_r\min(q_{ir},k_{jr})-\sum_r K_{jr}, that is 12∑rKjr−12∥Qi−Kj∥1\tfrac12\sum_r K_{jr}-\tfrac12\lVert Q_i-K_j\rVert_1 at half scale. The two coincide only when the coordinate sum of the key is constant across keys; otherwise Direct carries a key bias that the symmetric form lacks. That bias is the operator-level distinction, and the equal-budget ladder prices it: negative-ℓ1\ell_1 reaches 81.006% against Direct's 80.114%, so on this protocol the bias does not buy accuracy. ShiftAddViT and EcoTransformer train multiplication-free primitives across a whole network rather than substituting the scalar score of a fixed checkpoint, so a score-level comparison against them would confound the primitive with the training budget and the trainable parameter set; we therefore separate the operator-level argument, which we test, from the cross-design comparison, which we do not claim.

Our score keeps only the positive part of each coordinate difference, Sij=−∑rmax⁡(0,qir−kjr)S_{ij}=-\sum_r\max(0,q_{ir}-k_{jr}), so it records which of the two vectors is larger. The undirected alternative, −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|, is symmetric in qq and kk and removes that distinction. A post-submission experiment interpolates between the two directions, Sij(λ)=−∑r[(1−λ)max⁡(0,qir−kjr)+λmax⁡(0,kjr−qir)]S_{ij}(\lambda)=-\sum_r[(1-\lambda)\max(0,q_{ir}-k_{jr})+\lambda\max(0,k_{jr}-q_{ir})], with λ=0\lambda=0 the submitted score, λ=1\lambda=1 its reverse, and λ=0.5\lambda=0.5 the symmetric midpoint that equals negative-ℓ1\ell_1 at half scale. On Stable Diffusion UNet cross-attention at 10,000 COCO prompts, FID reads 194.453 for dot, 207.573 at λ=0\lambda=0, 237.499 at 0.25, 249.378 at 0.50, 233.690 at 0.75 and 228.849 at λ=1\lambda=1. The curve therefore has one peak and two better ends: the directed end is the best rectified setting, the reversed end is next, the symmetric midpoint is worst, and dot sits lowest. Relative to the dot anchor the directed end is 13.1 FID points away and the midpoint 54.9, so under this recipe the directed score is the closer of the two to dot.

(Reading Section 3.) The section is a chain. Proposition 1 shows the score is a representative of the coordinatewise-minimum class, so it is not a new similarity function. Proposition 2 turns both scores into a retrieval term plus a key bias, a coordinate sum for the submitted score and a squared norm for dot product, and the two-coordinate example shows the ranking consequence that follows.

(Memory and real-time efficiency.) Yes at the score unit, no end to end. At 16 bits the Direct score unit closes at 1.39 times the frequency of the dot unit, 92.42 against 66.45 MHz, each driven to its own maximum under the same flow, and it routes with 3,270 against 24,456 segments. It is also 8.67 times smaller in matched synthesis and 8.60 times smaller after routing, 20,958.0 against 180,283.0 μm2\mu\mathrm{m}^2. Those runs have no power distribution network and no DRC-clean verdict. Whether that reaches real time is not measured, because it would need a chip-level clock and memory system: under the model the same numbers give a 0.42% total-area reduction and at most 1.46% fewer whole-image cycles, since the score stage is 5.2% of layer cycles. Memory itself is unchanged: the attention matrix and the key-value cache have the same size.

(Evidence breadth.) On the vision-language model the vision-encoder-only arm reads 1.853 evaluation cross-entropy against a 1.841 baseline, the decoder-only arm 2.409, and the whole-model replacement 2.416; Appendix M tabulates those arms with both splits. On the translation model every arm stays above the 1.6976 baseline and the encoder-only arm is the best at 1.9326; Appendix N, Table N.1, carries the complete arm record. Table 1 reports 80.782% against 81.392%, a difference of 0.610 points, and the equal-budget ladder puts Direct at 80.114% against 81.444% for same-data dot.

Reviewer_wW3jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Attention, replacing query-key dot products with rectified differences while preserving softmax attention. It provides mathematical equivalences, evaluates Q/K-only checkpoint conversion on ImageNet-1K, and reports hardware-area reductions through RTL synthesis.

Critical strengths

S1. Simple and hardware-oriented design. The proposed scoring mechanism is straightforward and provides an interesting alternative to multiplication-based attention.

S2. Careful experimental reporting. The paper presents detailed checkpoint-conversion protocols, mathematical analyses, and hardware synthesis results, with transparent discussion of limitations.

Critical weaknesses

W1. Limited Methodological Novelty.

The proposed scoring function is closely related to existing distance-based and multiplication-free attention methods, such as Adder Attention and EcoTransformer. The mathematical equivalences largely follow from elementary identities, making the fundamental novelty insufficiently convincing.

W2. Limited Experimental Validation and Baseline Comparisons.

The main evaluation focuses on a single ViT-Small checkpoint and one adaptive training trajectory, with a 0.61-point accuracy drop. Comparisons with alternative distance-based scoring functions and matched training baselines are missing.

W3. Insufficient System-Level Efficiency Improvements.

Despite the substantial score-unit area reduction, the modeled SRAM-inclusive total area decreases by only 0.42%. The paper lacks end-to-end latency, energy, and throughput measurements, leaving the practical hardware benefits unclear.

W4. Limited Generalization Across Architectures and Tasks.

The transfer experiments show substantially degraded performance in causal decoding and cross-attention settings. This raises concerns about whether the proposed scoring mechanism can generalize beyond relatively small vision classification models.

Questions for authors

Q1. What fundamentally distinguishes Direct Attention from Adder Attention and EcoTransformer? Could the authors provide controlled comparisons with negative-L1 attention and other multiplication-free scoring methods?

Q2. Can the authors provide multi-seed experiments and matched Dot/Direct training controls to verify whether the observed accuracy gap is consistent and attributable to the scoring mechanism?

Q3. Can the authors report end-to-end latency, energy consumption, and throughput under matched precision, including softmax, value aggregation, and memory overhead?

Q4. Why does Direct Attention exhibit substantial degradation in causal and cross-attention settings? Can the authors demonstrate its effectiveness on larger models and more diverse downstream tasks?

2 comments

We thank the reviewer for these comments. The comparisons below were completed after the submission deadline, and all of them will be included in the revised paper.

(W1, Q1) The trained comparison. Our score keeps only the positive part of each coordinate difference, Sij=−∑rmax⁡(0,qir−kjr)S_{ij}=-\sum_r\max(0,q_{ir}-k_{jr}), so it records which of the two vectors is larger. The undirected alternative, −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|, is symmetric in qq and kk and removes that distinction. A post-submission experiment interpolates between the two directions, Sij(λ)=−∑r[(1−λ)max⁡(0,qir−kjr)+λmax⁡(0,kjr−qir)]S_{ij}(\lambda)=-\sum_r[(1-\lambda)\max(0,q_{ir}-k_{jr})+\lambda\max(0,k_{jr}-q_{ir})], with λ=0\lambda=0 the submitted score, λ=1\lambda=1 its reverse, and λ=0.5\lambda=0.5 the symmetric midpoint that equals negative-ℓ1\ell_1 at half scale. On Stable Diffusion UNet cross-attention at 10,000 COCO prompts, FID reads 194.453 for dot, 207.573 at λ=0\lambda=0, 237.499 at 0.25, 249.378 at 0.50, 233.690 at 0.75 and 228.849 at λ=1\lambda=1. The curve therefore has one peak and two better ends: the directed end is the best rectified setting, the reversed end is next, the symmetric midpoint is worst, and dot sits lowest. Relative to the dot anchor the directed end is 13.1 FID points away and the midpoint 54.9, so under this recipe the directed score is the closer of the two to dot. Each of the six settings is its own Q/K-LoRA arm trained under the same 5,000-step budget, rank 4 with 64 tensors and head-dimension scaling, so the symmetric form was trained exactly like the directed one and still came out worst.

(W2, Q2) Matched baselines and seeds. Three Direct seeds read 80.114%, 80.180% and 80.184% against 81.444% for the plain dot arm, so the gap is 1.26 to 1.33 points. The controlled comparison is a ladder of equal-budget arms, not the paper's primary endpoint, and the two must not be conflated. In the ladder each arm adapts only Q and K for 19,237 steps in fp16-mixed precision, and each cell is a single seed. Direct reads 80.114%, the negative-ℓ1\ell_1 score reads 80.842% at learning rate 10−410^{-4} and 81.006% at 10−310^{-3}, a dot score trained on the same data reads 81.444%, and a dot self-distillation control reads 81.490%. The two negative-ℓ1\ell_1 rows are one implementation at two learning rates, not two operators, because the Adder Attention score and negative-ℓ1\ell_1 both compute −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|. The paper's Table 1 reports 80.782% for the reported Phase-3 endpoint against 81.392% for the frozen reference, which is a different checkpoint from a longer adaptive trajectory. Both readings point the same way: on this protocol dot is stronger, and neither multiplication-free alternative closes the gap.

(W3, Q3) System-level efficiency. At 16 bits the Direct score unit is 8.67 times smaller in matched synthesis and 8.60 times smaller after routing and static timing, 20,958.0 against 180,283.0 μm2\mu\mathrm{m}^2, with 3,270 against 24,456 routed segments, and it closes at 1.39 times the dot unit's frequency, 92.42 against 66.45 MHz, each driven to its own maximum under the same flow; those runs have no power distribution network and no DRC-clean verdict. Holding SRAM and every other block fixed, the modeled ledger gives a 0.42% total-area reduction and 1.46% fewer whole-image cycles at best, because the score stage is 5.2% of layer cycles.

(W4, Q4) Generalisation and causal decoding. Both causal studies in the paper report a gap concentrated in the configurations that contain causal decoder attention, and both state that they do not identify masking as the cause: on the vision-language model the decoder-only arm reads 2.409 evaluation cross-entropy against 1.841 for the baseline and 1.853 for the vision-encoder-only arm, and on the translation model every arm stays above the 1.6976 baseline, with the encoder-only arm the best at 1.9326. The tests run since submission separate precision from mechanism. Across fp32, fp16 and bf16 the causal arm stays within 0.012 nat, at 4.677, 4.676 and 4.671, while the dot control reads 3.089 and an untrained identity swap reads 4.691, so reduced precision is not the mitigation; a 6,000-step Q/K-only retrain recovers 0.013 nat. The one mechanism we can demonstrate follows from the score being bounded above by zero: setting future positions to zero instead of minus infinity puts them at the maximum of the row, so future-position attention mass rises from 0.0 to 319.47 and mean negative log-likelihood to 9.53, against 7.46 and 3.07 for dot under the same convention. Whether the degradation is intrinsic to the score or an artefact of the conversion protocol is unresolved, subjecting to future work.

Reviewer_wW3jOpenReview ↗

Official Comment by Reviewer wW3j

Thank you for your response. The rebuttal portion addressed my concerns. However, apart from the methodology and experiments, I do not believe the paper's readability and completeness meet ICLR's acceptance standards. Therefore, I am keeping my score unchanged.

Reviewer_kLbaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Direct Attention, a multiplication‑free query‑key scoring function based on ReLU rectified differences. It provides mathematical analysis showing row‑softmax forward equivalence to coordinate‑minimum scoring. A Q/K‑only adapted ViT‑Small achieves 80.782% top‑1 on ImageNet‑1K (‑0.61% lower than dot‑product baseline). In circuit comparison, the results shows area reduction for the isolated Q‑K score unit.

Critical strengths

Investigating the attention mechanism is meaningful. The paper designs a new calculation of attention score.

Critical weaknesses
  1. The improvement is negative. The proposed method incurs a noticeable accuracy drop. And the practical value is unclear. The paper mentioned that the score-unit area can be largely reduced. However, the total-area reduction is about 0.42% since the attention module only accounts for a tiny fraction of the overall model compute/area budget. The large score‑unit area saving does not translate to large whole‑system benefits. Without energy/timing results, it is hard to assess the real hardware practicality of Direct Attention, yet hardware benefit is a central highlighted point of this work.

  2. The paper does not compare against other attention optimizations (e.g., sparse attention, Krause attention) that preserve or improve accuracy, making the advantage of the proposed method hard to identify.

  3. The readability of the paper is low. The appendix suffers from messy organization. Many experimental plots and tables are presented without sufficient descriptive context. Plenty of experiments are revealing unclear arguments in a way that is difficult to understand. Moreover, Appendices E, F, J, P, Q, and R are not mentioned anywhere in the manuscript. To improve readability, the authors are suggested to reorganize appendix contents, add captions/elaborations for each experiment settings and the investigating goals, and explicitly link every Appendix to relevant discussions in the main text, or delete them if they are unnecessary.

Overall, the paper has a simple new formulation of attention mechanism. The theoretical analysis is extensively conducted. However, the superiority of the proposed method to the previous works is unclear. Considering the weak model performance, confused experiments and unclear investigating goals, and low readability presentation, I vote for a clear rejection. I recommend the authors to simplify the experiments, concentrate resources to further optimize the performance of the method, and quantify the actual hardware advantages in specific scenarios, which will greatly enhance the solidity of this work.

Questions for authors

see weakness above

2 comments

We thank the reviewer for these comments. The comparisons below were completed after the submission deadline, and all of them will be included in the revised paper.

(Formulation.) Only the query–key score changes. Softmax, the value product, and every other layer stay as they are, so the paper's contribution is to explain what this substitution does to the similarity computation, not to introduce a new similarity function.

(The accuracy drop.) The paper's Table 1 endpoint is 80.782% top-1 against 81.392% for the frozen reference, a difference of 0.610 points, and we do not argue it away. What the review could not see at submission is a matched ladder in which the dot score is adapted under exactly the same intervention as the Direct arm: in that ladder each arm adapts only Q and K for 19,237 steps in fp16-mixed precision, and each cell is a single seed. Direct reads 80.114%, the negative-ℓ1\ell_1 score reads 80.842% at learning rate 10−410^{-4} and 81.006% at 10−310^{-3}, a dot score trained on the same data reads 81.444%, and a dot self-distillation control reads 81.490%. The two negative-ℓ1\ell_1 rows are one implementation at two learning rates, not two operators, because the Adder Attention score and negative-ℓ1\ell_1 both compute −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|. The ladder therefore makes the reference stronger, not weaker: the frozen checkpoint is not a matched ceiling, and the gap survives a dot arm that was adapted in the same way. Both readings point the same way: on this protocol dot is stronger, and neither multiplication-free alternative closes the gap.

(Score-unit saving versus the system budget.) The arithmetic in the review is right, and the advantage is larger than area alone. At 16 bits the Direct score unit occupies 20,784.9344 μ\mum$^2$ against 180,135.264 for the dot unit, a ratio of 8.67, and the recorded width sweep widens it as operands grow: 2.58 at 4 bits, 4.75 at 8, 8.67 at 16 and 16.2 at 32. Routing the same two units and timing them at one corner gives 20,958.0 against 180,283.0 μ\mum$^2$, a ratio of 8.60, with 3,270 against 24,456 routed segments, and the smaller unit also closes at 1.39 times the frequency, 92.42 against 66.45 MHz, each driven to its own maximum under the same flow; those runs have no power distribution network and no DRC-clean verdict, so the timing is a unit-level result rather than a signoff. The system ledger that holds every other block and the SRAM constant turns the area ratio into a 0.42% total-area reduction and, under the model's cycle breakdown, into 1.46% fewer whole-image cycles at best, because the score stage is 5.2% of layer cycles and only that stage speeds up. The reason is visible in the ledger: 22M weights at 16 bits occupy 222.6 mm$^2$ of the modeled die, and the score stage is 23.5% of the dot design's modeled logic against 3.4% of the Direct design's. We also implemented the design in which the score and the matrix-multiply path share one multiplier array, which is the obvious way to convert the local saving into a system saving: it costs 9.0% more area than the two units instantiated separately, so it does not.

(Other attention optimisations.) Sparse, low-rank and kernel attention change which pairs are computed, so they compose with any scalar score rather than compete with it. The comparison the review asks for is within the multiplication-free score family, and we ran it in a second setting: six Stable Diffusion UNet arms, one per point of the interpolation between the directed score and its reverse, each trained separately as a Q/K-LoRA adapter for 5,000 steps at rank 4 with 64 tensors and head-dimension scaling, and each evaluated on 10,000 COCO prompts with 50 DDIM steps at guidance 7.5 and seed 0. FID reads 194.453 for dot, 207.573 at the directed end, 237.499 at 0.25, 249.378 at the symmetric midpoint, 233.690 at 0.75 and 228.849 at the reversed end. The symmetric midpoint is exactly the negative-ℓ1\ell_1 score at half scale, and it received the same budget as ours, so the comparison is matched: the multiplication-free form the review names comes out worst of the six rather than better. The directed end sits 13.1 FID points from the dot anchor and the midpoint 54.9, so of the two rectified directions the submitted one is the closer to dot.

(Readability, appendices, and scope.) The review is right that Appendices E, F, J, P, Q and R are never referenced. They are supporting records, not claims: P, Q and R report further attempts, including the no-mask causal variants, while E, F and J report audits. The revision adds a guide that states each appendix's purpose, opens each of the six with the question it answers; no reported result is replaced, and every added comparison carries its protocol beside it.

I have read the rebuttal. My concerns are not addressed. The rebuttal is low readability, just like the content of the manuscript. I keep my original rating.

Reviewer_5D3WOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a method for multiplication-free attention scoring, motivated by reducing the chip area on a non-programmable inference engine. It proposes a rectified difference sum between q-k pairs, gives a recipe to fine-tune existing vision models to support it, and presents the accuracy and hardware area savings those provide.

Critical strengths

S1. The proposed rectified scoring is simple, distinct from previous methods, and clearly explained and motivated in the equations presented. In particular, the authors do a great job of showing how the rectified score reduced to a coordinate minmum and how it is different from other implementations of multiplication-free scoring.

S2. The hardware study attached is a strong demonstration of the method functioning and the gains associated. The authors went so far as running bit-matched simulations against reference to collect the area numbers and ratios for the comaprison.

S3. The study carefully details its experiments, their details, scopes, and limitations. It very clear about how it reuses the validation set and doesn't do a training only split for finetuning the model. It is clear about exlcuding timing and power numbers from its hardware details. It is clear about the two parallel lines of inquiry it is running for the hardware and inquiry. Further, the appendicies contain a lot of the detail making the details for reproduction clear.

While I commend the study on recognizing and clearly stating the gaps and weakness in some of its methods and test, some of them are still crucial to address with results and abalations rather than just acknowledgement.

Critical weaknesses

The study is quite long, so it is possible that I missed something or misunderstood something, especially if it was in the appendix. Please do guide me to the right path if the issues have already been addressed in the original manuscript.

W1. Accuracy and Fine-tuning Method: The authors themselves concede that during the fine-tuning there is "repeated validation use" making room for bias in the results and mention that they have not done but recognize "training-only selection split" as a solution. While it could be the case that the bias introduced is small, that is not checked by any experiment. This gap is furthere made severe since the authors focus heavily on the resulting accuracy.

W2. Range of Abalations: Continuing from the last point, the authors do not present a sufficient range of abalations to place their method against comparions. There are no suitable comparisons or abalations like:

  • A baseline where the dot-product score is also trained on the same data as the rectified direct score
  • Comparisions to other multiplication free scoring strategies discussed, that are also finetuned in the same as way the direct attention (to my understanding the comaprison in the appendix is direct inference test with no training)

W3. Area effciency and accuracy aren't cross-studied sutiably: The accuracy numbers presented come from training and doing inference in bf16 but the hardware area numbers come from fixed-point implementations. Further, it is unclear if the dot area is reduced for comaprison with 4, 8 bit fixed point scoring unit comparison. Conceretely the paper is missing:

  • Accuracy of the set-up with a fixed-point scoring unit or conversely the hardware area coverage of a fp unit
  • Comparison with the dot product at 4, 8, 16 bits (not just the direct unit) in the same type (floating or fixed point)

W4. The paper could stand to be more explicit about it's hardware study being focused on a fixed non-prorgammable inference engine that has separate dot units for the attention score and the other layers of the network. In particular since the energy or timing are not studied, being very clear about how much further narrow the results are in the main (not just the appendix) is an important limitation to highlight against the claims.

Questions for authors

These questions function in addition to the weaknesses above

Q1. Have the authors studied the training cost of fine-tuning the direct attention score comapred to the dot-product score ? Considering GPUs with TensorCores are much more capable with the dot-products than just float math - how much of an added cost do the authors expect there ?

Q2. Have the authors done any accuracy tests with fixed-point quantizations or any trainings with fixed-point q-k pairs for that matter ?

Q3. Have the authors considered an inference chip design where the dot unit is shared between the attention score and later linear layers ? How much overhead (in area and in timing) might that add compared to going with a separate direct scorer or a separate dot product scoring unit ?

Q4. Do the authors have an understanding of why this method struggles in causal / auto-regressive decoding set-ups ? Does changing to fp32 instead of bf16 in those cases make a difference ?

2 comments

We thank the reviewer for these comments. The comparisons below were completed after the submission deadline, and all of them will be included in the revised paper.

(W2) Directed versus undirected score. Our score keeps only the positive part of each coordinate difference, Sij=−∑rmax⁡(0,qir−kjr)S_{ij}=-\sum_r\max(0,q_{ir}-k_{jr}), so it records which of the two vectors is larger. The undirected alternative, −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|, is symmetric in qq and kk and removes that distinction. A post-submission experiment interpolates between the two directions, Sij(λ)=−∑r[(1−λ)max⁡(0,qir−kjr)+λmax⁡(0,kjr−qir)]S_{ij}(\lambda)=-\sum_r[(1-\lambda)\max(0,q_{ir}-k_{jr})+\lambda\max(0,k_{jr}-q_{ir})], with λ=0\lambda=0 the submitted score, λ=1\lambda=1 its reverse, and λ=0.5\lambda=0.5 the symmetric midpoint that equals negative-ℓ1\ell_1 at half scale. On Stable Diffusion UNet cross-attention at 10,000 COCO prompts, FID reads 194.453 for dot, 207.573 at λ=0\lambda=0, 237.499 at 0.25, 249.378 at 0.50, 233.690 at 0.75 and 228.849 at λ=1\lambda=1. The curve therefore has one peak and two better ends: the directed end is the best rectified setting, the reversed end is next, the symmetric midpoint is worst, and dot sits lowest. Relative to the dot anchor the directed end is 13.1 FID points away and the midpoint 54.9, so under this recipe the directed score is the closer of the two to dot.

(W2, W1) ImageNet comparison. The controlled comparison is a ladder of equal-budget arms, not the paper's primary endpoint, and the two must not be conflated. In the ladder each arm adapts only Q and K for 19,237 steps in fp16-mixed precision, and each cell is a single seed. Direct reads 80.114%, the negative-ℓ1\ell_1 score reads 80.842% at learning rate 10−410^{-4} and 81.006% at 10−310^{-3}, a dot score trained on the same data reads 81.444%, and a dot self-distillation control reads 81.490%. The two negative-ℓ1\ell_1 rows are one implementation at two learning rates, not two operators, because the Adder Attention score and negative-ℓ1\ell_1 both compute −∑r∣qir−kjr∣-\sum_r|q_{ir}-k_{jr}|. The paper's Table 1 reports 80.782% for the reported Phase-3 endpoint against 81.392% for the frozen reference, which is a different checkpoint from a longer adaptive trajectory. Both readings point the same way: on this protocol dot is stronger, and neither multiplication-free alternative closes the gap.

(W1) Selection. Checkpoint selection is now separate from evaluation for the arms that use it. The split is a disjoint 50,000-image subset of the training set, and we report both readings: negative-ℓ1\ell_1 at 10−310^{-3} scores 81.006% on official validation and 81.114% on the selection split, 16-bit quantization-aware training scores 80.124% and 80.182%, and the 8-bit arm scores 80.062% and 80.174%. This removes the evaluation split from the selection decision.

(W3, Q2) Quantized accuracy and matched-width area. Quantizing the score-unit operands and the score matrix to 4, 8, 16 and 32 bits with per-tensor symmetric scaling, keeping accumulation in fp32, and evaluating on all 50,000 validation images gives 76.158, 80.122, 80.116 and 80.122 for Direct and 78.204, 81.434, 81.448 and 81.448 for Dot. Against each arm's own fp32 anchor of 80.114 and 81.444, 16 bits is lossless to within 0.002 and 0.004 points, and the 4-bit collapse is a width effect present in both arms. These are accuracy measurements only: no datapath is claimed to execute that arithmetic, and no throughput, area or energy claim follows. We also trained with fake quantization of the query-key operands at the target width: quantization-aware training reads 80.124% at 16 bits and 80.062% at 8 bits, and those rows differ in bit width, device architecture and code revision, so their difference is not a width effect alone.

(Q3, W4) Shared hardware unit. The advantage is area and timing together. At 16 bits the matched synthesis comparison is 20,784.9344 μ\mum$^2$ for the Direct score unit against 180,135.264 for Dot, a ratio of 8.67, and the recorded width sweep widens it as operands grow: 2.58 at 4 bits, 4.75 at 8, 8.67 at 16 and 16.2 at 32. Taking the same two units through detailed routing and static timing at one corner gives 20,958.0 against 180,283.0 μ\mum$^2$, a ratio of 8.60, with 3,270 against 24,456 routed segments, and the smaller unit also closes at 1.39 times the frequency, 92.42 against 66.45 MHz, each unit driven to its own maximum under the same flow. What the advantage does not reach is the system: holding SRAM and every other block fixed, the modeled ledger falls by 0.42%, because the weights occupy most of the die. Sharing one multiplier array between the score and the matrix-multiply path does not pay either, since it synthesises to 218,978.77 μ\mum$^2$ against 200,920.19 for the two units instantiated separately, 9.0% more area, and its routed 9.0033 MHz is set by that implementation's ripple reduction rather than by sharing.

(W1) The added numbers address the validation split concern. Just as a follow up, what data-type are the 16 and 8 bit numbers reported here ?

(W2) I appreciate the clarification and urge the authors to highlight their finding that "on this protocol dot is stronger, and neither multiplication-free alternative closes the gap" in the main paper

(W3) I'm sorry if I missed it, but are these numbers for the fixed point implementations assumed for the area calculations or are these instead floating point and if so what is exact datatype (E vs M) ?

(W4) The original manuscript, if my memory serves correctly does not go into timing. Indeed this analysis would make the utility of the presented method stronger, but it was not highlighted. What the original manuscript does do in its very abstract is present the accuracy numbers from using bf16 and the area numbers from using a fixed point implementation. While the paper acknowledges this later on, these central numbers do not correspond to each other and that concern of mine still stands

Comments on the paper

1 comment
Area_Chair_k154OpenReview ↗

Reviewer-author discussion period is ending. Please feedback.

Dear Reviewers,

Thanks for your time and effort in reviewing ICLR2027 submissions. As the reviewer-author discussion period will end very soon, please take time to read the author’s rebuttal and give your feedback. Your continuous engagement is highly appreciated.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.