acceptodds
Under review as a conference paper at ICLR 2027

Shift-Accumulate Attention: Multiplier-Free Query-Key Products for Transformer Decoding

Abstract

Power-of-two quantisation turns a multiplication into a bit shift, but it has so far been applied primarily to the post-softmax attention–value product. We reformulate the earlier and larger score product, , by quantising the key cache to a signed power-of-two fixed-point representation, so that every scalar multiplication becomes a sign flip, a shift, and an integer accumulation. We call the resulting datapath ShiftAtten. A fixed-point headroom condition makes every shift a non-negative left shift, rendering the accumulation exact with respect to the quantised operands. A mantissa-extended power-of-two family then trades additional shift–add operations for accuracy across 4- to 8-bit representations, while a shift-exact online softmax extends the same treatment to the attention–value product. Fused CUDA kernels for a 1.1 B-parameter Llama decoder on an RTX 4090 reach the throughput of FP16 scaled dot-product attention at a 32k context length with a smaller KV cache. We separate the gain into from the narrower representation and from kernel engineering. On zero-shot reasoning, the 8-bit representation achieves percent versus percent for FP16, while at 4 bits it loses only percentage points compared with points for uniform INT4, despite worse perplexity and tensor-level error. A cost model calibrated by kernel disassembly shows that the remaining bottleneck is instruction issue rather than memory bandwidth: current GPUs provide a packed four-way integer multiply–accumulate but no corresponding packed shift primitive, and PTX scalar shift–accumulate is emulated. We therefore propose DS4A, a packed four-way shift-dot-accumulate instruction. Under the same kernel structure, DS4A is projected to reach FP16 throughput, compared with for an optimised INT8 MAC kernel, because the power-of-two representation moves 101 bytes per cached token versus 133 bytes for INT8.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Transformer inference, attention, power-of-two quantisation, fixed-point arithmetic, KV cache, CUDA kernels, hardware-software co-design.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KhCMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Shift-Accumulate Attention (ShiftAtten), a hardware-software co-designed attention kernel for LLM autoregressive decoding. The core idea is to quantize Key vectors into a signed power-of-two (PoT) format. Under this quantization, scalar multiplications in QK dot products are converted into bitwise shifts, sign flips, and integer accumulation, removing the need for multiplication units. The authors further propose DS4A, a hypothetical 4-way packed shift-dot-accumulate hardware primitive, inspired by the GPU’s __dp4a. They evaluate the attention kernel latency via simulation, showing reduced memory footprint from PoT KV quantization with controllable accuracy degradation.

Critical strengths
  1. The paper proposes a co-design between PoT key quantization and a dedicated hardware primitive DS4A. By encoding K values as signed power-of-two numbers, QK multiplication is replaced with cheaper shift-add operations, which eliminate multipliers inside the attention computation.

  2. The PoT quantization family (PoT-4, PoT-M1 to PoT-M4) provides flexible trade-offs: fewer bits reduce KV cache memory bandwidth, while additional mantissa bits recover model quality. Hadamard preprocessing is also introduced to mitigate quantization error.

  3. The hardware design of DS4A follows the existing GPU packed-instruction paradigm (__dp4a), making the proposed primitive conceptually easy to compare against the commodity GPU ISA. The analysis distinguishes two bottleneck regimes: issue-bound on current GPUs without DS4A, and memory-bound when DS4A is available.

Critical weaknesses
  1. Baseline comparisons are insufficient. The paper mainly compares against vanilla full-precision attention. It lacks comprehensive benchmarking against widely adopted KV compression/quantization baselines such as INT4/INT8 KV quantization, PagedAttention, and StreamingLLM, making it hard to quantify its advantages over existing state-of-the-art inference optimizations.

  2. The simulation adopts relatively idealized assumptions. System-level interference, including kernel launch overhead, cache contention, and multi-task preemption, is largely omitted. Real-world deployment will likely yield lower speedup than the simulated numbers.

  3. End-to-end evaluation is limited to a small 1.1B model. Although the paper provides real RTX4090 end-to-end throughput and perplexity for TinyLlama, there is no end-to-end measurement on larger models. It remains unclear whether the observed end-to-end benefits generalize to bigger LLMs, where MLP/GEMM layers occupy a larger fraction of total inference time and dilute the relative gain from attention optimization.

Questions for authors
  1. How does your PoT KV quantization compare quantitatively against existing KV compression schemes (e.g., INT4/INT8 KV, SWA, or sparse KV) under the same memory budget and model setup?

  2. Your end-to-end results are only demonstrated on TinyLlama-1.1B. Could you provide end-to-end throughput, latency, and perplexity for larger models? As model size grows, MLP layers dominate the total runtime, which may significantly reduce the overall end-to-end speedup from attention-only optimization.

Reviewer_EsVFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents ShiftAtten, which represents cached keys with signed power-of-two codes and computes query–key products via integer shifts and additions. The authors implement custom kernels and use an instruction-based cost model to project the benefit of a proposed packed shift instruction, which they refer to as DS4A. Experiments show speed gains over FP16 attention, but the implemented INT8 kernel remains faster, and end-to-end quality evaluation is limited to TinyLlama-1.1B.

Critical strengths
  • Restricting computation within attention to strictly shifts and adds is an interesting problem.
  • The paper provides implemented kernels and evaluates numerical error, downstream accuracy, and throughput.
  • The proposed method is demonstrated to provide performance improvement over FP16.
  • The end-to-end analysis considers the interaction with Hadamard rotations, although not for the baselines.
Critical weaknesses
  • The presentation makes the proposal and method unnecessarily difficult to follow. The roles of the key objects in the argumentation (e.g., "fixed-point headroom") should be explained before the derivation. Notation should also be defined when it is first introduced (as two of many examples, F>=emaxF >= e_{max} in the abstract are undefined, and the variables in Equation 1 leave one to assume the notation is the same as the one in Vaswami et al.)
  • There are grammatical issues; for two of many examples, "To first order" in line 243 is not a sentence and there is an extra period in line 065
  • The reference format is incorrect. For example, "Vaswani et al. (2017)" should be "(Vaswani et al., 2017)" and that is the difference between \cite and \citep{}.
  • The technical advance over established power-of-two and shift-based methods remains insufficiently justified. Section 2 explains the distinction from binary attention but does not establish why this extension is a substantial methodological contribution.
  • The headline speed comparison emphasizes FP16, although the paper’s INT8 kernel is faster throughout Table 5. No dedicated INT4 decoding kernel is evaluated. The results therefore do not establish a speed advantage over a competitive implementation at the same key bit width. Additionally, one would expect on a modern GPU to evaluate against MXFP4, which is accelerated on recent HW.
  • The DS4A proposal lacks hardware validation. The claim that the proposed unit is cheaper than the existing alternative needs supporting evidence.
  • The evaluation is narrow. End-to-end results cover one 1.1B model on one GPU, while the 8B results cover only tensor-level errors from five layers. These are substantial limits on the conclusions, even though the paper acknowledges them.
Questions for authors
  • What technical contribution, beyond applying established shift arithmetic to multi-level key representations, distinguishes this method from prior work?
  • How does ShiftAtten compare with an optimized INT4 or MXFP4 decoding kernel under the same workload, quantization granularity, and accuracy requirements?
  • What evidence supports the assumed DS4A throughput and the claim of lower hardware cost?
  • Why is a cost model necessary to locate a bottleneck? Profilers exist that can locate instruction-level bottlenecks. What is the gap that the cost model addresses?
Reviewer_2xJ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper (ShiftAtten) makes the query-key score product S = QK^T multiplier-free by quantizing the KV key cache to a signed power-of-two (PoT) code, so each scalar product reduces to a sign flip, a bit shift, and an integer add. A fixed-point head-room condition F >= E forces every shift to be a non-negative left shift, which makes the score accumulation exact with respect to the quantized operands (stronger than the usual truncation bound). A mantissa field (the PoT-Mk family, 4-8 stored bits) trades extra shift-adds for accuracy, with 8-bit PoT-M4 being iso-storage with an INT8 key cache; an integer-log online softmax optionally extends shifts to the AV product.

Critical strengths

First, the paper applies an exact and multiplier-free reformulation to the larger of the two attention products. Earlier shift-based work handled only the post-softmax AV product, whereas this paper targets QK^T, which dominates decoding. The head-room condition makes the accumulation exact with respect to the quantized operands, instead of letting a naive right shift discard low bits as it accumulates. Choosing the right product is what makes the method worth doing. Second, the measurements can be checked independently. The cost model is built on disassembled kernels, and the 4.60x is separated into 2.53x from the representation and 1.81x from kernel engineering, so the engineering gain is not folded into the method. There is a bit-exact reference implementation. The authors also report, as a measured observation, that the tensor-level proxies and task accuracy disagree at 4 bits, without claiming a mechanism.

Critical weaknesses

First, on the hardware the authors actually test, the method is neither the fastest nor the most accurate option. The tuned INT8 MAC kernel reaches 4.94x, above the best measured 4.60x of ShiftAtten. The 4-bit configuration, which should be the most attractive because the cache is smallest, has the worst perplexity, at 9.30 for PoT-4 against 8.39 for INT4 and 7.88 for FP16. Adding Hadamard rotation brings PoT-4 down to 8.51, which is still above both. The configuration that matches FP16, PoT-M4, is 8-bit, takes the same storage as INT8, and costs up to five shift-adds per term. The 6.51x that would actually beat INT8 depends on the DS4A instruction, which no shipping GPU provides. The present benefit therefore comes down to a smaller KV cache at 4 to 5 bits, and it is paid for in perplexity. Second, the 4-bit accuracy result, which carries the paper, is validated only at d=64. All end-to-end numbers come from the 1.1B, d=64 TinyLlama, while the d=128 Llama-3.1-8B appears only as ε_S and top-8 in Table 4. The difficulty is that on the 1.1B tensors the two proxies rank PoT-4 below INT4, with a score error of 0.130 versus 0.076 and a top-8 overlap of 0.777 versus 0.845, while zero-shot accuracy ranks it above, at -0.94 versus -2.68 points. The 8B proxy numbers therefore cannot support the claim that the 4-bit task-accuracy advantage holds at d=128. Computing perplexity and zero-shot accuracy at d=128 does not require the fast kernel, which is built only for d=64, since the existing bit-exact reference is sufficient, so this gap can be filled.

Questions for authors
  1. Can the authors report end-to-end perplexity and zero-shot accuracy for at least one 7 to 8B model at d=128, using the bit-exact PyTorch reference? This requires only a correct implementation and not the fast kernel, and throughput can come later.
  2. The drop-in 8-bit numbers in the abstract, 61.61% and perplexity 7.896, are actually the PoT-M4 plus Hadamard row, while plain PoT-M4 gives 7.904. Can the abstract state that the drop-in claim relies on Hadamard rotation, and can the paper report the runtime cost of rotation and whether it is already included in the throughput and KV-size figures?
  3. For the 4-bit inversion, the paper gives only the average zero-shot delta. Can it also give the per-task deltas and the spread across the five layers it has already captured, and does the paper have a hypothesis for why log-spaced codes help task accuracy while hurting epsilon_S and perplexity?
  4. The paper describes d=64 as an implementation limit. At d=128 with PoT-M4 and F=10, does the score accumulator still fit in INT32 without changing the tile size N, or is support for d=128 more than a recompile?
  5. On the measured tensors, how often does a negative rho trigger the truncating right-shift fallback, and how much does it contribute to ε_S? The paper calls this fallback benign, but this is asserted rather than quantified.
  6. Table 6 shows no value for INT4 KV throughput because no dedicated kernel was written. Since INT4 is the accuracy competitor at 4 bits, can the paper provide at least an estimate, or explain why a fair INT4 decode kernel is out of scope?
  7. Table 4 reports D_KL for the 1.1B tensors but not for the 8B ones. Can the paper explain the omission and add the values?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.