acceptodds
Under review as a conference paper at ICLR 2027

FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward

Abstract

Autoregressive decode repeatedly streams a growing KV cache, making attention a major cost at long context. Existing high-performance kernels use online softmax, which discovers a row's normalization reference as it scans keys. Earlier contributions therefore remain provisional and may require rescaling. We argue that the reference need not be discovered: softmax is invariant to a common shift, so the reference only has to keep the weights in range. We present FoldAttention, an additive formulation of softmax attention that fixes a finite reference before scanning the KV cache. Each weight is then final when computed, so contributions add across disjoint key ranges and their quotient equals softmax attention in real arithmetic. We use this property to develop two techniques for Hopper decode: (1) final weights gate key and value reads before the bytes are fetched, and a per-call depth cuts keys below while keeping their mass, and (2) additive partials compose split KV and shared-prefix cascades without rescaling. On H100 at , FoldAttention decodes seven real-model generations 1.36-2.30 faster than the fastest BF16 baseline, and up to 3.09 faster across MHA and GQA shapes, at an error within 1.5% of the lowest BF16 error on six of the seven; reading every key, it is 1.14-1.30 faster at matched error. We validate on Qwen3-8B that a whole decode step is up to 1.46 faster while likelihood and long-context accuracy match those under BF16 kernels. The same principle makes the backward deterministic: CTAs round bounded partial gradients onto an integer grid declared before the reduction and add them in any order. FoldAttention thereby removes the determinism tax: its deterministic backward is up to 1.84 faster than deterministic FlashAttention-3/4 and 1.05 faster than the fastest nondeterministic kernel.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
efficient attention, long-context inference, autoregressive decoding, KV-cache optimization, FlashAttention, GPU kernels, deterministic training, reproducible reductions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Uc5KOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a novel kernel implementation for attention calculations for Hopper architectures. In decode, it uses an additive formulation to calculate a fixed reference instead of updaing the reference online during softmax calculation. In prefill, thread blocks round the the partial gradients to the nearest grid to achieve determinism without reducing the performance of backward kernel.

Critical strengths
  • The paper addresses an important problem, the performance of kernels in attention calculations. I especially believe that developing deterministic kernels without performance penalties is an important but often overlooked direction, which this paper addresses.
Critical weaknesses
  • This paper is difficult to follow and understand. It feels like the manuscript is heavily generated by AI end-to-end, with little effort from the authors. Figures are not clear and lack in-text explanations. The quality of writing is not up to the standards of ICLR.

  • There is no explanation where the speedup comes from. There is no profiling data, trace timeline, performance breakdown to back up the claims that the speedup is coming from the novelties presented in the paper, not from implementation differences.

  • Method is implemented and evaluated only for Hopper. There is no evidence that this method would generalize to other more recent architectures.

Questions for authors

See the weaknesses above.

1 comment

We thank the reviewer, in particular for the point on deterministic kernels. We address the three weaknesses in turn.

W1. Writing and figures.

We rewrote the paper. The introduction now states the problem (exact kernels already stream the cache at 94% or more of measured bandwidth, so a faster decode must read fewer bytes), the two obstacles, and how each construction removes one. Every figure is explained in the text where it is cited, and each caption says what to read from it. Figure 2 works one example: four keys in two splits, with and without the declared reference. Figure 3 numbers the decode kernel's steps, and Section 4 describes them in that order. We removed internal terms and moved detail to the appendix.

W2. Where the speedup comes from, with profiling, traces and a breakdown.

The revision adds four kinds of evidence, all on the same model batches.

  • Bytes across all kernels. Figure 1a plots decode time per key against DRAM bytes per key for every kernel. Time follows bytes, and the fastest BF16 kernels already stream at 94% or more of the measured bandwidth from 16K keys up.
  • Hardware counters. Table 14 gives Nsight Compute's DRAM bytes per key, DRAM throughput and issue rate for our main kernel and FlashAttention-3. The speedup equals the byte ratio times the bandwidth ratio, for example 2.70 x 0.83 = 2.23 on Llama-3.1-8B. Our kernel reads 2.24 to 2.70 times fewer bytes and issues at 52 to 73% of slots against FlashAttention-3's 24 to 47%. At 16 rows a head and at d = 64, issue rather than bandwidth limits it, and Appendix F.2 gives instruction counts per tile.
  • Phases and traces. Table 15 splits the decode into the setup kernel, the main kernel and the tail. A trace build whose output matches the production kernel bit for bit stamps each tile and shows median data waits of 0 to 160 ns.
  • Same-kernel ablation. To separate the method from the implementation, we run our own kernel with each mechanism removed (Figure 5c). Reading every key at 16 bits, our kernel runs at 0.90 to 0.97x the fastest BF16 kernel, so the implementation alone gives no speed. The speed appears only when the precision levels read fewer bytes.

For the backward, Figure 4b runs our kernel with each way to add partials (ordered FP32, FP32 atomics, integer grid) under each tile order. The integer grid costs 2.6% over FP32 atomics, and the longest-first order it allows saves 15%. Table 21 and Figure 11 give in-kernel traces of each variant. Ordered adds leave the store warp waiting for its turn, and shortest-first order rereads queries and output gradients from DRAM (12.5 GB against 4.4 GB). So the speed over deterministic kernels comes from the tile order that only an order-free reduction can use. Against nondeterministic kernels the backward is as fast, and we claim no more.

W3. Hopper only.

Appendix F.9 runs the decode on a GeForce RTX 5070 Ti (Blackwell, sm_120) through the kernel's generic path, with no tuning for that GPU. FoldAttention reads 0.35 to 0.50 of the fastest baseline's bytes and decodes 1.21 to 2.20x faster, at 1.005 to 1.007 of the error of the most accurate exact kernel there. The mechanisms need integer tensor-core products, integer reductions to global memory and bulk copies, which recent NVIDIA GPUs have. The speed argument depends only on decode being bandwidth-bound, which holds on every recent GPU.

Reviewer_LsuxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes FoldAttention, an attention framework that adopts a predefined normalization reference to replace the dynamic maximum used in online softmax. Attention contributions from different key partitions can be directly accumulated without repeated rescaling. It incorporates selective KV fetching where attention weights are estimated with coarse-grained keys, as well as an integer-grid accumulation strategy for deterministic backpropagation. Benchmarks are conducted on NVIDIA H100 against FlashAttention, FlashInfer and cuDNN.

Critical strengths

The numerical formulation with a fixed reference enables additive attention accumulation and deterministic gradient computation. It achieves notable decoding speedups under tested settings while maintaining accuracy comparable to standard BF16 attention kernels.

Critical weaknesses

The paper does not sufficiently distinguish the performance contributions of the declared-reference formulation, quantization, and selective memory fetching. More rigorous ablation studies are needed to quantify their individual contributions. Furthermore, the full Llama model training experiments show only marginal performance differences, limiting the practical significance of the reported backward speedups.

Questions for authors

Could the authors provide more rigorous controlled experiments to separately quantify the performance contributions of declared-reference softmax, quantization, and selective memory fetching?

1 comment

We thank the reviewer for asking for the controlled ablation. The revision adds it for each part, in the same kernel.

Q1. Separate the declared reference, quantization and selective fetching.

Each ablation below changes one part of our kernel and keeps the rest, in the same timing job. The first five rows are speedups over the fastest BF16 kernel on four model rows at 16K keys and 256 GQA groups (Figure 5c, Table 10). The rest are at d = 128 with 256 GQA groups unless noted (Appendices F.4 and F.6).

PartAblationResult
Quantized format aloneread every key and value at 16 bits0.90 to 0.97x
Selective fetch: keysadd precision levels for keys only0.92 to 1.24x
Selective fetch: valuesadd precision levels for values only1.00 to 1.37x
Bothkey and value levels1.26 to 2.11x
Value samplingsample the lightest values (default kernel)1.26 to 2.12x
Declared referencethe mass a split has seen (Token-Picker, MassAlloc)1.20 to 1.95x the bytes, step 1.09 to 1.34x as long
FFD's shared maximum of recent scores1.03 to 1.15x the bytes
the exact row maximum0.99 to 1.12x the bytes
Grade instead of dropkeep-or-drop at the loosest threshold that keeps every head within 2% of FlashAttention-32.32 to 2.82x the bytes, step 1.83 to 2.43x as long
Ditherround-to-nearest codes, same estimatorerror 2.4 to 80x the bound on a two-cluster input
Integer gridinstead of FP32 partials and a combine kernelcosts at most 3.3% of the decode

The format alone gives no speed: our kernel reading every byte is slower than the fastest BF16 kernel. The speedup comes from the precision levels, and the declared reference is what lets a split set them, since a split-local reference reads up to 1.95 times the bytes. Hardware counters confirm the source: the speedup equals the byte ratio times the bandwidth ratio, for example 2.70 x 0.83 = 2.23 on Llama-3.1-8B (Table 14).

W2. The 1B training run shows only marginal differences.

The 1B run is not a speed claim. It tests that training with our backward is bit-reproducible and converges like the other kernels: three runs give bit-identical losses at every step and identical final weights, and the final validation loss, 3.9495, lies within the other kernels' 3.9450 to 3.9510. Matching the others is the intended result.

The backward's claim is that determinism becomes free. We now measure it directly as the determinism tax of a training step: deterministic step time over the fastest nondeterministic step time, minus one (Figure 7). On Qwen3-4B with 8K to 128K tokens, our deterministic step is as fast as the nondeterministic baseline at every length, while existing deterministic kernels add 10 to 27% at 128K. On packed batches the other deterministic kernels pay 1.4 to 2.7% and ours pays none. Megatron-LM treats a tax above 7% as a blocker, so this gap decides whether a production run can be reproducible. Against nondeterministic kernels the backward is as fast (1.03x), which is the goal. The gain is over deterministic kernels: 1.20x over 69 shapes.

Reviewer_cXNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FoldAttention, which enables the direct additive merging of attention numerators and denominators by fixing the softmax reference value before the KV scan. It also utilizes two-plane INT8 Key quantization and weight gating to reduce unnecessary KV reads. Furthermore, the paper achieves deterministic back-propagation without a fixed execution order by pre-determining an integer rounding grid. It can simultaneously improve long-context decoding and training efficiency.

Critical strengths
  1. The "Additive Contract" is essentially the translation invariance of softmax combined with the standard additive properties of unnormalized numerators and denominators; it is not a new mathematical structure for attention. Fixing or freezing reference values has already been explored in related works such as FlashDecoding++ [1] and VFA [2]. More importantly, there is no necessary algorithmic dependency between the fixed reference values in the forward pass and the integer grid in the backward pass. Therefore, the "unified principle" proposed in the paper feels more like a conceptual packaging of two existing numerical techniques rather than a new unified computational framework.

  2. The paper proves the equivalence of exact softmax under fixed reference values (Proposition 1), but the actual kernel uses incomplete Keys to calculate coarse logits and decides whether to read the remaining information based on these. This decision-making process might erroneously filter out important Keys; thus, theoretical equivalence does not guarantee the accuracy of the actual gated computation.

Critical weaknesses
  1. The "Additive Contract" is essentially the translation invariance of softmax combined with the standard additive properties of unnormalized numerators and denominators; it is not a new mathematical structure for attention. Fixing or freezing reference values has already been explored in related works such as FlashDecoding++ [1] and VFA [2]. More importantly, there is no necessary algorithmic dependency between the fixed reference values in the forward pass and the integer grid in the backward pass. Therefore, the "unified principle" proposed in the paper feels more like a conceptual packaging of two existing numerical techniques rather than a new unified computational framework.

  2. The paper proves the equivalence of exact softmax under fixed reference values (Proposition 1), but the actual kernel uses incomplete Keys to calculate coarse logits and decides whether to read the remaining information based on these. This decision-making process might erroneously filter out important Keys; thus, theoretical equivalence does not guarantee the accuracy of the actual gated computation.

Questions for authors
  1. What new algorithmic principles does FoldAttention actually introduce that cannot be directly derived from existing mathematical properties? Is there a more substantial connection between the forward and backward passes than the mere concept of sharing a pre-determined scale?

  2. Can the authors provide a theoretical upper bound for the actual attention output error when coarse logits incorrectly filter out important Keys, or when the contributions of a large number of low-weight Keys accumulate? Specifically, how does the threshold T constrain the total approximation error, rather than just the weight of an individual Key?

[1] Hong K, Dai G, Xu J, et al. Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics[J]. Proceedings of Machine Learning and Systems, 2024, 6: 148-161. [2] Sun Y, Li Y, Zou Z, et al. VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation[J]. arXiv preprint arXiv:2604.12798, 2026.

1 comment

We thank the reviewer for the two precise questions. The second one changed the decode kernel.

Q1. What is new beyond the shift invariance of softmax? Is there a real connection between the forward and the backward?

We agree that the identity itself is standard. Section 3 now says so: the additive form of the partial sums is the identity FlashDecoding++ uses with one profiled constant per model. What is new is what the reference is and what it is used for.

  • The reference is a bound, not an estimate. FlashDecoding++ fixes one constant per model and recomputes a row whose scores leave its FP32 window. VFA freezes a running maximum that starts from an estimate. Neither bounds anything. Our reference is a per-row lower bound F_i on the row's log-sum-exp, computed before the scan. Then every weight 2^(s_ij - F_i) is an upper bound on key j's share of the whole row, in every KV split, before any other key is read. This lets a split choose each key's read precision. An analog of VFA's estimate is not a lower bound: on our states it exceeds the log-sum-exp by up to 44 base-2 units, understates the shares, and puts rows over the bound (Appendix F.4).
  • A bound on the share gives a bound on the output. Over a dithered cache, read errors are unbiased and independent between keys, so they add in variance over keys (Lemma 1). Each key can then take 1/n of a squared-error budget, and Theorem 1 bounds the row's error for any input. This does not follow from shift invariance. On round-to-nearest codes the same estimator's error reaches 2.4 to 80 times the bound on a two-cluster input, because the errors of similar keys align (Appendix F.4).
  • The share must be measured against the whole row. A split-local reference, as in Token-Picker and MassAlloc, reads 1.20 to 1.95 times the bytes for the same bound (Appendix F.4).

On the connection: each kernel can be adopted alone, and the paper says so. The connection is one mechanism used twice, not a data dependency between the passes. A bound declared for each row before its reduction fixes a per-row integer grid on which partials add without rounding in any order (Proposition 1). In decode the bound is on the mass from below, so a fixed unit bounds the rounding relative to the row's mass for any scores. The splits' sums add as 64-bit integers inside the decode kernel, so no combine kernel runs, and an optional mode makes a request's bits independent of its batch. In the backward the bound is on the gradient from above (Eq. 7), so 32 bits suffice per dQ element and no CTA waits for another. That is what lets the deterministic kernel keep the longest-first tile order. Earlier pre-rounding derives the grid from the largest summand or widens it to 64 bits. Our kernel with a 64-bit grid costs 1.57x on model shapes (Table 2).

Q2. A bound on the output error when coarse logits filter out important keys, and on the accumulation of many low-weight keys.

The reviewer is right that the submitted 2^-T rule bounded each dropped key's weight, not the total. The revised kernel removes the cut. No key is dropped. Every key is scored from its 8-bit record, and the rule reads more of a key when its share bound times its score noise comes within a margin of its budget. The margin is 4.5 to 6 standard deviations and grows with the row length. The test does not see the error it removes, so it adds no bias (Lemma B.7). The lightest keys' values are sampled with Horvitz-Thompson weights, which also adds no bias.

Theorem 1 bounds the total, not one key. For each row,

E[(l_hat/l)^2 ||o_hat - o||^2] <= (G(1 + Delta) + Phi^2) Theta^2 ref,

where Theta is the row's unit (a declared multiple of the BF16 rounding of the predicted output) and ref averages the values' read variance and spread over the row's keys. Averaged over a GQA group, the factor 1 + Delta + Phi^2 is at most 1.003 at d = 128 with 4 to 16 rows a head and at most 1.017 at d = 64, for rows of 8K to 128K keys. Many light keys are covered: each key takes 1/n of the budget, and because the errors are independent they add in variance, so the sum over n keys stays within the budget. Corollaries B.1 and B.2 bound the normaliser's error and the unweighted output error in the same unit.

The bound is in expectation over the cache's dither seed and holds in exact arithmetic. The kernel's own arithmetic adds at most 0.012 of a BF16 rounding. Figure 6 plots every row's measured error over its bound on model states, stress inputs and a row built to attain the bound's order. Every row stays under its bound, and the built row reaches 0.9 of it. On real heads (Table 1) the median pooled error is 1.004 roundings against FlashAttention-3's 1.028, and every head is within 2% of FlashAttention-3. The worst single row, at 16 rows a head, reaches 3.0 roundings against FlashAttention-3's 1.5 on the same head, and it is under its bound.

Comments on the paper

2 comments

We thank the reviewers. The revision answers three requests that all reviews share: a bound on the output error, a breakdown of where the speed comes from, and a clearer paper. The main changes:

  1. Decode no longer drops keys, and its error has an a priori bound. The submitted kernel cut keys below a weight 2^-T. The revised kernel reads every key, at a precision set by a bound on that key's share of the row's softmax mass. The cache stores each key and value with a subtractive dither, so read errors are zero-mean and independent between keys. Theorem 1 bounds each row's mean-square error for any keys, values and query, with constants that are formulas of the row length, the group size and the format. No constant of the bound is fit to data.
  2. One mechanism in both kernels. A bound fixed for each row before its reduction sets an integer grid on which partial sums add in any order (Proposition 1, Section 3). In decode the bound is a lower bound on the mass, and the grid removes the combine kernel. In the backward it is an upper bound on the query gradient, and the grid frees the tile order.
  3. Ablations and profiling. Figure 5c adds one mechanism at a time to one kernel. Appendix F.2 gives Nsight Compute bytes, bandwidth and issue rates, the decode by launch, and per-tile traces. The backward separates the integer grid from the tile order (Figure 4b) and traces both (Table 21).
  4. Accuracy against every exact kernel. Table 1 scores FoldAttention, FlashAttention-3 and -4, SDPA and cuDNN in one unit over 396 KV heads of 46 layers from 16 models. On every head FoldAttention's error is within 2% of FlashAttention-3's.
  5. Writing. We rewrote the paper. Every figure is now explained in the text where it is cited, and each caption states what the figure shows.

The decode results now use Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, GLM-4-9B-Chat and gpt-oss-20b under the accuracy bar above. FoldAttention decodes the seven model batches 1.03 to 2.08x faster than the fastest BF16 kernel, and up to 2.8x on longer contexts. The backward results use the current FlashAttention-4, whose backward is faster than the version we compared against at submission. Against it our deterministic backward is as fast as the fastest nondeterministic kernel (1.03x over all shapes) and 1.20x faster than the fastest deterministic one.

Reviewer_Uc5KOpenReview ↗

Responses too late

Dear authors,

Thank you for providing responses to the reviewers. Unfortunately, since you added all your responses 15 minutes before the end of discussion period, I won't be able to read or review them, provide any further comments or change my score. Thank you for your understanding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.