acceptodds
Under review as a conference paper at ICLR 2027

LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

Abstract

While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that rerouting a selective minority of sub-blocks to high precision is sufficient to recover the baseline model performance.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
FlashAttention, numerical stability, mixed precision, transformer inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JBtvOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LampAttention introduces a Look-Ahead Mixed-Precision (LAMP) framework integrated into the FlashAttention kernel, targeting dedicated AI accelerators. The algorithm defaults to accumulating key-query matrix products and evaluating softmax exponentials in 8-bit formats. It utilizes a two-stage dynamic thresholding process guided by running maxima and an analytical knapsack error bound to identify numerically sensitive sub-blocks and adaptively recompute them in 16-bit precision. The authors propose a hardware co-design featuring 1 cycle 8 bit LUTs to evaluate exponentials and a compact packed register layout. Validation is conducted using simulated mixed-precision Triton kernels on Qwen3 and Gemma 3 models, showing recovery of 32 bit baseline metrics while computing the majority of tiles in 8 bit.

Critical strengths
  • SFU Bypass & Register Packing: The hardware proposal to evaluate 8 bit exponentials via a 1 cycle LUT is a pragmatic optimization that directly eliminates the multi-cycle SFU bottleneck present in standard attention computations. Furthermore, the compact layout packing four 8 bit accumulators into a single 32 bit register reduces the accumulator footprint. This effectively lowers register pressure, allowing for scaled-up SRAM tile sizes and higher warp occupancy.
  • The dynamic, intra-operational precision scaling operates independently of standard static quantization (e.g., GPTQ, FP8). The block-LAMP optimization mathematically bounds error propagation well enough to recover 32-bit baseline accuracy on models like Qwen3 while maintaining highly sparse 16-bit recomputations.
Critical weaknesses
  • The hardware co-design claims are undermined by the physical reality of the proposed Stage-2 knapsack solver. Evaluating the constraint requires executing floating-point arithmetic (Kronecker products, shifted exponentials) dynamically just to determine sub-block precision. Implementing this data-dependent decision block in physical RTL would introduce massive combinational logic depth and create routing congestion in the interconnects and starving the MAC arrays. The co-design is incomplete without proposing a lightweight hardware heuristic.
  • The compact register layout forces a sequential 16-bit fallback loop when a tile flags >= 2 sub-blocks for recomputation. In a SIMT execution model, if even a single thread in a warp triggers this fallback, the entire warp stalls. The Triton simulations using mantissa truncation fail to model the severe throughput degradation this warp divergence would cause.
  • The authors state in the text that recovering baseline performance requires only 20-30% of key tiles to be "disruptive" (requiring 2+ sub-blocks in 16-bit). However, Figure 1 clearly shows that for the Gemma-3-12b-pt model on the 5-shot MMLU task, the accuracy curve does not intersect the 32-bit baseline until the threshold is tightened to the point where nearly 75% of tiles require the 16-bit fallback.
Questions for authors
  • What is the estimated hardware area and clock-cycle overhead of the logic required to dynamically solve the multidimensional knapsack problem on the fly? Could this math be replaced by a fixed static hardware heuristic to prevent fetch/dispatch stalls?
  • Have you modeled the throughput degradation caused by SIMT warp divergence when only a subset of threads requires the 16-bit sequential fallback?
  • Could you clarify the discrepancy between the text's claim of a 20-30% disruptive tile rate and the empirical results for the Gemma 3 models in Figure 1, which seem to require a > 60% recomputation rate to fully recover baseline accuracy?
2 comments

We thank the reviewer for their thorough and insightful review.

LAMP's numerical value comes from selecting precision within FlashAttention's online, blockwise computation. The first two hardware objections misinterpret the selection arithmetic and the shared precision decisions. Resolving these points should give us a common basis for assessing the method's numerical benefits and its implementation requirements.

W1 and Q1 — Stage-2 computation and hardware cost

We agree that selection overhead matters for the hardware assessment. However, the selection step is simpler than this reading of Equation (8) suggests: it reuses existing exponentials and constructs no Kronecker product. Moreover, with four sub-blocks, there are only 2^4=16 precision assignments, called masks, which the simulator checks.

For each sub-block θ and query j, it forms

aθj=∑i∈Iθ(ω^j−e^ij)e^ij.a_{\theta j}=\sum_{i\in I_\theta}(\widehat\omega_j-\widehat e_{ij})\widehat e_{ij}.

Here IθI_θ contains the keys in sub-block θθ, e^ê denotes the existing exponentials, and $ω̂ $ is the updated normalization sum. Using these weights, each mask g is tested through �0��0� for every query j; gθ=1g_θ=1 means leaving sub-block θθ in low precision. Thus selection needs four weights per query and at most sixteen mask checks using additions, comparisons and reductions. The Kronecker notation repeats each block's precision choice across its keys; no Kronecker or diagonal matrix is constructed. We will expand Equation (8) into these weights and mask checks so that the selection cost can be assessed directly.

We have not synthesized this controller, so cannot give area or clock-cycle estimates. However, Eq. (8) does not require all sixteen candidates to be checked in one combinational circuit. The claimed inevitable massive logic depth and routing congestion therefore do not follow from the formulation alone. Assessing those costs requires an implementation study; the supporting comment provides operation counts and conditional overhead budgets. The sixteen-mask bound depends on retaining four sub-blocks.

At τ=1, Stage 2 selects no additional sub-blocks for recomputation. This gives a simpler policy using Stage 1 alone: Qwen3-32B MMLU rises from 75.55% to 82.46%, versus 83.44% baseline. Tighter thresholds can improve accuracy further (Section 4.2). Although the simulator still executes the mask search, hardware could omit it at τ=1. Thus the numerical result supports the simpler policy, but does not establish stall-free dispatch. A separate static heuristic has not been tested.

W2 and Q2 — Recomputations and warp divergence

Algorithm 1 makes the precision decision jointly for the warp, so its lanes need not take different precision branches. The simulator likewise shares one mask across all 128 query rows. Sequential recomputation still incurs extra work, whose throughput cost we have not modeled at clock-cycle level.

In particular, a warp can take longer to finish even when all its threads follow the same branch; other warps may finish earlier, and synchronization or shared resources can reduce throughput. These costs should be distinguished from lanes taking different branches within one warp. Since the simulator does not implement the compact register layout and its sequential fallback, its histogram identifies where fallback would be needed, without measuring that path's latency or numerical behavior.

W3 and Q3 — Correcting the recovery claim

Your comment identifies an important ambiguity in 'recovery'. Gemma requires a stricter threshold than Qwen to approach baseline accuracy; the result at τ=1/2 is one operating point, not LAMP's accuracy limit. However, tighter thresholds cost more recomputation, and 20–30% disruptive tiles is not generally sufficient to match the baseline. A disruptive tile has at least two sub-blocks selected for recomputation. For Gemma-3-12B MMLU, the baseline is 76.54% and pure low precision gives 36.18%. The supplementary results show:

Threshold τAccuracyDisruptive tiles
1/258.99%24.42%
1/1675.22%60.67%
1/3275.77%71.20%

Thus, tightening τ to 1/32 brings Gemma within 0.77 percentage points of baseline, but raises disruptive tiles to 71.20%. We will revise the abstract and Section 4.2 to report accuracy gaps alongside recomputation rates, distinguishing partial recovery from near-baseline results.

By contrast, Qwen3-32B at τ=1/2 gives 83.55% versus 83.44% on MMLU with 25.81% disruptive tiles, illustrating why the trade-off must be reported per model.

Finally, do these clarifications resolve the concerns about additional exponentials during selection and independent lane decisions? The controller and fallback costs remain important, and we hope the accompanying breakdown provides a useful basis for discussing them.

Thank you to the authors for the clarifications.

The warp-level masking sharing addresses one of the main concerns.

However, there are still a few limitations hindering practical benefits of this hw-algo co-design approach.

  • Gemma requires a high rate of 16-bit fallbacks, yielding much smaller speedup than originally claimed.
  • Dynamic floating-point comparators still remain in the critical path that is a hardware risk not proven with synthesized logic. Also pipeline stalls still exists (despite the lanes do not diverge) on the fallback loop that degrades overall throughput.
  • The other reviewer brought up the unconstrainted 32-bit exponent in the simulator. This means that the method's viability on low precision hardware remains untested.

I am keeping the current score.

Reviewer_WMumOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies mixed-precision FlashAttention combined with a customized data format. The goal is to reduce the precision of QK sub-blocks during accumulation while maintaining model performance.

They simulate a low-precision attention pipeline and their customized design using the Triton language, including custom data formats, LUT, and two-stage accumulation. The experiments are conducted on Qwen3 and Gemma3 models on perplexity, MMLU, and the ARC challenge benchmarks. The results show that recomputation, using higher precision, can recover performance on some of the downstream tasks. But the hardware benefits are not validated because the customized design cannot be implemented directly on GPUs, and the authors didn't provide a real implementation on other hardware types.

Critical strengths
  1. The method proposes a dedicated custom data format with a 16-bit width, which is neat for modern hardware architecture. They also consider register layouts and recomputation fallback, and provide promising connections between the algorithm and modern hardware design.
  2. The Experiments show that selective high-precision recomputation can improve pure low-precision Computation on Part of the downstream tasks. They also provide a sweep of hyperparameters, like the safety margin, which helps explain the role of the parameters. The supplemental material includes code and raw result logs, which increase the reproducibility of the proposed method.
Critical weaknesses
  1. The efficiency for practical deployment is not validated at all. The kernel is simulated, so the end-to-end time and kernel latency breakdown are difficult to provide from the current implementation. Therefore, It's hard to estimate the real speed-up benefit. And the trade-off between the recomputation ratio and the speed-up is also unclear.
  2. The simulator provided in the supplementary code uses round mantissa, which preserves the exponent range of FP32 and only changes the fraction precision, which is misaligned with e4m3/e4m11 described in the main text.
  3. In the code, they use FlashAttention 2, which natively uses a 16-bit width instead of a 32-bit width, but in their main table, FlashAttention 2 was marked as 32-bit width. I wonder about the actual implementation of FlashAttention-2 in their experiments, and why a 32-bit width was used instead of 16-bit. The experiments also lack 16-bit baselines, making it difficult to assess the benefits of the proposed method compared to directly using a pure 16-bit attention baseline.
  4. The proposed method uses a uniform hyperparameter across layers and heads. Although the authors have tested several parameter values, the paper does not explain why they use the same hyperparameter for all layers and heads, how σ\sigma and τ\tau are selected for different models or tasks, or whether these parameters would be changed. Different models and tasks, shallow and deep layers, and different heads may have different attention patterns, which might use different optimal hyperparameters.
  5. The evaluation benchmark is not comprehensive enough. It mainly covers language modeling tasks without complex tasks such as multi-step math reasoning, code generation, and long-context tasks. These tasks may have different attention patterns, and this evaluation can validate whether the preset hyperparameters are suitable for all these different downstream tasks.
  6. The motivation for error propagation lacks direct experimental evidence. They only provide theoretical analysis, but a direct experimental illustration could help readers understand the motivation. How do low-precision maximum errors affect normalization and attention outputs?
  7. The accuracy drop on some of the tasks is unacceptable. For example, Gemma-3-12B drops from 76.54% to 58.99% on MMLU. A strict τ\tau can improve accuracy performance, but requires a higher recomputation ratio. So, different hyperparameters for the accuracy and speed-up trade-off are suggested.
Questions for authors
  1. Compared to current mixed-position attention kernels, under what conditions would the proposed method have better hardware performance in terms of speed-up?
  2. An estimated time breakdown of the low-precision pass, maximum reduction, sorting, mask search, and high-precision recomputation would be helpful. It would be helpful to provide a cost explanation if it's difficult to have a real implementation during the limited rebuttal period.
  3. Please provide the actual maximum error and attention output, together with the predefined safety margin. It would be clear for the readers to compare them and assess whether the safety margin is correctly chosen.
2 comments

We thank the reviewer for their thorough and insightful review.

Several criticisms reflect misunderstandings of the computation, notably the incorrect claim about baseline precision. Working through these points together should help us reach common ground on LAMP's mechanism and results.

W3 — FlashAttention-2 baseline precision

The claimed inconsistency rests on an invalid comparison of input precision with intermediate arithmetic. Section 4.1's “32-bit” refers to logit accumulation and exponentials. We use standard FlashAttention-2 with torch_dtype="auto", without converting the model to FP32. We will label the two precisions separately. A uniform 16-bit accumulation/exponential baseline is not included.

Does separating these precisions resolve the concern about the baseline?

W2 — Exponent range

The simulator uses 3 or 11 mantissa bits but, as the reviewer notes, retains FP32's exponent range. It therefore does not fully emulate the proposed formats. Section 3.3 addresses overflow using QK normalization and pre-scaling; underflow and subnormal behavior still need validation. We will state this limitation in Section 4.1 and replace “truncated” with “rounded to nearest, with ties to even.”

W1 / Q1 — Conditions for hardware speedup

With native support, LAMP could make accumulation and exponentials cheaper than in kernels retaining high-precision intermediates, complementing input quantization. For illustration, suppose a full low-precision logit pass costs half a baseline pass and a high-precision pass costs the same as baseline. For Qwen3-32B at τ=1/2, recorded recomputation then leaves 26.17% of baseline logit time for other net costs before break-even, including selection, scheduling and changes in memory or exponential/output work. The supporting comment gives the assumptions; this is a conditional budget, not a measured speedup.

Q2 — Requested cost breakdown

A validated cycle-level model would require a detailed hardware design and a separate implementation study. For this numerical study, we provide the computation and memory costs the reviewer suggests.

For q queries, 64 keys in four sub-blocks, head dimension d and R recomputed sub-blocks:

StageArithmetic work
Low-precision logits64qd fused multiply-adds (FMAs)
Maxima and threat scores60q max comparisons; 4q differences; 4(q−1) score comparisons
Sorting and safety testsFive compare-and-swaps; at most 4q safety tests
Stage-2 weights64q subtractions, 64q products, 60q additions; 2q threshold products
Mask searchAt most 16 masks × q checks; sums and reductions
Recompute R sub-blocks16qdR extra FMAs; exponential and state updates

Selection reuses exponentials. Keeping Q/K on-chip avoids extra HBM reads, but consumes local bandwidth. The compact sequential fallback is not implemented in the simulator; details are in the supporting comment.

W4 — Parameter choices

We interpret σ as the paper's δ. We use δ=2^−8 and τ=1/2 in the main tables to test a shared policy without tuning per model, task, layer or head. Yet each tile's data still determines which sub-blocks need recomputation. Figure 1 varies τ and Appendix C compares δ values. The parameters are adjustable; we claim no universal optimum or automatic selection rule.

W5 — Evaluation coverage

We test five models from two families on C4, WikiText, MMLU and ARC, including few-shot ARC, covering perplexity and downstream tasks. These experiments establish accuracy–recomputation trade-offs in the tested settings; code, long-context and multi-step reasoning would test broader applicability.

W6 / Q3 — Maximum errors and attention output

Equations (2)–(4) explain how an error in the running maximum can affect both rescaling of earlier contributions and new exponentials. The resulting probability errors can change the attention output. Although a common shift cancels in exact arithmetic, rounding need not preserve that cancellation.

Stage 1's margin |μ̂_j|δ scales with the running maximum; δ is tunable, with its error-coverage condition in Footnote 3. The τ=1 experiment isolates Stage 1: Qwen3-32B reaches 82.46% MMLU from 75.55% in pure low precision. Tightening τ then activates Stage 2. Appendices C/D vary the margin and disable Stage 1. These task-level tests do not supply the requested direct comparison of maximum errors, margins and attention-output errors.

W7 — Gemma accuracy loss

Your observation clarifies that Gemma's result at τ=1/2 is not LAMP's accuracy limit: models and tasks can require different tolerances. We used a shared setting to test an untuned policy. For Gemma-3-12B MMLU, τ=1/32 gives 75.77% versus 76.54% baseline, with 71.20% of tiles needing at least two recomputations. By contrast, Qwen3-32B gives 83.55% versus 83.44% at τ=1/2, with 25.81% such tiles. We will clarify this dependence and report accuracy gaps alongside recomputation rates.

I thank the authors for a candid response. Several clarifications are useful in addressing my concerns.

The authors are right on W3: the 32-bit label in Section 4.1 is defensible once input precision and accumulation precision are separated. I acknowledge that the simulator retains the FP32 exponent range. However, the numerical evidence does not test the proposed format, and the authors also acknowledge that underflow and subnormal behavior still need validation. For W1, the authors respond without any actual measurement. On real hardware, sorting, compare-and-swap, mask search, tile scheduling and other on-chip operations are memory- and control-heavy. The Q2 table counts operations but does not measure a latency breakdown. I still hold that an end-to-end time breakdown on real hardware is necessary to show that the design brings real acceleration. For W7, reaching accuracy on Gemma-3-12B MMLU requires 71.2% of tiles to be recomputed at least twice, so the speed benefit is weakened by the large share of the high-precision path. The real speed-up therefore becomes questionable on accuracy-sensitive tasks.

I appreciate the honesty in the response to W2–W6. But since my concerns about the acceleration on real hardware remain unvalidated, and the speed-up and accuracy trade-off seems to be difficult, I stay unconvinced and keep my rating.

Reviewer_WvrYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LampAttention addresses the challenge of reducing the computational cost of attention while preserving numerical stability. The proposed method performs query–key product accumulation and softmax exponential evaluation in 8-bit formats, then selectively recomputes sensitive sub-blocks in 16-bit formats using running maxima and an analytical error bound. Experiments on various models show that selective high-precision recomputation substantially improves perplexity and downstream accuracy over a pure 8-bit baseline, generally approaching 32-bit baseline performance, with the degree of recovery depending on the model and task.

Critical strengths
  1. The mathematical formulation for selecting computational precision to minimize numerical error while primarily using 8-bit arithmetic is reasonable.
  2. The paper provides a detailed analysis of recomputation rates across various models and experimental settings.
Critical weaknesses
  1. To accelerate FlashAttention, the paper should provide a clear comparison of numerical error and model accuracy against methods using vector-wise quantization formats such as MXFP8 and NVFP4.
  2. The effectiveness of LampAttention should be demonstrated on actual GPU hardware. To propose LUT-based hardware rather than a GPU implementation, simply highlighting its small silicon area is insufficient. A detailed, quantitative analysis of the memory accesses and computational costs required for error compensation is also necessary.
  3. GPU floating-point arithmetic does not exactly reproduce real-number arithmetic, and the specialized bit-level behavior of GPU arithmetic units can introduce additional numerical discrepancies [“Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy,” Peichen Xie et al, MICRO 2026]. The authors should therefore clearly demonstrate whether LampAttention’s mathematical error guarantees remain valid when these hardware limitations are taken into account.
Questions for authors
  1. How does LampAttention compare with methods using vector-wise quantization formats, such as MXFP8 and NVFP4, in terms of numerical error and model accuracy?
  2. Can the authors demonstrate LampAttention’s performance benefits on actual GPU hardware or provide a quantitative analysis of the memory accesses and computational costs of error compensation for the proposed LUT-based accelerator, beyond its silicon-area requirements?
  3. Do LampAttention’s mathematical error guarantees remain valid under the bit-level behavior and numerical limitations of GPU arithmetic units, as discussed by Xie et al. in “Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy” (MICRO 2026), and how can this validity be verified?
2 comments

We thank the reviewer for their thorough and insightful review.

We believe that several concerns arise from misunderstandings of the proposed method, and we welcome the opportunity to work through them with the reviewer. We hope the clarifications below will establish a shared understanding of LAMP's mechanism and contribution. We will make the method and the rationale for our comparisons clearer in Sections 1.1 and 4.1.

W1 / Q1 — Comparison with MXFP8 and NVFP4

Input quantization and LAMP are essentially orthogonal because they act on different parts of the computation. MXFP8/NVFP4 change how matrix inputs are represented and scaled, whereas LAMP changes the precision used to accumulate dot products and evaluate softmax exponentials. Low-bit inputs do not imply low-bit accumulation: NVIDIA's block-scaled MXFP8/NVFP4 matrix operations use FP32 accumulators. LAMP could therefore complement these formats by reducing intermediate precision as well.

The purpose of our comparison is therefore to isolate LAMP's contribution by holding input representation fixed. We compare standard FlashAttention-2, uniformly low-precision intermediate arithmetic, and LAMP to test how much accuracy selective recomputation recovers. A comparison with MXFP8/NVFP4 would also change input quantization and assess a broader combination of numerical choices. Our controlled comparison does not establish relative numerical error or accuracy against those methods.

W2 / Q2 — GPU performance and cost accounting

We agree that hardware efficiency needs a quantitative account of computation and memory traffic. Because our kernel emulates the proposed arithmetic, its runtime cannot establish a GPU speedup. We therefore provide the cost analysis suggested by the reviewer. For context, LAMP achieves 83.55% MMLU accuracy on Qwen3-32B versus the 83.44% baseline, with 23.83% of sub-blocks selected for recomputation.

For q queries, 64 keys in four 16-key sub-blocks, and head dimension d, the initial pass needs 64qd fused multiply-adds (FMAs). Recomputing R sub-blocks adds 16qdR high-precision FMAs. In addition, Stage 2 forms four weights per query and checks at most 16 precision assignments. It reuses existing softmax exponentials and computes no Kronecker product.

If Q/K remain on-chip, recomputation needs no additional HBM reads of these inputs, although it still consumes on-chip bandwidth. Reading each required input once per recomputation gives R d(q+16) operand elements; however, physical SRAM traffic depends on reuse and instruction mapping. The supporting cost comment gives the remaining counts and a conditional timing model. For example, if a full low-precision logit pass costs half a baseline pass and a high-precision pass costs the same as baseline, Qwen's recomputation leaves 26.17% of baseline logit time for other net costs before break-even. This is an illustrative budget, not a measured speedup.

Does the accompanying accounting address the alternative analysis requested in Q2, or is there a particular cost that needs further explanation?

W3 / Q3 — Hardware arithmetic and the error analysis

The analysis separates two questions: how numerical errors affect normalization, and how large those errors are on a particular arithmetic unit. Proposition 1 addresses the first; a hardware-specific error model is needed for the second. The sensitivity result therefore remains valid, while a guarantee for a particular GPU requires additional validation.

Specifically, our analysis assumes that rounding preserves the largest-logit index, treats high precision as exact in a limiting case, and retains a higher-order Taylor remainder. In practice, the implemented rule uses computed probabilities and a tunable safety margin. Footnote 3 states the condition needed for this margin to cover logit errors. The rule therefore does not certify arbitrary GPU implementations or downstream accuracy.

Xie et al. provide a route to hardware-specific validation: characterize the arithmetic, obtain bounds covering summation order, rounding and exponent range, and use them in the normalization analysis. We thank the reviewer for highlighting this work and the importance of checking our assumptions against the arithmetic of the target hardware. We will list these assumptions and validation requirements alongside the error analysis in Section 2.

Thank you for the clarifications and the additional cost analysis. The explanation of mask sharing resolves my concern in Q2 and Q3.

Q1. Recent FlashAttention implementations already support block-scaled inputs such as MXFP8 and NVFP4, it remains unclear whether adding LAMP improves kernel throughput at comparable PPL when both methods use the same quantized inputs.

I am keeping the current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.