- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LampAttention introduces a Look-Ahead Mixed-Precision (LAMP) framework integrated into the FlashAttention kernel, targeting dedicated AI accelerators. The algorithm defaults to accumulating key-query matrix products and evaluating softmax exponentials in 8-bit formats. It utilizes a two-stage dynamic thresholding process guided by running maxima and an analytical knapsack error bound to identify numerically sensitive sub-blocks and adaptively recompute them in 16-bit precision. The authors propose a hardware co-design featuring 1 cycle 8 bit LUTs to evaluate exponentials and a compact packed register layout. Validation is conducted using simulated mixed-precision Triton kernels on Qwen3 and Gemma 3 models, showing recovery of 32 bit baseline metrics while computing the majority of tiles in 8 bit.
Critical strengths
- SFU Bypass & Register Packing: The hardware proposal to evaluate 8 bit exponentials via a 1 cycle LUT is a pragmatic optimization that directly eliminates the multi-cycle SFU bottleneck present in standard attention computations. Furthermore, the compact layout packing four 8 bit accumulators into a single 32 bit register reduces the accumulator footprint. This effectively lowers register pressure, allowing for scaled-up SRAM tile sizes and higher warp occupancy.
- The dynamic, intra-operational precision scaling operates independently of standard static quantization (e.g., GPTQ, FP8). The block-LAMP optimization mathematically bounds error propagation well enough to recover 32-bit baseline accuracy on models like Qwen3 while maintaining highly sparse 16-bit recomputations.
Critical weaknesses
- The hardware co-design claims are undermined by the physical reality of the proposed Stage-2 knapsack solver. Evaluating the constraint requires executing floating-point arithmetic (Kronecker products, shifted exponentials) dynamically just to determine sub-block precision. Implementing this data-dependent decision block in physical RTL would introduce massive combinational logic depth and create routing congestion in the interconnects and starving the MAC arrays. The co-design is incomplete without proposing a lightweight hardware heuristic.
- The compact register layout forces a sequential 16-bit fallback loop when a tile flags >= 2 sub-blocks for recomputation. In a SIMT execution model, if even a single thread in a warp triggers this fallback, the entire warp stalls. The Triton simulations using mantissa truncation fail to model the severe throughput degradation this warp divergence would cause.
- The authors state in the text that recovering baseline performance requires only 20-30% of key tiles to be "disruptive" (requiring 2+ sub-blocks in 16-bit). However, Figure 1 clearly shows that for the Gemma-3-12b-pt model on the 5-shot MMLU task, the accuracy curve does not intersect the 32-bit baseline until the threshold is tightened to the point where nearly 75% of tiles require the 16-bit fallback.
Questions for authors
- What is the estimated hardware area and clock-cycle overhead of the logic required to dynamically solve the multidimensional knapsack problem on the fly? Could this math be replaced by a fixed static hardware heuristic to prevent fetch/dispatch stalls?
- Have you modeled the throughput degradation caused by SIMT warp divergence when only a subset of threads requires the 16-bit sequential fallback?
- Could you clarify the discrepancy between the text's claim of a 20-30% disruptive tile rate and the empirical results for the Gemma 3 models in Figure 1, which seem to require a > 60% recomputation rate to fully recover baseline accuracy?
2 comments
Thank you to the authors for the clarifications.
The warp-level masking sharing addresses one of the main concerns.
However, there are still a few limitations hindering practical benefits of this hw-algo co-design approach.
- Gemma requires a high rate of 16-bit fallbacks, yielding much smaller speedup than originally claimed.
- Dynamic floating-point comparators still remain in the critical path that is a hardware risk not proven with synthesized logic. Also pipeline stalls still exists (despite the lanes do not diverge) on the fallback loop that degrades overall throughput.
- The other reviewer brought up the unconstrainted 32-bit exponent in the simulator. This means that the method's viability on low precision hardware remains untested.
I am keeping the current score.
We thank the reviewer for their thorough and insightful review.
LAMP's numerical value comes from selecting precision within FlashAttention's online, blockwise computation. The first two hardware objections misinterpret the selection arithmetic and the shared precision decisions. Resolving these points should give us a common basis for assessing the method's numerical benefits and its implementation requirements.
W1 and Q1 — Stage-2 computation and hardware cost
We agree that selection overhead matters for the hardware assessment. However, the selection step is simpler than this reading of Equation (8) suggests: it reuses existing exponentials and constructs no Kronecker product. Moreover, with four sub-blocks, there are only 2^4=16 precision assignments, called masks, which the simulator checks.
For each sub-block θ and query j, it forms
aθj=∑i∈Iθ(ωj−eij)eij.
Here Iθ contains the keys in sub-block θ, e^ denotes the existing exponentials, and $ω̂ $ is the updated normalization sum. Using these weights, each mask g is tested through �0� for every query j; gθ=1 means leaving sub-block θ in low precision. Thus selection needs four weights per query and at most sixteen mask checks using additions, comparisons and reductions. The Kronecker notation repeats each block's precision choice across its keys; no Kronecker or diagonal matrix is constructed. We will expand Equation (8) into these weights and mask checks so that the selection cost can be assessed directly.
We have not synthesized this controller, so cannot give area or clock-cycle estimates. However, Eq. (8) does not require all sixteen candidates to be checked in one combinational circuit. The claimed inevitable massive logic depth and routing congestion therefore do not follow from the formulation alone. Assessing those costs requires an implementation study; the supporting comment provides operation counts and conditional overhead budgets. The sixteen-mask bound depends on retaining four sub-blocks.
At τ=1, Stage 2 selects no additional sub-blocks for recomputation. This gives a simpler policy using Stage 1 alone: Qwen3-32B MMLU rises from 75.55% to 82.46%, versus 83.44% baseline. Tighter thresholds can improve accuracy further (Section 4.2). Although the simulator still executes the mask search, hardware could omit it at τ=1. Thus the numerical result supports the simpler policy, but does not establish stall-free dispatch. A separate static heuristic has not been tested.
W2 and Q2 — Recomputations and warp divergence
Algorithm 1 makes the precision decision jointly for the warp, so its lanes need not take different precision branches. The simulator likewise shares one mask across all 128 query rows. Sequential recomputation still incurs extra work, whose throughput cost we have not modeled at clock-cycle level.
In particular, a warp can take longer to finish even when all its threads follow the same branch; other warps may finish earlier, and synchronization or shared resources can reduce throughput. These costs should be distinguished from lanes taking different branches within one warp. Since the simulator does not implement the compact register layout and its sequential fallback, its histogram identifies where fallback would be needed, without measuring that path's latency or numerical behavior.
W3 and Q3 — Correcting the recovery claim
Your comment identifies an important ambiguity in 'recovery'. Gemma requires a stricter threshold than Qwen to approach baseline accuracy; the result at τ=1/2 is one operating point, not LAMP's accuracy limit. However, tighter thresholds cost more recomputation, and 20–30% disruptive tiles is not generally sufficient to match the baseline. A disruptive tile has at least two sub-blocks selected for recomputation. For Gemma-3-12B MMLU, the baseline is 76.54% and pure low precision gives 36.18%. The supplementary results show:
Thus, tightening τ to 1/32 brings Gemma within 0.77 percentage points of baseline, but raises disruptive tiles to 71.20%. We will revise the abstract and Section 4.2 to report accuracy gaps alongside recomputation rates, distinguishing partial recovery from near-baseline results.
By contrast, Qwen3-32B at τ=1/2 gives 83.55% versus 83.44% on MMLU with 25.81% disruptive tiles, illustrating why the trade-off must be reported per model.
Finally, do these clarifications resolve the concerns about additional exponentials during selection and independent lane decisions? The controller and fallback costs remain important, and we hope the accompanying breakdown provides a useful basis for discussing them.