- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a novel kernel implementation for attention calculations for Hopper architectures. In decode, it uses an additive formulation to calculate a fixed reference instead of updaing the reference online during softmax calculation. In prefill, thread blocks round the the partial gradients to the nearest grid to achieve determinism without reducing the performance of backward kernel.
Critical strengths
- The paper addresses an important problem, the performance of kernels in attention calculations. I especially believe that developing deterministic kernels without performance penalties is an important but often overlooked direction, which this paper addresses.
Critical weaknesses
-
This paper is difficult to follow and understand. It feels like the manuscript is heavily generated by AI end-to-end, with little effort from the authors. Figures are not clear and lack in-text explanations. The quality of writing is not up to the standards of ICLR.
-
There is no explanation where the speedup comes from. There is no profiling data, trace timeline, performance breakdown to back up the claims that the speedup is coming from the novelties presented in the paper, not from implementation differences.
-
Method is implemented and evaluated only for Hopper. There is no evidence that this method would generalize to other more recent architectures.
Questions for authors
See the weaknesses above.
We thank the reviewer, in particular for the point on deterministic kernels. We address the three weaknesses in turn.
W1. Writing and figures.
We rewrote the paper. The introduction now states the problem (exact kernels already stream the cache at 94% or more of measured bandwidth, so a faster decode must read fewer bytes), the two obstacles, and how each construction removes one. Every figure is explained in the text where it is cited, and each caption says what to read from it. Figure 2 works one example: four keys in two splits, with and without the declared reference. Figure 3 numbers the decode kernel's steps, and Section 4 describes them in that order. We removed internal terms and moved detail to the appendix.
W2. Where the speedup comes from, with profiling, traces and a breakdown.
The revision adds four kinds of evidence, all on the same model batches.
For the backward, Figure 4b runs our kernel with each way to add partials (ordered FP32, FP32 atomics, integer grid) under each tile order. The integer grid costs 2.6% over FP32 atomics, and the longest-first order it allows saves 15%. Table 21 and Figure 11 give in-kernel traces of each variant. Ordered adds leave the store warp waiting for its turn, and shortest-first order rereads queries and output gradients from DRAM (12.5 GB against 4.4 GB). So the speed over deterministic kernels comes from the tile order that only an order-free reduction can use. Against nondeterministic kernels the backward is as fast, and we claim no more.
W3. Hopper only.
Appendix F.9 runs the decode on a GeForce RTX 5070 Ti (Blackwell, sm_120) through the kernel's generic path, with no tuning for that GPU. FoldAttention reads 0.35 to 0.50 of the fastest baseline's bytes and decodes 1.21 to 2.20x faster, at 1.005 to 1.007 of the error of the most accurate exact kernel there. The mechanisms need integer tensor-core products, integer reductions to global memory and bulk copies, which recent NVIDIA GPUs have. The speed argument depends only on decode being bandwidth-bound, which holds on every recent GPU.