- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether the exponential inside the attention softmax can be replaced by a coarse approximation during pretraining, not only at inference. It defines K-interval attention: after subtracting the row maximum, the exponential is computed exactly at only K+1 grid points, and every other weight is reconstructed from them, either by linear interpolation (LERP) or by rounding to the nearest grid point (Nearest). The paper's point is that during training the approximation changes not only the forward pass but also the gradient the model learns from. It therefore asks whether the gradient must flow through the row maximum and minimum that set the grid, how calibration and surrogate placement interact for Nearest, and how close such operators can get to exact softmax, using GPT-2-style models. From what I understand, the paper is a study of how training behaves with an approximate exponential rather than a faster attention operator, since all attention arithmetic stays in fp32 and no kernels are measured (Section 1).
Critical strengths
- The question is well chosen. Prior approximate-softmax work targets inference or fine-tuning (Section 2), while this paper asks what happens when the model is trained from scratch with the approximation, where the operator also defines the learning rule (Section 1).
- The calibration-gradient result is clean. The same forward pass with a different backward fails, the result repeats on two further seeds, and the ablation that keeps only the maximum or only the minimum gradient shows which term matters (Section 5.1, Figure 2). It is a useful warning for anyone writing a custom backward for such an operator while also unique perspective that we can gain performance by using not exactly the same for forward and backward.
- FWM is a simple and clean design. Its bin width does not depend on the lower tail, and it needs only the row maximum (Section 3, Figure 1b). Which is make sense in my opinion due too value below certain level is hard to make impact in the training or inference.
Critical weaknesses
Weakness
-
The practical motivation is unclear. The introduction motivates the work with low-precision Transformer systems (Section 1), but the paper measures no kernels, claims no speed-up (Section 1), and does not say what K-interval attention would enable at inference or on which hardware. This matters because Section 5.5 shows that the trained model depends on its training operator (swapping in exact softmax raises the gap for LERP K=2 from +0.005 to +1.814 nats), so the approximate operator must also be used at deployment. Without at least a qualitative discussion of where a coarse exponential would be cheaper (for example, integer or lookup-table hardware) and also approximation of the speedup, it is hard to judge the impact of the results.
-
FlashAttention-4 are not cited. FlashAttention-4 (Zadouri et al., 2026, arXiv 2603.05451) already eases the exponential bottleneck on Blackwell GPUs by emulating the exponential with a polynomial on the FMA units, split with the hardware special-function unit, at BF16 accuracy and with no change to training. This is direct evidence for the bottleneck the paper is motivated by, and an existing approximate exponential in production attention kernels. In my opinion, author should discuss and argue of the positioning of the paper vs FA4, which is the main standard for current attention calculation (including the Softmax/exponent operation)
-
No existing approximate-softmax operator is trained as a baseline. Section 2 notes that EXAQ and BAPS were not tested in training and that IndexSoftmax is training-free, but the paper does not train any of them either. The only external baselines are BF16-exp and FP8-rounded exp, run on one seed at 124M and 100M tokens (Section 5.3), and they already end within 0.004 nats of softmax. FWM also looks very close to IndexSoftmax as described in Section 2 (a fixed window below the row maximum with a zero tail), so the operators themselves are not new and the contribution rests on the training analysis. Without such baselines, it is hard to see what the K-interval operators add.
Formatting and Presentations Issue
- The citation of "A framework for few-shot language model evaluation" (https://zenodo.org/records/10256836) has the wrong author list or version. Seems like the author list here is for version 1 but it is being cited as version 4.
- Pages 1, 2, 3, 5, 6 and 8 run one to two lines past the bottom margin, below the line-numbered area.
- "Four" means three different things in the experiments: four suites, four Nearest configurations, and K=4 intervals in Table 3. This makes Table 3 confusing to read at first.
- The legend of Figure 2d is confusing: the seed markers in the legend use the same pink as the detach row, while min_only and max_only are drawn in orange and blue. A neutral color for the legend markers would help.
- The suites are introduced in the order B, A, C, D (Section 4) but labelled A to D, which is confusing.
Questions for authors
- Since a BF16-accurate polynomial exponential (as in FlashAttention-4) already eases the exponential bottleneck at no cost in model quality, what setting do the authors have in mind where a K-interval operator would be preferable? On GPUs, the LERP variant appears to need a comparable number of operations while being less accurate.
- Why is LERP evaluated only with MinMax calibration? Since FWM improves the Nearest variants and LERP gives the best results, LERP with FWM seems the most natural configuration to test.
- Since Nearest's output changes with implementation and batch size, how sensitive are the reported gaps to the evaluation setup, and would a Nearest-trained model behave consistently when deployed with a different kernel?
- Since Nearest's rounding is sensitive to small numerical differences, how does training with TF32 matmuls but evaluating with TF32 disabled (Section 4) affect the Nearest results?
5 comments
Reply to Reviewer VyUR (2/5): Weakness 2
Weakness 2
Reviewer comment
FlashAttention-4 are not cited. FlashAttention-4 (Zadouri et al., 2026, arXiv 2603.05451) already eases the exponential bottleneck on Blackwell GPUs by emulating the exponential with a polynomial on the FMA units, split with the hardware special-function unit, at BF16 accuracy and with no change to training. This is direct evidence for the bottleneck the paper is motivated by, and an existing approximate exponential in production attention kernels. In my opinion, author should discuss and argue of the positioning of the paper vs FA4, which is the main standard for current attention calculation (including the Softmax/exponent operation)
Response
FlashAttention-4 (FA4) is now cited and positioned in §1, as follows. FA4 emulates part of the exponentials in software, with a polynomial whose result is nearly indistinguishable from the hardware one after rounding to BF16, so the attention it computes matches exact softmax at that precision and no learning-rule question arises. We study approximations coarse enough to change the attention probabilities (as few as 5 levels in the main results), where the backward rule becomes a design choice. The two are complementary, and FA4's measurements are direct evidence for the bottleneck that motivates our question.
Our BF16 baselines sit at FA4's precision tier and now have three seeds (Appendix I.5). With the exponential in bf16, ΔNLL is −0.0041 [−0.0110, +0.0027] at 100M tokens and +0.0011 at 250M. With exponential, row sum and division all in bf16 it is −0.0057 [−0.0331, +0.0218] and −0.0022. At that precision, pretraining from scratch is unaffected in our setting.
Reply to Reviewer VyUR (3/5): Weakness 3
Weakness 3
Reviewer comment
No existing approximate-softmax operator is trained as a baseline. Section 2 notes that EXAQ and BAPS were not tested in training and that IndexSoftmax is training-free, but the paper does not train any of them either. The only external baselines are BF16-exp and FP8-rounded exp, run on one seed at 124M and 100M tokens (Section 5.3), and they already end within 0.004 nats of softmax. FWM also looks very close to IndexSoftmax as described in Section 2 (a fixed window below the row maximum with a zero tail), so the operators themselves are not new and the contribution rests on the training analysis. Without such baselines, it is hard to see what the K-interval operators add.
Response
We agree on both counts. FWM uses the same construction as IndexSoftmax, a fixed window anchored at the row maximum with zero weight below it; the two differ in window width, number of levels, rounding and table precision. And the previous submission trained none of the operators it cited. The revision §2 now states the first directly, and Appendix H has a side-by-side table of FWM, IndexSoftmax and EXAQ. For the second we trained both (Appendix I.6, I.7). Both operators are defined for inference, so training them requires choosing a backward rule; we used the probability-level straight-through rule with the row maximum on the graph.
Implementation. IndexSoftmax follows the PyTorch implementation released with IntAttention. With float normalization our forward reproduces theirs bit for bit, and the UINT8 variant reproduces the integer codes of their integer path. Q, K and V stay in floating point. The original EXAQ code is no longer public; we follow the EXAQ paper and the reimplementation in the IntAttention artifact. Run on the same machine, our forward reproduces that reimplementation bit for bit; on the training machines its outputs differ from that reference by at most 2.0×10⁻⁶, with no change of grid index.
| ΔNLL vs softmax, nats | 100M, 3 seeds [95% t interval] | 250M | 2.5B |
|---|---|---|---|
| IndexSoftmax, float normalization | −0.010 [−0.045, +0.025] | +0.0014 | — |
| IndexSoftmax, UINT8 probabilities | −0.038 [−0.083, +0.007] | +0.0003 | +0.0172 [+0.0156, +0.0189] |
| EXAQ 3-bit, statistics on the graph | +0.067 [+0.057, +0.078] | +0.036 | — |
| EXAQ 3-bit, statistics detached | +0.088 [+0.076, +0.100] | +0.092 | — |
The BF16 and FP8 baselines you mentioned now have three seeds and a 250M run; all are within noise of softmax (Appendix I.5). These baselines still evaluate the exponential for every key; a fixed-window lookup evaluates none at run time.
On what K-interval adds: the family parameterizes such lookup operators along three axes, and the published operators land where it predicts. IndexSoftmax (32 levels, fixed window) behaves like FWM at K ≥ 16, near softmax at short horizons. EXAQ at 3 bits (8 levels, data-calibrated window) sits at the coarse, calibrated end where the paper locates the failures, and keeping its statistics on the graph is worth 0.02 nats at 100M and 0.055 at 250M.
At 2.5B, IndexSoftmax with UINT8 probabilities ends at +0.017, level with FWM–Prob K=4 (+0.019) and 0.012–0.013 behind MinMax–LERP K=4 and FWM–Weight K=16, which use fewer levels. That difference comes from one seed and is confounded (table rounding, truncation, UINT8 output, surrogate), so we report it without attributing it to any one of these factors. For reference scale, the two softmax seeds at 2.5B differ by 0.0011 nats in validation NLL (Appendix I.9), an order of magnitude below this difference. This result comes from an inference operator used outside its design: trained from scratch at 1024 context, with its forward unchanged and a backward of our choice. BAPS has no public code; the MXFP8-style baseline is the closest standard-format analogue.
Reply to Reviewer VyUR (1/5): Weakness 1 and Question 1
We highly appreciate your valuable and very detailed reviewing work. Thank you.
Your summary captures the paper's core intent: it studies how training behaves under an approximate exponential, not a faster attention operator.
Our reply has five parts: Weakness 1 with Question 1, Weakness 2, Weakness 3, Questions 2 to 4, and the formatting issues.
Weakness 1 and Question 1
Reviewer comment (Weakness 1)
Reviewer comment (Question 1)
Response
On FlashAttention-4's roofline (BF16 operands) a K-interval operator gains nothing; §6 now gives the estimate.
The roofline of FlashAttention-4 (FA4) gives, per tile, 1024 cycles for the matrix multiply, 1024 for the exponential unit and 768 for shared memory. Removing the exponential entirely leaves the bound at 1024, because the matrix multiply is co-limiting. Such a single-pass lookup kernel would also compute a different function. An online softmax rescales partial sums using the multiplicative property of exp, which a lookup grid does not have, so the grid would be anchored at the running maximum instead of the row maximum. On trained FWM endpoints this changes NLL by up to +0.0041 [+0.0038, +0.0046]. A kernel faithful to the trained operator needs a first pass for the row maximum, and its bound is 1.5× FA4's (Appendix I.10). These are estimates from FA4's published numbers.
A coarse exponential is cheaper on integer and lookup hardware. The fixed-window operator needs no exponential at run time: its K+1 values are constants shared by all rows, heads and layers, and the per-key work is a subtract, a scale, a round, one table read and a multiply. Appendix I.10 gives operation counts for every operator. IntAttention's IndexSoftmax uses the same fixed-window construction (6.6 nats, 32 levels). On an RK3588S2 (Armv8) CPU its authors report their integer attention pipeline, built on that operator, up to 2.0× faster than an INT8 attention pipeline that keeps a floating-point softmax, and up to 3.7× faster than FP16 attention. These are their measurements of attention latency at inference; the 3.7× includes the gain from INT8 matrix products.
What the paper contributes to that setting is the training side, for the reason you point out: the trained weights depend on the training operator, so the operator used at deployment must also be used in training, and then its backward rule decides whether training works. One new data point: the model we trained for 2.5B tokens with IndexSoftmax loses 0.008–0.009 nats when its operator is replaced at evaluation by exact softmax or by the float-normalized variant of the same lookup (Appendix I.7). Even a 32-level operator is not interchangeable after training.
§1 now states the scope in these terms, and §6 has a deployment paragraph that states these points, with the numbers in Appendix I.10.