acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Transformers with Quantized Softmax in Attention

Abstract

Low-precision Transformer systems increasingly quantize attention matrix multiplications, while softmax often remains at higher precision. During pretraining, an approximate softmax changes the gradients that train the model as well as its forward computation. We study this interaction with K-interval attention, which approximates the exponential using K+1 grid values. We vary per-row grid calibration, interpolation versus hard rounding, and the placement of a straight-through surrogate relative to normalization. We derive the corresponding backward rules, including calibration derivatives, and compare these choices in pretraining experiments matched on model, data, and optimizer. Detaching the row extrema leaves the forward computation unchanged but produces a delayed increase in validation loss. With hard rounding at K=4, min–max calibration and a pre-normalization surrogate incur a large loss gap; changing either choice substantially reduces it. At 124M parameters and 2.5B training tokens, fixed-window calibration with a post-normalization surrogate yields a validation loss gap of +0.019 nats relative to softmax at K=4; with a pre-normalization surrogate yields +0.004 nats at K=16.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
quantization-aware training, straight-through estimator, softmax approximation, low-precision training, language model pretraining, attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VyUROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the exponential inside the attention softmax can be replaced by a coarse approximation during pretraining, not only at inference. It defines K-interval attention: after subtracting the row maximum, the exponential is computed exactly at only K+1 grid points, and every other weight is reconstructed from them, either by linear interpolation (LERP) or by rounding to the nearest grid point (Nearest). The paper's point is that during training the approximation changes not only the forward pass but also the gradient the model learns from. It therefore asks whether the gradient must flow through the row maximum and minimum that set the grid, how calibration and surrogate placement interact for Nearest, and how close such operators can get to exact softmax, using GPT-2-style models. From what I understand, the paper is a study of how training behaves with an approximate exponential rather than a faster attention operator, since all attention arithmetic stays in fp32 and no kernels are measured (Section 1).

Critical strengths
  1. The question is well chosen. Prior approximate-softmax work targets inference or fine-tuning (Section 2), while this paper asks what happens when the model is trained from scratch with the approximation, where the operator also defines the learning rule (Section 1).
  2. The calibration-gradient result is clean. The same forward pass with a different backward fails, the result repeats on two further seeds, and the ablation that keeps only the maximum or only the minimum gradient shows which term matters (Section 5.1, Figure 2). It is a useful warning for anyone writing a custom backward for such an operator while also unique perspective that we can gain performance by using not exactly the same for forward and backward.
  3. FWM is a simple and clean design. Its bin width does not depend on the lower tail, and it needs only the row maximum (Section 3, Figure 1b). Which is make sense in my opinion due too value below certain level is hard to make impact in the training or inference.
Critical weaknesses

Weakness

  1. The practical motivation is unclear. The introduction motivates the work with low-precision Transformer systems (Section 1), but the paper measures no kernels, claims no speed-up (Section 1), and does not say what K-interval attention would enable at inference or on which hardware. This matters because Section 5.5 shows that the trained model depends on its training operator (swapping in exact softmax raises the gap for LERP K=2 from +0.005 to +1.814 nats), so the approximate operator must also be used at deployment. Without at least a qualitative discussion of where a coarse exponential would be cheaper (for example, integer or lookup-table hardware) and also approximation of the speedup, it is hard to judge the impact of the results.

  2. FlashAttention-4 are not cited. FlashAttention-4 (Zadouri et al., 2026, arXiv 2603.05451) already eases the exponential bottleneck on Blackwell GPUs by emulating the exponential with a polynomial on the FMA units, split with the hardware special-function unit, at BF16 accuracy and with no change to training. This is direct evidence for the bottleneck the paper is motivated by, and an existing approximate exponential in production attention kernels. In my opinion, author should discuss and argue of the positioning of the paper vs FA4, which is the main standard for current attention calculation (including the Softmax/exponent operation)

  3. No existing approximate-softmax operator is trained as a baseline. Section 2 notes that EXAQ and BAPS were not tested in training and that IndexSoftmax is training-free, but the paper does not train any of them either. The only external baselines are BF16-exp and FP8-rounded exp, run on one seed at 124M and 100M tokens (Section 5.3), and they already end within 0.004 nats of softmax. FWM also looks very close to IndexSoftmax as described in Section 2 (a fixed window below the row maximum with a zero tail), so the operators themselves are not new and the contribution rests on the training analysis. Without such baselines, it is hard to see what the K-interval operators add.

Formatting and Presentations Issue

  1. The citation of "A framework for few-shot language model evaluation" (https://zenodo.org/records/10256836) has the wrong author list or version. Seems like the author list here is for version 1 but it is being cited as version 4.
  2. Pages 1, 2, 3, 5, 6 and 8 run one to two lines past the bottom margin, below the line-numbered area.
  3. "Four" means three different things in the experiments: four suites, four Nearest configurations, and K=4 intervals in Table 3. This makes Table 3 confusing to read at first.
  4. The legend of Figure 2d is confusing: the seed markers in the legend use the same pink as the detach row, while min_only and max_only are drawn in orange and blue. A neutral color for the legend markers would help.
  5. The suites are introduced in the order B, A, C, D (Section 4) but labelled A to D, which is confusing.
Questions for authors
  1. Since a BF16-accurate polynomial exponential (as in FlashAttention-4) already eases the exponential bottleneck at no cost in model quality, what setting do the authors have in mind where a K-interval operator would be preferable? On GPUs, the LERP variant appears to need a comparable number of operations while being less accurate.
  2. Why is LERP evaluated only with MinMax calibration? Since FWM improves the Nearest variants and LERP gives the best results, LERP with FWM seems the most natural configuration to test.
  3. Since Nearest's output changes with implementation and batch size, how sensitive are the reported gaps to the evaluation setup, and would a Nearest-trained model behave consistently when deployed with a different kernel?
  4. Since Nearest's rounding is sensitive to small numerical differences, how does training with TF32 matmuls but evaluating with TF32 disabled (Section 4) affect the Nearest results?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer VyUR (1/5): Weakness 1 and Question 1

We highly appreciate your valuable and very detailed reviewing work. Thank you.

Your summary captures the paper's core intent: it studies how training behaves under an approximate exponential, not a faster attention operator.

Our reply has five parts: Weakness 1 with Question 1, Weakness 2, Weakness 3, Questions 2 to 4, and the formatting issues.

Weakness 1 and Question 1

Reviewer comment (Weakness 1)

The practical motivation is unclear. The introduction motivates the work with low-precision Transformer systems (Section 1), but the paper measures no kernels, claims no speed-up (Section 1), and does not say what K-interval attention would enable at inference or on which hardware. This matters because Section 5.5 shows that the trained model depends on its training operator (swapping in exact softmax raises the gap for LERP K=2 from +0.005 to +1.814 nats), so the approximate operator must also be used at deployment. Without at least a qualitative discussion of where a coarse exponential would be cheaper (for example, integer or lookup-table hardware) and also approximation of the speedup, it is hard to judge the impact of the results.

Reviewer comment (Question 1)

Since a BF16-accurate polynomial exponential (as in FlashAttention-4) already eases the exponential bottleneck at no cost in model quality, what setting do the authors have in mind where a K-interval operator would be preferable? On GPUs, the LERP variant appears to need a comparable number of operations while being less accurate.

Response

On FlashAttention-4's roofline (BF16 operands) a K-interval operator gains nothing; §6 now gives the estimate.

The roofline of FlashAttention-4 (FA4) gives, per tile, 1024 cycles for the matrix multiply, 1024 for the exponential unit and 768 for shared memory. Removing the exponential entirely leaves the bound at 1024, because the matrix multiply is co-limiting. Such a single-pass lookup kernel would also compute a different function. An online softmax rescales partial sums using the multiplicative property of exp, which a lookup grid does not have, so the grid would be anchored at the running maximum instead of the row maximum. On trained FWM endpoints this changes NLL by up to +0.0041 [+0.0038, +0.0046]. A kernel faithful to the trained operator needs a first pass for the row maximum, and its bound is 1.5× FA4's (Appendix I.10). These are estimates from FA4's published numbers.

A coarse exponential is cheaper on integer and lookup hardware. The fixed-window operator needs no exponential at run time: its K+1 values are constants shared by all rows, heads and layers, and the per-key work is a subtract, a scale, a round, one table read and a multiply. Appendix I.10 gives operation counts for every operator. IntAttention's IndexSoftmax uses the same fixed-window construction (6.6 nats, 32 levels). On an RK3588S2 (Armv8) CPU its authors report their integer attention pipeline, built on that operator, up to 2.0× faster than an INT8 attention pipeline that keeps a floating-point softmax, and up to 3.7× faster than FP16 attention. These are their measurements of attention latency at inference; the 3.7× includes the gain from INT8 matrix products.

What the paper contributes to that setting is the training side, for the reason you point out: the trained weights depend on the training operator, so the operator used at deployment must also be used in training, and then its backward rule decides whether training works. One new data point: the model we trained for 2.5B tokens with IndexSoftmax loses 0.008–0.009 nats when its operator is replaced at evaluation by exact softmax or by the float-normalized variant of the same lookup (Appendix I.7). Even a 32-level operator is not interchangeable after training.

§1 now states the scope in these terms, and §6 has a deployment paragraph that states these points, with the numbers in Appendix I.10.

AuthorsOpenReview ↗

Reply to Reviewer VyUR (2/5): Weakness 2

Weakness 2

Reviewer comment

FlashAttention-4 are not cited. FlashAttention-4 (Zadouri et al., 2026, arXiv 2603.05451) already eases the exponential bottleneck on Blackwell GPUs by emulating the exponential with a polynomial on the FMA units, split with the hardware special-function unit, at BF16 accuracy and with no change to training. This is direct evidence for the bottleneck the paper is motivated by, and an existing approximate exponential in production attention kernels. In my opinion, author should discuss and argue of the positioning of the paper vs FA4, which is the main standard for current attention calculation (including the Softmax/exponent operation)

Response

FlashAttention-4 (FA4) is now cited and positioned in §1, as follows. FA4 emulates part of the exponentials in software, with a polynomial whose result is nearly indistinguishable from the hardware one after rounding to BF16, so the attention it computes matches exact softmax at that precision and no learning-rule question arises. We study approximations coarse enough to change the attention probabilities (as few as 5 levels in the main results), where the backward rule becomes a design choice. The two are complementary, and FA4's measurements are direct evidence for the bottleneck that motivates our question.

Our BF16 baselines sit at FA4's precision tier and now have three seeds (Appendix I.5). With the exponential in bf16, ΔNLL is −0.0041 [−0.0110, +0.0027] at 100M tokens and +0.0011 at 250M. With exponential, row sum and division all in bf16 it is −0.0057 [−0.0331, +0.0218] and −0.0022. At that precision, pretraining from scratch is unaffected in our setting.

AuthorsOpenReview ↗

Reply to Reviewer VyUR (3/5): Weakness 3

Weakness 3

Reviewer comment

No existing approximate-softmax operator is trained as a baseline. Section 2 notes that EXAQ and BAPS were not tested in training and that IndexSoftmax is training-free, but the paper does not train any of them either. The only external baselines are BF16-exp and FP8-rounded exp, run on one seed at 124M and 100M tokens (Section 5.3), and they already end within 0.004 nats of softmax. FWM also looks very close to IndexSoftmax as described in Section 2 (a fixed window below the row maximum with a zero tail), so the operators themselves are not new and the contribution rests on the training analysis. Without such baselines, it is hard to see what the K-interval operators add.

Response

We agree on both counts. FWM uses the same construction as IndexSoftmax, a fixed window anchored at the row maximum with zero weight below it; the two differ in window width, number of levels, rounding and table precision. And the previous submission trained none of the operators it cited. The revision §2 now states the first directly, and Appendix H has a side-by-side table of FWM, IndexSoftmax and EXAQ. For the second we trained both (Appendix I.6, I.7). Both operators are defined for inference, so training them requires choosing a backward rule; we used the probability-level straight-through rule with the row maximum on the graph.

Implementation. IndexSoftmax follows the PyTorch implementation released with IntAttention. With float normalization our forward reproduces theirs bit for bit, and the UINT8 variant reproduces the integer codes of their integer path. Q, K and V stay in floating point. The original EXAQ code is no longer public; we follow the EXAQ paper and the reimplementation in the IntAttention artifact. Run on the same machine, our forward reproduces that reimplementation bit for bit; on the training machines its outputs differ from that reference by at most 2.0×10⁻⁶, with no change of grid index.

ΔNLL vs softmax, nats100M, 3 seeds [95% t interval]250M2.5B
IndexSoftmax, float normalization−0.010 [−0.045, +0.025]+0.0014—
IndexSoftmax, UINT8 probabilities−0.038 [−0.083, +0.007]+0.0003+0.0172 [+0.0156, +0.0189]
EXAQ 3-bit, statistics on the graph+0.067 [+0.057, +0.078]+0.036—
EXAQ 3-bit, statistics detached+0.088 [+0.076, +0.100]+0.092—

The BF16 and FP8 baselines you mentioned now have three seeds and a 250M run; all are within noise of softmax (Appendix I.5). These baselines still evaluate the exponential for every key; a fixed-window lookup evaluates none at run time.

On what K-interval adds: the family parameterizes such lookup operators along three axes, and the published operators land where it predicts. IndexSoftmax (32 levels, fixed window) behaves like FWM at K ≥ 16, near softmax at short horizons. EXAQ at 3 bits (8 levels, data-calibrated window) sits at the coarse, calibrated end where the paper locates the failures, and keeping its statistics on the graph is worth 0.02 nats at 100M and 0.055 at 250M.

At 2.5B, IndexSoftmax with UINT8 probabilities ends at +0.017, level with FWM–Prob K=4 (+0.019) and 0.012–0.013 behind MinMax–LERP K=4 and FWM–Weight K=16, which use fewer levels. That difference comes from one seed and is confounded (table rounding, truncation, UINT8 output, surrogate), so we report it without attributing it to any one of these factors. For reference scale, the two softmax seeds at 2.5B differ by 0.0011 nats in validation NLL (Appendix I.9), an order of magnitude below this difference. This result comes from an inference operator used outside its design: trained from scratch at 1024 context, with its forward unchanged and a backward of our choice. BAPS has no public code; the MXFP8-style baseline is the closest standard-format analogue.

Reviewer_2E27OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies trainability of K-interval approximate softmax/attention operators in GPT-2-style pretraining. It varies calibration (MinMax vs fixed-window FWM), reconstruction (LERP vs nearest), and surrogate placement (Weight-STE vs Prob-STE). The central finding is that the backward through per-row calibration matters: detaching row extrema causes delayed failure, and the maximum-dependent calibration gradient is the dominant channel. MinMax–Weight–Nearest with K=4 degrades badly, while moving the surrogate to probabilities or using FWM removes most of the gap. LERP is close to softmax. The paper supports this with four suites, multiple seeds, geometry diagnostics, and downstream evaluations.

Critical strengths

1.Controlled identification of the calibration-gradient defect. The detach vs full-gradient comparison, and especially the max-only/min-only ablation, isolate a concrete failure mode. This is consequential because it explains a delayed training failure that forward-only evaluation would miss. 2.Extensive and reproducible empirical suite. Multiple suites, seeds, confidence intervals, geometry probes, and downstream evaluations make the main empirical claims credible.

Critical weaknesses

1.Mechanism remains partly hypothetical. The “range–resolution feedback” story is supported by associations, not by a decisive intervention. This limits the mechanistic interpretation, though not the main empirical finding. 2.Presentation is heavy. The large operator family and many appendices make it hard to extract the main takeaway; some downstream near-baseline differences are small and single-seed.

Questions for authors

1.Can you run an intervention that directly tests the range–resolution feedback hypothesis, e.g. freezing h, QK-normalization, or swapping the surrogate from a saved checkpoint? 2.Which near-baseline downstream differences are robust across seeds, and which should be treated as noise? 3.What is the practical cost/accuracy trade-off of the best near-baseline operators relative to exact softmax?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 2E27 (1/2): Weakness 1 and Question 1

We highly appreciate your valuable, detailed reviewing work. Thank you.

Our reply has two parts here: Weakness 1 with Question 1 in this comment, then Weakness 2, Question 2 and Question 3.

Weakness 1 and Question 1

Reviewer comment (Weakness 1)

Mechanism remains partly hypothetical. The “range–resolution feedback” story is supported by associations, not by a decisive intervention. This limits the mechanistic interpretation, though not the main empirical finding.

Reviewer comment (Question 1)

Can you run an intervention that directly tests the range–resolution feedback hypothesis, e.g. freezing h, QK-normalization, or swapping the surrogate from a saved checkpoint?

Response

We ran two of the three interventions you name (Appendix I.3).

Switching the surrogate from saved checkpoints (Appendix I.3). From the 250M-token MinMax K=4 runs we branched at 50M and at 125M tokens, restored the optimizer state, and continued to 250M under either rule. Branches that keep their rule reproduce the original runs (within 0.0072 nats for Weight-STE, 0.0002 for Prob-STE).

branch pointWeight→Prob removesProb→Weight reproducesfrozen step removes
50M97.3%93.7%64.4%
125M65.6%53.8%19.5%

Percentages are of the 250M gap between the two unswitched branches (0.32 nats). The score span follows the rule in force. From 50M it ends at 55.8 for Weight→Weight, 21.6 after switching to Prob, 62.9 after switching to Weight, and 21.9 for Prob→Prob.

On three seeds of the 100M suite (branch at 40M) the effect has the same sign on every seed: Weight→Prob −0.074 [−0.098, −0.049], Prob→Weight +0.067 [+0.054, +0.080] (mean, 95% t interval). By 100M tokens each switched branch ends within 0.015 nats of the branch that used its new rule from the start.

On the 2.5B lineage (branch at 250M, continued to 625M) the Weight→Prob branch shows no loss spike and ends 1.45 nats below Weight→Weight. Weight→Weight spikes again (NLL 8.36 at 525M), at a later point than in the original run (highest evaluated NLL at 375M); we report this group as supporting evidence.

Freezing h. Continuing under Weight-STE with a per-layer constant step removes 64.4% of the gap from 50M and 19.5% from 125M, and from 50M the span stops growing (18.9 against 55.8). Two caveats are stated in the paper. The frozen-step operator extends the grid into the tail, so its forward differs from MinMax: evaluated on the branch-point weights, the swap alone changes NLL by +0.016 at 50M and +0.233 at 125M (Appendix I.3). And the effect is partial.

Detaching the row maximum on a static grid (Appendix I.2). Under FWM the step is constant, so detaching the row maximum removes only the derivative through the grid origin. Training still fails:

operator (100M tokens, two seeds)detach − full, natsspan, detach vs full
FWM–LERP K=4+0.52, +0.83660–785 vs 28
FWM–LERP K=16+0.09, +0.0656–57 vs 25
FWM–Nearest K=4, Weight-STE+1.08, +1.14762–810 vs 19

At 250M the FWM–LERP K=4 penalty is +1.71. The same pattern appears in two further operators whose calibration statistics are off the graph (EXAQ with its tensor-level statistics detached, and a per-head window whose width is an exponential moving average of the row span; Appendix I.4, I.6). On the static grid the failure occurs with a constant step, so it does not go through the interval width; the feedback hypothesis concerns MinMax–Weight only.

What this changes in the paper.

§5.4 no longer says that no intervention identifies the mechanism. It now states three things.

  1. The surrogate rule is a cause of the MinMax–Weight gap. Switching it from a saved checkpoint, from Weight-STE to Prob-STE or the reverse, moves loss and span toward the trajectory of the new rule, almost fully from 50M and partly from 125M.
  2. The interval width carries part of the gap: freezing the step removes 64.4% of it from 50M and 19.5% from 125M, shares that include the effect of the changed tail.
  3. The range–resolution feedback is stated as a hypothesis; points 1 and 2 are the two of its links that the interventions support. The three observations that limit it stay in the text.

What we did not run during the discussion period, and why.

QK-normalization changes the architecture: it needs its own softmax baseline and seeds, and it would test the feedback loop in a different model. We chose the two checkpoint interventions because they act on the runs the paper analyses.

A learning-rate sweep was not run because each learning rate would need its own softmax baseline and MinMax–Weight runs, which was beyond our time and compute in this discussion period.

AuthorsOpenReview ↗

Reply to Reviewer 2E27 (2/2): Weakness 2, Question 2 and Question 3

Weakness 2

Reviewer comment

Presentation is heavy. The large operator family and many appendices make it hard to extract the main takeaway; some downstream near-baseline differences are small and single-seed.

Response

Each of the three claims in the introduction now ends with the sections, figures and tables that support it to help to read and locate. §3 opens with a short quantization view, and we added the notation table. New results are in Appendix I and are summarized in the main text. Table 4 puts the calibration-gradient result for the four calibration types in one place, and Appendix I opens with a table of the new runs grouped by the question that each run answers (Table S28). Whether the small near-baseline downstream differences hold across seeds is answered under Question 2.

Question 2

Reviewer comment

Which near-baseline downstream differences are robust across seeds, and which should be treated as noise?

Response

We trained a second seed (7) of softmax and of LERP K=16 for 2.5B tokens and ran the full downstream evaluation (Appendix I.9).

What holds on both seeds:

  • The validation NLL gap. LERP K=16 − softmax is +0.0011 [−0.0003, +0.0024] on seed 7 and +0.0003 [−0.0013, +0.0020] on seed 1337.
  • The sign on BLiMP. The difference is −1.26 percentage points (pp) [−1.85, −0.66] on seed 1337 and −0.57 pp [−1.18, +0.01] on seed 7: same sign, about half the size. BLiMP is also the only benchmark on which every near-baseline run has a negative point estimate (seven runs, −0.5 to −1.6 pp).
  • The sign on held-out PTB. LERP K=16 − softmax is −0.058 nats on seed 1337 and −0.011 on seed 7, and both intervals exclude zero: same sign, about a fifth of the size.

What changes between the seeds:

  • Of the other five benchmarks, the LERP K=16 difference changes sign on four (HellaSwag, ARC-Easy, LAMBADA, SciQ) and keeps it on PIQA; on all five, both intervals include zero.
  • On held-out WikiText-103 it changes sign with both intervals excluding zero (−0.008 on 1337, +0.013 on 7).

For scale, the two softmax runs differ from each other by −1.57 pp [−2.50, −0.68] on LAMBADA, by 0.4 pp on BLiMP, and by −0.008 and −0.016 nats on held-out WikiText-103 and PTB (intervals exclude zero). Item-level intervals therefore do not cover run-to-run variation, and §5.5 now reports these differences between the two softmax runs alongside the near-baseline results.

The large-gap group has one seed. All 35 of its comparisons are negative and 27 reach q < 0.05 after Benjamini–Hochberg correction; MinMax–Weight K=4 loses 3.9–19.4 pp, against at most 1.6 pp between the two softmax runs. Softmax and LERP K=16 now have two seeds each and the other near-baseline conditions one. The near-baseline cells outside BLiMP that reached q < 0.05 (LERP K=2 on ARC-Easy and LAMBADA, FWM–Weight K=16 on the two SciQ configurations) have one seed and were not retested; we treat them as unconfirmed. A third seed was beyond our time and compute budget in this discussion period; we will add it in a later version.

Question 3

Reviewer comment

What is the practical cost/accuracy trade-off of the best near-baseline operators relative to exact softmax?

Response

Accuracy at 2.5B tokens (seed 1337):

operatorgrid levelsΔNLL vs softmax
LERP K=16 (MinMax)17+0.0003
FWM–Weight K=1617+0.0039
LERP K=4 (MinMax)5+0.0047
FWM–LERP K=4 (new)5+0.0069
FWM–Prob K=45+0.0193

Cost (Appendix I.10). MinMax needs the row minimum before any key can be indexed, and K+1 exponentials per row. FWM needs no exponential at run time: its K+1 grid values are constants shared by all rows. Per key it is a subtract, a scale, a round, one table read (two for LERP) and a multiply.

We do not claim a GPU speed-up. Using the published roofline of FlashAttention-4 (FA4), a single-pass lookup kernel at best ties FA4, because the matrix multiply is co-limiting. It also computes a different function: its grid would be anchored at the running maximum, not the row maximum (measured on trained endpoints: up to +0.0041 nats). A kernel faithful to the trained operator needs a first pass for the row maximum and is bounded at 1.5× FA4's cycle count. These are estimates from FA4's published numbers.

Where the exponential is expensive (integer and lookup hardware), IntAttention gives a measurement. It is an integer attention pipeline whose IndexSoftmax uses the same fixed-window construction as FWM. On an RK3588S2 (Armv8) CPU its authors report it up to 2.0× faster than an INT8 attention pipeline that keeps a floating-point softmax, and up to 3.7× faster than FP16 attention. These are their measurements of attention latency at inference; the 3.7× includes the gain from INT8 matrix products.


Thanks again for your review; your comments have helped us improve the paper. If you have further questions or comments, please let us know.

Thank you for the clarifications. Most of my concerns have been resolved. Therefore, I maintain my score.

Thank you for reading our response and for your feedback throughout the review. Your comments were a pleasure for us to work through, and they improved our revision. If there is a remaining concern we can still address, we would be glad to. Thanks again!

Reviewer_kgzcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the effect of quantizing the softmax during LLM pertaining, or more specifically quantizing the unnormalized exponential of the softmax using a lookup table based quantizer. It considers various choices of the quantization function (e.g. placement of STE, calibration, interpolation vs hard rounding) and then study the effect on the gradient and e2e training. The authors show that some combinations, e.g. MinMax grids with STE (weight-STE) has a significant loss gap after training, while FWM (fixed-window max) reduces this gap noticeably. Overall, they also notice that the backwards rule and calibration strategy have a larger impact than the forward approximation.

Critical strengths
  • The paper does study a fairly under explore topic, namely how does quantizing the softmax influence pertaining of LLMs. Quantizing softmax is studied in the inference context, but a detailed analysis on how choices effect pertaining with (simulated) softmax quantization is novel.
  • I find it interesting that they look both at forward quantization choices (calibration, reconstruction, granularity) and backward choices (gradients, STE …) and analyze each of their effects.
  • The detailed empirical study does rigorously isolate the effect of different aspects in a controlled manner. They also study multiple seeds and show that results are significant. The main limitation of the empirical study is the narrow scope and small scale in terms of model size and training tokens.
Critical weaknesses
  • The scope and applicability of the work is fairly limited and practical applicability to common quantization use-cases remains unclear. To be clear, with this I do not mean that they do not show any practical speedups, that is totally fine for a more analysis oriented paper, but rather the setting they study:
    • The work studies quantization of the softmax exponential, while leaving the normalization and reciprocal computation largely outside the scope.
    • They do not study commonly used quantization assumptions, e.g. MXFP8, NVFP4, INT8 etc, but rather focus on an fairly theoretical lookup based quantizers which is highly dynamic depending on the input data.
    • Because the quantization grid is recalibrated on every row statistics (M, m) many of the paper's findings are specific to highly dynamic calibration. It is therefore unclear to what extent the observed effects, particularly the importance of calibration gradients and the MinMax-vs-FWM behavior, transfer to quantization schemes with static or semi-static calibration parameters such as INT8 activation quantization, MXFP8 block scaling, or NVFP4-style scaling.
    • They never state for what reason or setup they quantize, e.g. inference time speedups, training speedups, memory savings etc. As they do not discuss low precision in the backwards part I assume the scope is limited to inference time applications.
  • The paper is very hard to follow. The language is overall good and on a sentence level it is easy to follow. However, the big picture and the overall contributions get lost in many detailed descriptions. A couple of specific other things that could also improve:
    • Notation is outsourced to “table 1” which is actually a table and then a hardly readable text paragraph unrelated to the table content.
    • It is hard to follow quantization assumptions, e.g. when reading it is for long unclear what quantization format they use and whether parameters are statically calibrated or dynamic at runtime. It is also mostly unclear how this connects to fixed-point/INT quantization or modern data types like MXFP or NVFP4.
    • For someone coming from an ML-related quantization background, the notation and terminology is very hard to follow, e.g. they nowhere introduce q(.) that clearly shows what they quantize or what the function q(.) is, use confusing terms like weight for unnormalized softmax scores which are activations in a neural network sense any many more.
  • Because of the limitations in w1 they also miss studying or discussing important softmax quantization issues such as vanishing probability mass of quantized softmax.
Questions for authors

How do your assumptions map to different bit-widths or quantization choices?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer kgzc (1/3): Weakness 1, first and second points

We highly appreciate your valuable, detailed review. Thank you.

Several of your points come from things the previous submission did not say clearly. We have rewritten those parts and added the experiments you asked about to the revision. Our reply has three parts: Weakness 1 in parts 1 and 2; Weakness 2, Weakness 3 and your question in part 3.

Weakness 1

Reviewer comment

The scope and applicability of the work is fairly limited and practical applicability to common quantization use-cases remains unclear. To be clear, with this I do not mean that they do not show any practical speedups, that is totally fine for a more analysis oriented paper, but rather the setting they study:

Response

The paper studies pretraining from scratch in which the forward pass uses the approximate softmax and the backward pass uses an explicitly defined rule. The abstract says this, and §1 now states it directly. Your four points are answered here in turn: the first two in this comment, the third and fourth in the next.

First point

Reviewer comment

The work studies quantization of the softmax exponential, while leaving the normalization and reciprocal computation largely outside the scope.

Response

Correct for K-interval attention, and now stated in §2: reconstruction is followed by an fp32 row sum and division. Two new data points go further (Appendix I.5, I.7; ΔNLL vs softmax, nats).

  • BF16 full chain (exponential, row sum and division all in bf16): −0.0057 [−0.0331, +0.0218] at 100M over 3 seeds, −0.0022 at 250M.
  • IndexSoftmax (the lookup operator of IntAttention) with its authors' UINT8 output normalization, round(255·p)/255 without renormalization: within noise at 100M, +0.0003 at 250M, but +0.0172 [+0.0156, +0.0189] at 2.5B. There the operator still sets to zero keys that carry 10% of the softmax mass (15% on rows longer than 512 keys).

We report the second as consistent with a cost of quantized normalization at long horizon; at 2.5B only the UINT8 variant was trained. Integer division and reciprocal lookup are left for future work.

Second point

Reviewer comment

They do not study commonly used quantization assumptions, e.g. MXFP8, NVFP4, INT8 etc, but rather focus on an fairly theoretical lookup based quantizers which is highly dynamic depending on the input data.

Response

We trained five format baselines, including MXFP8-style and NVFP4-style rules, in which e^z is rounded to the format and the backward is the exponential's own derivative (native autograd for BF16), not a rule built on the grid: three seeds at 100M tokens and one 250M run each (Appendix I.5). ΔNLL vs softmax, nats:

e^z rounded to100M, mean of 3 seeds [95% t interval]250M
BF16−0.0041 [−0.0110, +0.0027]+0.0011
FP8 E4M3+0.0004 [−0.0056, +0.0063]−0.0020
MXFP8-style (block 32, power-of-two scale)−0.0107 [−0.0265, +0.0051]−0.0008
NVFP4-style (block 16, E2M1, E4M3 scale)−0.0072 [−0.0300, +0.0157]−0.0021
UINT8 uniform−0.0084 [−0.0354, +0.0185]+0.0005

All five are within noise of softmax, including the NVFP4-style rule with 7 non-zero levels per block. Two points on reading the table.

  1. Each baseline applies a format's rounding rule to e^z in fp32. The recipes themselves keep softmax at its original precision. In all five the exponential is still evaluated for every key; a lookup on a fixed grid evaluates none at run time.
  2. The contrast with our K=4 failures is informative. In these baselines the gradient does not involve the grid, and no rule has a step that grows with the learned score span. The failures we report occur when a coarse grid is calibrated on the data and the backward is built on that grid. The paper's conclusions concern calibration and the backward rule rather than bit width; §5.3 now says so.
AuthorsOpenReview ↗

Reply to Reviewer kgzc (2/3): Weakness 1, third and fourth points

Weakness 1 (continued)

Third point

Reviewer comment

Because the quantization grid is recalibrated on every row statistics (M, m) many of the paper's findings are specific to highly dynamic calibration. It is therefore unclear to what extent the observed effects, particularly the importance of calibration gradients and the MinMax-vs-FWM behavior, transfer to quantization schemes with static or semi-static calibration parameters such as INT8 activation quantization, MXFP8 block scaling, or NVFP4-style scaling.

Response

FWM is static in the row-max-shifted coordinate: its step and all grid values are constants. The paper's main comparison, MinMax against FWM, is therefore per-row dynamic against static calibration, and the static side trains much better. What the previous submission did not test is whether the calibration gradient still matters there. We have now tested it in four settings in the new revised submission (Appendix I.2, I.4, I.6):

calibrationquantity taken off the grapheffect on NLL, natsscore span, off vs on graph
static grid (FWM), K=4row maximum (the origin)+0.52 to +1.14 at 100M; +1.71 at 250M660–810 vs 19–28
static grid (FWM), K=16row maximum+0.06 to +0.0956–57 vs 25
tensor-level statistics (EXAQ, 3-bit)σ and minimum+0.021 [+0.0002, +0.041] at 100M, 3 seeds; +0.055 at 250M114–118 vs 25–26
per-head window whose width is an exponential moving average (EMA) of the row span, K=4 (a buffer, as in the observers of quantization-aware training frameworks)the width, by constructionboth surrogates end +0.21 to +0.22 above softmax at 100M51–54 (Weight-STE), 677–800 (Prob-STE)

The finding that transfers is the one about the calibration gradient. In all four settings the backward is built on the grid. When the data-dependent quantity that positions or scales that grid is taken off the graph, the score span grows and a loss gap opens. On the static grid the mean penalty for FWM–LERP is +0.68 at K=4 against +0.08 at K=16.

For the surrogate placement, §5.2 now states the three calibrations under which it was tested. The probability-level surrogate removes most of the deficit under per-row MinMax (in the factorial runs, and when switched in from a saved checkpoint) and a smaller amount under FWM. Under the EMA window at K=4 its benefit is small: Weight − Prob is +0.010 [−0.014, +0.034] at 100M (per-row MinMax: +0.068) and +0.076 at 250M (per-row MinMax: +0.316), and both surrogates stay 0.2–0.3 nats above softmax (Appendix I.4).

For the block-scaled formats you name, see the second point (part 1): with the exponential's own derivative as backward, no calibration quantity enters the gradient, and they train at baseline. We make no claim about quantizers we did not train, such as INT8 activation quantization of linear layers.

Fourth point

Reviewer comment

They never state for what reason or setup they quantize, e.g. inference time speedups, training speedups, memory savings etc. As they do not discuss low precision in the backwards part I assume the scope is limited to inference time applications.

Response

Your assumption is right about where the saving is: it is in the forward pass at deployment, where a table read replaces the exponential on integer and lookup hardware (§6). The paper measures no training speed-up or memory saving, and the attention operator, including its backward pass, is computed in fp32 (§3, §4). Cost is now treated in §6 and Appendix I.10 through operation counts and a roofline estimate.

The scope is nevertheless pretraining, as stated in part 1. Training has to be studied because the trained weights depend on the training operator: substituting exact softmax at evaluation widens the gap in 37 of 42 cells (§5.5). The operator used at deployment therefore has to be used in training as well. This argument is now in §1.

On the training side, current low-precision pretraining recipes quantize the linear layers and keep the softmax path in high precision. The MXFP8 recipe keeps the two attention matrix products and the softmax in high precision and leaves reducing them "to future work". The NVFP4 recipe retains the original precision for "softmax and the query-key and attention score-value batched GEMMs". We ask what happens to training when that step is approximated.

AuthorsOpenReview ↗

Reply to Reviewer kgzc (3/3): Weakness 2, Weakness 3 and Question

Weakness 2

Reviewer comment

The paper is very hard to follow. The language is overall good and on a sentence level it is easy to follow. However, the big picture and the overall contributions get lost in many detailed descriptions. A couple of specific other things that could also improve:

Response

Each of the three claims in the introduction now ends with the sections, figures and tables that support it to help to read and locate. The reply to three specific points follow here.

First point

Reviewer comment

Notation is outsourced to “table 1” which is actually a table and then a hardly readable text paragraph unrelated to the table content.

Response

The notation table has been added in the new submission, separate from the method table.

Second point

Reviewer comment

It is hard to follow quantization assumptions, e.g. when reading it is for long unclear what quantization format they use and whether parameters are statically calibrated or dynamic at runtime. It is also mostly unclear how this connects to fixed-point/INT quantization or modern data types like MXFP or NVFP4.

Response

§3 now states which quantities are per-row and data-dependent. Under MinMax the row maximum and minimum set the origin and the step. Under FWM the step τ/K and all K+1 values are constants; the only per-row quantity is the row maximum, which every softmax implementation subtracts. For the relation to integer and block floating-point formats, see the format baselines under the second point of Weakness 1 (part 1) and the bit-width mapping under Question below.

Third point

Reviewer comment

For someone coming from an ML-related quantization background, the notation and terminology is very hard to follow, e.g. they nowhere introduce q(.) that clearly shows what they quantize or what the function q(.) is, use confusing terms like weight for unnormalized softmax scores which are activations in a neural network sense any many more.

Response

§3 opens with a quantization view. z = s − M is the row-max-shifted score, w = q(z) is the quantized unnormalized attention weight, and P = w / Σw. q has K+1 levels and three design choices: where the grid is placed (calibration), how z is mapped to a level (rounding or interpolation), and which surrogate derivative replaces q′.

The text now says "unnormalized attention weight" and notes at first use that it is an activation, not a parameter. The name Weight-STE is kept and explained.

Weakness 3

Reviewer comment

Because of the limitations in w1 they also miss studying or discussing important softmax quantization issues such as vanishing probability mass of quantized softmax.

Response

The revised submission collects this in one table (Appendix I.8): the softmax mass on keys that each operator sets to zero, at trained endpoints.

operatorsoftmax mass set to zero
FWM, τ=60.6–1.4%
FP8 E4M3 on e^z0.4–0.5%
UINT8 on e^z0.8–1.0%
NVFP4-style2.5–2.7%
IndexSoftmax, float normalization0.7–1.0%
IndexSoftmax, UINT8 probabilities10–13%

For the last row the loss rises with row length: 0.4% for rows up to 64 keys and 15–19% beyond 512 keys over the 2.5B run (Figure S14b), where up to 1.8% of rows were entirely zero during training. In the opposite direction, MinMax–Weight K=4 puts 99.5% of the softmax mass into its top bucket (Appendix D.6). In our runs a loss of 1–3% of the mass is compatible with baseline NLL at 100M and 250M tokens; the 10% case is the one with a gap at 2.5B.

Question

Reviewer comment

How do your assumptions map to different bit-widths or quantization choices?

Response

A K-interval grid has K+1 levels, an index of ⌈log2(K+1)⌉ bits: 3 bits for K=4 and 5 bits for K=16; K=3, 7 and 15 are exactly 2, 3 and 4 bits (Appendix I.10). Measured at 2.5B tokens, the gap to softmax of MinMax–Weight falls from +0.89 nats at 3 bits (K=4) to +0.04 at 7 bits (K=64), and that of FWM–Weight from +0.043 at 3 bits to +0.004 at 5 bits (K=16). On five seeds at 100M tokens the mean slope of the gap per doubling of K is −0.003 to −0.047 nats, depending on the operator (Table S9). For other quantization choices, see the five format baselines under the reply to the second point of Weakness 1 (Comment 1).

A limitation the reviewer noted under Strengths

Reviewer comment

The main limitation of the empirical study is the narrow scope and small scale in terms of model size and training tokens.

Response

We agree. Because of the time and compute available in the discussion period, model size and token budget are unchanged in this round: the largest runs are 124M parameters at 2.5B tokens and a 1B-parameter probe at 100M tokens (§4), and §6 states this limit. The revision adds four 2.5B-token runs, among them a second seed of softmax and of LERP K=16, used to check which downstream differences hold across seeds (Appendix I.9).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thanks and General response to all reviewers

We sincerely appreciate the three reviewers' time and their contribution to the community. Their comments were very helpful and have helped us improve the quality of this work.

We have added and run experiments that reviewers asked during the discussion period and revised the paper accordingly.

We also released the supplementary material that has code, manifests and per-block results for all experiments in our paper.

This general comment lists what is new; the individual replies would answer each question the reviewers have.

New experiments. 98 training runs (454 GPU-hours of training on RTX 5090), evaluated with the paper's protocol (full validation split, paired block bootstrap; paired t intervals over three seeds at 100M tokens). Appendix I.1 lists them by the question each one answers.

The new results added.

  1. The calibration-gradient result is general. On the static fixed-window grid, detaching only the row maximum costs +0.52 to +1.14 nats at K=4. EXAQ with detached statistics, and a window whose width is an exponential moving average (EMA) of the row span, show the same direction (for Reviewer kgzc).
  2. The surrogate rule is a cause, shown by intervention. Switching the surrogate at 50M of 250M tokens, from Weight-STE to Prob-STE or the reverse, moves the endpoint 94–97% of the way to the other rule's endpoint; on 3 seeds at 100M tokens the effect has the same sign on every seed (for Reviewer 2E27).
  3. A new near-baseline configuration. FWM with LERP at K=4 is near softmax at 100M, 250M and 2.5B tokens (+0.007 at 2.5B), with no straight-through estimator and no run-time exponential. It is the only K=4 configuration in the paper whose point estimate is within 0.01 nats of softmax at all three horizons (for Reviewer VyUR).
  4. Standard formats train at baseline. With e^z rounded to BF16, FP8-E4M3, an MXFP8-style or NVFP4-style block format, or UINT8, training is within noise of softmax on 3 seeds and at 250M. So is a full BF16 softmax chain (for Reviewer kgzc, VyUR).
  5. Published operators as baselines. IndexSoftmax, bit-aligned with its authors' PyTorch implementation, trains within noise of softmax at 100M and 250M and ends +0.017 at 2.5B; EXAQ 3-bit ends +0.04 to +0.09 (for Reviewer VyUR).
  6. Robustness. The validation gap of LERP K=16 reproduces on a second 2.5B seed (+0.0011 against +0.0003). Re-evaluating six endpoints under other settings moves NLL by at most 0.0015 nats when the kernel computes the same function (0.0041 with a single-pass kernel), and retraining with TF32 off by +0.003 and −0.006 (for Reviewer 2E27, VyUR).

Results that refine the submission.

  1. Downstream differences near the baseline. On the second seed the BLiMP difference of LERP K=16 has the same sign and about half the size (−0.57 percentage points against −1.26). Two softmax seeds differ by 1.6 percentage points on one benchmark. §5.5 now reports the differences between the two softmax runs alongside the near-baseline results.
  2. LERP at K=4. Its slow start under MinMax calibration is absent under FWM, so we now attribute it to the calibration, not to interpolation.
  3. Where the probability-level surrogate helps. It removes most of the deficit under per-row MinMax and a smaller amount under FWM. With a per-head EMA window held in a buffer its benefit is small (+0.010 nats at 100M). §5.2 now states the three calibrations under which it was tested.

Changes to the submitted paper.

  • Scope, in §1: an analysis of pretraining with an approximate softmax, in which the forward operator and the backward rule are both objects of study; all operators are computed in fp32 (§3). FlashAttention-4 is cited and positioned.
  • §3 opens with a quantization view that introduces q(·); the notation has its own table.
  • §5.4 is rewritten around the interventions; a new table in §5.1 (Table 4) shows the calibration-gradient result for per-row, static, tensor-level and EMA calibration.
  • §5.5 reports near-baseline downstream differences together with the differences between two softmax seeds.
  • The pre-specification of the downstream evaluation is stated exactly (§5.5, Appendix C.3, E.1). Fixed before any downstream run: the three condition groups, four of the five predictions, the ±0.01-nat margin and the decision rules. The fifth prediction was added after the held-out-corpus results. The three benchmarks on which softmax scored below chance were not carried further, a choice made on softmax-only results; the submission attributed it to a pre-registered screen.
  • §6 has a deployment paragraph; operation counts and a roofline estimate are in Appendix I.10.
  • Page overflows, the lm-evaluation-harness reference and the labelling issues are fixed.
  • New results are in Appendix I and summarized in the main text.
AuthorsOpenReview ↗

Summary of the paper's contributions and of the discussion

We summarise the paper's contributions and what changed in the discussion period. Details and numbers are in the general response and the individual replies.

What the paper shows. Low-precision training recipes such as MXFP8 and NVFP4 keep the attention softmax in high precision. This paper asks whether a coarse softmax, a lookup with K+1 grid values, can be used in pretraining from scratch. The results show that the backward rule and the grid calibration matter more than how coarse the forward is, which gives practical guidance for training with a coarse or quantized softmax. With the right choices, a five-level lookup trains to within 0.007 nats of the softmax baseline at 2.5B tokens.

  1. The full calibration gradient matters. Detaching the row extrema (quantization-aware training frameworks likewise keep their observer statistics out of the gradient computation) leaves the forward computation unchanged but produces a delayed increase in validation loss, under both min–max and fixed-window calibration (+0.52 to +1.14 nats at K=4 on the fixed-window grid). Tensor-level and moving-average calibration show the same direction (Table 4).
  2. Calibration and surrogate placement interact. With hard rounding at K=4, min–max calibration with a pre-normalization surrogate ends +0.89 nats above softmax at 2.5B tokens; changing either choice (the calibration or the surrogate placement) removes most of the gap at the end of training. Switching the surrogate between pre- and post-normalization, in either direction, at a checkpoint saved at 50M of 250M tokens, moves the final loss across 94–97% of the 0.32-nat endpoint gap between the two placements, so the surrogate placement is a cause of the gap.
  3. Models trained with very coarse operators can reach near-baseline loss. At 124M parameters and 2.5B tokens, a fixed-window grid with interpolation ends +0.007 nats from softmax at K=4 (five levels), and hard rounding ends +0.019 at K=4 and +0.004 at K=16.

What the discussion period added (98 training runs, 454 GPU-hours of training; Appendix I).

  • For Reviewer 2E27 (mechanism, seeds): interventions from saved checkpoints, and a second seed at 2.5B tokens. The reviewer has replied that most concerns are resolved.
  • For Reviewer kgzc (scope, standard formats, static calibration, presentation): five format baselines (BF16, FP8, MXFP8-style, NVFP4-style, UINT8) train within noise of softmax; the calibration-gradient result is tested under static, tensor-level and moving-average calibration; §1 and §3 state the scope and the quantization view; a table reports the probability mass each operator sets to zero.
  • For Reviewer VyUR (motivation, FlashAttention-4, baselines): FlashAttention-4 is cited and positioned, with a roofline estimate; IndexSoftmax and EXAQ are trained as baselines; fixed-window calibration with interpolation is trained at three horizons; the reported gaps move by at most 0.0015 nats under other evaluation settings that compute the same function; the formatting issues are fixed.

Thank you all for your valuable time and work.

Regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.