acceptodds
Withdrawn from ICLR 2027

Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On NVIDIA Blackwell B200, tensor-core throughput outpaces special-function exponential throughput by more than two orders of magnitude, exposing exponential evaluation in fused attention kernels. A pretrained Transformer, however, may not need it evaluated accurately at every element. We characterize what a pretrained model does need by approximating softmax at inference in ten frozen decoder-only models (0.5B–72B). The number of positions the softmax map assigns probability to and within-row resolution can be cut substantially, yet uniform weighting of the same positions is damaging. Where a fixed resolution budget is placed matters as much as its size, with resolution near the row maximum consistently favored. Perturbations matched on scalar distortion produce model-dependent responses of opposite sign. These findings motivate Rowmax-PoT, a coarse logarithmic weight representation anchored at each row maximum, and Rowmax-H15, its hardware specialization in FlashAttention-4. On B200, the patched FP8 attention forward is 12.4% faster at causal 8K and 25.8% faster at non-causal 8K in host-side call-latency measurements; board energy per forward falls by 8.4% at causal 16K. Measured separately on the BF16 kernel path at 2K, Rowmax-H15 increases perplexity by 0.091–0.492% across five models from three families.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
attention, softmax approximation, pretrained language models, efficient inference, FlashAttention, GPU kernels

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rQQMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It appears to be an auto-research work, and the authors did not even proofread the manuscript. I am not able to understand the content and message of this paper. At a very high level, this work proposes Rowmax-H15, an implementation that patches flash attention and approximates softmax.

Critical strengths

At least the authors are honest that "Generative AI assistants (code-capable large language models) were used in the design, implementation, analysis, and writing of this work."

Critical weaknesses
  • This is purely AI-generated and not readable. For example, what is JSD? Is it Jensen-Shannon Divergence? This terminology appears in the second paragraph of the introduction and is never defined, and it also serves as a core component in the analysis. The paper is full of unnecessary details that no human researcher would put in the main text. For example, sec. 3.1, "the test stream is cut into 97 aligned blocks of 2048 tokens (198,559 predictions)", and line 362, FA4 "commit 0dc2cb48"
Questions for authors

N/A. Just withdraw your paper.

1 comment

We respond to each point factually.

On “JSD”. You are right that the main text does not expand the acronym. It is the Jensen–Shannon divergence, as you inferred. Its formula is Eq. (21) in Appendix A.7, and Sec. 3.2 points to Appendices A.6–A.7, but it should have been defined at first use (line 044). We will revise the presentation.

On the details you call unnecessary. The block and prediction counts in Sec. 3.1 (97 blocks × 2,047 predictions = 198,559, the sample behind our primary NLL results and their bootstrap intervals) and the FlashAttention-4 commit (line 362) are there so that readers can reproduce and check our numbers. For the same reason, the Reproducibility Statement (lines 488–499) says that the code and results would be provided during the discussion period.

On AI use. We disclosed it because the ICLR 2027 AI Policy for Authors requires disclosure. The statement you quote continues (lines 503–520): the reported measurements, figures and tables derive from executed code and recorded experimental outputs, and no experimental result was supplied by an AI model; the authors chose the research questions and experiments, ran or supervised the evaluations and hardware measurements, checked the reported results against stored outputs, reviewed the manuscript before submission, and take responsibility for its content.

On whether the numbers are real. They are real. The supplementary material, uploaded during the discussion period as the Reproducibility Statement promised, contains the code, the per-block and per-run result files behind every figure and table, and a provenance map from each figure and table to its input files. One included script recomputes 794 numeric table cells of the paper from those files in a few seconds, without a GPU (python paper/tables/check_tables.py).

Your review does not identify any number that is inconsistent, implausible or irreproducible. If you have one in mind, please name it; the provenance map points to the exact files behind it. Otherwise, we respectfully ask you to withdraw the ethics flag. We ask this even though we are withdrawing the submission, because the review will remain public. We have also asked the Area Chair to assess it on the evidence.

For the public record, we intend to withdraw this submission, for two reasons. Reviewers o371 and M7it identified comparisons that require new kernel experiments, and we accept those points. And this review is a significant part of why we will not continue in this cycle: it alleges fabrication without identifying a single number, and it gives us nothing technical that a revision or an experiment could address. Withdrawing does not mean we accept your unsubstantiated allegation.

We stand by the reported results.

Reviewer_M7itOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how frozen pretrained LLMs respond to softmax approximations, examining the effects of attention support, relative weights, and numerical resolution across multiple models. The authors propose Rowmax-PoT and its hardware-specialized variant, Rowmax-H15, and integrate Rowmax-H15 into FlashAttention-4. Experiments on NVIDIA B200 GPUs demonstrate attention-kernel speedups, while separate BF16 evaluations at a 2K context length report small perplexity increases.

Critical strengths

S1. The paper uses controlled experiments to disentangle factors affecting softmax approximation. In particular, single-layer experiments with the same attention support set help isolate the importance of relative attention weights, while cross-model experiments support the conclusion that more numerical resolution should be allocated near the row maximum.

S2. The authors implement Rowmax-H15 in an actual fused attention kernel and compare it with stock FlashAttention-4 under matched benchmarking conditions. In addition, the paper directly evaluates the BF16 kernel on five models and clearly distinguishes direct measurements from simulation results.

Critical weaknesses

W1. The paper's main model-fidelity and speedup results are obtained under different experimental configurations. Direct model-fidelity evaluations use BF16 at a 2K context length, whereas the main speedup results use FP8 at 8K–16K context lengths. Consequently, the accuracy–speed trade-off under the actual accelerated configuration remains unverified. Moreover, the FP8 semantic simulation does not quantize the attention operands to FP8 and therefore cannot fully address this concern.

W2. The choice of approximation resolution is not sufficiently justified. Appendix B.2 shows that higher-resolution variants achieve smaller NLL degradation in offline experiments and can use a similar addition-and-shift operation pattern. However, the authors have not evaluated these variants inside FlashAttention-4. Without a comparison of their actual kernel performance, it remains unclear what performance benefit justifies the additional approximation error introduced by H15's coarser resolution.

Questions for authors

Q1. Can the authors report model fidelity and latency for Rowmax-H15 under the same configuration and context length, particularly FP8 at 8K or 16K? Alternatively, direct BF16 fidelity measurements at these lengths would enable a more directly comparable accuracy–speed assessment.

Q2. How do the finer (k=2,3) variants in Appendix B.2 compare with H15 in terms of actual kernel latency and model fidelity when implemented inside FlashAttention-4?

1 comment
AuthorsOpenReview ↗

Response to Reviewer M7it

Thank you for the responsible, careful and accurate review work. We agree with both weaknesses you give.

W1 and Q1

Reviewer comment

W1. The paper's main model-fidelity and speedup results are obtained under different experimental configurations. Direct model-fidelity evaluations use BF16 at a 2K context length, whereas the main speedup results use FP8 at 8K–16K context lengths. Consequently, the accuracy–speed trade-off under the actual accelerated configuration remains unverified. Moreover, the FP8 semantic simulation does not quantize the attention operands to FP8 and therefore cannot fully address this concern.

Q1. Can the authors report model fidelity and latency for Rowmax-H15 under the same configuration and context length, particularly FP8 at 8K or 16K? Alternatively, direct BF16 fidelity measurements at these lengths would enable a more directly comparable assessment.

Response

The paper contains no configuration in which kernel fidelity and speed are both measured in this round. What exists is:

  • BF16 fidelity: kernel fidelity at 2K only (five models).
  • BF16 timing: 8K and 16K (+5.4% and +7.1% causal, Table B5a), with no BF16 fidelity at those lengths.
  • FP8: timing at 8K–16K, with fidelity only as a semantic simulation that does not quantize Q, K, V and P.
  • 2K, where fidelity was measured: FP8 call latency shows no reliable gain. The three-round sweep, the retest and the traced session are negative (−2.6%, −3.7%, −4.7%), and one single-round run is positive (+7.8%); see Tables B4 and B5.

A direct FP8 fidelity run was attempted and was not interpretable, because per-tensor E4M3 conversion of the BF16 activations added about 0.49 nats of unrelated error (Appendix B.2). So the accuracy–speed trade-off in the accelerated configuration is unverified.

W2 and Q2

Reviewer comment

W2. The choice of approximation resolution is not sufficiently justified. Appendix B.2 shows that higher-resolution variants achieve smaller NLL degradation in offline experiments and can use a similar addition-and-shift operation pattern. However, the authors have not evaluated these variants inside FlashAttention-4. Without a comparison of their actual kernel performance, it remains unclear what performance benefit justifies the additional approximation error introduced by H15's coarser resolution.

Q2. How do the finer (k=2,3) variants in Appendix B.2 compare with H15 in terms of actual kernel latency and model fidelity when implemented inside FlashAttention-4?

Response

Not measured. Appendix B.2 shows that these variants lower offline NLL and follow the same add-and-shift pattern on the tested inputs, so we cannot claim that H15's coarser resolution buys a kernel-level benefit over them. This needs a matched in-kernel comparison.

Presentation

We accept that the main text demands too much effort and will revise it.

Next steps

We will withdraw our submission, and we thank you for your work.

These points need new kernel experiments that we cannot complete in this cycle given our time and compute budget. This is one of the two reasons we intend to withdraw; the other is stated in our response to Reviewer rQQM. We have also uploaded the supplementary material promised in the Reproducibility Statement (code, result files and a provenance map).

Thanks again for the review.

Reviewer_o371OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the sensitivity of frozen pretrained language models to approximations of attention softmax. Controlled interventions examine support size, relative weighting, quantization resolution, resolution allocation, and distributional distortion across ten models. Motivated by these observations, the authors introduce Rowmax-PoT and implement Rowmax-H15, a coarse exponential approximation using values of the form {1, 1.5} times an integer power of two, inside FlashAttention-4. The paper reports B200 attention-forward acceleration and separately evaluates model fidelity using the BF16 kernel.

Critical strengths
  1. The controlled characterization provides useful empirical evidence about softmax approximation. In particular, the single-layer same-support comparison separates relative weighting from support selection, while the fixed-resolution experiments isolate where quantization intervals are allocated. The cross-model results and paired confidence intervals strengthen the observation that finer resolution near the row maximum often preserves model quality better than uniform allocation.

  2. The approximation is implemented within a competitive fused attention kernel rather than evaluated only as an isolated operator. The paper reports actual B200 measurements, includes controls over FlashAttention-4’s existing approximation settings, and directly measures BF16-kernel fidelity in five models. The manuscript also clearly distinguishes measured results from simulations and acknowledges important limitations.

Critical weaknesses
  1. The contribution beyond existing approximation methods is limited. As the paper acknowledges, row-maximum-relative quantization already appears in prior work, and the ideal Rowmax-H15 operator is a half-integer-input specialization of Schraudolph’s classical exponential construction. The characterization adds useful measurements, but much of it refines established intuitions about preserving relative weights and allocating precision to important entries. A distinctive quality–efficiency advantage is therefore important to justify the methodological contribution, yet it is not established against closely related alternatives. Appendix B.2 shows that quarter- and eighth-octave variants reduce offline perplexity degradation from approximately 0.105% to 0.033% and 0.008%, respectively, and can use a similar add-and-shift construction on the tested inputs. Their actual kernel costs are unmeasured, leaving the motivation for the particular H15 operating point insufficiently supported.

  2. The principal speed and fidelity results are obtained under different conditions, leaving the practical tradeoff incomplete. The headline acceleration is measured on FP8 attention at 8K–16K, whereas direct model-fidelity measurements use BF16 at 2K. The FP8 fidelity estimate is a semantic simulation on one model that does not quantize Q, K, V, and P to FP8, so it does not validate the accelerated FP8 configuration. The long-context and zero-shot checks primarily evaluate other characterization operators rather than the H15 kernel. Furthermore, timing covers attention forward rather than full-model inference. Consequently, the paper does not yet demonstrate the claimed combination of useful acceleration and small model degradation in a common deployment setting.

Overall assessment: The controlled experiments are informative and the kernel implementation is useful. However, the methodological novelty is modest, the selected approximation is not sufficiently distinguished from closely related alternatives, and the principal quality–efficiency tradeoff remains incompletely validated.

Questions for authors
  1. What quality–latency advantage does H15 offer over the quarter- and eighth-octave variants in Appendix B.2 when all are implemented inside the same FA4 kernel? A matched comparison would clarify whether the chosen coarse resolution provides a meaningful computational benefit.

  2. Can you directly evaluate the FP8 kernel’s model fidelity at the 8K and 16K settings used for the main speed measurements, with an appropriate FP8 baseline, and report the corresponding full-model prefill latency? This would establish whether the numerical and performance benefits coexist in the same configuration.

1 comment
AuthorsOpenReview ↗

Response to Reviewer o371

Thank you for the responsible, careful and accurate review work.

Weakness 1 and Question 1

Reviewer comment

Weakness 1. The contribution beyond existing approximation methods is limited. As the paper acknowledges, row-maximum-relative quantization already appears in prior work, and the ideal Rowmax-H15 operator is a half-integer-input specialization of Schraudolph's classical exponential construction. The characterization adds useful measurements, but much of it refines established intuitions about preserving relative weights and allocating precision to important entries. A distinctive quality–efficiency advantage is therefore important to justify the methodological contribution, yet it is not established against closely related alternatives. Appendix B.2 shows that quarter- and eighth-octave variants reduce offline perplexity degradation from approximately 0.105% to 0.033% and 0.008%, respectively, and can use a similar add-and-shift construction on the tested inputs. Their actual kernel costs are unmeasured, leaving the motivation for the particular H15 operating point insufficiently supported.

Question 1. What quality–latency advantage does H15 offer over the quarter- and eighth-octave variants in Appendix B.2 when all are implemented inside the same FA4 kernel? A matched comparison would clarify whether the chosen coarse resolution provides a meaningful computational benefit.

Response

We have no in-kernel measurement for k = 2 or 3. Our bit-level tests (Appendix B.2) show that these variants follow the same add-and-shift pattern as k = 1 on the tested inputs, and their offline NLL is lower, so we cannot claim that H15's coarser resolution is cheaper in the kernel. H15 is the point we implemented and measured in FA4; the k-ablation was a later offline follow-up. A matched comparison inside the same FA4 kernel is the right experiment.

On novelty, we agree with your account of the operator's lineage (row-maximum anchoring in EXAQ, Schraudolph's construction), which we state in Sec. 2 and Appendix B.1.

Weakness 2 and Question 2

Reviewer comment

Weakness 2. The principal speed and fidelity results are obtained under different conditions, leaving the practical tradeoff incomplete. The headline acceleration is measured on FP8 attention at 8K–16K, whereas direct model-fidelity measurements use BF16 at 2K. The FP8 fidelity estimate is a semantic simulation on one model that does not quantize Q, K, V, and P to FP8, so it does not validate the accelerated FP8 configuration. The long-context and zero-shot checks primarily evaluate other characterization operators rather than the H15 kernel. Furthermore, timing covers attention forward rather than full-model inference. Consequently, the paper does not yet demonstrate the claimed combination of useful acceleration and small model degradation in a common deployment setting.

Question 2. Can you directly evaluate the FP8 kernel's model fidelity at the 8K and 16K settings used for the main speed measurements, with an appropriate FP8 baseline, and report the corresponding full-model prefill latency? This would establish whether the numerical and performance benefits coexist in the same configuration.

Response

We agree; Sec. 7 lists this as a limitation.

Two clarifications:

  • FP8 fidelity. We attempted a direct FP8 fidelity run, but per-tensor E4M3 conversion of the BF16 activations added about 0.49 nats of unrelated error (Appendix B.2). A meaningful FP8 measurement needs a proper FP8 operand-quantization baseline, which we do not have.
  • Full-model latency. Prefill latency was not measured. Appendix B.5 gives only the attention share of prefill FLOPs for Qwen2.5-1.5B (21% at 8K, 35% at 16K), which is a workload share, not a latency.

Next steps

We will withdraw our submission. However, we thank you for your work.

These points need new kernel experiments that we cannot complete in this cycle but we will take them to improve our work. This is one of the two reasons we intend to withdraw in this round; the other is stated in our response to Reviewer rQQM. We have also uploaded the supplementary material promised in the Reproducibility Statement (code, result files and a provenance map).

Thanks again for the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.