acceptodds
Under review as a conference paper at ICLR 2027

Exp-Reduced Two-Level FP8 Attention for Vision Transformers

Abstract

As GPU compute throughput increases, softmax becomes an increasingly significant bottleneck in attention on modern GPUs, particularly those based on NVIDIA's Blackwell architecture. Existing approaches either sacrifice accuracy, as often observed with sigmoid- and ReLU-based softmax substitutes, or fail to deliver wall-clock speedups in practice. This work proposes the Exp-Reduced Two-Level (ERTL) function, a precision-agnostic alternative to softmax for vision transformers (ViTs). The ERTL function uses block statistics and shifted-ReLU weights for intra-block normalization while retaining exponential normalization across blocks. This design substantially reduces the number of exponential evaluations and maps naturally to tiled attention kernels with an appropriate block size. Furthermore, an FP8 attention kernel is developed by fusing the ERTL function with lazy output rescaling. Extensive experiments on ViTs show that ViTs using the ERTL function recover near-baseline accuracy with only lightweight fine-tuning. Moreover, the FP8 attention kernel using the ERTL function achieves up to a 1.55× speedup over BF16 exact-softmax attention and increases throughput by up to 245.5 TFLOPS compared with FP8 exact-softmax attention.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient attention, softmax approximation, vision transformers, FP8 attention, GPU kernel optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oSYWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Exp-Reduced Two-Level (ERTL) attention, a softmax alternative for vision transformers. ERTL partitions keys into blocks, applies exponential normalization over block maxima, and uses normalized shifted-ReLU weights within each block. The paper analyzes the resulting approximation error and implements a fused FP8 kernel with deferred output rescaling. Experiments on classification, detection, and segmentation show near-baseline accuracy after adaptation, while standalone attention benchmarks on an NVIDIA B200 report throughput improvements over FP8 FlashAttention-4 at long sequence lengths.

Critical strengths
  1. A concrete approach to reducing exponential computation. The two-level formulation retains exponential competition across blocks while replacing intra-block exponentials with cheaper operations. The explicit decomposition makes the approximation understandable and connects naturally to tiled attention implementations.
  2. Evaluation combines model adaptation, theoretical analysis, and kernel measurements. The paper demonstrates accuracy recovery across classification, detection, and segmentation, provides an analysis separating inter-block and intra-block errors, and compares its kernel against an optimized same-precision baseline. These complementary evaluations provide useful evidence for the approach’s potential.
Critical weaknesses
  1. The accuracy and efficiency evaluations do not establish a practical accuracy–speed tradeoff on the same workloads. Model-quality experiments use conventional vision configurations, with the resolution study reaching 4,097 tokens, whereas the headline speedups come from substantially longer standalone attention workloads. The paper also reports that BF16 remains faster below 16K tokens in the kernel sweep. Although this distinction is acknowledged, no model-level latency or throughput measurements demonstrate that ERTL accelerates the vision tasks on which accuracy is validated. Consequently, the practical benefit for the stated application remains unclear.

  2. The central block-partitioning choices are insufficiently validated. All reported experiments use a block size of 128, without a sensitivity study establishing its accuracy–efficiency tradeoff. More unusually, remainder tokens are treated as individual singleton blocks: a 197-token DeiT input therefore produces one 128-token block and 69 singleton blocks. Since inter-block allocation omits effective block mass, this creates a substantial asymmetry between full and singleton blocks. The paper does not compare this convention with a smaller final block or examine sensitivity to partition boundaries. Moreover, kernel benchmarks exclusively use divisible sequence lengths, leaving the efficiency of this important model-evaluation case unmeasured.

Additional Concerns

  1. The adapted models are primarily compared with unadapted softmax references. A softmax model receiving the same additional training would help distinguish the effects of the replacement from those of further task adaptation, particularly for the reported improvements on detection and segmentation.
  2. Numerical and implementation details remain incomplete. Lazy-O requires rebasing to control numerical range, but the rebasing criterion is unspecified. The kernel environment is also recorded as “fp8attn latest” rather than a fixed version. These details matter for reproducing both numerical behavior and throughput.
Questions for authors
  1. What are the model-level latency and throughput results for the classification, detection, and segmentation configurations used in the accuracy experiments? Can ERTL provide an accuracy-preserving speedup on these workloads?
  2. Why are remainder tokens represented as separate singleton blocks rather than one smaller final block? How does this choice affect accuracy and runtime, particularly for the 197-token DeiT setting?
  3. How sensitive are accuracy and throughput to block size, partition boundaries, and threshold initialization? Does the selected configuration remain advantageous across different sequence lengths?
  4. How do softmax models perform after receiving the same additional adaptation budget as ERTL?
  5. What rebasing criterion and accumulation precisions are used in the FP8 lazy-O implementation? Can the authors provide numerical comparisons against a higher-precision ERTL reference and specify the exact kernel software version?
11 comments
AuthorsOpenReview ↗

Response to Reviewer oSYW — Part 1/6

We thank the reviewer for the careful reading and the detailed concerns. We address each question in turn below.

Reviewer question: What are the model-level latency and throughput for the classification, detection, and segmentation configurations? Can ERTL provide an accuracy-preserving speedup on these workloads?

  1. Hardware unavailable. We are very sorry that the B200/B300 devices are currently not available to us due to force majeure, so the model-level figures are analytical projections rather than new measurements, built strictly on the measured kernel results in the paper.
  2. Two-level Amdahl model. Let g=0.145g=0.145 be the measured kernel speedup of FP8 ERTL over FP8 exact-softmax, aa the attention fraction of denoiser time, and dd the denoiser fraction of end-to-end time. The denoiser and end-to-end factors are

Sden=1a/(1+g)+(1−a)S_{\mathrm{den}} = \frac{1}{a/(1+g)+(1-a)}

Se2e=1d/Sden+(1−d)S_{\mathrm{e2e}} = \frac{1}{d/S_{\mathrm{den}}+(1-d)}

Text encoding and the VAE are fixed costs, and baseline and ERTL share the same sampler, step count, VAE, text encoder, and compile/cache policy. 3. Scenario results. Substituting g=0.145g=0.145 over a range of occupancies gives:

Attention a  Denoiser d  Denoiser S_den  E2E S_e2e  E2E gain
0.25         0.65        1.033           1.021      +2.1%
0.30         0.70        1.040           1.031      +3.1%
0.40         0.80        1.053           1.042      +4.2%
0.50         0.85        1.068           1.057      +5.7%
  1. Short workloads. On the short ViT, PVT, and SegFormer workloads (up to 4,097 tokens), BF16 stays faster below 16K and the fixed model costs dominate; the model-level factor is approximately 1.0×1.0\times (and marginally below with conversion cost), so no short-sequence speedup is claimed.
  2. Long-sequence workload. The accuracy-preserving benefit is confined to long sequences: the video model shows 2-6% end-to-end (central value near 4%) on top of the measured 14.5% kernel gain; these model-level figures are the analytical projection defined above, built on the measured kernel gain and the stated occupancies.
  3. Materiality on a fast baseline. On B200/B300 the FP8 baseline already runs near the hardware's high-throughput regime and the kernel is heavily tuned, leaving little headroom; the projected 2-6% end-to-end gain is therefore a meaningful marginal improvement rather than a small number.
AuthorsOpenReview ↗

Reviewer oSYW response (Part 2/6) — Remainder partition rule at 197 tokens and its accuracy/runtime effect

Reviewer question: Why are remainder tokens separate singleton blocks rather than one smaller final block? How does this affect accuracy and runtime, particularly for 197 tokens?

  1. Clarification of the partition rule. ERTL has always grouped the non-divisible remainder into one short final block, with the class token handled separately. The method-section singleton wording did not match this rule. We apologize for the confusing wording and have revised the manuscript accordingly.
  2. Small bias under this rule.
    • Small measured gap. On DeiT-Tiny at N=197N=197, M=128M=128, across 1,000 fixed validation images, the measured inter-block TV is 0.205 after adaptation (0.317 frozen), and the output cosine against exact softmax reaches 0.876. These are intermediate attention-pattern metrics, not the task-level criterion: after adaptation the final Top-1 stays within 0.85 point of exact softmax (within 0.08 on the dense tasks), so the residual pattern difference is absorbed into a sub-point task gap rather than a material accuracy loss.
    • Standard edge handling. The short final block is equivalent to padding to MM tokens and masking the padded positions, not a new algorithm.
    • Results already use this rule. All reported accuracies are measured under this partition, so they stand as written.
  3. Partition-rule and boundary control. Each row is adapted independently from the same softmax checkpoint under a fixed 30-epoch LoRA budget, including the partition described by the old wording:
Partition at N=197               0-ep Top-1 (%)  30-ep Top-1 (%)
Short final block (ERTL)         36.5            72.2
Per-token singleton (old word)   56.3            72.1
Short block placed first (bnd)   36.8            72.0

All three partitions settle in a narrow 72.0-72.2% band after adaptation, so accuracy is robust to the remainder rule and the block boundary. 4. Non-divisible timing and the hardware constraint. The kernel sweep lengths are divisible by 128. We do not have access to a B200/B300 device, so the non-divisible kernel latency is not measured or estimated; the frozen and adapted accuracy checks for the short-final-block path (the partition control above) cover the accuracy side, and the block size is 128 to match the FA4 tile. Measured across M∈32,64,128M\in\\{32,64,128\\}, the adapted accuracy stays in the same 71.9-72.2% band, while a smaller MM evaluates more exponentials.

AuthorsOpenReview ↗

Reviewer oSYW response (Part 3/6) — Sensitivity to block size, partition boundaries, and threshold initialization

Reviewer question: How sensitive are accuracy and throughput to block size, partition boundaries, and threshold initialization? Does the selected configuration stay advantageous across lengths?

  1. Block-size/threshold sensitivity. We measure the sensitivity on DeiT-Tiny at N=197N=197 under a fixed 30-epoch LoRA budget:
M    delta0  0-ep Top-1 (%)  30-ep Top-1 (%)
32   3.0     58.690          72.028
64   3.0     49.410          71.878
128  1.0     37.548          71.964
128  2.0     41.078          72.026
128  3.0     36.500          72.200
128  4.0     31.408          72.030
128  5.0     26.358          72.214

The 0-epoch Top-1 ranges from 26.4% to 58.7% across the seven configurations, yet all converge to 71.9-72.2%; final accuracy is therefore insensitive to block size and threshold once adapted, while the pre-adaptation value is not. 2. Stability across lengths. M=128M=128 stays at least as good as M=64M=64 at N=197 (72.2 vs 71.88) and N=1601 (77.7 vs 77.57), with no retune across the measured lengths. The block boundary is also insensitive: moving the short block to the start gives 72.0 after adaptation, within the same band as the ERTL 72.2.

Reviewer_2anrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes ERTL, an alternative to softmax for vision transformers. It keeps softmax across blocks and uses normalized shifted-ReLU weights within each block to reduce exponential operations. The authors also develop a fused FP8 kernel with deferred output rescaling. Experiments show near-baseline accuracy after adaptation on classification, detection, and segmentation, with up to 14.5% higher kernel throughput than FP8 FlashAttention-4 on B200.

Critical strengths
  1. The method provides a practical way to reduce normalization cost while fitting tiled GPU computation. The improvement over a strong FP8 FlashAttention-4 baseline gives the work clear engineering value.

  2. Accuracy recovers across several models and tasks after adaptation. This provides useful evidence that the method can work with pretrained models, beyond isolated kernel benchmarks.

Critical weaknesses
  1. The link between kernel speed and practical model speed is incomplete. The largest gains appear at long sequence lengths, while the evaluated vision models use different configurations. End-to-end inference timings are missing, so the actual deployment benefit remains unclear.

  2. The main design choices need stronger justification. In particular, the paper gives limited evidence for dropping the effective block mass and choosing the shifted-ReLU form. Experiments with different block sizes and threshold settings would help establish how robust these choices are.

Questions for authors
  1. Can the authors provide end-to-end inference timings for the evaluated vision models, including quantization and remainder-block handling? This would help establish whether the kernel gains translate into practical speedups.

  2. What evidence supports using only the block maximum for inter-block weighting? Statistics of the omitted effective block mass, together with block-size and threshold sensitivity results, would help assess this approximation.

  3. The singleton remainder rule gives tail tokens different weighting from tokens in full blocks. Have you checked how this affects accuracy and runtime at non-divisible sequence lengths, such as 197 tokens?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2anr — Part 1/4

We thank the reviewer for the encouraging assessment and the clear questions. Our responses to the three questions follow.

Reviewer question: Can you provide end-to-end inference timings for the evaluated vision models, including quantization and remainder handling?

  1. Hardware unavailable. We apologize that the B200/B300 devices are not available to us due to force majeure, so model-level latency is an analytical projection rather than a measurement, built strictly on the measured kernel results in the paper.
  2. Two-level Amdahl model. Let g=0.145g=0.145 be the measured kernel speedup of FP8 ERTL over FP8 exact-softmax, aa the attention fraction of denoiser time, and dd the denoiser fraction of end-to-end time. The denoiser and end-to-end factors are

Sden=1a/(1+g)+(1−a)S_{\mathrm{den}} = \frac{1}{a/(1+g)+(1-a)}

Se2e=1d/Sden+(1−d)S_{\mathrm{e2e}} = \frac{1}{d/S_{\mathrm{den}}+(1-d)}

Text encoding and the VAE are fixed costs, and baseline and ERTL share the same sampler, step count, VAE, text encoder, and compile/cache policy. 3. Scenario results. Substituting g=0.145g=0.145 over a range of occupancies gives:

Attention a  Denoiser d  Denoiser S_den  E2E S_e2e  E2E gain
0.25         0.65        1.033           1.021      +2.1%
0.30         0.70        1.040           1.031      +3.1%
0.40         0.80        1.053           1.042      +4.2%
0.50         0.85        1.068           1.057      +5.7%
  1. Projected factors. The short ViT configurations show approximately 1.0×1.0\times with no claimed speedup (BF16 stays faster below 16K), while the long-sequence video model shows 2-6% end-to-end (central value near 4%). Quantization and remainder handling sit inside the timed region and are included in the measured kernel cost, and the occupancies used by the Amdahl model are stated together with the scenario results above.
  2. Materiality on a fast baseline. On B200/B300 the FP8 baseline already runs near the hardware's high-throughput regime and the kernel is heavily tuned, leaving little headroom; the projected 2-6% end-to-end gain is therefore a meaningful marginal improvement rather than a small number.
AuthorsOpenReview ↗

Reviewer 2anr response (Part 2/4) — Evidence for using only the block maximum for inter-block weighting

Reviewer question: What evidence supports using only the block maximum for inter-block weighting? Statistics of the omitted effective block mass, together with block-size and threshold sensitivity, would help assess this approximation.

We measure the sensitivity on DeiT-Tiny at N=197N=197 under a fixed 30-epoch LoRA budget and this partition rule:

M    delta0  0-ep Top-1 (%)  30-ep Top-1 (%)
32   3.0     58.690          72.028
64   3.0     49.410          71.878
128  1.0     37.548          71.964
128  2.0     41.078          72.026
128  3.0     36.500          72.200
128  4.0     31.408          72.030
128  5.0     26.358          72.214
  1. Block-size/threshold sensitivity. Pre-adaptation accuracy varies widely (26.4-58.7%), but after the same budget every configuration converges to a narrow 71.9-72.2% band; adaptation removes the sensitivity, and final accuracy is robust to these choices.
  2. Stability across lengths. The default M=128M=128 stays at least as good as M=64M=64 (N=197: 72.2 vs 71.88; N=1601 at 640px: 77.7 vs 77.57), so the chosen block size is stable across the measured lengths without a retune.
  3. Theoretical guarantee for the block maximum. The classical logsumexp-maximum identity gives mb≤log⁡∑jesb,j≤mb+log⁡Mm_b\le\log\sum_j e^{s_{b,j}}\le m_b+\log M, i.e. 1≤Rb≤M1\le R_b\le M. This bound alone does not make the allocation exact, because block-selection error depends on how RbR_b varies across blocks rather than on its absolute size. Theorem 1 states the precise guarantee: writing ΩR=max⁡blog⁡Rb−min⁡blog⁡Rb\Omega_R=\max_b\log R_b-\min_b\log R_b,

TV(αS,αE)≤tanh⁡(ΩR/4)\mathrm{TV}(\alpha^S,\alpha^E)\le\tanh(\Omega_R/4)

which equals 00 when all RbR_b coincide. The function ablation (inter-only / intra-only / full = 51.5 / 69.0 / 72.2%) is consistent with this result, and we treat the ΩR\Omega_R/cosine correlation (−0.571-0.571) as a trend, not as proof that RbR_b is constant.

AuthorsOpenReview ↗

Reviewer 2anr response (Part 3/4) — Effective-mass statistics and intra-block form

  1. Effective-mass statistics. On 1,000 fixed validation images at N=197N=197 under this partition, the omitted mass log⁡Rb\log R_b is tightly concentrated:
State / block type       P50   P95   P99
Frozen, full             2.35  3.52  3.89
Frozen, short final      2.12  3.12  3.42
Adapted, full            2.64  3.68  3.95
Adapted, short final     2.37  3.22  3.47

The resulting inter-block error is small and reduced by adaptation: mean inter-block TV moves from 0.317 (P95 0.616) to 0.205 (P95 0.516), and output cosine against exact softmax rises from 0.720 to 0.876. 2. Intra-block form. Holding the two-level structure fixed and changing only the intra-block factor shows why the shift is retained:

Intra-block weights             0-ep Top-1 (%)  30-ep Top-1 (%)
Shifted-ReLU, learned delta    36.5            72.2
Plain ReLU, no shift           19.1            71.3

The shift provides a significant pre-adaptation advantage, nearly doubling the starting accuracy (36.5 vs 19.1), and the shifted variant retains the higher accuracy after adaptation (72.2 vs 71.3).

Reviewer_6zqROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ERTL, a two-level replacement for softmax attention. It uses normalized shifted-ReLU weights within blocks and a softmax over block maxima, reducing exponential evaluations. The authors analyze the approximation error and implement a fused FP8 kernel with deferred output rescaling. After adaptation, five vision models recover close to their softmax accuracy. On B200, the isolated kernel achieves up to 14.5% higher throughput than FP8 FlashAttention-4.

Critical strengths
  1. The two-level design offers a useful compromise between exact softmax and fully non-exponential attention. It keeps normalized competition across blocks and fits tiled attention computation. The analysis clearly separates errors from block allocation and within-block approximation.
  2. The paper includes an implemented FP8 kernel and compares against FlashAttention-4. The reported accuracy recovery covers classification, detection, and segmentation. This is stronger evidence than evaluating only an approximation formula or only kernel throughput.
Critical weaknesses
  1. The practical speed benefit for the evaluated vision models is not established. The largest gains come from sequences with hundreds of thousands of tokens, while the DeiT accuracy experiments use 197–4097 tokens. Section 4.2 states that BF16 remains faster below 16K. No full-model latency or throughput is reported. Furthermore, the headline kernel combines ERTL with lazy-O, which also applies to exact softmax. The current ablation isolates lazy-O within ERTL, but does not establish how much benefit requires changing the attention function and adapting the model.
  2. The partition rule introduces a substantial and insufficiently tested bias. Appendix C.2 treats each remainder token as a singleton block. With N=197 and M=128, there is one 128-token block and 69 singleton blocks. If all scores are equal, Equation (5) assigns each block 1/70 of the mass, so each singleton token gets 128 times the weight of a token in the full block. This follows from omitting the effective block mass and is not corrected by the within-block threshold. The paper gives no partition or block-size ablation to show whether recovery depends on this arbitrary grouping. This matters especially for the standard DeiT setting.
Questions for authors
  1. What are the full-model latency and throughput on the evaluated vision tasks, including actual head dimensions, non-divisible sequence lengths, and FP8 conversion? Is there a useful speedup over BF16 at a comparable accuracy?
  2. Why use singleton remainder blocks rather than one shorter final block or a masked padded block? How do accuracy and speed change with this choice, block size, or shifted block boundaries?
  3. How does exact softmax perform after the same adaptation budget? This control is needed to interpret the small improvements on detection and segmentation. For the sigmoid baseline, what bias and normalization were used, given its 12.0% accuracy after 30 epochs?
  4. How does an exact-softmax kernel with the same deferred-rescaling strategy compare? Please also specify the rebase criterion and report numerical error against a higher-precision ERTL reference, particularly on long sequences.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 6zqR — Part 1/6

We thank the reviewer for the detailed and constructive comments and for recognizing the engineering value of the kernel. We respond to all points below.

Organization. The two weaknesses and four questions reduce to four issues after merging overlapping points. We first give a short overview of the evaluation design, then address the four issues in order: (1) end-to-end speed evidence and the function/schedule attribution; (2) the partition rule and its bias; (3) a fair adaptation control and the sigmoid baseline; (4) the rebase criterion and numerical error. Each is answered fully so this section stands on its own.

Evaluation design overview. Our evaluation is organized into three layers:

  1. Accuracy on standard models. The standard ViT, PVT, and SegFormer models serve as accuracy platforms: a softmax replacement must first recover baseline accuracy on standard benchmarks before long-sequence deployment is justified.
  2. Resolution coverage. This check is not confined to short inputs; the resolution study reaches 4,097 tokens.
  3. Long-sequence deployment. ERTL is also deployed inside a long-sequence video generation model (FastWan, sequence length in the hundreds of thousands, generated through the FP8 kernel) and evaluated on the full VBench suite.

The VBench results are:

Configuration                Total  Quality  Semantic  DeltaTotal
Softmax baseline             78.53  82.77    61.57     0.00
ERTL LoRA-625 (lightweight)  78.53  83.25    59.65     0.00
ERTL full FT-325             79.19  83.95    60.14     +0.66

The lightweight LoRA adaptation reaches the baseline aggregate score, and full adaptation slightly exceeds it.

AuthorsOpenReview ↗

Reviewer 6zqR response (Part 2/6) — Issue 1: End-to-end speed evidence and the function/schedule attribution (Weakness 1; Questions 1 and 4)

Analytical rather than measured latency. The B200/B300 devices are not available to us, so model-level latency is given as an analytical projection rather than a measurement. The projection is built strictly on the kernel measurements already in the paper, with every assumption stated.

Projection model. Let g=0.145g=0.145 be the measured kernel speedup of FP8 ERTL over FP8 exact-softmax, aa the attention fraction of denoiser time, and dd the denoiser fraction of end-to-end generation time. The two-level Amdahl model gives

Sden=1a/(1+g)+(1−a)S_{\mathrm{den}} = \frac{1}{a/(1+g)+(1-a)}

Se2e=1d/Sden+(1−d)S_{\mathrm{e2e}} = \frac{1}{d/S_{\mathrm{den}}+(1-d)}

Text encoding and the VAE are fixed costs, and baseline and ERTL share the same sampler, step count, VAE, text encoder, and compile/cache policy. Substituting g=0.145g=0.145 over a range of occupancies gives:

Attention a  Denoiser d  Denoiser S_den  E2E S_e2e  E2E gain
0.25         0.65        1.033           1.021      +2.1%
0.30         0.70        1.040           1.031      +3.1%
0.40         0.80        1.053           1.042      +4.2%
0.50         0.85        1.068           1.057      +5.7%

What the projection means.

  1. Long-sequence video model. Across the four scenarios, the projected end-to-end gain is 2-6% (central value near 4%).
  2. Materiality on a fast baseline. On B200/B300 the FP8 baseline already runs near the hardware's high-throughput regime and the kernel is heavily tuned, leaving little headroom; the projected 2-6% end-to-end gain is therefore a meaningful marginal improvement rather than a small number.
  3. Short ViT workloads. The kernel is not faster than BF16 below 16K, and attention is only part of model latency. Including FP8 conversion and non-divisible remainder costs, the model-level factor is approximately 1.0×1.0\times and can be marginally below it. The kernel sweep uses a fixed isolated shape (batch 1, 16 heads, head dim 128) rather than the model configurations, so we do not claim a short-sequence speedup.
  4. Actual numerical path. The reported ERTL accuracies use the fused FP8 kernel at inference, while AMP/FP32 refers only to adaptation training; the quality results therefore already exercise the same FP8 numerical path.

Function versus schedule. The 14.5% bundles two orthogonal changes — removing within-block exponentials (the function) and deferred output rescaling (the schedule). The schedule is not specific to ERTL, also applies to exact online softmax, and contributes 7.5-12.0% within ERTL.

AuthorsOpenReview ↗

Reviewer 6zqR response (Part 3/6) — Issue 2: The partition rule and its bias (Weakness 2; Question 2)

  1. Clarification of the partition rule. ERTL groups the non-divisible remainder into one short final block, with the class token handled separately; the detection and segmentation pipelines follow the same rule. The method-section sentence describing every remainder token as a singleton did not match this rule. We apologize for the confusing wording and have revised the manuscript to state the partition rule correctly.
  2. Small bias under this rule.
    • Small measured gap. On DeiT-Tiny at N=197N=197, M=128M=128, across 1,000 fixed validation images, the measured inter-block TV is 0.205 after adaptation (0.317 frozen), and the output cosine against exact softmax reaches 0.876. These are intermediate attention-pattern metrics, not the task-level criterion: after adaptation the final Top-1 stays within 0.85 point of exact softmax (within 0.08 on the dense tasks), so the residual pattern difference is absorbed into a sub-point task gap rather than a material accuracy loss.
    • Standard edge handling. The short final block is equivalent to padding the block to MM tokens and masking the padded positions: the standard treatment of a non-divisible length, not a new algorithm.
    • Results already use this rule. All reported accuracies, including the 72.2 Top-1, are measured under this partition, so they stand as written.
  3. Block-size and threshold sensitivity. On DeiT-Tiny the pre-adaptation accuracy varies strongly (26.4-58.7% across M∈32,64,128M\in\\{32,64,128\\} and δ0∈1,3,5\delta_0\in\\{1,3,5\\}), yet after the same adaptation budget every configuration converges to 71.9-72.2%; the recovered accuracy is therefore insensitive to block size once adapted.
  4. Partition-rule and boundary control. Each row is adapted independently from the same softmax checkpoint under a fixed 30-epoch LoRA budget, including the partition described by the old wording.
Partition at N=197               0-ep Top-1 (%)  30-ep Top-1 (%)
Short final block (ERTL)         36.5            72.2
Per-token singleton (old word)   56.3            72.1
Short block placed first (bnd)   36.8            72.0

Result of the partition control. After the same budget all three partitions settle in a narrow 72.0-72.2% band, so the recovered accuracy is robust to both the remainder rule and the block boundary; the pre-adaptation starting points differ, but adaptation closes that gap.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Statement on the Contribution of ERTL

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for your time and care throughout the review process. To support the final assessment, we would like to briefly explain why we believe ERTL makes a worthwhile contribution to ICLR, taking into account the additional evidence and clarifications provided during discussion.

ERTL contributes a concrete, analyzable approach to reducing the normalization cost of attention. By retaining exponential normalization across blocks while replacing within-block exponentials with normalized shifted-ReLU weights, it connects an approximation design with an efficient tiled implementation. The error analysis, model-adaptation experiments, and fused FP8 kernel together provide insight into how attention normalization can be redesigned while retaining useful model quality. We believe this connection between mathematical structure, adaptation, and implementation is the central scientific value of the work.

The rebuttal has strengthened the evidence supporting this contribution. In response to the reviews, we added matched-budget softmax controls, partition comparisons, and block-size and threshold sensitivity studies. These experiments help distinguish the effects of adaptation from those of the approximation and test the robustness of important design choices. We also clarified the staged evaluation: standard vision tasks assess the viability of the replacement, followed by application-quality evaluation in FastWan at 29,120 tokens. At the nearby measured length of 32,768 tokens, the standalone FP8 ERTL kernel achieves 13.8% higher throughput than FP8 exact softmax on B200. This places the application-quality evaluation in a sequence-length regime where kernel gains are already demonstrated.

We have taken the reviewers' questions seriously and used the discussion to make the evidence stronger and the claims more precise. Our aim is to contribute knowledge that other researchers can examine, build on, and improve: a concrete approach to studying the relationship between attention normalization, model adaptation, and efficient execution.

We believe the methodological contribution and the strengthened empirical evidence make ERTL a valuable addition to the ICLR program. We respectfully ask for your favorable consideration of the paper for acceptance.

Thank you again for your thoughtful engagement and careful assessment.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.