- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Exp-Reduced Two-Level (ERTL) attention, a softmax alternative for vision transformers. ERTL partitions keys into blocks, applies exponential normalization over block maxima, and uses normalized shifted-ReLU weights within each block. The paper analyzes the resulting approximation error and implements a fused FP8 kernel with deferred output rescaling. Experiments on classification, detection, and segmentation show near-baseline accuracy after adaptation, while standalone attention benchmarks on an NVIDIA B200 report throughput improvements over FP8 FlashAttention-4 at long sequence lengths.
Critical strengths
- A concrete approach to reducing exponential computation. The two-level formulation retains exponential competition across blocks while replacing intra-block exponentials with cheaper operations. The explicit decomposition makes the approximation understandable and connects naturally to tiled attention implementations.
- Evaluation combines model adaptation, theoretical analysis, and kernel measurements. The paper demonstrates accuracy recovery across classification, detection, and segmentation, provides an analysis separating inter-block and intra-block errors, and compares its kernel against an optimized same-precision baseline. These complementary evaluations provide useful evidence for the approach’s potential.
Critical weaknesses
-
The accuracy and efficiency evaluations do not establish a practical accuracy–speed tradeoff on the same workloads. Model-quality experiments use conventional vision configurations, with the resolution study reaching 4,097 tokens, whereas the headline speedups come from substantially longer standalone attention workloads. The paper also reports that BF16 remains faster below 16K tokens in the kernel sweep. Although this distinction is acknowledged, no model-level latency or throughput measurements demonstrate that ERTL accelerates the vision tasks on which accuracy is validated. Consequently, the practical benefit for the stated application remains unclear.
-
The central block-partitioning choices are insufficiently validated. All reported experiments use a block size of 128, without a sensitivity study establishing its accuracy–efficiency tradeoff. More unusually, remainder tokens are treated as individual singleton blocks: a 197-token DeiT input therefore produces one 128-token block and 69 singleton blocks. Since inter-block allocation omits effective block mass, this creates a substantial asymmetry between full and singleton blocks. The paper does not compare this convention with a smaller final block or examine sensitivity to partition boundaries. Moreover, kernel benchmarks exclusively use divisible sequence lengths, leaving the efficiency of this important model-evaluation case unmeasured.
Additional Concerns
- The adapted models are primarily compared with unadapted softmax references. A softmax model receiving the same additional training would help distinguish the effects of the replacement from those of further task adaptation, particularly for the reported improvements on detection and segmentation.
- Numerical and implementation details remain incomplete. Lazy-O requires rebasing to control numerical range, but the rebasing criterion is unspecified. The kernel environment is also recorded as “fp8attn latest” rather than a fixed version. These details matter for reproducing both numerical behavior and throughput.
Questions for authors
- What are the model-level latency and throughput results for the classification, detection, and segmentation configurations used in the accuracy experiments? Can ERTL provide an accuracy-preserving speedup on these workloads?
- Why are remainder tokens represented as separate singleton blocks rather than one smaller final block? How does this choice affect accuracy and runtime, particularly for the 197-token DeiT setting?
- How sensitive are accuracy and throughput to block size, partition boundaries, and threshold initialization? Does the selected configuration remain advantageous across different sequence lengths?
- How do softmax models perform after receiving the same additional adaptation budget as ERTL?
- What rebasing criterion and accumulation precisions are used in the FP8 lazy-O implementation? Can the authors provide numerical comparisons against a higher-precision ERTL reference and specify the exact kernel software version?
11 comments
Reviewer oSYW response (Part 2/6) — Remainder partition rule at 197 tokens and its accuracy/runtime effect
Reviewer question: Why are remainder tokens separate singleton blocks rather than one smaller final block? How does this affect accuracy and runtime, particularly for 197 tokens?
- Clarification of the partition rule. ERTL has always grouped the non-divisible remainder into one short final block, with the class token handled separately. The method-section singleton wording did not match this rule. We apologize for the confusing wording and have revised the manuscript accordingly.
- Small bias under this rule.
- Small measured gap. On DeiT-Tiny at , , across 1,000 fixed validation images, the measured inter-block TV is 0.205 after adaptation (0.317 frozen), and the output cosine against exact softmax reaches 0.876. These are intermediate attention-pattern metrics, not the task-level criterion: after adaptation the final Top-1 stays within 0.85 point of exact softmax (within 0.08 on the dense tasks), so the residual pattern difference is absorbed into a sub-point task gap rather than a material accuracy loss.
- Standard edge handling. The short final block is equivalent to padding to tokens and masking the padded positions, not a new algorithm.
- Results already use this rule. All reported accuracies are measured under this partition, so they stand as written.
- Partition-rule and boundary control. Each row is adapted independently from the same softmax checkpoint under a fixed 30-epoch LoRA budget, including the partition described by the old wording:
Partition at N=197 0-ep Top-1 (%) 30-ep Top-1 (%)
Short final block (ERTL) 36.5 72.2
Per-token singleton (old word) 56.3 72.1
Short block placed first (bnd) 36.8 72.0
All three partitions settle in a narrow 72.0-72.2% band after adaptation, so accuracy is robust to the remainder rule and the block boundary. 4. Non-divisible timing and the hardware constraint. The kernel sweep lengths are divisible by 128. We do not have access to a B200/B300 device, so the non-divisible kernel latency is not measured or estimated; the frozen and adapted accuracy checks for the short-final-block path (the partition control above) cover the accuracy side, and the block size is 128 to match the FA4 tile. Measured across , the adapted accuracy stays in the same 71.9-72.2% band, while a smaller evaluates more exponentials.
Reviewer oSYW response (Part 3/6) — Sensitivity to block size, partition boundaries, and threshold initialization
Reviewer question: How sensitive are accuracy and throughput to block size, partition boundaries, and threshold initialization? Does the selected configuration stay advantageous across lengths?
- Block-size/threshold sensitivity. We measure the sensitivity on DeiT-Tiny at under a fixed 30-epoch LoRA budget:
M delta0 0-ep Top-1 (%) 30-ep Top-1 (%)
32 3.0 58.690 72.028
64 3.0 49.410 71.878
128 1.0 37.548 71.964
128 2.0 41.078 72.026
128 3.0 36.500 72.200
128 4.0 31.408 72.030
128 5.0 26.358 72.214
The 0-epoch Top-1 ranges from 26.4% to 58.7% across the seven configurations, yet all converge to 71.9-72.2%; final accuracy is therefore insensitive to block size and threshold once adapted, while the pre-adaptation value is not. 2. Stability across lengths. stays at least as good as at N=197 (72.2 vs 71.88) and N=1601 (77.7 vs 77.57), with no retune across the measured lengths. The block boundary is also insensitive: moving the short block to the start gives 72.0 after adaptation, within the same band as the ERTL 72.2.
Response to Reviewer oSYW — Part 1/6
We thank the reviewer for the careful reading and the detailed concerns. We address each question in turn below.
Sden=a/(1+g)+(1−a)1
Se2e=d/Sden+(1−d)1
Text encoding and the VAE are fixed costs, and baseline and ERTL share the same sampler, step count, VAE, text encoder, and compile/cache policy. 3. Scenario results. Substituting g=0.145 over a range of occupancies gives: