acceptodds
Under review as a conference paper at ICLR 2027

FreeAttn: Unlocking Native E3M4 for Efficient and Accurate Attention

Abstract

The quadratic cost of attention limits long-context prefill and video generation. Low-bit quantization increases throughput but introduces numerical error. At the same eight-bit width, E3M4 trades a narrower dynamic range for finer normal-number spacing than E4M3. We independently identify and validate native packed F2FP conversion and matrix multiply–accumulate (MMA) modes for E3M4 on Blackwell SM120, and connect them into the native attention path of FreeAttn. The framework relates QK/PV quantization errors to attention outputs under shared scaling, organizes composable corrections through attention identities, and reduces their cost through shared statistics and fused execution. Matched-format controls show lower aggregate BF16-relative probability and pixel error with E3 than E4. Evaluated for both fidelity and latency, optimized all-FP8 E3 accelerates Qwen3-8B/32B prefill by 1.15–1.16× and Wan2.1-14B video delivery by 1.43× over Torch SDPA. The FP4 path with Basic correction achieves 1.45–1.51× complete-attention-call speedup over Torch SDPA on seven common shapes, with lower aggregate operator output error than same-bitwidth SageAttention3. These results establish native E3M4 as a useful representation and execution option for attention inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
efficient attention, low-precision inference, FP8 E3M4, GPU systems, hardware-software co-design

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_e3CTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FreeAttn connects native E3M4 conversion and matrix instructions on Blackwell SM120 GPUs into an attention implementation. It also combines error corrections with fused execution. Matched E3/E4 experiments show lower aggregate numerical error at similar runtime. Operator benchmarks and Qwen/Wan experiments evaluate accuracy and speed.

Critical strengths
  1. The matched-format comparisons make a useful case for E3M4: attention can benefit from its finer precision without increasing bitwidth or runtime. This gives practitioners another option beyond E4M3.

  2. The native conversion path is backed by careful checks. In particular, the native/software comparison preserves outputs bitwise while reducing complete-call time by 4.6–17.4%. This helps establish where the implementation gains come from.

Critical weaknesses
  1. The practical gain over SageAttention2++ is small. The current E3 configuration saves roughly 2% of Qwen first-token time and 4% of video delivery time, while showing larger deviations from BF16. The paper does not yet establish a clear advantage at comparable model fidelity.

  2. Deployment requires rewriting GPU machine instructions. The paper provides validation checks, but compatibility and maintenance across GPU and toolchain versions remain unclear. This makes the modest end-to-end gains harder to judge in practice.

Questions for authors
  1. At a matched model-fidelity target, how much speed advantage remains over SageAttention2++? This comparison would help me judge the practical benefit.

  2. Which parts of the instruction-rewriting process depend on the GPU and compiler version? Does the released package detect incompatible binaries before running them?

5 comments
AuthorsOpenReview ↗

Response to Reviewer e3CT (Part 1/2: W1 & Q1)

Response to W1 & Q1 (Speed Advantage Over SageAttention2++ at Matched or Higher Fidelity, and Separating DiT Step vs. Full Video Generation Time)

Free 8/8 E3M4 (+Free) matches or exceeds SageAttention2++ model fidelity while retaining a clear speed advantage. As detailed in Global Response Item 3, unified hardware QMMA.SF plus in-register F2FP removes Sage2++'s per-tile INT8/E4M3 software scale conversions inside the main loop, while single-pass GDDR7 fusion keeps +Free preprocessing to 1.4%–4.1% of full operator time on long sequences (9.7% on short-sequence Qwen3-8B L21):

  • End-to-End Speedup at Higher Model Fidelity (Appendix Table 11, p. 26; Tables 13–14, p. 28): On 50-step 720p Wan2.1-14B Turntable generation, Free 8/8 E3M4 (+Free) exceeds Sage2++ F16 by +2.74+2.74 dB PSNR (24.0224.02 vs. 21.2821.28 dB, LPIPS 0.04810.0481 vs. 0.07750.0775) while running self-attention 1.07×1.07\times faster (6.536.53 vs. 6.966.96 s/step) and finishing 50-step generation 69 s faster (1,569.71{,}569.7 vs. 1,638.61{,}638.6 s; +3.95+3.95 dB and 47 s faster than Sage2++ F32). Across the 4-scene 720p suite (Appendix Table 14), it outperforms Sage2++ F16 on all three metrics (mean PSNR 20.4420.44 vs. 18.9818.98 dB, SSIM 0.79390.7939 vs. 0.78290.7829, LPIPS 0.14500.1450 vs. 0.14990.1499) while finishing 66 s faster (1,565.81{,}565.8 vs. 1,631.61{,}631.6 s; 43 s faster with +1.16+1.16 dB higher mean PSNR than Sage2++ F32, at mean SSIM/LPIPS 0.7939/0.14500.7939/0.1450 vs. 0.8054/0.13930.8054/0.1393), and wins all metrics at lower latency on 6-scene VBench and 4-scene 480p. On 32K LongBench-v2, it achieves higher answer agreement (100.0% vs. 96.9%) and lower KL (Qwen3-8B: 0.00330.0033 vs. 0.00650.0065; Qwen3-32B: 0.00410.0041 vs. 0.00430.0043) while running attention 7.5%–11.9% faster than Sage2++ F16 (Qwen3-8B: 0.2450.245 vs. 0.2780.278 s; Qwen3-32B: 0.8760.876 vs. 0.9470.947 s).
  • Operator Latency Including Preprocessing (Figure 4, p. 8; Appendix Table 5, p. 21; Table 7, p. 23): Across the 12 outlier-free Gaussian synthetic shapes co-supported with Sage2++ (Appendix Table 7), Free 8/8 E3M4 (+Free) and hybrid Free 4/8 E3M4 (+Free) pass all 12/1212/12 accuracy gates (5% relative L2L_2 error threshold vs. FP32) and run 1.14×1.14\times–1.16×1.16\times and 1.27×1.27\times–1.29×1.29\times faster than Sage2++ including preprocessing (1.03×1.03\times–1.05×1.05\times and 1.15×1.15\times–1.17×1.17\times faster than PyTorch SDPA). Across the four captured model layers (Appendix Table 5), full operator calls including preprocessing run 1.56×1.56\times–1.94×1.94\times faster than PyTorch SDPA and 1.05×1.05\times–1.53×1.53\times faster than Sage2++ F16. Specifically, 8/8 E3M4 (+Free) is both faster and more accurate on Qwen3-8B L21 (0.1240.124 vs. 0.1900.190 ms, relative L2L_2 error 0.020260.02026 vs. 0.033250.03325) and Qwen3-32B L0 (24.8324.83 vs. 26.0926.09 ms, error 0.022830.02283 vs. 0.041270.04127). On Wan1.3B L18 (0.1630.163 vs. 0.2170.217 ms, error 0.009290.00929 vs. 0.009010.00901) and Wan14B L19 (169.54169.54 vs. 181.07181.07 ms, error 0.010850.01085 vs. 0.009740.00974), +Free runs faster, while +Free+Paid lowers error below Sage2++ F16 (0.2140.214 ms and 0.008020.00802 on Wan1.3B L18, both faster and more accurate; 0.009710.00971 vs. 0.009740.00974 on Wan14B L19 at 225.49225.49 vs. 181.07181.07 ms, +24.5% latency). Against 4-bit Sage3, Free 4/4 (+Free+Paid) runs 1.25×1.25\times–1.44×1.44\times faster including preprocessing across the 7 co-supported synthetic shapes (where all 4-bit PVPV paths exceed the 5% Gaussian gate at 5.99%–15.33% vs. 14.58%–33.24% for Sage3) and is both faster and more accurate on Wan14B L19 (144.98144.98 vs. 149.62149.62 ms, error 0.063960.06396 vs. 0.072320.07232).
  • Separate Self-Attention, DiT Step, and Full Generation Columns (Appendix Table 13, p. 28): Appendix Table 13 reports three separate latency columns—Attn (s), 1-Step DiT (s), and Full Gen (s)—across all video models (e.g., on 720p Wan2.1-14B, E3M4 +Free runs in 6.536.53 s, 15.3915.39 s, and 1,569.71{,}569.7 s vs. 6.966.96 s, 15.8115.81 s, and 1,638.61{,}638.6 s for Sage2++ F16, achieving 1.96×1.96\times, 1.41×1.41\times, and 1.42×1.42\times speedups over BF16). [Revised Paper: Section 4.3 Figure 4 (p. 8), Appendix D Tables 5 & 7 (pp. 21, 23), Appendix F Tables 13–14 (p. 28)]
AuthorsOpenReview ↗

Response to Reviewer e3CT (Part 2/2: W2 & Q2)

Response to W2 & Q2 (GPU vs. Compiler-Version Dependence in Instruction Rewriting, and Pre-Run Binary Checks)

We separate GPU ISA dependence from compiler lowering and enforce three-layer pre-execution checks (Appendix A & Table 2, p. 14; Appendix G, pp. 29–31):

  • GPU Architecture vs. Compiler Version Dependence: The rewritten format-select bit positions—F2FP bits {76,85}\lbrace 76, 85\rbrace and QMMA.SF.16832 bits {78,79,82,83,84,85}\lbrace 78, 79, 82, 83, 84, 85\rbrace (XOR mask 282+2842^{82}+2^{84})—are fixed by the Blackwell SM120 128-bit SASS encoding. Compiler (ptxas) dependence is limited to lowering our PTX placeholder instructions into F2FP and QMMA.SF.16832 mnemonics (verified across CUDA 12.8–13.x). Because our patchers (tools/patch/sass_f2fp_e3m4_patch.py and tools/patch/sass_mxf8f6f4_patch.py) locate instruction words from nvdisasm ELF code-section disassembly rather than fixed byte offsets, changes in register allocation, instruction scheduling, or branch offsets do not affect rewriting.
  • Detecting Incompatible Binaries Before Execution: Yes, across three layers before any attention kernel runs. (1) Build-time CMake fail-fast: CMake runs --inspect across every patched cubin (unexpected_bits=none) and performs a full-file byte diff (complete_file_diff=PASS, confirming zero bit changes outside the target format-select fields). If a compiler update alters expected mnemonics/opcodes, or if an input is missing, a corrupted non-ELF binary, or a wrong-architecture ELF object, the patcher exits non-zero without writing output (100% rejection across the 3 negative binary tests in Appendix A, p. 14 and bench/run_reviewer_ablations.py --patcher-only). (2) Runtime module-load guard (csrc/host/api.cpp): before calling cuModuleLoad or launching any kernel, the C++ host loader queries CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR/MINOR (aborting immediately with an explicit RuntimeError unless compute capability is 12.0 / SM120) and verifies cuModuleLoad and all kernel symbol handles. (3) Pre-run on-device probe (tests/cuda/check_probe.py): before running benchmarks, GPU probes verify hardware F2FP (81,93281{,}932 values) and E2M1 (81,94481{,}944 values) bit-for-bit against IEEE-754 software decoding and, alongside the 2,0682{,}068 exhaustive byte-encoding sweeps in Table 2, check QMMA.SF on exact-grid and random non-NaN inputs (relative error ≤10−6\le 10^{-6}, 0.000e+00 max error on E3M4). [Revised Paper: Section 3.2 (p. 4), Appendix A & Table 2 (p. 14), Appendix G (pp. 29–31)]

Thank you for your review; please let us know if any further details on latency or binary verification would be helpful.

Thanks to the authors for the clarification. My concerns have been addressed. The revised results show better fidelity in several settings, although the speed gains remain small. Modifying the compiled GPU code also adds deployment complexity. I view this as an incremental contribution that may be useful to other researchers.

Reviewer_MgRXOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents FreeAttn, an inference-time framework for low-precision attention that integrates native E3M4 FP8 conversion and matrix multiplication on Blackwell SM120 GPUs. E3M4 trades dynamic range for finer numerical resolution compared with E4M3. The paper analyzes how QK and PV quantization errors propagate to attention outputs and organizes existing correction techniques into a composable framework with fused execution. Matched-format experiments show improved aggregate fidelity over E4M3, while the optimized implementation achieves 1.15–1.16× faster Qwen3 prefill and 1.43× faster Wan2.1 video delivery than BF16 Torch SDPA. The results also demonstrate workload-dependent accuracy–speed tradeoffs and show that lower operator error does not necessarily translate into better model-output fidelity.

Critical strengths
  1. A useful systems contribution supported by controlled validation. FreeAttn integrates native E3M4 conversion and matrix multiplication into streaming attention, with independent checks of both instruction interfaces. Native conversion reduces complete-call latency by 4.6–17.4% relative to optimized software encoding while preserving bitwise-identical outputs. This directly substantiates the implementation’s practical value.

  2. Convincing evidence for E3M4 as an alternative attention format. Matched E3/E4 comparisons isolate the representation effect and show improved aggregate fidelity, including a 47.7% reduction in output RMSE on held-out Qwen states. Together with complete-call measurements and transparent reporting of unfavorable results, these experiments provide credible evidence that format selection can improve low-precision attention.

Critical weaknesses
  1. The advantage over the strongest baseline remains unclear. Relative to SageAttention2++ F32, FreeAttn reduces Qwen latency by only 1.7–2.3%, below the paper’s own 3% timing-resolution criterion, while producing higher KL divergence from BF16. Video delivery is approximately 3.6% faster, but BF16-relative SSIM decreases from 0.810 to 0.692. These results do not establish a compelling improvement at comparable fidelity, limiting the demonstrated practical advantage.

  2. The correction framework lacks convincing end-to-end validation. The main optimized model results use raw E3 without the correction menu, and some corrections improve operator error while worsening model fidelity. Consequently, these results primarily validate the native E3M4 implementation rather than the broader correction framework. Given that many correction components are established techniques, demonstrating their combined model-level benefit is important to substantiate this additional contribution.

Questions for authors
  1. Clarification of novelty. Could you distinguish precisely what FreeAttn adds beyond the cited public E3M4 ISA work, particularly regarding packed conversion, semantic validation, and integration into online attention? A concise comparison in the rebuttal would help establish the scope of the systems contribution.

  2. Evidence of the accuracy–speed tradeoff. Can you provide a focused comparison with SageAttention2++ at comparable fidelity on representative workloads? The current optimized configuration offers small latency gains but worse BF16 agreement. Establishing a useful operating point at matched fidelity would materially strengthen my assessment.

  3. Clarification and validation of the correction framework. Since the main model experiments use raw E3, which proposed corrections provide an end-to-end benefit after accounting for their overhead? A comparison between raw E3 and one fixed correction configuration on identical inputs would help substantiate this contribution. If no such benefit is established, please clarify the framework’s demonstrated scope and revise the claims accordingly.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MgRX (Part 1/2: Q1)

Response to Q1 (Specific Contributions Beyond Public blackwell-isa Notes)

Public blackwell-isa notes probe QMMA.SF operand format bits in isolated microbenchmarks on static pre-quantized matrices. Beyond blackwell-isa, FreeAttn builds a complete block-scaled streaming attention framework with zero inner-loop overhead on the default path. (1) Native E3M4 Conversion, Semantic Validation, and Block-Scaled Fit: we independently recover native E3M4 conversion in the 128-bit SASS dual-element instruction F2FP (bits {76,85}\lbrace 76, 85\rbrace), validate both F2FP and QMMA.SF bit-for-bit across all byte encodings and block exponents (Paper Table 2, p. 14), show that 32-element block scaling places over 99.93% of energy within each block across 18 real Q,K,VQ,K,V tensors inside E3M4's normal range [0.25,30.0][0.25, 30.0] (Appendix Table 3, p. 16), and pack online softmax probabilities PP in registers in one instruction (4.6%–17.4% faster complete operator calls vs. software bit-packing, Figure 5c & Table 8, p. 24). (2) Fused Softmax-Invariant Transforms (+Free): we prove that Smooth-K, Scale-QK, Hadamard-QK, and Smooth-V (which cancels probability truncation row-sum bias (ΔP1)μV⊤(\Delta P \mathbf{1})\mu_V^\top) require zero extra main-loop instructions and fuse into one streaming GDDR7 pass (1,4671{,}467–1,5591{,}559 GB/s vs. 420420–445445 GB/s unfused). (3) Streaming Library & Model Validation: we deliver a complete library supporting D∈{64,128,256}D \in \lbrace 64, 128, 256\rbrace, native MHA/GQA, and optional +Paid 4-bit tiers across 7 end-to-end models. [Revised Paper: Section 1 (p. 2), Sections 3.2–3.4 (pp. 4–6), Section 5 (p. 10), Appendices A–B (pp. 14–16)]

AuthorsOpenReview ↗

Response to Reviewer MgRX (Part 2/2: W1, W2 & Q2, Q3)

Response to W1, W2 & Q2, Q3 (Direct Comparison of Raw E3, Fixed +Free, and SageAttention2++ on Identical Inputs, and Why FreeAttn Runs Faster)

On identical end-to-end inputs (Table R4), our fixed Free 8/8 E3M4 (+Free) policy yields large model-level gains over E3M4 (Raw) at minimal overhead—adding up to 5.2% LongBench-v2 attention time (raising agreement from 90.6% on Qwen3-8B and 96.9% on Qwen3-32B to 100.0%, 32/32) and 1.0%–1.1% full 50-step 720p video generation time (+7.22+7.22 dB PSNR on primary Turntable, 16.80→24.0216.80 \to 24.02 dB; +1.79+1.79 dB across the 4-scene mean, 18.65→20.4418.65 \to 20.44 dB). Our simultaneous speed and fidelity advantage over Sage2++ follows directly from this zero-inner-loop-overhead design (Global Response Item 3): unified hardware QMMA.SF (4 mantissa bits, full FP32 accumulation) plus single-instruction register F2FP eliminates the per-tile software scale conversions in Sage2++'s split INT8/E4M3 main loop, while fusing all four +Free transforms into one GDDR7 pass. Against Sage2++ F16, E3M4 (+Free) wins every LLM and video metric while running LongBench-v2 attention 7.5%–11.9% faster and 720p generation 6666–6969 s faster (full operator calls run 1.14×1.14\times–1.16×1.16\times faster across 12 synthetic shapes and 1.05×1.05\times–1.53×1.53\times faster across 4 captured layers). Against Sage2++ F32, it achieves higher LLM fidelity while running LongBench-v2 attention faster on Qwen3-8B (0.2450.245 vs. 0.2570.257 s) and matching Qwen3-32B (0.8760.876 vs. 0.8770.877 s), and finishes 50-step 720p video 4747 s faster at +3.95+3.95 dB PSNR on Turntable and 4343 s faster at +1.16+1.16 dB mean PSNR across the 4-scene suite (with slightly lower mean SSIM: 0.79390.7939 vs. 0.80540.8054). [Revised Paper: Section 4.3–4.5 Figures 4 & 6 (pp. 8–10), Appendix D Tables 5 & 7 (pp. 21, 23), Appendix E Tables 10–11 (pp. 25–26), Appendix F Tables 13–14 (p. 28)]

Table R4: Head-to-head comparison of SageAttention2++ (F16 / F32), E3M4 (Raw), and fixed Free 8/8 E3M4 (+Free) on identical end-to-end inputs (Appendix Table 11, p. 26; Tables 13–14, p. 28; and bench/results_wan14b_720p_50step.json).

WorkloadMetric (incl. preprocessing)Sage2++ F16Sage2++ F32E3M4 (Raw)E3M4 (+Free)
Qwen3-8B (32K)Mean KL ↓\downarrow / Answer Agrmt. ↑\uparrow0.0065 / 96.9%0.0074 / 96.9%0.0123 / 90.6%0.0033 / 100.0%
Qwen3-8B (32K)Greedy Sequence Match (128t) ↑\uparrow78.1% (25/32)81.2% (26/32)78.1% (25/32)93.8% (30/32)
Qwen3-8B (32K)Attn Time / TTFT (s) ↓\downarrow0.278 / 1.490.257 / 1.470.233 / 1.450.245 / 1.46
Qwen3-32B (32K)Mean KL ↓\downarrow / Answer Agrmt. ↑\uparrow0.0043 / 96.9%0.0078 / 96.9%0.0087 / 96.9%0.0041 / 100.0%
Qwen3-32B (32K)Greedy Sequence Match (128t) ↑\uparrow75.0% (24/32)81.2% (26/32)75.0% (24/32)90.6% (29/32)
Qwen3-32B (32K)Attn Time / TTFT (s) ↓\downarrow0.947 / 5.680.877 / 5.620.833 / 5.560.876 / 5.59
Wan14B (720p primary)50-Step PSNR (dB) ↑\uparrow / LPIPS ↓\downarrow21.28 / 0.077520.07 / 0.090016.80 / 0.171324.02 / 0.0481
Wan14B (720p primary)50-Step SSIM ↑\uparrow / Time (s) ↓\downarrow0.9164 / 1638.60.9071 / 1616.40.8551 / 1553.80.9435 / 1569.7
Wan14B (720p 4-scene)Mean PSNR (dB) ↑\uparrow / LPIPS ↓\downarrow18.98 / 0.149919.28 / 0.139318.65 / 0.160520.44 / 0.1450
Wan14B (720p 4-scene)Mean SSIM ↑\uparrow / Time (s) ↓\downarrow0.7829 / 1631.60.8054 / 1608.80.7736 / 1548.10.7939 / 1565.8

Thank you for your positive assessment and helpful questions.

Reviewer_MgRXOpenReview ↗

Response to Rebuttal

Thank you for the detailed response and additional experiments, which strengthen the empirical evidence and demonstrate the fidelity benefits of the correction framework. I will maintain my score and increase my confidence.

AuthorsOpenReview ↗

Thank you for your review and updated confidence

Thank you for reviewing our additional experiments and increasing your confidence! We are glad the end-to-end comparisons under our fixed +Free configuration clearly demonstrated its model-level fidelity benefit, and we truly appreciate your support.

Reviewer_CTtQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FreeAttn integrates native E3M4 conversion and MMA instructions into attention on NVIDIA SM120. The paper validates the instruction semantics, analyzes QK/PV quantization-error propagation, and develops composable corrections with fused execution. Controlled experiments compare E3M4 with E4M3, while optimized configurations demonstrate language-model prefill and video-generation speedups.

Critical strengths
  1. Practical native E3M4 attention implementation. The paper integrates native E3M4 conversion and MMA into fused attention,.

  2. Analysis of quantization errors. The paper explains how quantization errors affect attention outputs and identifies greater sensitivity to QK errors in the evaluated model captures.

  3. Evaluation of error-correction methods. The paper evaluates several error-correction techniques and examines their effects on attention accuracy and computational cost.

Critical weaknesses
  1. Novelty: Native E3M4 MMA support has been publicly documented [1], and prior PTQ research has reported advantages of E3M4 over E4M3 on vision tasks [2]. The paper should clarify its incremental contributions beyond these precedents.

  2. Experimental settings: The main end-to-end experiments use raw E3 without the corrections proposed in Section 3.4, while Sage2++ enables Smooth-K. Their quantization granularities also differ. Although separate matched-format controls are provided, a comparison of E4M3 and E3M4 with matched smoothing and quantization granularity under the current optimized configuration would better isolate the format's contribution.

  3. End-to-end results: The advantage at comparable fidelity remains unclear. FreeAttn is faster than Sage2++ but has higher output-distribution KL and lower video SSIM (Tables 8 and 11), although standard-task accuracy remains close to BF16.

[1] blackwell-isa: Public research artifacts for the NVIDIA Blackwell SASS instruction set. Revision 8fe1478, as cited by the submission. https://github.com/kacper-daftcode/blackwell-isa/blob/8fe1478/README.md

[2] Haihao Shen et al. Efficient Post-training Quantization with FP8 Formats. arXiv:2309.14592v2, 2024.

Questions for authors
  1. In Fig.8, why Free8/8 get high error compared with Sage2++?

  2. Are the best-performing correction methods the same for E3M4 and E4M3? Can you compare E3M4 and E4M3 using their best-performing correction configuration with Sage2++, in end-to-end tasks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CTtQ (Part 1/2: W1 & Q1)

Response to W1 (Novelty Beyond Public ISA Notes [1] and Prior FP8 PTQ Studies [2])

Rather than an engineering wrapper around an instruction probe, FreeAttn combines block-scaled native E3M4 execution with zero-inner-loop-overhead equivalent transforms (+Free), differing fundamentally from non-block-scaled FP8 PTQ (Shen et al., 2024 [2], cited in Sections 1 and 5, pp. 2, 10) and static blackwell-isa probes [1]:

  • Native Block-Scaled E3M4 vs. Non-Block-Scaled FP8 [2] (Figure 5a,b, p. 9; Appendix Table 3, p. 16; Table 4, p. 20): Without hardware 32-element block scaling [2], cross-block outliers clip E3M4's narrower range. Under SM120's K=32K=32 hardware block scaling (UE8M0), each 32-element block has its own power-of-two exponent to absorb dynamic range. Across 18 real Q,K,VQ,K,V tensors (Appendix Table 3, p. 16), over 99.93% of ℓ2\ell_2 energy within each block lies inside E3M4's normal range [0.25,30.0][0.25, 30.0], turning E4M3's idle fourth exponent bit into a fourth mantissa bit for 3.82×3.82\times–4.62×4.62\times lower tensor MSE at identical Tensor Core throughput (1.000×1.000\times). Single-operand isolation (Figure 5b; Table 4) confirms that E3M4 halves error on QK⊤QK^\top, VV, and online probabilities PP individually.
  • In-Register F2FP Packing & Softmax-Invariant +Free Transforms vs. Static Probes [1] (Figure 5c,d, p. 9; Appendix B, p. 15; Table 8, p. 24): Public blackwell-isa notes [1] test QMMA.SF format bits on static pre-quantized matrices without packed conversion (F2FP) or streaming attention dynamics, where O(BHN2)O(BHN^2) probabilities PP are generated online in registers and truncating PP biases outputs by (ΔP1)μV⊤(\Delta P \mathbf{1})\mu_V^\top. FreeAttn first recovers native dual-element F2FP E3M4 conversion (128-bit SASS bits {76,85}\lbrace 76, 85\rbrace) with log-domain full-scale folding to pack online PP in one register instruction (4.6%–17.4% faster complete operator calls vs. software bit-packing). Second, it proves that key centering (Smooth-K), balanced scaling and Hadamard rotation (Scale-QK + Hadamard-QK), and value centering with epilogue +μV+\mu_V (Smooth-V, which zeroes (ΔP1)μV⊤(\Delta P \mathbf{1})\mu_V^\top) cancel algebraically across online softmax with zero extra inner-loop matrix multiplies, fusing into one streaming GDDR7 pass (1,4671{,}467–1,5591{,}559 GB/s vs. 420420–445445 GB/s unfused) cleanly separated from optional +Paid compensations. [Revised Paper: Section 1 (p. 2), Sections 3.2–3.4 (pp. 4–6), Figure 5a–d (p. 9), Section 5 (p. 10), Appendices A–B (pp. 14–16), Table 8 (p. 24)]

Response to Q1 (Why Free 8/8 Had Higher Error Than Sage2++ in Initial Figure 8)

Initial Figure 8 evaluated uncorrected Free 8/8 E3M4 on the older non-block-scaled D=128D=128 path, whereas Sage2++ enabled Smooth-K key centering and per-tile Smooth-Q query centering. On our hardware K=32K=32 block-scaled kernel with +Free (revised Appendix Figure 7, p. 17; Appendix Table 5, p. 21), Free 8/8 E3M4 (+Free) is both faster and more accurate than Sage2++ F16 on Qwen layers (Qwen3-8B L21: 0.1240.124 vs. 0.1900.190 ms, relative L2L_2 error 0.020260.02026 vs. 0.033250.03325; Qwen3-32B L0: 24.8324.83 vs. 26.0926.09 ms, 0.022830.02283 vs. 0.041270.04127). On Wan diffusion layers with milder channel outliers, +Free omits Smooth-Q (whose key-varying residual ΔSij=μg(i)⊤Kc,j/D\Delta S_{ij}=\mu_{g(i)}^\top K_{c,j}/\sqrt{D} cannot cancel in row-wise softmax like Smooth-K and requires inner-loop compensation; Appendix C.2, Eqs. (15)--(16), p. 19), running faster within ≤0.0011\le 0.0011 relative L2L_2 error (Wan1.3B L18: 0.1630.163 vs. 0.2170.217 ms, 0.009290.00929 vs. 0.009010.00901; Wan14B L19: 169.54169.54 vs. 181.07181.07 ms, 0.010850.01085 vs. 0.009740.00974). Enabling +Free+Paid in Table 5 (16-channel BF16 score protection OC-QK) lowers error below Sage2++ F16 on both layers: 0.008020.00802 at 0.2140.214 ms on Wan1.3B L18 (both faster and more accurate) and 0.009710.00971 vs. 0.009740.00974 on Wan14B L19 at 225.49225.49 vs. 181.07181.07 ms (+24.5% latency; tiled query centering Smooth-Q-Tile similarly takes 203.13203.13–205.73205.73 ms on Wan14B L19 for a <0.001<0.001 single-layer gain). Inner-loop compensations thus pay off mainly in high-fluctuation query or 4-bit regimes, whereas default +Free already improves both multi-step video quality and speed (24.0224.02 vs. 21.2821.28 dB). [Revised Paper: Appendix Figure 7 (p. 17), Appendix C.2 Eqs. (15)--(16) (p. 19), Appendix D.1 Table 5 (p. 21)]

AuthorsOpenReview ↗

Response to Reviewer CTtQ (Part 2/2: W2, W3 & Q2)

Response to W2, W3 & Q2 (Whether Best Corrections Match for E3M4 and E4M3, and End-to-End Comparison Under Their Best Policy vs. Sage2++)

Yes: E3M4 and E4M3 share the exact same correction ranking (Appendix Table 4, Part B, p. 20). For minimum single-layer error regardless of cost, unrotated explicit channel protection (+oc_qk or +smk_smv_scqk+prob_norm+sink_v+oc_qk) achieves the lowest error for both formats (0.01460.0146–0.01640.0164 E3M4 vs. 0.02860.0286 E4M3 on Qwen3-32B L0; 0.00940.0094 vs. 0.01770.0177 on Wan14B L19). For end-to-end deployment balancing speed and fidelity, single-pass fused +Free (SmK+SmV+ScQK+HadQK) is the shared optimal policy for both. Under this shared +Free policy and identical K=32K=32 block scaling (Table R1; Figure 6, p. 9; Appendix Tables 9–14, pp. 25–28), 8/8 E3M4 (+Free) outperforms both 8/8 E4M3 (+Free) and Sage2++ F16. Across four 32K LLMs, it cuts prefill KL by 2.4×2.4\times–4.3×4.3\times over E4M3 (+Free). On LongBench-v2, it achieves lower mean KL than both Sage2++ F16 and E4M3 (+Free) (Qwen3-8B: 0.00330.0033 vs. 0.00650.0065 and 0.00790.0079; Qwen3-32B: 0.00410.0041 vs. 0.00430.0043 and 0.00930.0093, with faster attention) alongside 100.0% (32/3232/32) answer agreement (vs. 87.5% / 93.8% for E4M3 +Free and 96.9% for Sage2++ F16). On 50-step 720p Wan2.1-14B Turntable, it reaches 24.0224.02 dB PSNR and 0.94350.9435 SSIM (+3.86+3.86 dB over E4M3 +Free; +2.74+2.74 dB and 6969 s faster than Sage2++ F16). [Revised Paper: Section 4.2–4.5 Figures 5–6 (pp. 7–10), Appendix Figure 7 (p. 17), Appendix Tables 4, 5, 9–14 (pp. 20–21, 25–28)]

References: [1] kacper-daftcode, "Blackwell ISA: Public Research Artifacts for the NVIDIA Blackwell SASS Instruction Set," GitHub, 2026 (blackwellisa in revised paper). [2] H. Shen et al., "Efficient Post-training Quantization with FP8 Formats," arXiv:2309.14592v2, 2024 (shen2024fp8 in revised paper).

Thank you for your review; we are glad to discuss the matched E3/E4 comparisons further if helpful.

Reviewer_CTtQOpenReview ↗

Response to Rebuttal

Thank you for the updates and detailed explanations. The matched E3M4/E4M3 comparisons and end-to-end results with corrections address my main concerns and clarify the practical accuracy–latency trade-off. I also appreciate the improved organization of the experiments.

While I still view the contribution as primarily systems and implementation work, the revised results demonstrate its practical value more clearly.

Overall, I think it's a good paper and I have therefore raised my score.

Here are a few suggestions for future work, rather than requests for additional experiments in this revision:

  1. Combine E3M4 quantization with sparse attention and evaluate the best achievable end-to-end speedup.
  2. Compare E3M4 and E4M3 under QAT.
  3. Explore reusing or selectively updating K/V quantization statistics across denoising steps, balancing numerical fidelity against statistics-computation overhead to assess potential end-to-end speedups.

Note: The experiment part has changed notably from the original submission. My increased score is based on these revised settings and results.

AuthorsOpenReview ↗

Thank you for your reassessment and suggestions

Thank you for carefully reviewing our revision and raising your score! Once hardware block scaling handles dynamic range, mantissa precision becomes the main bottleneck, making native E3M4 equally promising for sparse attention, cross-step diffusion reuse, and QAT. We will include these directions in the final version and hope our results encourage upstream toolchains like PTX to expose E3M4 directly.

Reviewer_vsqBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FreeAttn, a low-precision attention implementation for Blackwell SM120. It validates native E3M4 conversion and matrix computation modes, analyzes how QK and PV quantization errors affect attention outputs, and proposes composable correction methods. The paper evaluates both operator-level and model-level performance. Its latency measurements distinguish kernel time from complete-call time, which includes preparation, corrections, and output processing. The model-level experiments cover Qwen long-context prefill and Wan video generation. The results indicate that FreeAttn can improve speed, while output fidelity relative to BF16 varies across configurations and workloads.

Critical strengths

The paper provides useful analysis of low-precision error propagation through attention. It separates the effects of QK quantization on attention weights from the effects of PV quantization on the weighted sum. It also shows that probability error alone does not determine output error, since the effect depends on the value vectors. This analysis helps explain why precision allocation can depend on the workload. The performance evaluation distinguishes kernel latency from complete-call latency. Including quantization preparation, corrections, and output processing gives a more practical view of the realized speedup and shows how these costs affect performance. The authors also validate the native conversion path and compare it with software conversion under fixed arithmetic, providing evidence for the performance contribution of native conversion.

Critical weaknesses

The end-to-end results are difficult to attribute to specific design choices. FreeAttn and Sage2++ differ in quantization methods, QK/PV precision, grouping, corrections, and execution implementation. Although the paper includes format comparisons, correction ablations, and precision-configuration experiments, these experiments do not fully match the end-to-end evaluation settings. It is therefore difficult to determine which factors account for the observed differences in end-to-end quality and speed. The evidence for end-to-end quality remains limited. In the Qwen experiments, FreeAttn has higher KL divergence than BF16 and Sage2++. In the Wan video experiments, FreeAttn is faster, but its SSIM relative to BF16 is substantially lower; the evaluation includes only four scene/noise pairs. The paper notes that reducing operator error does not necessarily improve model outputs, but the model-level evidence is not sufficient to support broad claims about accuracy.

Questions for authors

Attributing end-to-end differences. Could the authors clarify how much of the end-to-end quality and speed differences comes from quantization choices, QK/PV precision, correction methods, and system-level optimizations? A more direct connection between the controlled experiments and the end-to-end configurations would help readers interpret the results. Scope of the quality claims. Given the higher KL divergence on Qwen and lower BF16-relative SSIM on Wan, how should readers interpret the paper’s accuracy claims? Could the authors either expand the model-level evaluation with more independent requests, scenes, and random seeds, or qualify the conclusions as workload-specific speed-fidelity trade-offs?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vsqB (Part 1/2: W1 & Q1, Table R3 Part A)

Response to W1 & Q1 (Connecting Controlled Single-Factor Ablations Directly to End-to-End Quality and Speed)

End-to-end fidelity gains come jointly from switching format from E4M3 to E3M4 and single-pass +Free smoothing, while speedups at matched bitwidth come from native F2FP hardware packing and single-pass preprocessing fusion (Table R3; Figures 5–6, p. 9; Appendix Table 4, p. 20; Table 8, p. 24; Table 11, p. 26; Table 13, p. 28):

  1. Format and Corrections from Operator to End-to-End: At the operator level (Table R3), switching any single 8-bit operand from E4M3 to E3M4 halves its output error, while adding +all_free on Qwen3-32B L0 cuts E3M4 error by 78.1% (0.1043→0.02280.1043 \to 0.0228) at +3.8% latency (24.07→24.9924.07 \to 24.99 ms). This progression carries directly to identical end-to-end LongBench-v2 inputs on Qwen3-8B (Appendix Table 11, p. 26): starting from Raw E4M3 (KL 0.05350.0535, 81.2% agreement, 0.2310.231 s attention), switching format alone to Raw E3M4 improves results to KL 0.01230.0123 and 90.6% (0.2330.233 s), adding +Free alone on E4M3 reaches KL 0.00790.0079 and 87.5% (0.2420.242 s), and combining both in E3M4 (+Free) reaches KL 0.00330.0033 and 100.0% agreement (0.2450.245 s vs. 0.2780.278 s for Sage2++ F16). On 50-step 720p Wan2.1-14B Turntable (Appendix Table 13, p. 28; bench/results_wan14b_720p_50step.json), PSNR likewise progresses from 16.8016.80 dB (Raw E3M4, 1,553.81{,}553.8 s) and 20.1720.17 dB (E4M3 +Free, 1,561.91{,}561.9 s) to 24.02 dB (E3M4 +Free, 1,569.71{,}569.7 s vs. 21.2821.28 dB / 1,638.61{,}638.6 s for Sage2++ F16).
  2. QK⊤/PVQK^\top / PV Precision Allocation (Appendix Tables 5–6, p. 21; Tables 11–13, pp. 26–28): Under +Free alone, keeping 8-bit QK⊤QK^\top / 4-bit PVPV (8/4 E3M4) beats unprotected 4-bit QK⊤QK^\top / 8-bit PVPV (4/8); on outlier-heavy Qwen layers, adding 16-channel BF16 score protection makes 4/8 (+Free+Paid) lower in relative L2L_2 error than unprotected 8/4 (+Free) (0.046580.04658 at 25.9125.91 ms vs. 0.081860.08186 at 20.2320.23 ms on Qwen3-32B L0). End-to-end, lowering 8/8 to 8/4 or 4/4 reduces single-forward attention time (e.g., 6.536.53 s to 5.745.74 s / 5.815.81 s on 720p Wan14B) at a clear multi-step fidelity cost (Appendix Tables 11 and 13).
  3. Why E3M4 (+Free) Runs Faster Than Sage2++ (Figure 5c,d; Appendix Table 8, p. 24): At bitwise-identical outputs, pairing unified hardware QMMA.SF block scaling with single-instruction register F2FP packing cuts full operator call time by 17.4%/10.1%/4.6% at D=64/128/256D=64/128/256 over software bit-packing and avoids the per-tile software scale conversions in Sage2++'s split INT8→\toE4M3 pipeline. Outside the main loop, all four equivalent +Free transforms add zero inner-loop instructions and fuse into one streaming pass at 1,4671{,}467–1,5591{,}559 GB/s (vs. 420420–445445 GB/s multi-pass, only 1.4%–4.1% of long-sequence call time), keeping end-to-end E3M4 (+Free) attention 7.5%–11.9% faster than Sage2++ F16. [Revised Paper: Section 4.2–4.5 Figures 5–6 (pp. 7–10), Appendix Tables 4 (p. 20), 5–6 (p. 21), 8 (p. 24), 11 (p. 26), 13 (p. 28)]

Table R3: Single-operand isolation (Part A, native K=16K=16 UE4M3 4-bit scaling; other operands in exact FP32) and streaming-kernel correction breakdown (Part B, from Appendix Table 4, p. 20).

Part A: Single-operand quantization error isolation (Rel-L2L_2 ↓\downarrow, Unfixed / Corrected)

LayerIsolated Operand8/8 E4M38/8 E3M44/4 NVFP4
Qwen3-32B L0QK⊤QK^\top-only (Unfixed / +Free)0.2016 / 0.03230.1035 / 0.01860.2593 / 0.1114
Qwen3-32B L0VV-only (Unfixed / +Smooth-V)0.0243 / 0.02340.0123 / 0.01190.0791 / 0.0775
Qwen3-32B L0PP-only (Unfixed / +SmV+Prob-Norm)0.0044 / 0.00440.0023 / 0.00290.0263 / 0.0313
Qwen3-32B L0PVPV-only (Unfixed / +SmV+Prob-Norm)0.0247 / 0.02370.0125 / 0.01210.0834 / 0.0837
Qwen3-32B L0All operands (Unfixed / +Free)0.2031 / 0.04020.1043 / 0.02220.2717 / 0.1382
Wan2.1-14B L19QK⊤QK^\top-only (Unfixed / +Free)0.0216 / 0.01860.0111 / 0.00990.0762 / 0.0658
Wan2.1-14B L19VV-only (Unfixed / +Smooth-V)0.0047 / 0.00470.0024 / 0.00240.0179 / 0.0173
Wan2.1-14B L19PP-only (Unfixed / +SmV+Prob-Norm)0.0027 / 0.00220.0014 / 0.00130.0255 / 0.0277
Wan2.1-14B L19PVPV-only (Unfixed / +SmV+Prob-Norm)0.0056 / 0.00520.0028 / 0.00260.0338 / 0.0287
Wan2.1-14B L19All operands (Unfixed / +Free)0.0223 / 0.01950.0114 / 0.01030.0831 / 0.0707
AuthorsOpenReview ↗

Response to Reviewer vsqB (Part 2/2: Table R3 Part B & W2, Q2)

Part B: Full streaming kernel Free / Paid component breakdown (Rel-L2L_2 ↓\downarrow / Call ms ↓\downarrow)

LayerCorrection Config8/8 E4M38/8 E3M44/4 NVFP4
Qwen3-32B L0none (Unfixed)0.2031 / 23.420.1043 / 24.070.2717 / 15.34
Qwen3-32B L0+scale_qk0.2011 / 23.730.1033 / 24.250.2447 / 15.52
Qwen3-32B L0+smooth_k0.1234 / 23.750.0607 / 24.230.1896 / 15.46
Qwen3-32B L0+smooth_v0.2033 / 24.340.1042 / 24.770.2716 / 16.05
Qwen3-32B L0+smooth_k+smooth_v0.1233 / 24.440.0605 / 24.900.1897 / 16.13
Qwen3-32B L0+all_free (SmK+SmV+ScQK+HadQK)0.0417 / 24.540.0228 / 24.990.1453 / 16.46
Qwen3-32B L0+all_free+prob_norm+oc_qk (HadQK kept)0.0400 / 31.720.0222 / 32.260.1377 / 24.80
Qwen3-32B L0+all_free+oc_qk (HadQK kept)0.0401 / 30.870.0222 / 31.380.1364 / 21.51
Qwen3-32B L0+all_free+sink_v+oc_qk (HadQK kept)0.0399 / 30.870.0222 / 31.410.1356 / 21.54
Qwen3-32B L0+smk_smv_scqk+prob_norm+oc_qk (no HadQK)0.0288 / 32.350.0165 / 32.780.0944 / 24.74
Qwen3-32B L0+smk_smv_scqk+prob_norm+sink_v+oc_qk (no HadQK)0.0286 / 32.380.0164 / 32.810.0932 / 24.75
Wan2.1-14B L19none (Unfixed)0.0224 / 165.270.0116 / 169.840.0831 / 98.31
Wan2.1-14B L19+all_free (SmK+SmV+ScQK+HadQK)0.0199 / 169.270.0103 / 171.830.0704 / 100.61
Wan2.1-14B L19+all_free+prob_norm+oc_qk (HadQK kept)0.0185 / 233.100.0098 / 234.360.0657 / 165.24
Wan2.1-14B L19+all_free+oc_qk (HadQK kept)0.0187 / 228.520.0098 / 229.910.0658 / 145.13
Wan2.1-14B L19+all_free+sink_v+oc_qk (HadQK kept)0.0187 / 227.470.0098 / 228.860.0657 / 145.18
Wan2.1-14B L19+smk_smv_scqk+prob_norm+oc_qk (no HadQK)0.0177 / 239.210.0094 / 240.340.0632 / 166.07
Wan2.1-14B L19+smk_smv_scqk+prob_norm+sink_v+oc_qk (no HadQK)0.0177 / 238.340.0094 / 239.730.0632 / 166.28

Response to W2 & Q2 (Expanded Multi-Request, Multi-Scene, and Multi-Seed Evaluation, and Workload-Specific Speed–Fidelity Framing)

Across our expanded multi-request, multi-scene, and multi-seed suites, 8/8 E3M4 (+Free) beats Sage2++ F16 in both quality and speed on multi-step video and LongBench-v2, while 4-bit paths serve as higher-throughput lossy options (Section 4.5, pp. 9–10; Appendix Tables 9–14, pp. 25–28):

  • Expanded Independent Requests, Scenes, and Seeds: Language evaluation now spans four 32K GQA models (Appendix Table 9), all 32 LongBench-v2 requests (Appendix Table 11), 5,120 downstream items with paired 95% confidence intervals (Appendix Table 10), and 8 decode-KV requests plus 128-step thinking chains (Appendix Table 12). Video evaluation spans 6 VBench scenes on Wan2.1-1.3B, 6 distinct Wan2.1-14B prompts across 12 scene/seed/resolution runs—four 480p scenes, four full 50-step 720p scenes, six paired prompt/seed runs (Appendix Tables 13–14, geometric-mean E3M4/E4M3 (+Free) pixel MSE ratio 0.4150.415 across six runs and 0.5270.527 across four 720p scenes), and two additional prompts on our anonymous project page (https://freeattn-iclr27.pages.dev/)—plus single-step forward passes on Wan2.2-5B.
  • Explicit Workload- and Bitwidth-Specific Framing: On our unified K=32K=32 block-scaled kernel, 8/8 E3M4 (+Free) outperforms Sage2++ F16 in both quality and speed across all three multi-step video suites (+0.39+0.39 to +2.74+2.74 dB PSNR) and LongBench-v2 (32/3232/32 vs. 31/3231/32, 7.5%–11.9% faster attention). On 32K prefill KL (Appendix Table 9), 8/8 E3M4 (+Free) achieves lower KL than Sage2++ F16 on Qwen3-32B (0.001440.00144 vs. 0.001520.00152) and Ministral-3-8B (0.000610.00061 vs. 0.000720.00072), while trading a small KL difference for 9%–12% faster attention on Qwen3-8B (0.001870.00187 vs. 0.001790.00179) and Llama-3.1-8B (0.001460.00146 vs. 0.001370.00137). Enabling +Paid closes or beats Sage2++ F16 across all four models at 20%–27% higher attention latency, while 8/4 and 4/4 NVFP4 serve as higher-throughput lossy options. [Revised Paper: Section 4.5 (pp. 9–10), Appendix E Tables 9–12 (pp. 25–27), Appendix F Tables 13–14 (p. 28)]

We appreciate your constructive comments and welcome any further questions on the ablation breakdown.

AuthorsOpenReview ↗

Checking in on our new ablations and expanded evaluation

Dear Reviewer vsqB,

With the public discussion phase wrapping up soon, we wanted to briefly check in on our two-part response and revised manuscript above. To address your two questions, we added step-by-step ablations on identical inputs linking operator choices to end-to-end quality and speed, along with expanded multi-request, multi-seed, and multi-scene evaluations. Please let us know whether these additions clarify your questions—we welcome further discussion and any feedback to help us refine the paper.

Reviewer_KvUWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FreeAttn integrates native E3M4 (an FP8 format with 3 exponent and 4 mantissa bits) into fused attention on Blackwell SM120 GPUs. The authors identify and validate the E3M4 modes of the F2FP conversion and MMA instructions, which are not exposed through PTX types, by rewriting format bits in compiled instructions and checking them against independent numerical references. They decompose the attention output error into QK and PV components, organize existing training-free corrections (centering/smoothing, Hadamard rotation, channel retention, sink protection) into a composable menu with composition conditions under online softmax, and fuse preparation and recovery into the kernel. Validation includes operator benchmarks on captured Qwen3/Wan layers and synthetic shapes; Qwen3-8B/32B prefill (1.15–1.16× TTFT over Torch SDPA, KL to BF16, five short-context tasks); Wan2.1-14B video generation (1.43× delivery time, SSIM/MSE to BF16); matched E3/E4 format controls; and an FP4 path (1.45–1.51× complete-call speedup over Torch SDPA on seven shapes).

Critical strengths

Timely native E3M4 integration, with the value of the native path isolated. E3M4 is gaining hardware support (SM120, and SM110a per App. A.3), and bringing it into fused attention is a novel attempt. It is not a mere combination: at fixed E3 arithmetic with bitwise-identical outputs, native F2FP encoding reduces complete-call time by 4.6–17.4% over an optimized software encoding (Fig. 5a, App. D.4), which shows that the native path itself matters.

A careful, candid study of SageAttention-style training-free corrections. The corrections are ablated under controlled, paired settings, and both favorable and unfavorable results are reported. In particular, the QK/PV error decomposition is backed by controls (e.g., under raw QK4 the isolated-PV E3/E4 RMSE ratio goes from 0.499 to 0.997, App. B.3), and the paper shows that lower layer-wise error need not improve end-to-end fidelity (FineQ/OC lower layer RMSE but reduce video SSIM, Fig. 5b,c; Smooth-K lowers operator error but raises KL, App. E.3). These are questions the community cares about, and the evidence is useful to others working on low-bit attention.

Critical weaknesses

No perceptual evaluation for video, despite a large fidelity gap. On Wan2.1-14B, the optimized 8-bit E3 path reaches SSIM 0.692 vs. 0.810 for the 8-bit SageAttention2++ baseline (RGB MSE 1575 vs. 901; Table 11), for a ~4% delivery-time gain (1557 vs. 1616 s). It is also below the paper's own corrected FP4 results (~0.85 on Turntable, Table 12/Fig. 13; separate cohort). The paper notes that SSIM measures reference agreement rather than perceptual quality, but it reports no perceptual metric for this comparison, and only four scene/noise pairs. I consider a perceptual benchmark such as VBench essential for this line of work: even if imperfect, it is far more informative than SSIM here. Without it, the video results cannot show that the E3 path preserves generation quality. This is the main reason for my rating.

The LLM evaluation does not reflect realistic long-context serving. Only prefill attention is quantized; decoding and KV caches remain BF16 (L327–328), although in serving the KV cache persists and decode is a large part of the cost. Long inputs (11K–32K tokens) are used, but quality is measured by KL on the last 32 teacher-forced positions and a greedy generation pass with thinking disabled and a 128-token cap on 32 requests (13–16/32 correct, Table 8); the standard-task panel uses inputs of at most 2K tokens. No dedicated long-context benchmark is reported, so it remains unclear whether the method is safe in practical LLM serving.

Questions for authors

Please report VBench (or a comparable perceptual benchmark) for BF16, SageAttention2++, and the optimized E3 path on Wan2.1-14B, with more prompts than four scene/noise pairs. Parity with SageAttention2++ would resolve my main concern.

For LLMs, please evaluate settings closer to real serving, even if the results are unfavorable: thinking-enabled generation without the 128-token cap, a full long-context benchmark (e.g., LongBench v2 or RULER), and E3M4/FP4 KV caches used during decode.

Corrections also change the kernel schedule (Table 3). Could the authors report per-correction operator latency, separating the cost of the correction from that of the schedule change? Separately optimized kernels are not required.

In my own experiments, QK quantization has a comparatively small effect. The QK-dominance result (App. B.2) pairs 4-bit QK with 8-bit PV, and Table 5 shows the opposite ordering on synthetic inputs. Could the authors report the decomposition at matched bitwidths (QK8PV8, QK4PV4) and state when QK dominance is expected to hold?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KvUW (Part 1/3: W1 & Q1)

Response to W1 & Q1 (Perceptual Video Quality PSNR/SSIM/LPIPS, VBench Multi-Scene Evaluation, and Surpassing SageAttention2++)

The initial 0.6920.692 SSIM on Wan2.1-14B occurred because our evaluation script dispatched an older branch without hardware block scaling or +Free transforms. Once unified on our hardware K=32K=32 block-scaled QMMA.SF kernel with Free 8/8 E3M4 (+Free), we evaluate 6 distinct Wan2.1-14B prompts across 12 scene/seed/resolution runs (plus 6 VBench scenes on Wan2.1-1.3B), outperforming SageAttention2++ in both perceptual quality and speed (Table R2):

  • Suite-level wins: Free 8/8 E3M4 (+Free) beats SageAttention2++ under both FP16 and FP32 accumulators (Sage2++ F16 / F32) on PSNR, SSIM, and LPIPS (lower is better) across Wan2.1-1.3B (6 VBench scenes), Wan2.1-14B (4-scene 480p), and 50-step 720p Turntable (24.02 dB PSNR, 0.9435 SSIM, 0.0481 LPIPS; 6969 s faster than Sage2++ F16 and 4747 s faster than Sage2++ F32).
  • 4-scene 720p mean & seed stability (Appendix Table 14, p. 28): E3M4 (+Free) beats Sage2++ F16 on all three metrics (20.44 vs. 18.98 dB PSNR, 0.7939 vs. 0.7829 SSIM, 0.1450 vs. 0.1499 LPIPS; 6666 s faster). Against Sage2++ F32, it finishes 4343 s faster with +1.16+1.16 dB higher PSNR (20.4420.44 vs. 19.2819.28 dB) while trailing slightly on 4-scene mean SSIM/LPIPS (0.7939/0.14500.7939/0.1450 vs. 0.8054/0.13930.8054/0.1393, driven by smoke evolution on Train). Across six paired prompt/seed runs, its geometric-mean pixel MSE is 0.415×0.415\times that of E4M3 (+Free).
  • Semantic/temporal alignment & additional Wan2.1-14B prompts (bench/results_wan*.json): Across the VBench, 480p, and 720p suites, E3M4 (+Free) matches BF16 prompt-video CLIP similarity (0.32120.3212, 0.34940.3494, 0.33440.3344 vs. 0.32020.3202, 0.35090.3509, 0.33770.3377) and lowers temporal frame-difference MAE relative to Sage2++ F16 (4.714.71, 6.416.41, 7.607.60 vs. 5.115.11, 6.726.72, 8.698.69). Alongside our 5-scene 2×42\times 4 comparison video (videos/FreeAttn_ICLR27_Supplementary_Video.mp4), our anonymous project page (https://freeattn-iclr27.pages.dev/) adds two further Wan2.1-14B prompts (65-frame 3D Spiral Atrium Orbit: 19.0019.00 vs. 18.2618.26 dB, SSIM 0.4790.479 vs. 0.4320.432; Kitten on Grass: 26.3726.37 vs. 23.7323.73 dB, SSIM 0.9310.931 vs. 0.9200.920; both with faster attention than Sage2++ F16), providing 4 interactive split-screen sliders and 9 synchronized scene grids where E3M4 (+Free) preserves object proportions, motion direction, and fine textures closest to BF16. [Revised Paper: Figure 1 (p. 2), Section 4.5 & Figure 6b,c,f (p. 9), Appendix F Tables 13–14 (p. 28), Supplementary videos/FreeAttn_ICLR27_Supplementary_Video.mp4]

Table R2: Single-forward relative L2L_2 error, multi-step perceptual video quality (PSNR, SSIM, LPIPS), and latency across Wan diffusion models (Figure 1, Figure 6, and Appendix Tables 13–14). For the 20-step rows, Attn (s) is cumulative 20-step attention time and Step (s) is standalone single-forward DiT time; for 720p, Attn (s) and Step (s) are single-forward latencies and Full Gen (s) is 50-step wall-clock time (-- indicates single-forward-only or full-gen-only rows).

|Video Workload|Attention Config|Rel-L2↓L_2 \downarrow|PSNR ↑|SSIM ↑|LPIPS ↓|Attn (s)|Step / Full Gen (s)| |:-|:-|-:|-:|-:|-:|-:|-:|-:| |Wan1.3B (6 VBench)|Sage2++ F16 (8b)|0.00593|23.37|0.8444|0.0877|2.79 (1.69×)|0.98 (1.25×) / 17.53 (1.11×)| ||Free E4M3 (+Free)|0.00567|21.01|0.7747|0.1461|2.44 (1.93×)|0.93 (1.31×) / 17.18 (1.13×)| ||Free E3M4 (+Free)|0.00536|23.85|0.8605|0.0821|2.46 (1.92×)|0.94 (1.30×) / 17.20 (1.13×)| |Wan14B (4-scene 480p)|Sage2++ F16 (8b)|0.00643|22.02|0.8314|0.0961|11.60 (1.63×)|5.16 (1.20×) / 81.61 (1.09×)| ||Free E4M3 (+Free)|0.00730|20.25|0.7888|0.1247|10.81 (1.75×)|5.04 (1.23×) / 80.89 (1.10×)| ||Free E3M4 (+Free)|0.00595|22.41|0.8501|0.0811|10.92 (1.74×)|5.06 (1.23×) / 80.99 (1.10×)| |Wan14B (720p 50-step)|PyTorch BF16|0.00000|Ref|1.0000|0.0000|12.77 (1.00×)|21.63 (1.00×) / 2235.7 (1.00×)| ||Sage2++ F16 (8b)|0.00572|21.28|0.9164|0.0775|6.96 (1.83×)|15.81 (1.37×) / 1638.6 (1.36×)| ||Sage2++ F32 (8b)|0.00578|20.07|0.9071|0.0900|6.84 (1.87×)|15.69 (1.38×) / 1616.4 (1.38×)| ||Free E4M3 (+Free)|0.00545|20.17|0.9103|0.0767|6.45 (1.98×)|15.29 (1.41×) / 1561.9 (1.43×)| ||Free E3M4 (+Free)|0.00517|24.02|0.9435|0.0481|6.53 (1.96×)|15.39 (1.41×) / 1569.7 (1.42×)| ||Free 4/4 (+Free+Paid)|0.00572|--|--|--|5.81 (2.20×)|14.52 (1.49×) / --| ||Free 4/4 (+Smooth-QKV+Had)|--|17.69|0.8822|0.1494|--|-- / 1439.7 (1.55×)| ||Free 4/4 (Raw)|0.01263|12.56|0.8069|0.3580|3.64 (3.50×)|12.46 (1.74×) / 1430.0 (1.56×)|

AuthorsOpenReview ↗

Response to Reviewer KvUW (Part 2/3: W2, Q2 & Q3)

Response to W2 & Q2 (LongBench-v2, Thinking-Enabled Evaluation, and Decode-Phase E3M4/FP4 KV Caches)

Across LongBench-v2, 128-step thinking chains, and B=1B=1 decode KV caches, 8/8 E3M4 (+Free) matches or outperforms Sage2++ F16, whereas 4-bit KV caches degrade on smaller models (Section 4.5, pp. 9–10; Appendix E Tables 9–12, pp. 25–27):

  • LongBench-v2 and 32K / 5,120-Question Suites (Appendix Tables 9–11, pp. 25–26): Across all 32 LongBench-v2 requests (11.5K–31.9K input tokens, greedy max_new_tokens=128), Free 8/8 E3M4 (+Free) achieves 100.0% (32/32) final-answer agreement with BF16 on both Qwen3-8B and Qwen3-32B (vs. 31/3231/32, or 96.9%, for Sage2++ F16 and 87.5% / 93.8% for E4M3 +Free), with 93.8% (30/3230/32) and 90.6% (29/3229/32) exact sequence match and 1.14×1.14\times–1.16×1.16\times TTFT speedup over BF16. Across four 32K models, prefill KL drops 2.4×2.4\times–4.3×4.3\times vs. E4M3 (+Free) (1.07×1.07\times–1.14×1.14\times faster attention than Sage2++ F16), and 5,120-item macro-accuracy on Qwen3-8B and Qwen3-32B (70.16%, 76.37%) matches BF16 (70.04%, 76.27%).
  • Thinking-Enabled Evaluation (enable_thinking=True, Appendix Table 12, p. 27): We measure next-token KL divergence along 128-step teacher-forced BF16 reasoning chains (long-horizon free-running generation is left to future work). Free 8/8 E3M4 (+Free) cuts trajectory mean KL by 2.93×2.93\times on Qwen3-8B (0.002290.00229 vs. 0.006710.00671) and 2.49×2.49\times on Qwen3-32B (0.001730.00173 vs. 0.004310.00431) over E4M3 (+Free), outperforming Sage2++ F16 (0.003290.00329, 0.002200.00220).
  • Decode-Phase E3M4 and FP4 KV Caches (Appendix Table 12, p. 27): We test numerical decode-KV fidelity across 8 sequential B=1B=1 long-context requests using a PyTorch K=32K=32 power-of-two (UE8M0) quantize-dequantize simulation (effective bitwidths of 8.258.25, 6.256.25, and 4.254.25 bits/element for 8/8, 8/4, and 4/4 including the 8-bit scale per 32 elements, excluding per-head metadata). Over E4M3 (+Free), 8/8 E3M4 (+Free) cuts 32-step decode KL by 3.6×3.6\times on Qwen3-8B (0.008670.00867 vs. 0.031270.03127, with 6/86/8 exact matches [75.0%] and 90.23% token agreement) and 2.9×2.9\times on Qwen3-32B (0.004570.00457 vs. 0.013230.01323, with 7/87/8 exact matches [87.5%] and 98.44% token agreement, matching Sage2++ F16's 7/87/8 on uncompressed 16-bit KV). Under 4-bit KV paths (8/4 E3M4 +Free+Paid at 6.256.25b and 4/4 NVFP4 +Free+Paid at 4.254.25b), Qwen3-32B holds 87.5% (7/87/8, 97.27% / 98.44% token agreement), whereas smaller Qwen3-8B degrades to 62.5% (5/85/8, 75.71% token agreement) and 37.5% (3/83/8, 57.89% token agreement, decode KL 0.237210.23721), showing that 8-bit E3M4 is substantially safer than 4-bit KV on smaller models. [Revised Paper: Section 4.5 & Figure 6a,d,e (p. 9), Appendix E Tables 9–12 (pp. 25–27)]

Response to Q3 (Per-Correction Operator Latency Separated from Kernel Schedule Changes)

All +Free variants execute the exact same main streaming QMMA.SF kernel schedule (zero added main-loop instructions); only +Paid compensations switch to schedules with main-loop probability tracking or BF16 channel bypass (Appendix Table 4, Part B, p. 20):

  • Pure Correction Cost Under a Fixed Kernel Schedule (+Free breakdown): On Qwen3-32B L0 (S=31,852S=31{,}852, 8/8 E3M4), uncorrected none runs in 24.0724.07 ms. Keeping the identical streaming main loop, fused single-pass preparation and epilogue +μV+\mu_V add only +0.18+0.18 ms for +scale_qk (24.2524.25 ms), +0.16+0.16 ms for +smooth_k (24.2324.23 ms), +0.70+0.70 ms for +smooth_v (24.7724.77 ms), +0.83+0.83 ms for +smooth_k+smooth_v (24.9024.90 ms), and +0.92+0.92 ms (+3.8%) for full +all_free including HadQK (24.9924.99 ms, cutting relative L2L_2 error from 0.10430.1043 to 0.02280.0228). On Wan2.1-14B L19 (S=75,600S=75{,}600), +all_free adds +1.99+1.99 ms (169.84→171.83169.84 \to 171.83 ms, +1.2%) under the same schedule.
  • Schedule and Bypass Cost (+Paid breakdown): On Qwen3-32B L0 (8/8 E3M4), retaining BF16 sink tokens (+sink_v only) adds +1.31+1.31 ms (25.3825.38 ms); switching to the main-loop quantized-weight tracking kernel (+prob_norm only) adds +2.44+2.44 ms (26.5126.51 ms); and running the 16-channel BF16 score bypass (+oc_qk only) adds +6.40+6.40 ms (30.4730.47 ms; 32.7832.78 ms combined in +smk_smv_scqk+prob_norm+oc_qk). This separation is why +Free serves as our default fast path, while +Paid is reserved for outlier-heavy or 4-bit regimes. [Revised Paper: Section 4.3–4.4 Figures 4–5 (pp. 8–9), Appendix C.3 Table 4 Part B (p. 20)]
AuthorsOpenReview ↗

Response to Reviewer KvUW (Part 3/3: Q4)

Response to Q4 (Matched-Bitwidth QK8PV8 and QK4PV4 Error Decomposition, and When QK⊤QK^\top Dominance Holds)

At matched bitwidths (QK8PV8 and QK4PV4), PVPV quantization dominates on isotropic inputs without channel outliers, whereas QK⊤QK^\top quantization dominates on unsmoothed real-model activations with concentrated channel means and outlier channels (Section 4.2, pp. 7–8; Appendix Table 4, Part A, p. 20; Table 6, p. 21):

  • Why QK⊤QK^\top error is small on synthetic inputs (matching the reviewer's observation): On isotropic zero-mean Gaussian inputs under native K=16K=16 UE4M3 4-bit scaling across three seeds, matched QK4PV4 single-operand isolation yields only 3.32%–3.41% relative L2L_2 error for QK⊤QK^\top-only, compared with 9.84%–10.28% for PP-only, 9.61%–10.97% for VV-only, and 13.76%–14.96% for PVPV-only (all four operands FP4: 14.25%–15.30%). Because channel magnitudes are uniform, 16/32-element block scales stay tight on QQ and KK, leaving unstructured PVPV aggregation as the dominant error source (and explaining why synthetic inputs favor raw QK4PV8 over QK8PV4).
  • When QK⊤QK^\top dominance holds on real layers—and how smoothing shifts the bottleneck: Real attention layers exhibit large per-channel mean offsets (e.g., key means hold 94.0% of KK energy in Qwen3-8B L0, Appendix Figure 8, p. 18) and concentrated outlier channels that inflate shared 32/16-element block scales and perturb softmax scores exponentially. Consequently, before smoothing (Unfixed in Appendix Table 4, Part A, p. 20), QK⊤QK^\top-only dominates PVPV-only at both matched QK8PV8 (E3M4: 0.10350.1035 vs. 0.01250.0125 on Qwen3-32B L0, 0.01110.0111 vs. 0.00280.0028 on Wan14B L19; E4M3: 0.20160.2016 vs. 0.02470.0247) and matched QK4PV4 (NVFP4: 0.25930.2593 vs. 0.08340.0834 on Qwen3-32B L0, 0.07620.0762 vs. 0.03380.0338 on Wan14B L19). Once +Free (Smooth-K + Scale-QK + Hadamard-QK) centers keys and spreads channel outliers, QK⊤QK^\top-only error on Qwen3-32B L0 drops 5.6×5.6\times in E3M4 (0.1035→0.01860.1035 \to 0.0186, approaching PVPV-only 0.01210.0121) and 2.3×2.3\times in FP4 (0.2593→0.11140.2593 \to 0.1114 vs. PVPV-only 0.08370.0837; adding 16-channel OC-QK brings full QK4PV4 to 0.09320.0932, near the PVPV-only floor). [Revised Paper: Section 4.2 (pp. 7–8), Appendix Figure 8 (p. 18), Appendix C.3 Table 4 Part A (p. 20), Appendix D.1 Table 6 (p. 21)]

Thank you for the thorough review. All side-by-side comparison videos and interactive sliders are available on our anonymous project page (https://freeattn-iclr27.pages.dev/) and in videos/, and we welcome any follow-up questions.

AuthorsOpenReview ↗

Looking forward to discussing our revision with you

Dear Reviewer KvUW,

As the discussion period nears its close, we wanted to briefly follow up on our response and revised manuscript above. We added dedicated experiments addressing your main questions on multi-prompt perceptual video evaluation, LongBench-v2 and decode-KV serving tests, fixed-schedule correction latency, and matched-bitwidth QK/PV error analysis. We would love to hear whether these new results address your concerns, and we look forward to any discussion or suggestions to help us further improve the work.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers (Part 3/3)

  1. Binary Checks & Reproducibility: Three-layer pre-execution verification with a self-contained reproduction package
    • Three-Layer Automated Binary Checks: Before any patched SM120 attention kernel executes, (1) build-time CMake checks run --inspect bitfield verification and full-file byte diffing (zero bit changes outside format-select fields, 100% rejection on 3 negative-test binaries), (2) the C++ module loader (csrc/host/api.cpp) enforces compute capability 12.0 (SM120) and symbol checks prior to cuModuleLoad, and (3) pre-run on-device probes (tests/cuda/check_probe.py) verify 81,93281{,}932 F2FP and 81,94481{,}944 E2M1 conversions bit-for-bit alongside 2,0682{,}068 exhaustive QMMA.SF sweeps in Table 2 and random inputs (relative error ≤10−6\le 10^{-6}, 0.000e+00 max error on E3M4).

Table R1: Representative end-to-end model quality and latency on one NVIDIA RTX PRO 6000 Blackwell GPU (96 GB GDDR7) across identical inputs and initial noise seeds, including preprocessing.

WorkloadEvaluation MetricPyTorch BF16Sage2++ F16E4M3 (+Free)E3M4 (+Free)
Wan14B (720p primary)PSNR (dB) ↑\uparrow / SSIM ↑\uparrowRef / 1.000021.28 / 0.916420.17 / 0.910324.02 / 0.9435
Wan14B (720p primary)LPIPS ↓\downarrow / 50-Step (s) ↓\downarrow0.0000 / 2235.70.0775 / 1638.60.0767 / 1561.90.0481 / 1569.7
Wan14B (720p 4-scene)Mean PSNR ↑\uparrow / LPIPS ↓\downarrowRef / 0.000018.98 / 0.149917.66 / 0.200220.44 / 0.1450
Wan1.3B (6 VBench)Mean PSNR ↑\uparrow / LPIPS ↓\downarrowRef / 0.000023.37 / 0.087721.01 / 0.146123.85 / 0.0821
Qwen3-8B (LongBench-v2)Mean KL ↓\downarrow / Agrmt. ↑\uparrow0.0000 / 100%0.0065 / 96.9%0.0079 / 87.5%0.0033 / 100.0%
Qwen3-32B (LongBench-v2)Mean KL ↓\downarrow / Agrmt. ↑\uparrow0.0000 / 100%0.0043 / 96.9%0.0093 / 93.8%0.0041 / 100.0%
Qwen3-8B/32B (KV Quant)32-Step KL (8B / 32B) ↓\downarrow0.0000 / 0.0000-- (BF16 KV)0.0313 / 0.01320.0087 / 0.0046

Full CUDA/Python source, reproduction commands (20–25 min self-contained build, probes, and synthetic benchmarks without external weights), result JSONs, and our 5-scene 2×42\times 4 comparison video are in FreeAttn_ICLR2027_Supplementary.zip, alongside our anonymous project page (https://freeattn-iclr27.pages.dev/). Reviewers can inspect the side-by-side videos there or run the provided code.

AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers (Part 2/3)

  1. Long-Context LLMs: 100% final-answer agreement on LongBench-v2, plus decode KV quantization and thinking-mode evaluation

    • 32K Prefill & 5,120-Question Suite: Across four 32K GQA models, Free 8/8 E3M4 (+Free) cuts mean KL divergence by 2.4×2.4\times–4.3×4.3\times over matched E4M3 while running attention 1.07×1.07\times–1.14×1.14\times faster than Sage2++ F16 (1.89×1.89\times–1.97×1.97\times attention and 1.18×1.18\times–1.19×1.19\times TTFT speedup over BF16). Its 32K prefill KL is directly below Sage2++ F16 on Qwen3-32B and Ministral-3-8B, while trading a tiny KL gap on Qwen3-8B (0.001870.00187 vs. 0.001790.00179) and Llama-3.1-8B for faster attention. Enabling +Paid beats Sage2++ F16 on 3 of 4 models (0.001800.00180 on Qwen3-8B) at 20%–27% higher attention time. On Qwen3-8B/32B, 5,120-question macro-accuracy matches BF16 (70.16%/76.37% vs. 70.04%/76.27%).
    • Real Long-Context & Thinking-Mode Evaluation: Across all 32 LongBench-v2 requests (up to 32K input tokens, max_new_tokens=128), it reaches 100.0% (32/32) final-answer agreement with BF16, and achieves lower mean KL divergence than Sage2++ F16 along 128-step teacher-forced thinking-mode (enable_thinking=True) reference chains.
    • Decode-Phase KV-Cache Quantization: We evaluate numerical decode-KV fidelity across 8 sequential B=1B=1 long-context requests using a PyTorch K=32K=32 power-of-two (UE8M0) quantize-dequantize simulation (effective bitwidths of 8.258.25, 6.256.25, and 4.254.25 bits/element for 8/8, 8/4, and 4/4 including the 8-bit scale per 32 elements, excluding per-head metadata; 1.94×1.94\times, 2.56×2.56\times, and 3.76×3.76\times compression vs. BF16). Over E4M3 (+Free), Free 8/8 E3M4 (+Free) cuts 32-step decode KL by 2.9×2.9\times–3.6×3.6\times, reaching 6/86/8 (75.0%, 90.23% token match) exact 32-token matches on Qwen3-8B and 7/87/8 (87.5%, 98.44% token match) on Qwen3-32B (vs. 5/85/8 and 7/87/8 for E4M3 +Free; Sage2++ F16 reaches 7/87/8 on uncompressed 16-bit KV).
  2. Mechanism & Prior-Work Positioning: Zero-Main-Loop-Overhead Block-Scaled Attention, Distinction Beyond Public ISA Probes, and Why It Beats Sage2++

    • Block-Scaled Streaming Attention Beyond Isolated ISA Probes: Public blackwell-isa notes probe QMMA.SF format bits on static pre-quantized matrices; they do not address streaming attention, where O(BHN2)O(BHN^2) probabilities PP are quantized online in registers, channel outliers inflate block scales, and truncating PP biases outputs by (ΔP1)μV⊤(\Delta P \mathbf{1})\mu_V^\top. FreeAttn eliminates these errors without adding main-loop instructions through two mechanisms. (1) Zero-overhead native E3M4 representation and register packing: under 32-element power-of-two block scaling (UE8M0), over 99.93% of energy within each block across 18 real Q,K,VQ,K,V tensors falls inside E3M4's normal range [0.25,30.0][0.25, 30.0], unlike non-block-scaled FP8 [Kuzmin et al., 2022; Shen et al., 2024]. Turning E4M3's unused fourth exponent bit into a fourth mantissa bit cuts tensor MSE by 3.82×3.82\times–4.62×4.62\times over matched E4M3 at identical Tensor Core throughput (1.000×1.000\times), while our recovered dual-element F2FP instruction packs online PP in registers in one instruction (4.6%–17.4% faster complete operator calls vs. software bit-packing). (2) Single-pass softmax-invariant transforms (+Free): subtracting key means (Smooth-K), applying balanced scaling and block-diagonal Hadamard rotation (Scale-QK + Hadamard-QK), and centering values with epilogue +μV+\mu_V restoration (Smooth-V) algebraically cancel score shifts and row-sum truncation bias (ΔP1)μV⊤(\Delta P \mathbf{1})\mu_V^\top outside the O(N2)O(N^2) inner loop in one streaming GDDR7 pass (1,4671{,}467–1,5591{,}559 GB/s vs. 420420–445445 GB/s unfused, 92%–98% peak). By contrast, query centering (Smooth-Q) leaves a key-varying residual ΔSij=μg(i)⊤Kc,j/D\Delta S_{ij}=\mu_{g(i)}^\top K_{c,j}/\sqrt{D} that does not cancel in row-wise softmax and requires inner-loop compensation, so we reserve it for optional +Paid paths in high-fluctuation query or 4-bit regimes.
    • Why Free 8/8 E3M4 (+Free) Beats SageAttention2++ in Both Speed and Accuracy: Sage2++ splits QK⊤QK^\top (per-warp INT8) and PVPV (3-mantissa-bit E4M3 with reduced FP16/FP22 accumulation), requiring per-tile software scale conversion inside the main loop and multi-pass Smooth-K-only preparation. FreeAttn removes main-loop software rescaling via unified hardware QMMA.SF (full FP32 accumulation, 4 mantissa bits) plus in-register F2FP, while collapsing all four equivalent +Free transforms into one streaming pass taking 1.4%–4.1% of long-sequence call time.
AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers (Part 1/3)

We thank all five reviewers (KvUW, CTtQ, vsqB, MgRX, and e3CT) for their constructive feedback and for recognizing our hardware discovery of native E3M4 packed conversion (F2FP) and block-scaled matrix multiply (QMMA.SF) on Blackwell SM120, our online probability encoding ablation, and our transparent experimental analysis.

Root-Cause Identification and Summary of Revisions: Tracing the gap between our operator controls and initial model numbers revealed a dispatch bug in our end-to-end evaluation scripts: in the initial submission, D=128D=128 model runs were routed to an older branch without hardware block scaling or fused +Free transforms. In the revision, all D=128D=128 operator and end-to-end evaluations on a single NVIDIA RTX PRO 6000 Blackwell GPU (96 GB GDDR7) run on our unified K=32K=32 block-scaled QMMA.SF kernel. Our default 8-bit policy Free 8/8 E3M4 (+Free) fuses key and value channel centering, QKQK channel scaling, and block-diagonal Hadamard rotation into one preprocessing pass with zero added main-loop instructions. For 4-bit tiers (Free 4/4 NVFP4) or extreme-outlier layers, optional +Paid compensations add main-loop probability normalization (Prob-Norm), tiled query centering (Smooth-Q-Tile), or a 16-channel BF16 score bypass (OC-QK). All updates below are incorporated into the uploaded revised PDF (Table R1; exact section and page pointers are in each individual response):

  1. Video Generation: Outperforming SageAttention2++ in both perceptual quality and speed, with side-by-side comparison videos
    • 50-Step 720p Generation (Wan2.1-14B): On the primary Turntable scene, Free 8/8 E3M4 (+Free) reaches 24.02 dB PSNR (+3.86 dB over matched E4M3 and +2.74 dB over Sage2++ F16, which uses FP16 PVPV accumulation) while finishing 50-step generation 69 s faster than Sage2++ F16 (1.42×1.42\times end-to-end speedup over BF16).
    • Multi-Scene and VBench Suites: Across all four 720p scenes (mean PSNR 20.44 dB, 0.527×0.527\times geometric-mean pixel MSE vs. E4M3 (+Free)), four 480p scenes, and Wan2.1-1.3B (6 VBench scenes), Free 8/8 E3M4 (+Free) outperforms Sage2++ F16 on PSNR, SSIM, and LPIPS (lower is better).
    • Comparison Videos & Anonymous Project Page (https://freeattn-iclr27.pages.dev/): Alongside our 5-scene 2×42\times 4 8-method comparison video (videos/FreeAttn_ICLR27_Supplementary_Video.mp4), our anonymous project page provides 4 interactive Wan2.1-14B split-screen sliders (720p Turntable and Waterfall, 3D Spiral Atrium Orbit, Kitten on Grass), 9 synchronized 2×42\times 4 scene grids, and verbatim 32K 8/8, 8/4, and 4/4 NVFP4 LLM outputs. While several baselines look plausible in isolation, frame-by-frame comparison with BF16 shows that E3M4 (+Free) matches BF16 most closely in object scale, motion direction, and fine details (toy bus front, cat whiskers and chest markings, tram windows and reflections, leaf raindrops).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.