- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
FreeAttn connects native E3M4 conversion and matrix instructions on Blackwell SM120 GPUs into an attention implementation. It also combines error corrections with fused execution. Matched E3/E4 experiments show lower aggregate numerical error at similar runtime. Operator benchmarks and Qwen/Wan experiments evaluate accuracy and speed.
Critical strengths
-
The matched-format comparisons make a useful case for E3M4: attention can benefit from its finer precision without increasing bitwidth or runtime. This gives practitioners another option beyond E4M3.
-
The native conversion path is backed by careful checks. In particular, the native/software comparison preserves outputs bitwise while reducing complete-call time by 4.6–17.4%. This helps establish where the implementation gains come from.
Critical weaknesses
-
The practical gain over SageAttention2++ is small. The current E3 configuration saves roughly 2% of Qwen first-token time and 4% of video delivery time, while showing larger deviations from BF16. The paper does not yet establish a clear advantage at comparable model fidelity.
-
Deployment requires rewriting GPU machine instructions. The paper provides validation checks, but compatibility and maintenance across GPU and toolchain versions remain unclear. This makes the modest end-to-end gains harder to judge in practice.
Questions for authors
-
At a matched model-fidelity target, how much speed advantage remains over SageAttention2++? This comparison would help me judge the practical benefit.
-
Which parts of the instruction-rewriting process depend on the GPU and compiler version? Does the released package detect incompatible binaries before running them?
5 comments
Response to Reviewer e3CT (Part 2/2: W2 & Q2)
Response to W2 & Q2 (GPU vs. Compiler-Version Dependence in Instruction Rewriting, and Pre-Run Binary Checks)
We separate GPU ISA dependence from compiler lowering and enforce three-layer pre-execution checks (Appendix A & Table 2, p. 14; Appendix G, pp. 29–31):
- GPU Architecture vs. Compiler Version Dependence: The rewritten format-select bit positions—
F2FPbits andQMMA.SF.16832bits (XOR mask )—are fixed by the BlackwellSM120128-bit SASS encoding. Compiler (ptxas) dependence is limited to lowering our PTX placeholder instructions intoF2FPandQMMA.SF.16832mnemonics (verified across CUDA 12.8–13.x). Because our patchers (tools/patch/sass_f2fp_e3m4_patch.pyandtools/patch/sass_mxf8f6f4_patch.py) locate instruction words fromnvdisasmELF code-section disassembly rather than fixed byte offsets, changes in register allocation, instruction scheduling, or branch offsets do not affect rewriting. - Detecting Incompatible Binaries Before Execution: Yes, across three layers before any attention kernel runs. (1) Build-time CMake fail-fast: CMake runs
--inspectacross every patched cubin (unexpected_bits=none) and performs a full-file byte diff (complete_file_diff=PASS, confirming zero bit changes outside the target format-select fields). If a compiler update alters expected mnemonics/opcodes, or if an input is missing, a corrupted non-ELF binary, or a wrong-architecture ELF object, the patcher exits non-zero without writing output (100% rejection across the 3 negative binary tests in Appendix A, p. 14 andbench/run_reviewer_ablations.py--patcher-only). (2) Runtime module-load guard (csrc/host/api.cpp): before callingcuModuleLoador launching any kernel, the C++ host loader queriesCU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR/MINOR(aborting immediately with an explicitRuntimeErrorunless compute capability is12.0/SM120) and verifiescuModuleLoadand all kernel symbol handles. (3) Pre-run on-device probe (tests/cuda/check_probe.py): before running benchmarks, GPU probes verify hardwareF2FP( values) andE2M1( values) bit-for-bit against IEEE-754 software decoding and, alongside the exhaustive byte-encoding sweeps in Table 2, checkQMMA.SFon exact-grid and random non-NaN inputs (relative error ,0.000e+00max error on E3M4). [Revised Paper: Section 3.2 (p. 4), Appendix A & Table 2 (p. 14), Appendix G (pp. 29–31)]
Thank you for your review; please let us know if any further details on latency or binary verification would be helpful.
Thanks to the authors for the clarification. My concerns have been addressed. The revised results show better fidelity in several settings, although the speed gains remain small. Modifying the compiled GPU code also adds deployment complexity. I view this as an incremental contribution that may be useful to other researchers.
Response to Reviewer e3CT (Part 1/2: W1 & Q1)
Response to W1 & Q1 (Speed Advantage Over
SageAttention2++at Matched or Higher Fidelity, and Separating DiT Step vs. Full Video Generation Time)Free 8/8 E3M4 (+Free)matches or exceedsSageAttention2++model fidelity while retaining a clear speed advantage. As detailed in Global Response Item 3, unified hardwareQMMA.SFplus in-registerF2FPremovesSage2++'s per-tileINT8/E4M3software scale conversions inside the main loop, while single-pass GDDR7 fusion keeps+Freepreprocessing to 1.4%–4.1% of full operator time on long sequences (9.7% on short-sequence Qwen3-8B L21):Wan2.1-14BTurntable generation,Free 8/8 E3M4 (+Free)exceedsSage2++ F16by +2.74 dB PSNR (24.02 vs. 21.28 dB, LPIPS 0.0481 vs. 0.0775) while running self-attention 1.07× faster (6.53 vs. 6.96 s/step) and finishing 50-step generation 69 s faster (1,569.7 vs. 1,638.6 s; +3.95 dB and 47 s faster thanSage2++ F32). Across the 4-scene 720p suite (Appendix Table 14), it outperformsSage2++ F16on all three metrics (mean PSNR 20.44 vs. 18.98 dB, SSIM 0.7939 vs. 0.7829, LPIPS 0.1450 vs. 0.1499) while finishing 66 s faster (1,565.8 vs. 1,631.6 s; 43 s faster with +1.16 dB higher mean PSNR thanSage2++ F32, at mean SSIM/LPIPS 0.7939/0.1450 vs. 0.8054/0.1393), and wins all metrics at lower latency on 6-scene VBench and 4-scene 480p. On 32KLongBench-v2, it achieves higher answer agreement (100.0% vs. 96.9%) and lower KL (Qwen3-8B: 0.0033 vs. 0.0065; Qwen3-32B: 0.0041 vs. 0.0043) while running attention 7.5%–11.9% faster thanSage2++ F16(Qwen3-8B: 0.245 vs. 0.278 s; Qwen3-32B: 0.876 vs. 0.947 s).Sage2++(Appendix Table 7),Free 8/8 E3M4 (+Free)and hybridFree 4/8 E3M4 (+Free)pass all 12/12 accuracy gates (5% relative L2 error threshold vs. FP32) and run 1.14×–1.16× and 1.27×–1.29× faster thanSage2++including preprocessing (1.03×–1.05× and 1.15×–1.17× faster than PyTorch SDPA). Across the four captured model layers (Appendix Table 5), full operator calls including preprocessing run 1.56×–1.94× faster than PyTorch SDPA and 1.05×–1.53× faster thanSage2++ F16. Specifically,8/8 E3M4 (+Free)is both faster and more accurate on Qwen3-8B L21 (0.124 vs. 0.190 ms, relative L2 error 0.02026 vs. 0.03325) and Qwen3-32B L0 (24.83 vs. 26.09 ms, error 0.02283 vs. 0.04127). On Wan1.3B L18 (0.163 vs. 0.217 ms, error 0.00929 vs. 0.00901) and Wan14B L19 (169.54 vs. 181.07 ms, error 0.01085 vs. 0.00974),+Freeruns faster, while+Free+Paidlowers error belowSage2++ F16(0.214 ms and 0.00802 on Wan1.3B L18, both faster and more accurate; 0.00971 vs. 0.00974 on Wan14B L19 at 225.49 vs. 181.07 ms, +24.5% latency). Against 4-bitSage3,Free 4/4 (+Free+Paid)runs 1.25×–1.44× faster including preprocessing across the 7 co-supported synthetic shapes (where all 4-bit PV paths exceed the 5% Gaussian gate at 5.99%–15.33% vs. 14.58%–33.24% forSage3) and is both faster and more accurate on Wan14B L19 (144.98 vs. 149.62 ms, error 0.06396 vs. 0.07232).Attn (s),1-Step DiT (s), andFull Gen (s)—across all video models (e.g., on 720pWan2.1-14B,E3M4 +Freeruns in 6.53 s, 15.39 s, and 1,569.7 s vs. 6.96 s, 15.81 s, and 1,638.6 s forSage2++ F16, achieving 1.96×, 1.41×, and 1.42× speedups over BF16). [Revised Paper: Section 4.3 Figure 4 (p. 8), Appendix D Tables 5 & 7 (pp. 21, 23), Appendix F Tables 13–14 (p. 28)]