- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents HyperHadamard, a CUDA kernel for exact normalized Hadamard transforms, aimed at online rotations in quantized LLM inference and at randomized Hadamard sketching. Its central observation is that the d butterfly stages of a length-2^d FWHT are rotations along independent hypercube axes and therefore commute (Lemma 1). Conventional tensor-core kernels such as HadaCore follow the stage order of the recursive algorithm, so they must shuffle or transpose each MMA output back into the layout the next stage expects. HyperHadamard instead keeps a typed register Fragment holding a compile-time permutation P (which logical axis occupies each lane or register bit) and an axis state S (pending or complete). Each mma.m16n8k16 (one H16, i.e., four axes) is applied to whichever pending axes the previous instruction left in the operand slots. Layout changes become metadata, and only thread-local register swaps are emitted: Appendix A computes H256 with four MMAs and no shuffles or shared-memory transposes. For N ≥ 4096 the axes are split into two register phases separated by one shared-memory exchange. Its strided view places the phase-2 axes directly into MMA slots and removes HadaCore's final output realignment. A Kronecker factorization H_q ⊗ H_{2^k} (H28 as a padded tensor-core GEMM) covers non-power-of-two orders such as 14336, and a true-FP32 CUDA-core variant reuses the same schedule.
Validation:
- Operator level (H100, 2^30 elements): FP16/BF16 throughput stays within 1.7% of an empirical 3.03 TB/s roofline for N = 256–32768. Against HadaCore it is at parity for N ≤ 4096 (0.988–1.003×), 1.05× and 1.10× at N = 8192 and 16384, and 1.52–1.59× at N = 32768. The mechanism is supported by Nsight profiles (3.47–3.60× fewer warp instructions; memory-I/O-queue stalls fall from about 5.7% to 0.1%), a forced-materialization ablation (+1.75% / +5.89% latency), an arithmetic-free speed-of-light kernel (only 1.055–1.058× faster) and an A100 replication. The FP32 kernel is 1.461× faster than Dao-AILab at N = 32768, which the paper attributes to lower register use and higher occupancy.
- vLLM: a Llama-3.1-8B QuIP W4A16 checkpoint uses catalog Hadamards of orders 14336, 4096 and 1024 (built from H224 and H256). The kernel matches them through a q×q basis correction that is exact in exact arithmetic. End to end, it runs 14.70× faster than vLLM's dense FP64 path and 1.915× faster than a Dao-AILab + cuBLAS baseline with the same factorization, with unchanged perplexity and ARC accuracy. For a power-of-two Llama-3.2-1B checkpoint, replacing HadaCore gives 1.022–1.052× prefill throughput and 0.978× in batch-8 decode.
- Influence Distillation: with gather remapping and fused Rademacher signs, the 2^30-element FP32 projector becomes 1.747× faster and remains bit-exact.
- Quantization study (Mistral-7B, GPTQ INT3/INT2 weights, dynamic per-token INT4 activations, rotations applied as FP64 hooks): relative to H128 blocks, full-dimension rotations lower WikiText-2 perplexity by 6.37% (INT3) and 39.6% (INT2). With BF16 or INT8 activations they change perplexity by +0.2% to +0.5%.
Critical strengths
S1. Credible, mechanism-level evidence for the speed claim. The paper goes beyond reporting throughput. HyperHadamard stays within 1.7% of the achievable roofline at every length. The gap to HadaCore is traced to instruction count and memory-I/O-queue stalls. Fewer instructions translate into speed only once HadaCore falls below the roofline (N ≥ 8192). A within-kernel ablation that forces materialization isolates the layout effect, an arithmetic-free kernel bounds the remaining headroom, and the result replicates on A100. This matters most to my assessment: a kernel paper stands or falls on whether its gain is real and explained, and here it is, with little headroom left.
S2. Practical coverage that existing fast kernels lack. The Kronecker factorization with an exact q×q basis correction gives catalog-Hadamard QuIP checkpoints (orders 14336, 4096 and 1024) a fast path in vLLM. End to end, this is 1.915× faster than an operation-matched Dao-AILab + cuBLAS baseline, with perplexity changing by 0.0026% and identical ARC accuracy. The true-FP32 variant, combined with gather remapping and fused signs, makes a released billion-coordinate projector 1.747× faster with bit-exact output. This shows the design carries over to real workloads, not only to a microbenchmark.
Critical weaknesses
W1. The benefit is confined to a narrow regime, and one end-to-end result does not agree with the operator sweep.
- Regime. FP16/BF16 gains require N ≥ 8192 and many rows. The kernel is at parity for N ≤ 4096. At N = 32768 it needs at least 2^23 elements to overtake HadaCore, and in batch-8 decode it reaches only 0.978× of HadaCore. The supported range starts at N = 256, which excludes the 16–128-element blocks used for head-dimension rotations and for group-aligned MXFP4/NVFP4 rotations, so many current deployments would see no benefit. The 14.70× headline mostly measures dense FP64 against any fast transform; the operation-matched comparison is 1.915×.
- Inconsistency. For Llama-3.2-1B, transform time falls from 126.2 ms to 85.6 ms (1.47×), and H8192 accounts for 73.3% of the saving (Appendix F). The operator sweep, however, reports 1.05× at N = 8192 and parity at N ≤ 4096. Even if all transform time were spent at N = 8192, a 1.05× kernel speedup would save at most about 5%, not 32%. Possible causes are that vLLM's embedded HadaCore differs from the standalone version, or that the much smaller tensors (about 2^24 elements at a 2048-token budget, versus 2^30 in the sweep) behave differently. Either way the gap needs an explanation, because it determines how well the operator results predict deployment.
W2. The application-level case for full-dimension rotations is thin and leaves out FP4 and adaptive transforms. Section 4.4 is the paper's argument that fast full-dimension rotations are worth having. It covers one model (Mistral-7B) and one recipe (GPTQ INT3/INT2 weights, dynamic per-token INT4 activations), with rotations run as FP64 hooks. The gain depends on activation precision rather than weight precision: with BF16 or INT8 activations, full-dimension rotations slightly hurt (+0.2% to +0.5%) for both INT3 and INT2 weights. Appendix H shows that H14336 alone hurts and helps only together with H4096, which is not explained. None of these quality-sensitive settings is run end to end with the kernel. The vLLM experiments use W4A16, where Table 4 suggests rotation size makes no difference.
- No FP4 evaluation. MXFP4 and NVFP4 are arguably the main 4-bit activation targets on current hardware, and the paper itself notes (Sec. 2, L119–121) that for microscaling FP4 formats rotations are usually confined to blocks matching the scaling groups. Under per-16- or per-32-element scales, an outlier inflates only its own group's scale. Full-dimension rotation helps per-token INT4 by diluting an outlier that would otherwise set a single row-wide scale, and that mechanism is largely absent under microscaling. Whether full-dimension rotations help, are neutral or hurt there is untested.
- No comparison with data-aware or learned transforms. WUSH (arXiv:2512.00956, ICML 2026) derives closed-form, data-aware blockwise transforms for joint weight–activation quantization. Each combines a Hadamard backbone with a second-moment-based component, is non-orthogonal, and uses a block size tied to the quantization group. WUSH reports W4A4 gains over the strongest Hadamard baselines in MXFP4, e.g., +2.8 average points with RTN and +0.7 with GPTQ on Llama-3.1-8B-Instruct. The submission does not cite WUSH, and it cites but does not compare against SpinQuant or ButterflyQuant. Section 4.4 compares only fixed H_k blocks with a full fixed Hadamard. It therefore cannot show that larger exact rotations beat better small-block transforms at equal cost, and such transforms (non-orthogonal, data-dependent, with blocks below N = 256) fall outside the kernel's supported structure and range.
This weakness concerns the significance of the application, not the correctness of the kernel. It matters because the paper presents full-dimension rotations as the main use case for its kernel (Sec. 1, 4.4, 5). If FP4 microscaling or adaptive block transforms are where practice is heading, the kernel's significance narrows to integer per-token schemes, QuIP-style catalog checkpoints and randomized sketching.
Questions for authors
- Llama-3.2-1B (W1). Please reconcile the 1.47× reduction in transform time with the operator sweep's 1.0–1.05× at N ≤ 8192. Could you report a matched operator-level measurement at the shapes and dtype vLLM actually uses (e.g., 2048 × 8192, BF16) for HyperHadamard, standalone HadaCore and vLLM's embedded
C::hadacore_transform? - Small-batch regime. How does throughput compare with HadaCore below 2^23 elements at each N, and in decode at batch sizes 1–64? Would dispatching to HadaCore below a threshold remove the 0.978× decode regression?
- FP32 bit-exactness. Lemma 1 holds in exact arithmetic, but reordering floating-point butterflies changes rounding in general ((a+b)+(c+d) need not equal (a+c)+(b+d)). Is the FP32 kernel bit-exact with Dao-AILab by construction, i.e., with the same addition order and normalization point at every length, or only on the inputs tested? The byte-identical selection result in Sec. 4.3 depends on the answer.
- Large-N schedule and artifacts. For each N ≥ 4096, what are A1, A2 and the phase-2 fragment layout, and what register footprint, shared-memory footprint and occupancy result? Can the code be shared with reviewers anonymously during the discussion period?
Suggestions:
- Evaluate MXFP4/NVFP4 activations (and weights), comparing group-aligned H16/H32 rotations, larger blocks and full-dimension rotations, to test whether the Section 4.4 trend carries over to microscaling formats.
- Discuss WUSH and, if feasible, compare a full-dimension Hadamard against WUSH-style data-aware block transforms and learned rotations (SpinQuant) at matched cost, ideally in an end-to-end W4A4 setting that uses the kernel.
- Explain why H14336 alone raises perplexity while H4096 alone lowers it (Appendix H), and add at least one more model family to Section 4.4.
- Position the work against classical FFT/WHT formulations that reorder or self-sort stages and against layout systems (CuTe, linear layouts), separating what is new (typed tracking of axis state through tensor-core layouts) from what is known (Kronecker factors commute).
7 comments
Response to Reviewer oTr2 (2/5)
W1b. The Llama-3.2-1B result is consistent with the operator sweep.
We thank the reviewer for checking this carefully and we note that the second explanation the reviewer proposes is the correct one. The operator sweep fixes 2^30 elements per point, whereas at a 2048-token budget vLLM calls H8192 on 2048 x 8192 = 2^24 elements and H512 on 2^20. At these sizes HadaCore does not saturate bandwidth, and HyperHadamard's lower instruction count shows up as time even at small N. Following the reviewer's request, we measured the exact shapes against vLLM's embedded _C::hadacore_transform (Table R3.2).
Table R3.2. Speedup over vLLM's embedded _C::hadacore_transform at vLLM call shapes (H100, ten fresh processes, eager timing as in Figure 1).
| Shape (rows x N) | BF16 | FP16 |
|---|---|---|
| 2048 x 8192 | 1.57x [1.568, 1.577] | 1.44x [1.439, 1.445] |
| 2048 x 2048 | 1.30x [1.278, 1.323] | 1.29x [1.267, 1.312] |
| 2048 x 512 | 1.30x [1.288, 1.314] | 1.29x [1.273, 1.304] |
Against standalone HadaCore, the 2048 x 8192 call, which accounts for 73% of the transform-time saving in vLLM (Appendix F), is likewise 1.55x faster in BF16. At N = 8192 the advantage peaks at 2^23-2^24 elements and falls to 1.06-1.08x at 2^25-2^27, meeting the 1.05x of Figure 1 at 2^30. These per-call speedups match the in-vLLM trace (1.53x for H8192, 1.37x for H2048, 1.35x for H512) and account for the measured 1.47x reduction in transform time, so the operator results do predict deployment once they are measured at deployment shapes. We will replace the sentence in Section 4.2 with this shape-matched comparison.
W2. Full-dimension rotations matter where the activation scale is coarse, and the kernel preserves their quality
W2a. The kernel in the loop, a second model family, and W4A16 (also Suggestion 3).
Following the reviewer's request, we replaced the FP64 hooks with HyperHadamard in BF16 (all 448 rotation modules, same calibrated model, same process).
Table R3.3. Mistral-7B WikiText-2 perplexity with HyperHadamard in place of the FP64 hooks.
| Setting | FP64 hooks | HyperHadamard BF16 | H128 blocks |
|---|---|---|---|
| W3A4, full rotation | 6.5405 | 6.5339 | 6.9856 |
| W2A4, full rotation | 34.50 | 36.83 | 57.15 |
The kernel reproduces the FP64 result at W3A4 and keeps most of the full-rotation gain at W2A4, so the quality benefit holds with the kernel itself. On Llama-3.1-8B-Instruct, the same pipeline reproduces the Mistral-7B pattern (Table R3.4): with 4-bit per-token activations, full rotations improve perplexity by 12.4%, and with BF16 or INT8 activations rotation size matters little, as expected since rotations exist to suppress activation outliers.
Table R3.4. Llama-3.1-8B-Instruct WikiText-2 perplexity, INT3 weights.
| Rotation | BF16 act. | INT8 act. | INT4 act. |
|---|---|---|---|
| H128 blocks | 8.826 | 8.838 | 12.082 |
| Full dimension | 8.905 | 8.908 | 10.582 |
The vLLM experiments use W4A16 because that is what the released QuIP checkpoints are: the checkpoint fixes its full-dimension rotations, and any serving kernel must reproduce them exactly, whether or not a smaller rotation would have served at W4A16. End-to-end W3A4 latency would additionally require a low-bit activation GEMM, which vLLM does not provide for this recipe with any rotation kernel.
W2b. FP4: microscaling formats and block rotations are co-designed, full-dimension rotations are complementary, and HyperHadamard removes the cost argument for small blocks (also Suggestion 1).
We agree with the reviewer's mechanism. MXFP4 and NVFP4 were designed together with block rotations matched to their scaling groups: as the reviewer notes, when one scale covers 16 or 32 values, an outlier inflates only its own group's scale, and spreading it across the whole dimension can raise the scales of groups that were already accurate. Block rotations are also chosen for cost rather than accuracy. Tseng et al. [10] apply their random Hadamard transform for MXFP4 training blockwise, as a dense multiply whose runtime grows with the block size; in their ablation, larger blocks lower validation perplexity (12.02 at block size 32 to 11.98 at 256, attributed to lower stochastic-rounding variance), yet they use 64 to keep the transform memory bound; applied over the full dimension, the same multiply costs O(N^2) per row. That is the assumption our work removes (Section 1): HyperHadamard applies an exact full-dimension rotation in O(N log N) within 1.7% of the memory roofline, so a full rotation moves the same bytes as a block rotation, rotation size becomes a choice made for accuracy alone, and transforms previously ruled out by cost, such as data-aware or learned transforms on a full-dimension Hadamard backbone, become practical to explore for new and aggressive quantization methods.
Response to Reviewer oTr2 (3/5)
Following the reviewer's suggestion, we evaluated llm-compressor's W4A4 MXFP4 and NVFP4 presets (round-to-nearest) with no rotation, group-aligned H16/H32, H128 and full rotations (Table R3.5).
Table R3.5. Mistral-7B WikiText-2 perplexity under W4A4 FP4 microscaling.
| Format | None | H16 | H32 | H128 | Full |
|---|---|---|---|---|---|
| MXFP4 (group 32, power-of-two scales) | 5.979 | 5.877 | 5.962 | 5.948 | 5.831 |
| NVFP4 (group 16, FP8 scales) | 5.833 | 5.866 | 5.840 | 5.825 | 6.058 |
The results follow scale granularity. Under MXFP4, full rotation gives the best perplexity, 2.0% better than H128 and 2.5% better than no rotation. Full-dimension rotation is therefore complementary to block rotation even within a microscaling format, rather than harmful to it. Under NVFP4, the finest-grained format, no rotation size is needed, consistent with the co-design. We will add these results to Section 4.4.
W2c. Adaptive transforms build on the Hadamard transform rather than replace it (also Suggestion 2).
The adaptive transforms the reviewer lists are of two kinds. Learned rotations: SpinQuant [2] optimizes its residual-stream rotations offline, but its online rotations R3 and R4, including R4 at the input of the down projection over the full MLP width, are fixed Hadamard transforms, exactly the operation HyperHadamard accelerates; ButterflyQuant [5] learns butterfly-structured orthogonal transforms in place of the Hadamard. Data-aware transforms: WUSH [4] combines a fixed Hadamard backbone with a second-moment-based component, blockwise with blocks tied to the quantization group, the co-designed microscaling regime of W2b. Both kinds keep a Hadamard or butterfly backbone and stay at small blocks because of cost, so a fast exact full-dimension backbone is an enabler for them rather than a competitor. We will cite WUSH and discuss these methods in Section 4.4. Our other applications, exact catalog rotations in released QuIP checkpoints and randomized sketching, do not depend on this question.
W2d. The decomposition of Appendix H across random rotation draws (also Suggestion 3).
Following the reviewer's suggestion, we repeated the decomposition with five additional random rotation draws. Relative to H1024 blocks, H4096 alone and both full rotations change W3A4 perplexity by -0.9% to +0.5% and -0.8% to +0.2% across draws, while H14336 alone raises it on every draw (+0.6% to +3.1%). At the H1024 block size these effects are therefore small and depend on the draw, and we will revise Appendix H to report the seed range rather than a single draw. The gain the paper relies on is relative to small blocks, where full rotations improve W3A4 perplexity by 6.4% on Mistral-7B and 12.4% on Llama-3.1-8B, well outside this range.
Responses to Questions
Q1. Reconcile the Llama-3.2-1B result with a matched operator-level measurement. Please see W1b and Table R3.2: 1.57x at 2048 x 8192 BF16 against vLLM's embedded HadaCore and 1.55x against standalone HadaCore.
Q2. Small batch: throughput below 2^23 elements and in decode at batch sizes 1-64; would dispatching remove the regression? Table R3.2 covers the 2^20-2^24-element calls vLLM issues. In decode we found two causes and removed both without dispatching to a second kernel. First, H512 is processed two rows per block, and a single-row call was padded on the host with an extra allocation and copy; we now handle the partial last block inside the kernel. Second, for large transforms our kernel assigns one warp per row, which maximizes bandwidth when a call has thousands of rows, whereas a decode call has 1-64 rows, so its time is the latency of a single row, and HadaCore spreads each row over 256-512 threads. Splitting each row across four warps, the launch configuration of W1a, changes only the launch: the stage schedule, the register layouts and the output (bit-identical) are unchanged.
Table R3.6. Llama-3.2-1B decode throughput relative to vLLM's embedded HadaCore (H100, input 32, output 128, six paired runs with alternating order).
| Batch | 1 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|
| Submitted kernel | 0.917x | 0.974x | 0.975x | 0.979x | 0.980x |
| + partial last block in kernel | 0.969x | 0.980x | 0.970x | 0.972x | 0.977x |
| + four warps per row | 1.019x | 1.014x | 1.018x | 1.016x | 1.018x |
HyperHadamard is now faster than HadaCore in decode at every batch size from 1 to 64, with every 95% interval above 1.
Response to Reviewer oTr2 (1/5)
We thank the reviewer for the detailed and technically precise review. We appreciate the recognition of our "credible, mechanism-level evidence for the speed claim", that "a kernel paper stands or falls on whether its gain is real and explained, and here it is", and of the "practical coverage that existing fast kernels lack" for catalog-Hadamard checkpoints and the bit-exact projector. We address the two weaknesses first, then the four questions and the remaining suggestions. Additionally, we have added new experiments: (i) shape-matched operator measurements at vLLM call sizes, (ii) a decode fix that makes HyperHadamard faster than HadaCore at every batch size, (iii) an NVIDIA B200, (iv) adversarial FP32 bit-exactness tests, (v) a second model family, (vi) NVFP4 and MXFP4 rotation sweeps, (vii) the quantization study with the kernel in the loop, and (viii) the rotation decomposition over additional random seeds.
W1. The supported regime is where full-dimension transforms are expensive, and the Llama-3.2-1B result agrees with the operator sweep at vLLM's tensor sizes
W1a. Parity at small N on H100 is a memory-roofline effect, and on newer hardware the advantage holds at every length.
On H100, parity at N <= 4096 does not reflect a limit of the method: at these lengths both kernels reach the H100's 3.0 TB/s memory roofline, so neither can be faster. The instruction savings appear as time only once HadaCore falls below the roofline, which on H100 happens at N >= 8192. Newer GPUs raise bandwidth faster than the per-SM work available to consume it: the B200 has about twice the H100's bandwidth with a similar number of SMs, which the submission's H100 and A100 results could not show. We have now run the same paired protocol on an NVIDIA B200, with about 6.8 TB/s: there HadaCore no longer saturates memory at any length, reaching at most 5.6 TB/s, while HyperHadamard reaches 6.8-7.0 TB/s for N = 512-8192. HyperHadamard is therefore faster at every length, including the small transforms where H100 shows parity: 1.24-1.75x for N = 256-4096 and 1.97-2.35x for N >= 8192 (Table R3.1). With an optimized launch configuration for Blackwell (four warps per row, with each warp issuing all of its asynchronous loads before waiting; algorithm and output unchanged, H100 performance on large transforms unchanged within 0.5%), it reaches 6.7-7.0 TB/s at every length and up to 2.97x. The same configuration also removes the decode gap on H100 (Q2).
Table R3.1. Speedup over HadaCore on an NVIDIA B200 (five processes, 2^30 elements).
At the call shapes inside vLLM, HyperHadamard is also faster at N = 512 and 2048 on H100 (Table R3.2), and in decode it is now faster at every batch size (Q2). We agree that the kernel targets N >= 256 and will state this in Section 4.1 and the conclusion: rotations of 16-128 elements fit in a few registers and are served by existing kernels, whereas the expensive transforms are the full-dimension online rotations of QuaRot, SpinQuant (R4), QuIP# and llm-compressor's QuIP transforms, of order 4096-28672 [1-3]. Within that range, the regime is not narrow: it covers every length on B200, every call shape vLLM issues in prefill, and decode.
On the headline: we agree that the 14.70x is measured against a dense path. It is the path stock vLLM executes for these released checkpoints, since its only fast Hadamard kernel, HadaCore, supports powers of two only, and it is the reference that llm-compressor's own documentation uses for its transforms, reporting HadaCore against dense GEMM latency. No prior fast kernel computes these checkpoints' catalog factorizations exactly, so we report the operation-matched 1.915x alongside it for exactly the reason the reviewer gives. Against HadaCore + cuBLAS, on a Llama-3.1-8B checkpoint whose 14336-wide rotation uses QuaRot's factorization H28 x H512, HyperHadamard is 1.123x faster end to end in vLLM. In online serving with continuous batching, it raises the released checkpoint's throughput over stock vLLM 13.1x (please see our response to Reviewer HykP, W2b and W4b). Both other reviewers note the practical value of this path: Reviewer vdsM finds that the 1.915x "demonstrates value beyond replacing the stock dense fallback," and Reviewer HykP that the catalog factors "fill a real gap."