acceptodds
Under review as a conference paper at ICLR 2027

HyperHadamard: Layout-Aware Hadamard Transforms on Tensor Cores

Abstract

Low-bit quantization is an appealing way to cut the memory and bandwidth cost of large language models. But activation outliers make few-bit quantization unstable, and rotation-based methods address them by mixing channels before quantization. The Hadamard transform is the rotation of choice because it admits a fast algorithm. Tensor-core FWHT kernels made these rotations cheap enough to deploy, but at large rotation dimensions existing kernels sustain only 42–77% of achievable memory bandwidth, even though the transform is memory-bound. We identify intermediate-layout materialization as a source of excess instructions and dependency stalls. We propose HyperHadamard, which treats butterfly stages as rotations along the binary axes of a hypercube and assigns those axes to tensor-core register layouts at compile time, so intermediate transposes become metadata rather than data movement. At rotation dimensions large enough for these overheads to dominate, HyperHadamard is up to 1.59 faster than HadaCore, the tensor-core FWHT used in vLLM, and stays within 1.7% of achievable memory bandwidth. Together with a gather remap and a fused sign load, a true-FP32 instantiation of the same schedule speeds up a billion-dimensional randomized gradient projection used for training-data selection by 1.75 while remaining bit-exact. The same design supports non-power-of-two transforms built from available catalog Hadamard matrices, whereas prior fast kernels expose only power-of-two and selected fixed-factor sizes. Integrated into vLLM, a Llama-3.1-8B QuIP W4A16 model produced by llm-compressor runs 14.7 faster end to end than vLLM's dense GEMM path and 1.9 faster than the same factorization assembled from Dao-AILab kernels and cuBLAS, with no measurable change in perplexity or zero-shot accuracy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Hadamard Transform, GPU kernel optimization, tensor cores, quantized LLM inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oTr2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents HyperHadamard, a CUDA kernel for exact normalized Hadamard transforms, aimed at online rotations in quantized LLM inference and at randomized Hadamard sketching. Its central observation is that the d butterfly stages of a length-2^d FWHT are rotations along independent hypercube axes and therefore commute (Lemma 1). Conventional tensor-core kernels such as HadaCore follow the stage order of the recursive algorithm, so they must shuffle or transpose each MMA output back into the layout the next stage expects. HyperHadamard instead keeps a typed register Fragment holding a compile-time permutation P (which logical axis occupies each lane or register bit) and an axis state S (pending or complete). Each mma.m16n8k16 (one H16, i.e., four axes) is applied to whichever pending axes the previous instruction left in the operand slots. Layout changes become metadata, and only thread-local register swaps are emitted: Appendix A computes H256 with four MMAs and no shuffles or shared-memory transposes. For N ≥ 4096 the axes are split into two register phases separated by one shared-memory exchange. Its strided view places the phase-2 axes directly into MMA slots and removes HadaCore's final output realignment. A Kronecker factorization H_q ⊗ H_{2^k} (H28 as a padded tensor-core GEMM) covers non-power-of-two orders such as 14336, and a true-FP32 CUDA-core variant reuses the same schedule.

Validation:

  • Operator level (H100, 2^30 elements): FP16/BF16 throughput stays within 1.7% of an empirical 3.03 TB/s roofline for N = 256–32768. Against HadaCore it is at parity for N ≤ 4096 (0.988–1.003×), 1.05× and 1.10× at N = 8192 and 16384, and 1.52–1.59× at N = 32768. The mechanism is supported by Nsight profiles (3.47–3.60× fewer warp instructions; memory-I/O-queue stalls fall from about 5.7% to 0.1%), a forced-materialization ablation (+1.75% / +5.89% latency), an arithmetic-free speed-of-light kernel (only 1.055–1.058× faster) and an A100 replication. The FP32 kernel is 1.461× faster than Dao-AILab at N = 32768, which the paper attributes to lower register use and higher occupancy.
  • vLLM: a Llama-3.1-8B QuIP W4A16 checkpoint uses catalog Hadamards of orders 14336, 4096 and 1024 (built from H224 and H256). The kernel matches them through a q×q basis correction that is exact in exact arithmetic. End to end, it runs 14.70× faster than vLLM's dense FP64 path and 1.915× faster than a Dao-AILab + cuBLAS baseline with the same factorization, with unchanged perplexity and ARC accuracy. For a power-of-two Llama-3.2-1B checkpoint, replacing HadaCore gives 1.022–1.052× prefill throughput and 0.978× in batch-8 decode.
  • Influence Distillation: with gather remapping and fused Rademacher signs, the 2^30-element FP32 projector becomes 1.747× faster and remains bit-exact.
  • Quantization study (Mistral-7B, GPTQ INT3/INT2 weights, dynamic per-token INT4 activations, rotations applied as FP64 hooks): relative to H128 blocks, full-dimension rotations lower WikiText-2 perplexity by 6.37% (INT3) and 39.6% (INT2). With BF16 or INT8 activations they change perplexity by +0.2% to +0.5%.
Critical strengths

S1. Credible, mechanism-level evidence for the speed claim. The paper goes beyond reporting throughput. HyperHadamard stays within 1.7% of the achievable roofline at every length. The gap to HadaCore is traced to instruction count and memory-I/O-queue stalls. Fewer instructions translate into speed only once HadaCore falls below the roofline (N ≥ 8192). A within-kernel ablation that forces materialization isolates the layout effect, an arithmetic-free kernel bounds the remaining headroom, and the result replicates on A100. This matters most to my assessment: a kernel paper stands or falls on whether its gain is real and explained, and here it is, with little headroom left.

S2. Practical coverage that existing fast kernels lack. The Kronecker factorization with an exact q×q basis correction gives catalog-Hadamard QuIP checkpoints (orders 14336, 4096 and 1024) a fast path in vLLM. End to end, this is 1.915× faster than an operation-matched Dao-AILab + cuBLAS baseline, with perplexity changing by 0.0026% and identical ARC accuracy. The true-FP32 variant, combined with gather remapping and fused signs, makes a released billion-coordinate projector 1.747× faster with bit-exact output. This shows the design carries over to real workloads, not only to a microbenchmark.

Critical weaknesses

W1. The benefit is confined to a narrow regime, and one end-to-end result does not agree with the operator sweep.

  • Regime. FP16/BF16 gains require N ≥ 8192 and many rows. The kernel is at parity for N ≤ 4096. At N = 32768 it needs at least 2^23 elements to overtake HadaCore, and in batch-8 decode it reaches only 0.978× of HadaCore. The supported range starts at N = 256, which excludes the 16–128-element blocks used for head-dimension rotations and for group-aligned MXFP4/NVFP4 rotations, so many current deployments would see no benefit. The 14.70× headline mostly measures dense FP64 against any fast transform; the operation-matched comparison is 1.915×.
  • Inconsistency. For Llama-3.2-1B, transform time falls from 126.2 ms to 85.6 ms (1.47×), and H8192 accounts for 73.3% of the saving (Appendix F). The operator sweep, however, reports 1.05× at N = 8192 and parity at N ≤ 4096. Even if all transform time were spent at N = 8192, a 1.05× kernel speedup would save at most about 5%, not 32%. Possible causes are that vLLM's embedded HadaCore differs from the standalone version, or that the much smaller tensors (about 2^24 elements at a 2048-token budget, versus 2^30 in the sweep) behave differently. Either way the gap needs an explanation, because it determines how well the operator results predict deployment.

W2. The application-level case for full-dimension rotations is thin and leaves out FP4 and adaptive transforms. Section 4.4 is the paper's argument that fast full-dimension rotations are worth having. It covers one model (Mistral-7B) and one recipe (GPTQ INT3/INT2 weights, dynamic per-token INT4 activations), with rotations run as FP64 hooks. The gain depends on activation precision rather than weight precision: with BF16 or INT8 activations, full-dimension rotations slightly hurt (+0.2% to +0.5%) for both INT3 and INT2 weights. Appendix H shows that H14336 alone hurts and helps only together with H4096, which is not explained. None of these quality-sensitive settings is run end to end with the kernel. The vLLM experiments use W4A16, where Table 4 suggests rotation size makes no difference.

  • No FP4 evaluation. MXFP4 and NVFP4 are arguably the main 4-bit activation targets on current hardware, and the paper itself notes (Sec. 2, L119–121) that for microscaling FP4 formats rotations are usually confined to blocks matching the scaling groups. Under per-16- or per-32-element scales, an outlier inflates only its own group's scale. Full-dimension rotation helps per-token INT4 by diluting an outlier that would otherwise set a single row-wide scale, and that mechanism is largely absent under microscaling. Whether full-dimension rotations help, are neutral or hurt there is untested.
  • No comparison with data-aware or learned transforms. WUSH (arXiv:2512.00956, ICML 2026) derives closed-form, data-aware blockwise transforms for joint weight–activation quantization. Each combines a Hadamard backbone with a second-moment-based component, is non-orthogonal, and uses a block size tied to the quantization group. WUSH reports W4A4 gains over the strongest Hadamard baselines in MXFP4, e.g., +2.8 average points with RTN and +0.7 with GPTQ on Llama-3.1-8B-Instruct. The submission does not cite WUSH, and it cites but does not compare against SpinQuant or ButterflyQuant. Section 4.4 compares only fixed H_k blocks with a full fixed Hadamard. It therefore cannot show that larger exact rotations beat better small-block transforms at equal cost, and such transforms (non-orthogonal, data-dependent, with blocks below N = 256) fall outside the kernel's supported structure and range.

This weakness concerns the significance of the application, not the correctness of the kernel. It matters because the paper presents full-dimension rotations as the main use case for its kernel (Sec. 1, 4.4, 5). If FP4 microscaling or adaptive block transforms are where practice is heading, the kernel's significance narrows to integer per-token schemes, QuIP-style catalog checkpoints and randomized sketching.

Questions for authors
  1. Llama-3.2-1B (W1). Please reconcile the 1.47× reduction in transform time with the operator sweep's 1.0–1.05× at N ≤ 8192. Could you report a matched operator-level measurement at the shapes and dtype vLLM actually uses (e.g., 2048 × 8192, BF16) for HyperHadamard, standalone HadaCore and vLLM's embedded C::hadacore_transform?
  2. Small-batch regime. How does throughput compare with HadaCore below 2^23 elements at each N, and in decode at batch sizes 1–64? Would dispatching to HadaCore below a threshold remove the 0.978× decode regression?
  3. FP32 bit-exactness. Lemma 1 holds in exact arithmetic, but reordering floating-point butterflies changes rounding in general ((a+b)+(c+d) need not equal (a+c)+(b+d)). Is the FP32 kernel bit-exact with Dao-AILab by construction, i.e., with the same addition order and normalization point at every length, or only on the inputs tested? The byte-identical selection result in Sec. 4.3 depends on the answer.
  4. Large-N schedule and artifacts. For each N ≥ 4096, what are A1, A2 and the phase-2 fragment layout, and what register footprint, shared-memory footprint and occupancy result? Can the code be shared with reviewers anonymously during the discussion period?

Suggestions:

  1. Evaluate MXFP4/NVFP4 activations (and weights), comparing group-aligned H16/H32 rotations, larger blocks and full-dimension rotations, to test whether the Section 4.4 trend carries over to microscaling formats.
  2. Discuss WUSH and, if feasible, compare a full-dimension Hadamard against WUSH-style data-aware block transforms and learned rotations (SpinQuant) at matched cost, ideally in an end-to-end W4A4 setting that uses the kernel.
  3. Explain why H14336 alone raises perplexity while H4096 alone lowers it (Appendix H), and add at least one more model family to Section 4.4.
  4. Position the work against classical FFT/WHT formulations that reorder or self-sort stages and against layout systems (CuTe, linear layouts), separating what is new (typed tracking of axis state through tensor-core layouts) from what is known (Kronecker factors commute).
7 comments
AuthorsOpenReview ↗

Response to Reviewer oTr2 (1/5)

We thank the reviewer for the detailed and technically precise review. We appreciate the recognition of our "credible, mechanism-level evidence for the speed claim", that "a kernel paper stands or falls on whether its gain is real and explained, and here it is", and of the "practical coverage that existing fast kernels lack" for catalog-Hadamard checkpoints and the bit-exact projector. We address the two weaknesses first, then the four questions and the remaining suggestions. Additionally, we have added new experiments: (i) shape-matched operator measurements at vLLM call sizes, (ii) a decode fix that makes HyperHadamard faster than HadaCore at every batch size, (iii) an NVIDIA B200, (iv) adversarial FP32 bit-exactness tests, (v) a second model family, (vi) NVFP4 and MXFP4 rotation sweeps, (vii) the quantization study with the kernel in the loop, and (viii) the rotation decomposition over additional random seeds.

W1. The supported regime is where full-dimension transforms are expensive, and the Llama-3.2-1B result agrees with the operator sweep at vLLM's tensor sizes

W1a. Parity at small N on H100 is a memory-roofline effect, and on newer hardware the advantage holds at every length.

On H100, parity at N <= 4096 does not reflect a limit of the method: at these lengths both kernels reach the H100's 3.0 TB/s memory roofline, so neither can be faster. The instruction savings appear as time only once HadaCore falls below the roofline, which on H100 happens at N >= 8192. Newer GPUs raise bandwidth faster than the per-SM work available to consume it: the B200 has about twice the H100's bandwidth with a similar number of SMs, which the submission's H100 and A100 results could not show. We have now run the same paired protocol on an NVIDIA B200, with about 6.8 TB/s: there HadaCore no longer saturates memory at any length, reaching at most 5.6 TB/s, while HyperHadamard reaches 6.8-7.0 TB/s for N = 512-8192. HyperHadamard is therefore faster at every length, including the small transforms where H100 shows parity: 1.24-1.75x for N = 256-4096 and 1.97-2.35x for N >= 8192 (Table R3.1). With an optimized launch configuration for Blackwell (four warps per row, with each warp issuing all of its asynchronous loads before waiting; algorithm and output unchanged, H100 performance on large transforms unchanged within 0.5%), it reaches 6.7-7.0 TB/s at every length and up to 2.97x. The same configuration also removes the decode gap on H100 (Q2).

Table R3.1. Speedup over HadaCore on an NVIDIA B200 (five processes, 2^30 elements).

N25651210242048409681921638432768
BF161.341.751.401.461.572.312.171.97
FP161.341.741.241.331.622.352.032.13
BF16, optimized launch1.361.771.391.471.712.462.432.97
FP16, optimized launch1.361.771.221.341.642.352.242.91

At the call shapes inside vLLM, HyperHadamard is also faster at N = 512 and 2048 on H100 (Table R3.2), and in decode it is now faster at every batch size (Q2). We agree that the kernel targets N >= 256 and will state this in Section 4.1 and the conclusion: rotations of 16-128 elements fit in a few registers and are served by existing kernels, whereas the expensive transforms are the full-dimension online rotations of QuaRot, SpinQuant (R4), QuIP# and llm-compressor's QuIP transforms, of order 4096-28672 [1-3]. Within that range, the regime is not narrow: it covers every length on B200, every call shape vLLM issues in prefill, and decode.

On the headline: we agree that the 14.70x is measured against a dense path. It is the path stock vLLM executes for these released checkpoints, since its only fast Hadamard kernel, HadaCore, supports powers of two only, and it is the reference that llm-compressor's own documentation uses for its transforms, reporting HadaCore against dense GEMM latency. No prior fast kernel computes these checkpoints' catalog factorizations exactly, so we report the operation-matched 1.915x alongside it for exactly the reason the reviewer gives. Against HadaCore + cuBLAS, on a Llama-3.1-8B checkpoint whose 14336-wide rotation uses QuaRot's factorization H28 x H512, HyperHadamard is 1.123x faster end to end in vLLM. In online serving with continuous batching, it raises the released checkpoint's throughput over stock vLLM 13.1x (please see our response to Reviewer HykP, W2b and W4b). Both other reviewers note the practical value of this path: Reviewer vdsM finds that the 1.915x "demonstrates value beyond replacing the stock dense fallback," and Reviewer HykP that the catalog factors "fill a real gap."

AuthorsOpenReview ↗

Response to Reviewer oTr2 (2/5)

W1b. The Llama-3.2-1B result is consistent with the operator sweep.

We thank the reviewer for checking this carefully and we note that the second explanation the reviewer proposes is the correct one. The operator sweep fixes 2^30 elements per point, whereas at a 2048-token budget vLLM calls H8192 on 2048 x 8192 = 2^24 elements and H512 on 2^20. At these sizes HadaCore does not saturate bandwidth, and HyperHadamard's lower instruction count shows up as time even at small N. Following the reviewer's request, we measured the exact shapes against vLLM's embedded _C::hadacore_transform (Table R3.2).

Table R3.2. Speedup over vLLM's embedded _C::hadacore_transform at vLLM call shapes (H100, ten fresh processes, eager timing as in Figure 1).

Shape (rows x N)BF16FP16
2048 x 81921.57x [1.568, 1.577]1.44x [1.439, 1.445]
2048 x 20481.30x [1.278, 1.323]1.29x [1.267, 1.312]
2048 x 5121.30x [1.288, 1.314]1.29x [1.273, 1.304]

Against standalone HadaCore, the 2048 x 8192 call, which accounts for 73% of the transform-time saving in vLLM (Appendix F), is likewise 1.55x faster in BF16. At N = 8192 the advantage peaks at 2^23-2^24 elements and falls to 1.06-1.08x at 2^25-2^27, meeting the 1.05x of Figure 1 at 2^30. These per-call speedups match the in-vLLM trace (1.53x for H8192, 1.37x for H2048, 1.35x for H512) and account for the measured 1.47x reduction in transform time, so the operator results do predict deployment once they are measured at deployment shapes. We will replace the sentence in Section 4.2 with this shape-matched comparison.

W2. Full-dimension rotations matter where the activation scale is coarse, and the kernel preserves their quality

W2a. The kernel in the loop, a second model family, and W4A16 (also Suggestion 3).

Following the reviewer's request, we replaced the FP64 hooks with HyperHadamard in BF16 (all 448 rotation modules, same calibrated model, same process).

Table R3.3. Mistral-7B WikiText-2 perplexity with HyperHadamard in place of the FP64 hooks.

SettingFP64 hooksHyperHadamard BF16H128 blocks
W3A4, full rotation6.54056.53396.9856
W2A4, full rotation34.5036.8357.15

The kernel reproduces the FP64 result at W3A4 and keeps most of the full-rotation gain at W2A4, so the quality benefit holds with the kernel itself. On Llama-3.1-8B-Instruct, the same pipeline reproduces the Mistral-7B pattern (Table R3.4): with 4-bit per-token activations, full rotations improve perplexity by 12.4%, and with BF16 or INT8 activations rotation size matters little, as expected since rotations exist to suppress activation outliers.

Table R3.4. Llama-3.1-8B-Instruct WikiText-2 perplexity, INT3 weights.

RotationBF16 act.INT8 act.INT4 act.
H128 blocks8.8268.83812.082
Full dimension8.9058.90810.582

The vLLM experiments use W4A16 because that is what the released QuIP checkpoints are: the checkpoint fixes its full-dimension rotations, and any serving kernel must reproduce them exactly, whether or not a smaller rotation would have served at W4A16. End-to-end W3A4 latency would additionally require a low-bit activation GEMM, which vLLM does not provide for this recipe with any rotation kernel.

W2b. FP4: microscaling formats and block rotations are co-designed, full-dimension rotations are complementary, and HyperHadamard removes the cost argument for small blocks (also Suggestion 1).

We agree with the reviewer's mechanism. MXFP4 and NVFP4 were designed together with block rotations matched to their scaling groups: as the reviewer notes, when one scale covers 16 or 32 values, an outlier inflates only its own group's scale, and spreading it across the whole dimension can raise the scales of groups that were already accurate. Block rotations are also chosen for cost rather than accuracy. Tseng et al. [10] apply their random Hadamard transform for MXFP4 training blockwise, as a dense multiply whose runtime grows with the block size; in their ablation, larger blocks lower validation perplexity (12.02 at block size 32 to 11.98 at 256, attributed to lower stochastic-rounding variance), yet they use 64 to keep the transform memory bound; applied over the full dimension, the same multiply costs O(N^2) per row. That is the assumption our work removes (Section 1): HyperHadamard applies an exact full-dimension rotation in O(N log N) within 1.7% of the memory roofline, so a full rotation moves the same bytes as a block rotation, rotation size becomes a choice made for accuracy alone, and transforms previously ruled out by cost, such as data-aware or learned transforms on a full-dimension Hadamard backbone, become practical to explore for new and aggressive quantization methods.

AuthorsOpenReview ↗

Response to Reviewer oTr2 (3/5)

Following the reviewer's suggestion, we evaluated llm-compressor's W4A4 MXFP4 and NVFP4 presets (round-to-nearest) with no rotation, group-aligned H16/H32, H128 and full rotations (Table R3.5).

Table R3.5. Mistral-7B WikiText-2 perplexity under W4A4 FP4 microscaling.

FormatNoneH16H32H128Full
MXFP4 (group 32, power-of-two scales)5.9795.8775.9625.9485.831
NVFP4 (group 16, FP8 scales)5.8335.8665.8405.8256.058

The results follow scale granularity. Under MXFP4, full rotation gives the best perplexity, 2.0% better than H128 and 2.5% better than no rotation. Full-dimension rotation is therefore complementary to block rotation even within a microscaling format, rather than harmful to it. Under NVFP4, the finest-grained format, no rotation size is needed, consistent with the co-design. We will add these results to Section 4.4.

W2c. Adaptive transforms build on the Hadamard transform rather than replace it (also Suggestion 2).

The adaptive transforms the reviewer lists are of two kinds. Learned rotations: SpinQuant [2] optimizes its residual-stream rotations offline, but its online rotations R3 and R4, including R4 at the input of the down projection over the full MLP width, are fixed Hadamard transforms, exactly the operation HyperHadamard accelerates; ButterflyQuant [5] learns butterfly-structured orthogonal transforms in place of the Hadamard. Data-aware transforms: WUSH [4] combines a fixed Hadamard backbone with a second-moment-based component, blockwise with blocks tied to the quantization group, the co-designed microscaling regime of W2b. Both kinds keep a Hadamard or butterfly backbone and stay at small blocks because of cost, so a fast exact full-dimension backbone is an enabler for them rather than a competitor. We will cite WUSH and discuss these methods in Section 4.4. Our other applications, exact catalog rotations in released QuIP checkpoints and randomized sketching, do not depend on this question.

W2d. The decomposition of Appendix H across random rotation draws (also Suggestion 3).

Following the reviewer's suggestion, we repeated the decomposition with five additional random rotation draws. Relative to H1024 blocks, H4096 alone and both full rotations change W3A4 perplexity by -0.9% to +0.5% and -0.8% to +0.2% across draws, while H14336 alone raises it on every draw (+0.6% to +3.1%). At the H1024 block size these effects are therefore small and depend on the draw, and we will revise Appendix H to report the seed range rather than a single draw. The gain the paper relies on is relative to small blocks, where full rotations improve W3A4 perplexity by 6.4% on Mistral-7B and 12.4% on Llama-3.1-8B, well outside this range.

Responses to Questions

Q1. Reconcile the Llama-3.2-1B result with a matched operator-level measurement. Please see W1b and Table R3.2: 1.57x at 2048 x 8192 BF16 against vLLM's embedded HadaCore and 1.55x against standalone HadaCore.

Q2. Small batch: throughput below 2^23 elements and in decode at batch sizes 1-64; would dispatching remove the regression? Table R3.2 covers the 2^20-2^24-element calls vLLM issues. In decode we found two causes and removed both without dispatching to a second kernel. First, H512 is processed two rows per block, and a single-row call was padded on the host with an extra allocation and copy; we now handle the partial last block inside the kernel. Second, for large transforms our kernel assigns one warp per row, which maximizes bandwidth when a call has thousands of rows, whereas a decode call has 1-64 rows, so its time is the latency of a single row, and HadaCore spreads each row over 256-512 threads. Splitting each row across four warps, the launch configuration of W1a, changes only the launch: the stage schedule, the register layouts and the output (bit-identical) are unchanged.

Table R3.6. Llama-3.2-1B decode throughput relative to vLLM's embedded HadaCore (H100, input 32, output 128, six paired runs with alternating order).

Batch18163264
Submitted kernel0.917x0.974x0.975x0.979x0.980x
+ partial last block in kernel0.969x0.980x0.970x0.972x0.977x
+ four warps per row1.019x1.014x1.018x1.016x1.018x

HyperHadamard is now faster than HadaCore in decode at every batch size from 1 to 64, with every 95% interval above 1.

Reviewer_vdsMOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents HyperHadamard, a GPU implementation of Hadamard transforms that exploits the commutativity of butterfly stages to choose an execution order compatible with Tensor Core register layouts. A compile-time representation tracks logical axes and completed stages, allowing subsequent computation to use hardware-produced layouts and reducing intermediate layout materialization. The design also supports catalog-based non-power-of-two transforms and a separate true-FP32 path for randomized gradient projection. The authors report up to 1.59x speedup over HadaCore on H100 for long FP16/BF16 transforms, reaching within 1.7% of an empirical memory-bandwidth roofline. Application studies include quantized vLLM inference and Influence Distillation. The evaluation combines operator benchmarks, profiling, local ablations, numerical checks, and application measurements.

Critical strengths
  1. A coherent algorithm-hardware contribution with convincing operator evidence. The central contribution is the use of butterfly-stage ordering to accommodate Tensor Core output layouts. Tracking axis placement and completion at compile time gives a concrete way to avoid restoring a canonical layout between computations. The worked schedule and the two-phase organization make this an implementable design. On H100, the long-transform gains reach approximately 1.52x in FP16 and 1.59x in BF16 over HadaCore, while effective bandwidth approaches the independently measured 3.03 TB/s reference. The A100 results reproduce the long-transform advantage. Instruction and memory-system profiles, together with the forced-materialization ablation, support the proposed mechanism. This combination of a clear design principle and measured hardware behavior is the main reason for my acceptance recommendation. 2. Practical integration supported by meaningful application baselines. The catalog-transform support addresses a concrete deployment problem in an existing quantized checkpoint. The evaluation includes a baseline using the same factorization assembled from Dao-AILab kernels and BF16 cuBLAS. The reported 1.915x end-to-end improvement over this structured baseline demonstrates value beyond replacing the stock dense fallback. The measured model-quality change is negligible: WikiText-2 perplexity changes by approximately 0.0026%, and the combined ARC accuracy is unchanged. The FP32 projector study provides another useful application and separates kernel replacement, gather remapping, and sign fusion. Its full 1.747x improvement includes all three optimizations and preserves bit-exact output. These studies establish practical utility in two different workloads.
Critical weaknesses

I do not identify a weakness that warrants rejection. Two limitations constrain how broadly I interpret the results: 1. Application benefits depend strongly on the workload and existing path. The 14.70x inference result is measured against a stock FP64 dense catalog-transform fallback. For a checkpoint already using HadaCore, replacing only the CUDA transform yields 1.022-1.052x end-to-end improvements in the tested prefill workload, and batch-8 decode reaches 0.978x the baseline throughput. The large operator sweeps also require enough independent rows. These results support specific long-transform and prefill use cases; broader serving gains remain unestablished. Similarly, the W3A4 rotation-size experiment uses FP64 hooks and demonstrates quality rather than efficient end-to-end W3A4 execution. The paper generally acknowledges these boundaries, which limits their impact on my recommendation. 2. The central mechanism is supported, but its share of the total gain is only partially isolated. Forcing phase-2 layout materialization increases latency by 1.75% in FP16 and 5.89% in BF16, whereas the complete implementation improves over HadaCore by up to 1.59x. The local intervention validates a cost of materialization, but it does not account for the full implementation difference. Scheduling, register usage, memory access organization, and pipelining can contribute jointly. A clearer decomposition would strengthen the causal explanation. This is a limitation of attribution; the measured overall speedup and the supporting profiles remain persuasive.

Questions for authors
  1. Clarification of performance attribution: Which implementation changes account for the remaining gap to HadaCore beyond the phase-2 materialization intervention? Can the existing profiles or ablations distinguish the effects of stage ordering, resource usage, and memory pipelining? This would sharpen my understanding of which parts of the design are essential to the main performance result. 2. Clarification of reproducibility: The PDF states that code and benchmarks will be released upon publication. Does the supplementary archive already contain the complete implementation and executable benchmark scripts, including the structured vLLM baseline and catalog basis correction? If so, please identify the entry points and pinned versions. This would strengthen confidence in independent verification of the central comparisons. I have not inspected that archive.
2 comments
AuthorsOpenReview ↗

Response to Reviewer vdsM (1/2)

We thank the reviewer for the unwavering positive assessment. We appreciate the recognition of "a coherent algorithm-hardware contribution with convincing operator evidence", that "this combination of a clear design principle and measured hardware behavior is the main reason for my acceptance recommendation", and that our applications "establish practical utility in two different workloads". The reviewer also noted that they did not identify a weakness that would warrant rejection. Nevertheless, we provide some clarification and answers to some of the questions and limitations raised by the reviewer.

Limitation 1. The application gains hold beyond the tested prefill workload

We agree that the end-to-end gain depends on the path it replaces, and we therefore measured each case the reviewer raises: the stock catalog path, a checkpoint already using HadaCore, the number of rows per call, small decode calls, online serving, and W3A4 with the kernel itself.

  • Baselines. Dense FP64 is the path stock vLLM executes for these checkpoints, because HadaCore supports powers of two only; the operation-matched comparison is 1.915x. Against HadaCore + cuBLAS on QuaRot's factorization H28 x H512, HyperHadamard is 1.123x faster end to end (please see our response to Reviewer HykP, W2b).
  • Rows per call. At the shapes vLLM issues (2048 rows, 2^20-2^24 elements), HyperHadamard is 1.30x faster than vLLM's embedded HadaCore at N = 512 and 2048, and 1.57x at N = 8192, where the 2^30 sweep shows parity to 1.05x (response to Reviewer oTr2, W1b).
  • Serving. In vLLM online serving with continuous batching on the released Llama-3.1-8B QuIP checkpoint, HyperHadamard raises output throughput over stock vLLM from 138 to 1,806 tokens/s (13.1x; response to Reviewer HykP, W4b).
  • Power-of-two checkpoints and small calls. The 1.02-1.05x end-to-end gain is bounded by the transforms' share of the workload; the transform time itself falls 1.47x. We traced the decode gap to per-row latency and now split each row across four warps, a launch-only change with bit-identical output: HyperHadamard is now 1.014-1.019x faster than HadaCore in decode at every batch size from 1 to 64 (response to Reviewer HykP, W2c).
  • W3A4 with the kernel. Replacing the FP64 hooks with HyperHadamard in BF16 gives W3A4 perplexity 6.5339, versus 6.5405 with the hooks.

Q1 / Limitation 2. The existing profiles and ablations separate stage ordering, resource usage, and memory pipelining

Yes. Each of the three effects has its own measurement on the H100 of the paper, and together they point to the instruction reduction from layout-aware stage ordering. We profile one NVTX-marked launch after warmup with Nsight Compute 2025.2.1, matched against HadaCore at the same shape, and time every ablation in fresh processes in paired order.

  1. Stage ordering. We compare dynamic warp instructions and physical DRAM throughput across N = 4096-32768. HyperHadamard executes 2.0-2.3x fewer warp instructions than HadaCore at N = 4096 and 3.5-3.6x fewer at N = 32768, and its DRAM-throughput advantage rises with it, from 1.03-1.09x to 1.72-1.79x. The N = 32768 profile, repeated in three fresh processes, gives identical instruction ratios and a 1.68x (FP16) and 1.78x (BF16) faster kernel. We then force phase 2 to read and write its strided views through memory instead of consuming them in place, the one materialization that can be reinserted without changing the rest of the schedule. Over ten processes, this costs 1.75% (FP16) and 5.89% (BF16) at N = 32768, with 3.6-7.4% more instructions and 5.6-14.7% lower DRAM throughput; on an A100 it costs 12.6% and 16.7%.

  2. Resource usage. From the same profiles, at N = 32768 our kernel keeps at most three one-warp blocks per SM, against one 16-warp block for HadaCore, yet sustains 2.43-2.45 versus 1.36-1.43 TB/s of DRAM throughput. The gain therefore does not come from higher occupancy.

  3. Memory pipelining. We build a null kernel that reuses our exact launch configuration and data-movement code but performs no arithmetic; it moves the same DRAM bytes within 0.1%. Over ten processes, HyperHadamard is only 1.055x (FP16) and 1.058x (BF16) slower than this floor while executing 7.0-7.7x its instructions. We also change the pipelining directly, splitting each row across four warps and issuing all asynchronous loads before waiting: H100 performance on large transforms is unchanged (within 0.5%), whereas on an NVIDIA B200, with about twice the bandwidth, the same change raises our kernel from 4.5-6.3 to 6.7-7.0 TB/s at N = 16384-32768.

Therefore, the H100 gain comes mainly from the instructions that layout-aware stage ordering removes, not from occupancy or memory pipelining. Isolating stage ordering completely would require running HadaCore's recursive order inside our kernel, which we note as future work.

AuthorsOpenReview ↗

Response to Reviewer vdsM (2/2)

Q2. Reproducibility

Yes, the original supplementary archive did contain the code, and we have updated it to reflect the rebuttal evaluations. The supplementary archive submitted with the paper contains the full implementation and executable benchmarks:

  • Build: csrc/setup.py; smoke test: scripts/smoke_test.py.
  • Figure 1: scripts/benchmark_operator.py, scripts/run_operator_repetitions.sh; roofline: scripts/benchmark_roofline.py.
  • vLLM integration with the catalog basis correction: vllm_patch/sitecustomize.py, vllm_patch/had28.py; paired runs: scripts/run_vllm_pair.sh; checkpoint audit: scripts/audit_vllm_checkpoint.py; quality: scripts/eval_wikitext2.py.
  • Influence Distillation projector: scripts/benchmark_influence_projector.py.

Pinned versions: vLLM 0.14.0 (official container), PyTorch 2.9.1, CUDA 12.9.1, H100 80GB HBM3. The revised archive adds a rebuttal/ folder with the structured Dao-AILab + cuBLAS and HadaCore + cuBLAS vLLM baselines and the scripts for every new experiment; rebuttal/README.md lists the commands.

We thank the reviewer again for the support and supporting clear acceptance of the paper.

Reviewer_HykPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents HyperHadamard, a layout-aware GPU FWHT kernel that schedules butterfly stages as commuting hypercube-axis rotations so they match tensor-core register layouts, eliminating intermediate materialization. On H100 it stays within 1.7% of the memory roofline and is up to 1.59× faster than HadaCore at N=32768. With catalog Hadamard factors it accelerates QuIP rotations in vLLM; a true-FP32 variant speeds a randomized gradient projector.

Critical strengths

Clean systems insight (axis commutation + compile-time fragment/layout tracking) with careful roofline-based evaluation, profiler attribution, and confidence intervals. Non-power-of-two catalog factors (e.g. H224) fill a real gap: stock vLLM falls back to dense FP64 GEMMs for released QuIP checkpoints. Bit-exact FP32 path and correctness checks against baselines are done properly.

Critical weaknesses

Limited ML contribution. The core contribution is a CUDA kernel design. Lemma 1 (stage commutation) is classical FWHT; Kronecker factors for non-powers of two are standard. For ICLR the ML novelty is thin — this reads as MLSys/GPU-architecture material. Headline speedups overstate practical impact. The 14.7× number is vs a pathological dense-FP64 fallback, not SOTA. Against a fair structured baseline the gain is 1.9× at operator level; replacing HadaCore in vLLM is only 1.02–1.05× e2e, and batch-8 decode is 0.98× (slightly worse). When large rotations matter is under-validated. The rotation-size study is one model (Mistral-7B) and one GPTQ recipe, run with FP64 hooks (quality only). Gains vanish or reverse with BF16/INT8 activations; there is no e2e quality+latency result with the proposed kernel under aggressive quantization. Narrow evaluation. Primary results on a single H100; no serving/continuous-batching traces. Influence Distillation is only the released projector microbench (pipeline gain 1.015×); the full experiment is unavailable. Code is promised, not released.

Questions for authors

What fraction of realistic deployments use full-dimension / non-power-of-two rotations (vs H128/H256 blocks under FP4/MXFP4)? Where does HyperHadamard matter beyond these QuIP checkpoints? Can you report e2e latency + quality for the INT3/INT2 recipes using HyperHadamard itself, not FP64 quality hooks? How do the numbers change against the strongest available fallbacks (HadaCore with padding; cuBLAS only for the catalog factor), rather than stock dense FP64? Any results on Blackwell or multi-GPU serving?

5 comments
AuthorsOpenReview ↗

Response to Reviewer HykP (1/4)

We thank the reviewer for the positive assessment and constructive questions. We appreciate the recognition of our "clean systems insight (axis commutation + compile-time fragment/layout tracking) with careful roofline-based evaluation, profiler attribution, and confidence intervals", that our non-power-of-two catalog factors "fill a real gap", and that the "bit-exact FP32 path and correctness checks against baselines are done properly". We address the four concerns the reviewer raises, on the ML contribution, the headline speedups, when large rotations matter, and the breadth of the evaluation, and answer each question at the end. Additionally, we have added new experiments: (i) the HadaCore + cuBLAS composition the reviewer proposes, end to end in vLLM, (ii) online serving with continuous batching, (iii) an NVIDIA B200, (iv) the quantization study with the kernel in the loop, and (v) a decode fix that makes HyperHadamard faster than HadaCore at every batch size.

W1. HyperHadamard is a practical algorithm with concrete ML applications, the kind of contribution ICLR publishes

Limited ML contribution. The core contribution is a CUDA kernel design. Lemma 1 (stage commutation) is classical FWHT; Kronecker factors for non-powers of two are standard. For ICLR the ML novelty is thin — this reads as MLSys/GPU-architecture material.

ICLR welcomes diverse contributions that advance ML research and lists infrastructure, software libraries, and hardware among its subject areas. Kernels whose value lies in the ML workloads they make practical are among the most widely used works in ML practice, and their main contribution is the kernel design and algorithm. Numerous highly cited works follow this pattern, such as FlashAttention [1], FlashAttention-2 (ICLR 2024) [2], and FlashFFTConv (ICLR 2024) [3], which maps FFT convolutions onto tensor cores and is the closest precedent to our setting. On the quantization side, rotation-based methods such as SpinQuant (ICLR 2025) [4] and QuaRot [5] depend on exactly the online Hadamard transforms we accelerate.

The other reviewers assess the contribution in these terms: Reviewer vdsM recommends clear acceptance for "a coherent algorithm-hardware contribution with convincing operator evidence", and Reviewer oTr2 notes that "a kernel paper stands or falls on whether its gain is real and explained, and here it is."

Following this line, we develop a practical algorithm with concrete ML applications and evaluate it extensively: exact catalog rotations in a released quantized model (1.915x end to end over an operation-matched baseline, unchanged quality, 13.1x serving throughput over stock vLLM), and a 1.747x bit-exact projector for training-data selection.

We agree that stage commutation (Lemma 1) and Kronecker factorization are classical results. Our contribution is how we build on them: a compile-time representation that tracks each hypercube axis's position and completion state through tensor-core fragment layouts, which we use to co-design an algorithm that is parallelism-friendly and hardware-amenable, so the stage order follows the layouts the MMA produces and no layout is restored between instructions. We isolate what this removes: it cuts the instruction count by up to 3.5x and keeps the kernel within 1.7% of the memory roofline. The same design extends to non-power-of-two orders, running exact catalog transforms such as H28 x H512 in a single kernel.

W2. The headline comparisons cover the deployed path, operation-matched baselines, and the strongest fallback

Headline speedups overstate practical impact. The 14.7× number is vs a pathological dense-FP64 fallback, not SOTA. Against a fair structured baseline the gain is 1.9× at operator level; replacing HadaCore in vLLM is only 1.02–1.05× e2e, and batch-8 decode is 0.98× (slightly worse).

We agree that the 14.70x is measured against a dense path. It is the path stock vLLM executes for these checkpoints, since its only fast Hadamard kernel, HadaCore, supports powers of two only, and it is the reference that llm-compressor's own documentation uses for its transforms, reporting HadaCore against dense GEMM latency. No prior fast kernel computes these checkpoints' catalog factorizations exactly. We therefore also report operation-matched comparisons.

W2a. End to end and operator level. We note one correction: the 1.915x is an end-to-end result in vLLM, against the same factorization assembled from Dao-AILab kernels and cuBLAS. At the operator level, HyperHadamard is 1.52-1.59x faster than HadaCore at N = 32768 (Figure 1) and 1.57x faster at the 2048 x 8192 BF16 calls that vLLM issues for Llama-3.2-1B (95% CI 1.568-1.577x, shape-matched against vLLM's embedded HadaCore). On an NVIDIA B200 it is faster than HadaCore at every length, by up to 2.97x (W4a).

AuthorsOpenReview ↗

Response to Reviewer HykP (2/4)

W2b. The strongest fallback. HadaCore with padding cannot serve these checkpoints: the online rotation must reproduce the checkpoint's matrix exactly, and H16384 restricted to 14336 coordinates is a different matrix. The remaining fallback is two kernels, HadaCore on the power-of-two factor followed by a cuBLAS GEMM on the catalog factor; it writes the intermediate result to memory and reads it back, whereas HyperHadamard applies both factors in one pass. Following the reviewer's suggestion, we measured this composition end to end: on a Llama-3.1-8B checkpoint, HyperHadamard is 1.123x faster end to end than HadaCore + cuBLAS in vLLM (mean of six paired runs with alternating order; 95% CI 1.122-1.125x), with every rotation checked against stock vLLM at load. Our structured baseline in the paper (1.915x) uses the same split, with Dao-AILab on the power-of-two factor.

W2c. Replacing HadaCore on a power-of-two checkpoint, and decode. This experiment replaces the strongest existing kernel in its own regime, power-of-two sizes, so the end-to-end gain is bounded by the transforms' share of the workload. On this Llama-3.2-1B workload, the paper's trace shows the transforms take 126.2 ms of an 819.3 ms interval, so even an infinitely fast transform would give at most 1.18x end to end (Amdahl's law). HyperHadamard reduces the transform time 1.47x (to 85.6 ms), which predicts exactly the measured 1.052x, so the end-to-end gain is fully accounted for by the kernel speedup. At the exact shapes vLLM issues, the kernel is 1.57x faster than vLLM's embedded HadaCore (W2a), and on a B200 the same 2048 x 8192 call is 1.99x faster than HadaCore (2.39x with the optimized launch, W4a), so the share the transform can recover grows on newer hardware.

For decode, we traced the small-batch gap to per-row latency. For large transforms our kernel assigns one warp to each row, which maximizes bandwidth when a call has thousands of rows; a decode call has only 1-64 rows, so its time is the latency of a single row, and HadaCore, which spreads each row over 256-512 threads, finishes it sooner. We therefore split each row across four warps, the same launch configuration as on Blackwell (W4a), and also removed a host-side padding copy for single-row calls. This is a launch-configuration change only: the stage schedule, the register layouts, and the output (bit-identical) are unchanged. In Llama-3.2-1B decode, HyperHadamard is now 1.014-1.019x faster than HadaCore at every batch size from 1 to 64, up from 0.92-0.98x, while large-transform performance on H100 is unchanged (within 0.5%).

W3. Large rotations matter where one activation scale covers a whole token, and the kernel preserves their quality

When large rotations matter is under-validated. The rotation-size study is one model (Mistral-7B) and one GPTQ recipe, run with FP64 hooks (quality only). Gains vanish or reverse with BF16/INT8 activations; there is no e2e quality+latency result with the proposed kernel under aggressive quantization.

We have extended the rotation-size study along the three axes the reviewer names: the kernel itself in place of the FP64 hooks, a second model family, and activation precision.

W3a. Quality with the kernel in the loop. We replaced the FP64 hooks with HyperHadamard in BF16 (all 448 rotation modules, same calibrated model, same process).

Table R1.1. Mistral-7B WikiText-2 perplexity with HyperHadamard in place of the FP64 hooks.

SettingFP64 hooksHyperHadamard BF16H128 blocks
W3A4, full rotation6.54056.53396.9856
W2A4, full rotation34.5036.8357.15

The kernel reproduces the FP64 result at W3A4 and keeps most of the full-rotation gain at W2A4 (36.83 versus 57.15 for H128 blocks), so the quality benefit of full-dimension rotations holds with the kernel itself, not only with FP64 hooks. End-to-end W3A4 latency would additionally require a low-bit activation GEMM, which vLLM does not provide for this recipe with any rotation kernel; the latency of the rotation itself is the operator, and end-to-end speedups in W2.

W3b. When full-dimension rotations matter. The gain depends on the granularity of the activation scale, and our results now show this on two model families. With 4-bit per-token activations, where one scale covers a whole token, full-dimension rotations improve W3A4 perplexity over H128 blocks by 6.4% on Mistral-7B and by 12.4% on Llama-3.1-8B-Instruct (12.08 to 10.58). With BF16 or INT8 activations, rotation size changes perplexity by less than 1% on both models. This is the expected behavior, since rotations exist to suppress activation outliers and matter only once the activations are quantized coarsely. Microscaling FP4 formats were co-designed with block rotations matched to their scaling groups, where group-aligned blocks are the established choice; even there, under MXFP4, full rotation improves Mistral-7B perplexity over H128 by 2.0%.

AuthorsOpenReview ↗

Response to Reviewer HykP (3/4)

The per-token 4-bit setting is exactly the one QuaRot [5], SpinQuant (R4) [4], QuIP# [6], and the llm-compressor QuIP transforms behind the released checkpoints target with full-dimension online rotations. Large rotations therefore matter in the deployed rotation-based recipes, and HyperHadamard makes them exact and fast; beyond quantization, it also serves randomized sketching (Section 4.3).

W4. The evaluation now spans three GPU generations, online serving, and code available to reviewers

Narrow evaluation. Primary results on a single H100; no serving/continuous-batching traces. Influence Distillation is only the released projector microbench (pipeline gain 1.015×); the full experiment is unavailable. Code is promised, not released.

We have broadened the evaluation along each axis the reviewer lists: hardware, serving, the Influence Distillation scope, and code.

W4a. Hardware beyond a single H100. Appendix D already repeats the operator sweep on an A100 (1.34-1.37x over HadaCore at N = 32768). We now also ran the kernels on an NVIDIA B200 with the same paired protocol as Figure 1 (five processes, 2^30 elements), and optimized the launch configuration for Blackwell (four warps per row, with each warp issuing all of its asynchronous loads before waiting); the algorithm and output are unchanged, H100 performance on large transforms is unchanged (within 0.5%), and on H100 it also removes the decode gap (W2c).

Table R1.2. Speedup over HadaCore on an NVIDIA B200.

N25651210242048409681921638432768
BF161.341.751.401.461.572.312.171.97
FP161.341.741.241.331.622.352.032.13
BF16, optimized launch1.361.771.391.471.712.462.432.97
FP16, optimized launch1.361.771.221.341.642.352.242.91

HyperHadamard is faster than HadaCore at every length on the B200, by up to 2.97x, because HadaCore stays at or below 5.6 TB/s against a 6.8 TB/s roofline reference while HyperHadamard reaches 6.7-7.0 TB/s with the optimized launch. The FP32 kernel is bit-exact with Dao-AILab on the B200 and 1.30x, 1.37x and 1.58x faster at N = 4096, 16384 and 32768. The advantage therefore holds across A100, H100 and B200, and it grows with memory bandwidth.

Multi-GPU serving of these checkpoints is not possible in vLLM with any rotation kernel, since vLLM 0.14.0 does not support online transforms with tensor parallelism. Single-GPU serving is the supported deployment, and data-parallel replicas repeat the single-GPU result.

W4b. Online serving with continuous batching. We ran vLLM's online serving benchmark (vllm serve with vllm bench serve) on the released Llama-3.1-8B QuIP checkpoint (H100, 200 requests of 1,024 input and 128 output tokens, three paired runs with alternating order), comparing HyperHadamard with stock vLLM, which runs the checkpoint's catalog rotations as dense FP64 GEMMs. With no arrival limit, HyperHadamard raises output throughput from 138 to 1,806 tokens/s (13.1x) and lowers mean time to first token from 87.3 to 6.1 s. At 4 requests/s, stock vLLM completes only 1.08 requests/s and its queue grows throughout the run, whereas HyperHadamard sustains the load (3.90 requests/s, mean time to first token 0.10 s). The offline 14.70x therefore carries over to online serving: with HyperHadamard these checkpoints sustain a realistic request rate that stock vLLM cannot.

W4c. Influence Distillation. The full 200k-candidate experiment is not public, so we evaluate the projector its authors released. Its 1.747x speedup on H100 (1.80x on B200) is bit-exact, so the selected training set is unchanged. The 1.015x pipeline gain is bounded by the projector's share of that pipeline; the projector, the component a fast transform can accelerate, is where the gain is measured.

W4d. Code. The complete implementation and benchmark scripts were included in the supplementary material submitted with the paper (build: csrc/setup.py; operator benchmark: scripts/benchmark_operator.py; vLLM integration: vllm_patch/). We have updated the supplementary material to add the baselines and scripts for the new experiments in this response, and we will release the code publicly upon acceptance.

Comments on the paper

2 comments

Dear reviewers,

The discussion with authors is closing soon. If you haven't already, please review the rebuttal to see if the authors have addressed your concerns, acknowledge to the authors that you have read their response, and make any necessary adjustments to the score as needed. Thanks

AuthorsOpenReview ↗

Final Remarks by Authors

Dear Area Chair, Program Chairs, and Reviewers,

As the discussion period has ended, we thank the reviewers for their constructive feedback and the Area Chair for encouraging discussion. Below, we summarize HyperHadamard's contributions and key rebuttal results.

HyperHadamard is a state-of-the-art GPU Hadamard transform implementation with practical applications in quantized LLM inference and training-data selection. It introduces a layout-aware algorithm that schedules butterfly stages around Tensor Core register layouts, eliminating intermediate materialization. This reduces warp instructions by up to 3.5× and reaches within 1.7% of the H100 memory bandwidth roofline. The design also supports exact non-power-of-two transforms and bit-exact FP32 projections.

The framework demonstrates improvements across hardware and ML applications:

  • GPU performance. Up to 1.59× faster than HadaCore on H100, with gains across A100 and B200. New B200 results reach 2.97× over HadaCore.
  • Quantized LLM inference (vLLM). HyperHadamard runs released QuIP checkpoints 14.7× faster than stock vLLM, 1.915× faster than Dao-AILab + cuBLAS, and 1.123× faster than HadaCore + cuBLAS (added during rebuttal), with unchanged quality.
  • Online serving (new). Continuous-batching throughput increases from 138 to 1,806 output tokens/s (13.1×) over stock vLLM.
  • Training-data selection. The FP32 projector achieves a 1.747× speedup on a billion-dimensional projection with bit-exact outputs.

All three reviewers recognized the technical contribution or practical value of the work. Reviewer vdsM recommended clear acceptance (4), while HykP and oTr2 raised concerns about deployment, quantization, and evaluation breadth. We performed substantial experiments to address these concerns.

Reviewer vdsM described HyperHadamard as "a coherent algorithm-hardware contribution with convincing operator evidence" and noted that its applications "establish practical utility in two different workloads." Their questions concerned performance attribution and reproducibility. We provided profiler comparisons separating stage ordering, occupancy, and pipelining, and identified the complete implementation, benchmarks, and pinned dependencies in the supplementary archive. The reviewer maintained a rating of 4.

Reviewer HykP recognized the "clean systems insight" and noted that catalog transforms "fill a real gap." Their concerns involved practical speedups, stronger baselines, hardware generalization, serving, and quantization. We added HadaCore + cuBLAS (1.123× end-to-end), B200 benchmarks (up to 2.97×), online serving (13.1× throughput), and quantization evaluations using the actual kernel. We also eliminated the decode regression, with HyperHadamard now faster than HadaCore across batch sizes 1–64.

Reviewer oTr2 acknowledged "credible, mechanism-level evidence for the speed claim" and "practical coverage that existing fast kernels lack." Their main concerns were the operator/deployment speedup inconsistency and limited evidence for full-dimension rotations in modern quantization.

  • The operator/deployment inconsistency is resolved. Shape-matched benchmarks at actual vLLM call sizes (2048 × 8192, BF16) show a 1.57× speedup over embedded HadaCore, consistent with the 1.47× reduction in transform time. The discrepancy arose from different tensor sizes.
  • The decode regression is eliminated. HyperHadamard is now 1.014–1.019× faster than HadaCore across batch sizes 1–64.
  • The quantization evaluation is expanded. We added MXFP4/NVFP4, Llama-3.1-8B-Instruct, and evaluations using HyperHadamard instead of FP64 hooks. Full rotations improve W3A4 perplexity over H128 blocks by 6.4% on Mistral-7B and 12.4% on Llama-3.1-8B-Instruct. Full rotation also achieves the best MXFP4 perplexity, while NVFP4 shows the dependence on scale granularity.
  • Other questions were addressed, including adversarial FP32 bit-exactness, resource footprints, additional random seeds, and positioning relative to FFT/WHT, WUSH, and learned transforms.

Unfortunately, none of the three reviewers engaged with our rebuttals, despite responses submitted on November 9–10, follow-up reminders, and the Area Chair's request on November 18. We therefore could not learn whether the new evidence resolved their concerns or clarify any remaining questions.

We hope the Area Chair and Program Chairs will consider the substantial new evidence in their final assessment, particularly the resolution of reviewer oTr2's central concerns underlying their rating of 2. The rebuttal produced new benchmarks, implementation improvements, and evaluations that materially strengthen the paper.

We appreciate the reviewers' feedback and will incorporate the results and promised revisions into the final manuscript.

Thank you for your consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.