acceptodds
Under review as a conference paper at ICLR 2027

AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

Abstract

Hadamard transforms have become a standard tool for stabilizing MXFP4 LLM training by spreading outliers within its 32-element scaling blocks. We show that their effectiveness depends on a simple geometric factor: how outliers are oriented relative to the scaling axis. Outliers perpendicular to the axis create within-block concentration that Hadamard mixing can effectively reduce, whereas outliers aligned with the axis enlarge the shared block scales, leaving a scale-driven component that rotation does not directly address. We call this the scaling-axis blind spot. Measurements of LLM training tensors reveal stable Row-wise, Column-wise, and None outlier patterns whose combinations vary across GEMMs; notably, 97.8–100% of weight-gradient GEMMs contain a blind-spot operand. We introduce AdaHOP, an outlier-pattern-aware MXFP4 training framework that identifies these patterns in a short calibration phase, uses Inner Hadamard Transform (IHT) as the low-precision backbone, and selectively extracts only the blind-spot outlier rows or columns into a small BF16 path. With fused Triton kernels, AdaHOP-Lv1 executes less than 1% of GEMM FLOPs in BF16 while trailing BF16 by only 0.23 points in average zero-shot accuracy across models from 1B to 8B. On Llama3.1-8B, it reduces memory by and accelerates end-to-end training by over BF16.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Low-Precision Training, Quantization for training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sFYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies MXFP4 LLM training and identifies outlier patterns that Hadamard transforms cannot adequately address. AdaHOP calibrates these patterns for individual GEMM operand pairs, combines Hadamard transforms with selective BF16 outlier correction, and updates the selected outlier indices during training. Fused kernels reduce the correction overhead on AMD GPUs. Experiments evaluate training quality, downstream accuracy, and end-to-end training efficiency.

Critical strengths
  1. The analysis provides a concrete explanation for why Hadamard transforms are insufficient for certain outlier patterns. Selecting treatments at the GEMM operand-pair level is a reasonable way to address these differences during training.

  2. The paper demonstrates practical quality and efficiency benefits. AdaHOP-Lv1 trails BF16 by only 0.23 percentage points in average zero-shot accuracy across five 1B–8B models and achieves a 1.45× end-to-end training speedup on Llama3.1-8B.

Critical weaknesses
  1. The quality comparison with Quartet is limited relative to the scope of the main results. Quartet is a closely related FP4 training method, but it is evaluated only on Qwen3-0.6B, while the main quality results cover models from 1B to 8B. Different hardware backends complicate direct speed comparisons, but do not prevent larger-model quality comparisons using the numerical simulation already adopted in this paper. It therefore remains unclear how AdaHOP compares with this strong baseline at larger scales.

  2. The ablations provide evidence for pattern-aware assignment, but do not fully establish its advantage over simpler policies with comparable high-precision computation. Table 10 matches the number of GEMMs receiving outlier correction, which does not by itself guarantee equal BF16 FLOPs or runtime when GEMM shapes and extraction sides differ. Since selective high-precision correction is an established approach, comparisons against simple fixed policies under a matched computation budget are needed to determine whether the proposed assignment rule offers a better quality–efficiency trade-off.

I would consider raising my score if these concerns are addressed.

Questions for authors
  1. How does AdaHOP compare with Quartet in training quality on a representative larger model under matched settings?

  2. Does the proposed assignment rule outperform simpler fixed policies at a comparable BF16 computation budget or measured overhead?

3 comments

W1, Q1. Comparison with Quartet on a larger model

We sincerely thank the reviewer for this constructive feedback and for the willingness to raise the score if our responses address the concerns. To evaluate Quartet beyond the 0.6B scale within the rebuttal window, we conducted a controlled 2B-token numerical simulation on Llama3.2-1B using the same optimizer, learning-rate schedule, and C4 dataset across BF16, Quartet, and AdaHOP-Lv1. As shown in Table R1 below, AdaHOP-Lv1 reduces Quartet's training loss gap to BF16 by more than two-and-a-half times. As the reviewer noted, while hardware backend differences prevent a direct wall-clock speed comparison, our precision simulation environment provides a like-for-like algorithmic comparison under identical numerical conditions. These new results confirm that the optimization advantage AdaHOP demonstrated at the 0.6B scale transfers consistently to larger models.

[Table R1] Comparison with Quartet on Llama3.2-1B (C4 2B tokens, precision simulation, same optimizer and schedule, loss gap to BF16)

MethodLoss gap
BF160
Quartet0.0389
AdaHOP-Lv10.0152

Furthermore, our extended 10B-token evaluation on Qwen3-0.6B in Table 3, which trains ten times longer than the ablation runs in Table 10, demonstrates that a smaller training loss gap translates directly into stronger downstream task performance. Across all four zero-shot benchmarks, AdaHOP-Lv1 consistently outperforms Quartet, reducing the average accuracy drop relative to BF16 to less than a third of Quartet's. We will incorporate the new Llama3.2-1B simulation results into the revised manuscript to provide a thorough comparison across model scales.

W2, Q2. Advantage over simple fixed policies at a comparable BF16 compute budget

We appreciate the reviewer's precise observation that selective high-precision policies must be compared under their actual BF16 FLOPs share rather than merely by matching the number of treated GEMMs. When we compute the BF16 FLOPs on Qwen3-0.6B using the true matrix dimensions of each layer, the count-matched Random OE baseline from Table 10 actually consumes more BF16 FLOPs than AdaHOP-Lv1 because the randomly selected GEMMs happen to have larger average dimensions. Despite using a larger high-precision compute budget, Random OE suffers roughly three times the three-seed loss gap of AdaHOP-Lv1. This confirms that the quality recovery stems from our pattern-aware placement rather than the sheer volume of BF16 computation.

To further test AdaHOP against a strong fixed path-level policy, we evaluated a heuristic called Path rule A. Because blind-spot operands concentrate heavily in the weight-gradient (∇W\nabla W) pass, Path rule A uniformly applies right-side outlier extraction (OE-R) to every GEMM on the ∇W\nabla W path while applying standard IHT everywhere else. Although Path rule A consumes nearly the same BF16 FLOPs as AdaHOP-Lv1, it misses the blind-spot GEMMs that appear on the input-gradient (∇X\nabla X) path and extracts along the suboptimal axis for certain ∇W\nabla W layers. When evaluated across three random seeds under the identical setup of Table 10, AdaHOP-Lv1 outperforms Path rule A by a margin that exceeds the across-seed standard deviation, as shown in Table R2 below. Adapting at the operand-pair level therefore achieves a superior quality-efficiency trade-off compared to fixed path-level heuristics, and we will add this FLOPs analysis and multi-seed comparison to the final manuscript.

[Table R2] Training comparison under the same BF16 budget (Qwen3-0.6B, C4 1B tokens, 3 seeds, gap to BF16 averaged over the last 100 steps, ±\pm is the 3-seed std of each arm)

ArmBF16 FLOPs (%)3-Seed Gap Mean ±\pm Std
Path rule A1.6670.0294 ±\pm 0.0065
Count-matched Random OE2.0080.0538 ±\pm 0.0082
AdaHOP-Lv11.8150.0175 ±\pm 0.0077

Thank you for the additional experiments and clarifications. The larger-model comparison with Quartet and the three-seed comparison against a fixed path-level policy at a comparable BF16 FLOPs budget substantially address my original concerns.

My remaining concern is the generality of the findings. AdaHOP relies on identifying stable outlier patterns and assigning suitable treatments, but it remains unclear whether these patterns and the effectiveness of the assignment policy extend to MoE or hybrid architectures. It would also be valuable to understand whether the observed benefits persist at larger scales and over longer training horizons, where quantization effects may differ from those captured by the current controlled comparisons. I view these as remaining limitations of the evidence rather than demonstrated failures of the method.

We thank the reviewer for reviewing our additional experiments and clarifications and for confirming that they address the original concerns. We also agree that the remaining points reflect limitations in our empirical evidence rather than failures of the method itself. Below we summarize the supporting evidence for each point.

MoE and hybrid architectures. We ran a preliminary check on the public DeepSeek-V2-Lite checkpoint (64 routed experts per layer) to verify that AdaHOP's calibration remains valid for MoE. Applying our 30-step calibration to two disjoint sets of C4 batches yielded 97.2% agreement in detected patterns across all 9,984 routed-expert GEMMs. This high consistency confirms that expert patterns remain stable under the same data distribution despite batch-to-batch token routing variations.

We further trained a small MoE model (Qwen2-MoE architecture, 494M parameters, 32 routed experts per layer) from scratch on C4, and the detected patterns of its routed-expert GEMMs agreed at 96.6% during 3,000 step training. We will include full MoE training results in the camera-ready version since the rebuttal timeframe does not permit full training runs.

In hybrid architectures, state-space components consume only a small fraction of compute and are known to be sensitive to reduced precision. Keeping these components in BF16 is therefore a natural choice. This approach aligns naturally with AdaHOP because our method already retains a small share of sensitive GEMMs in BF16. AdaHOP can then focus on accelerating the attention and MLP GEMMs that dominate the overall compute.

Scale and training length. We agree it is important to examine whether our observed benefits persist at larger scales and over longer training horizons. Across our Chinchilla-optimal training runs from 1B to 8B, AdaHOP delivers consistent gains throughout the entire tested compute range. Even on our largest Llama3.1-8B model trained on 160B tokens, AdaHOP-Lv1 stays within 0.05 points of BF16 in zero-shot accuracy whereas HALO falls 1.31 points behind (Section 6.1, Table 3). Beyond preserving accuracy at scale, AdaHOP becomes more efficient as models grow because the Lv1 BF16 share drops from 0.73% at 1B to 0.45% at 8B (Appendix N, Table 16).

Extending these benefits to longer training horizons relies on the stability of the detected outlier patterns over time. When we re-detect patterns 26% into the Llama3.2-3B run at step 30,000, the results still show a 95.8% agreement with our initial 30-step calibration (Appendix F). This long-term stability suggests that model architecture largely determines these patterns rather than training history. Indeed, our from-scratch weight and gradient patterns almost exactly match those of the public Llama3.2-1B checkpoint despite different training data, optimizers, learning-rate schedules, and token counts (Appendix G, Table 8). Even if the patterns were to drift during an exceptionally long run, periodic re-calibration easily acts as a safeguard at a negligible cost of 0.026% of training FLOPs per invocation (Table 5).

Training beyond 8B exceeded our compute budget, so we have stated validation at these scales as future work in the Limitations of the revision. We thank the reviewer again for the careful review and constructive feedback.

Reviewer_teQ5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaHOP starts from the observation that the benefit of Hadamard rotation in MXFP4 training depends on the orientation of outliers relative to the quantization scaling axis. It calibrates each operand pair's pattern over the first 30 BF16 steps, applies an inner Hadamard transform (IHT) as the base, and for pairs whose outliers lie along the scaling axis additionally extracts the top-r rows or columns into a thin BF16 path. Experiments span 0.6B–8B models, reporting an average 0.23 points below BF16 across five models and 1.45× end-to-end speedup on Llama3.1-8B.

Critical strengths

S1. The measurement study is concrete and reusable. Outlier orientation is measured for every operand pair in three model families and is stable over training (95.8% agreement after 30k steps, Table 5), and the transposition observation, that one output gradient behaves differently in ∇X and ∇W, is correct.

S2. The training and systems validation carry real weight. Unlike most low-precision-training papers, which report simulated quality only, this work ships a fused MXFP4+BF16 kernel with a per-stage latency breakdown, trains six models up to 8B / 160B tokens, and measures speedups on real hardware.

Critical weaknesses

W1. The theoretical claims do not establish the proposed mechanism. Proposition 4(c) lower-bounds the error bound, not the actual quantization error. In Proposition 1 the sign diagonal is applied after mixing, which leaves γ_b unchanged; the Hoeffding argument requires A D H_b. Empirically, the no-transform error is identical across the four patterns (Φ = 2.25), and three outlier-targeted baselines all beat uniform IHT in training (Quartet 0.0416, HALO 0.0445 vs. 0.0618; Tables 10–11). The closely related NVFP4 recipe (arXiv 2509.25149) is not cited.

W2. The source of the pattern-aware gain is unclear and within noise. "Pattern-aware IHT" accounts for 75% of the improvement without any BF16 computation, yet this arm is never defined. OE then improves the gap only from 0.0233 to 0.0167 in a single run, below the seed std of 0.0088. A simple path-level rule agrees with Lv1 on about 90% of GEMMs but is never evaluated, and the headline 0.0041 gap is single-seed while the three-seed gap is 0.0175 (Table 10; §4.2). Downstream, every per-model margin over the best baseline is a single run within an SE of ≈0.6 pt.

Questions for authors
  1. Is Proposition 4 intended only to compare error bounds, and where exactly should the random sign diagonal sit in Proposition 1? In what order does the deployed kernel apply it?
  2. What operations do "Pattern-aware IHT" and "Tseng et al. + PA" actually change, which GEMMs receive IHT in the deployed Lv1, and do random OE and AdaHOP consume identical BF16 FLOPs? What value of κ was used?
  3. At matched BF16 budget and over three seeds, how does AdaHOP compare with the path-level rule above and with random rows/columns inside the same selected GEMMs?
  4. Please give the memory breakdown behind the 3.6× claim with the distributed/recomputation configuration, and explain how the forward cache supports backward computation along a different quantization axis.
2 comments

W1, Q1. Prop 4, the sign matrix of Prop 1, and the deployed kernel

We thank the reviewer for this rigorous reading of our theoretical analysis. Regarding Proposition 4(c), we agree that the quantity ϵq∥A∥F\epsilon_q\Vert A\Vert_F represents a floor on the upper bound of the quantization error that no orthogonal transform can lower, and we will revise the phrasing to state this explicitly. This theoretical floor aligns closely with our empirical measurements in Table 4, which confirm that IHT does not reduce the actual quantization error on the blind-spot pairs RN and RC.

Regarding Proposition 1, the reviewer is right that the random sign diagonal DD must precede the Hadamard mixing as ADHbADH_b for Hoeffding's inequality to hold. We will correct the AHbDAH_bD typo in the statement of Proposition 1 and align the accompanying text in Appendix L.1. In our actual training pipeline, to avoid the runtime overhead of an extra element-wise multiplication, the deployed kernel executes the deterministic transform A↦AHbA\mapsto AH_b. This corresponds to the fixed-HbH_b case of Proposition 1 under conditions C1 and C2, which suffices because worst-case sign alignment between the outlier support and the Hadamard rows does not arise in practice.

W1. Table 11, outlier-targeted baselines, NVFP4 recipe

In Table 11, the four untransformed patterns share the same value Φ=2.25\Phi=2.25 because the table reports the normalized error bound of Theorem 1, where any outlier spanning a full row or column yields a within-block concentration of γb=1\gamma_b=1. In actual unnormalized terms, those outliers inflate the overall Frobenius norms ∥A∥F\Vert A\Vert_F or ∥B∥F\Vert B\Vert_F, causing the true quantization errors to vary substantially across patterns as shown in Table 4. We will clarify this distinction in the caption of Table 11.

This scale-driven error also explains the performance gap between path-level baselines and AdaHOP. While HALO and Quartet improve upon Uniform IHT by applying path-level rotations, Proposition 4 shows that rotation alone cannot eliminate the scale-driven error of blind-spot operands. As a result, HALO's three-seed loss gap remains roughly three times larger than that of AdaHOP-Lv1. Conversely, when we apply our pattern-aware assignment to the components of Tseng et al., its loss gap drops by more than five-fold, demonstrating that the primary quality gain comes from where the treatments are placed.

Finally, we thank the reviewer for pointing out the NVFP4 recipe, which we will cite and compare in Section 2 and Table 14 together with TetraJet-v2.

W2, Q2. Ablation arms, Lv1 assignment, BF16 FLOPs, and κ\kappa

We are glad to clarify the definitions of our ablation arms and policy levels. In Table 10, the Pattern-aware IHT arm applies IHT exclusively to non-blind-spot operand pairs while leaving blind-spot pairs in plain MXFP4 without any outlier extraction. Similarly, the Tseng et al. + PA arm applies the same pattern-aware placement to Tseng's random Hadamard transform and stochastic rounding. Simply omitting rotation on blind-spot pairs yields a notable improvement because IHT fails to reduce blind-spot error and can even increase it on some pairs, as shown in Figure 6.

In contrast, the full AdaHOP-Lv1 policy follows Table 2 across all computation paths. Non-blind-spot pairs receive IHT alone, while the blind-spot pairs RN, RC, NC, and CC receive outlier extraction along the designated axis together with IHT on the residual tensor. Keeping IHT on the residual is beneficial because once OE removes the dominant rows or columns that inflate the shared scales, the remaining error is governed by within-block imbalance that IHT effectively smooths. When we compute the BF16 FLOPs share using the true matrix dimensions of each layer, the count-matched Random OE baseline actually consumes slightly more BF16 FLOPs than AdaHOP-Lv1 while suffering a much larger loss gap, as shown in Table R1 below. Finally, the threshold κ\kappa governs when CC pairs are promoted to full BF16 computation. In our experiments, we evaluate the two endpoints of this rule, namely κ=∞\kappa=\infty (Lv1, where no CC pairs are promoted) and κ=0\kappa=0 (Lv2, where all CC pairs are promoted), and we will report the per-model CC counts in the appendix.

W2, Q3. Equal-budget baselines over 3 seeds

To verify that AdaHOP's advantage holds over simple fixed rules and random selection under an equal compute budget, we evaluated the three baselines defined in Table R1 across three random seeds.

As shown in Table R1, Random index OE matches AdaHOP-Lv1 in GEMM locations, extraction directions, and exact BF16 FLOPs, yet it nearly triples the three-seed loss gap. This confirms that FOID's variance-based index selection is essential for capturing the true outlier support. Furthermore, while Path rule A is a strong heuristic with a comparable BF16 budget, it misses the blind-spot GEMMs on the input-gradient (∇X\nabla X) path and extracts along the suboptimal axis on certain weight-gradient (∇W\nabla W) layers. Consequently, AdaHOP-Lv1 outperforms Path rule A by a margin that exceeds the across-seed standard deviation, just as its margin over HALO exceeds four times the largest across-seed standard deviation in Section 6.3.

[Table R1] Training comparison under the same BF16 budget (Qwen3-0.6B, C4 1B tokens, 3 seeds, gap to BF16 averaged over the last 100 steps, ±\pm is the 3-seed std of each arm)

ArmDefinitionBF16 FLOPs (%)3-Seed Gap Mean ±\pm Std
Path rule AOE-R on every ∇W\nabla W, IHT elsewhere1.6670.0294 ±\pm 0.0065
Count-matched Random OESame setting as the Table 10 arm2.0080.0538 ±\pm 0.0082
Random index OESame GEMMs and operands as AdaHOP, rr random indices1.8150.0495 ±\pm 0.0074
AdaHOP-Lv1Policy of Table 21.8150.0175 ±\pm 0.0077

Q4. Memory reduction and the forward-cache axis

We resolve the quantization axis mismatch between the forward and backward passes by re-quantizing the activation along the token dimension during the forward pass. Specifically, the forward GEMM quantizes the activation XX along the input channel dimension dind_{\text{in}}, whereas the weight-gradient GEMM GW=GY⊤XG_W=G_Y^\top X reduces along the token dimension. Rather than keeping a full-precision copy of XX to handle this transpose, our forward pass frees the dind_{\text{in}}-quantized tensor immediately after the forward multiplication and, within the same pass, applies IHT and MXFP4 quantization to XX along the token axis to cache for the backward pass. For GEMMs that use OE-R, we additionally cache only the compact [M,64][M, 64] BF16 outlier slice and its indices, ensuring that a full BF16 activation tensor is never stored.

Regarding the 3.6× memory reduction reported in Figure 1a, this ratio measures the activation memory stored for backpropagation, excluding model weights, gradients, and optimizer states. Because all compared methods in our Torchtitan setup use selective activation recomputation for the nonlinear regions, the stored activations are dominated by the inputs to the linear projections. Compressing these linear inputs from BF16 to MXFP4 therefore drives the overall activation memory savings, as broken down in Table R2. We will add this detailed memory breakdown and the selective recomputation configuration to Appendices H and L.

[Table R2] Activation memory breakdown (Llama3.1-8B, BS 8, seq 4096, 8-way FSDP, selective recomputation of the nonlinear regions, GB/GPU, based on Figure 1a of the manuscript)

MethodActivation memory (GB)Source of the increase over the previous row
BF1676.00Linear inputs stored in BF16
MXFP4 + Had.20.60Linear inputs stored in MXFP4
AdaHOP-Lv120.94+ 0.34 of r=64r=64 BF16 slices for OE GEMMs
AdaHOP-Lv228.04+ 7.10 of BF16 inputs for the CC GEMMs promoted to BF16
Reviewer_TB4hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AdaHOP, an MXFP4 training framework that addresses the scaling-axis blind spot of Hadamard rotations, where certain outlier patterns remain difficult to quantize accurately. AdaHOP combines Inner Hadamard Transform (IHT) with pattern-aware Outlier Extraction (OE), selectively computing dominant outlier rows or columns in BF16. Experiments on 1B–8B LLMs demonstrate near-BF16 accuracy with 1.45× training speedup and 3.6× memory reduction.

Critical strengths
  1. Meaningful theoretical contribution. The paper provides an insightful analysis of the operand characteristics in FPROP, WGRAD, and DGRAD GEMMs, identifying the scaling-axis blind spot that limits the effectiveness of Hadamard transforms. Based on this observation, the authors propose pattern-aware Outlier Extraction (OE) to reduce the training loss gap between MXFP4 and BF16.

  2. Convincing empirical evaluation. The experiments across multiple model architectures and scales, together with extensive ablation studies, provide strong evidence for the effectiveness of the proposed approach under the evaluated settings.

Critical weaknesses
  1. Unclear efficiency advantage over FP8. AdaHOP introduces additional computational overhead, including Outlier Extraction, Hadamard transforms, and high-precision correction. While the proposed method achieves higher training throughput than BF16, it remains unclear whether it offers a meaningful efficiency advantage over well-optimized FP8/MXFP8 training.

  2. Limited novelty of Outlier Extraction. TetraJet-v2 has already introduced a similar approach, called OutControl, to preserve outlier accuracy through selective high-precision computation. The main novelty of AdaHOP appears to lie in its adaptive, GEMM-aware treatment of different outlier patterns. However, the incremental benefits over existing outlier-control techniques are not directly evaluated.

Questions for authors
  1. Efficiency compared with FP8/MXFP8. AdaHOP achieves better accuracy than the evaluated FP4 baselines and approaches BF16 training quality. However, optimized FP8/MXFP8 training can also achieve near-BF16 accuracy. Could the authors provide a theoretical complexity analysis or optimized kernel benchmarks to estimate whether AdaHOP offers meaningful end-to-end speedups over FP8/MXFP8, accounting for its additional computational overhead?

  2. Ablation of IHT across GEMM paths. Prior studies using FP4 E2M1, such as TetraJet-v2 and NVIDIA's NVFP4 training recipe, have reported that applying Random Hadamard Transforms (RHT) to FPROP can degrade training quality under their experimental settings. Have the authors conducted similar ablation studies to evaluate the individual contributions of IHT on FPROP, WGRAD, and DGRAD? In particular, does applying IHT to all three GEMMs consistently improve training quality compared with selectively applying it to specific GEMMs?

3 comments

W1, Q1. Overhead vs. FP8/MXFP8: theory, kernels, and end-to-end

We thank the reviewer for raising this central question. Even when every auxiliary operation is included, AdaHOP trains 1.12× faster end-to-end than an optimized FP8 baseline while consuming 2.8× less activation memory.

Theory. The auxiliary operations outside the main matrix multiplication add negligible compute. On the two representative Llama3.1-8B GEMM shapes, the thin BF16 correction GEMM, IHT, quantization, FOID, and scatter-add together account for less than two percent of the main GEMM operations. Almost all of this small overhead comes from the thin BF16 GEMM over the extracted rows or columns, whereas IHT and FOID scale linearly with the matrix area rather than with the full M×N×KM\times N\times K product. Because peak MXFP4 throughput on MI350X is twice that of FP8, the tensor-core lower bound of the MLP GEMM stays at about half that of FP8 even with this overhead added serially.

Kernels. Our hardware-aware design hides this correction cost in practice. We benchmarked hipBLASLt FP8 and plain MXFP4 GEMMs on the same MI350X hardware using the dimensions of Table 12, as summarized in Table R1. Because our kernel interleaves BF16 outlier tiles with MXFP4 residual tiles across Compute Units, the fused AdaHOP GEMM runs at nearly the same speed as a plain MXFP4 GEMM, confirming that the BF16 correction latency is almost entirely hidden. When we include the memory-bound preprocessing stages such as IHT and quantization, the full AdaHOP pipeline is still 1.28× faster than FP8 on the large MLP projections. On the smaller attention projections, the fixed preprocessing share is proportionally larger and makes the full pipeline slightly slower than FP8, but the MLP layers dominate overall model FLOPs and drive the net speedup.

[Table R1] GEMM kernel latency comparison (MI350X, ms, same setting as Table 12)

GEMMBF16FP8MXFP4 GEMM without transformsAdaHOP Fused GEMM (full pipeline)
MLP4.762.851.621.65 (2.22)
Attention1.050.760.610.64 (0.84)

End-to-end. This kernel efficiency translates directly into faster and lighter training. Under the settings of Figure 1a using Torchtitan's standard torchao float8 configuration, AdaHOP-Lv1 achieves a 1.12× higher token throughput than FP8 while cutting activation memory by nearly three-fold and maintaining near-BF16 accuracy. We will include this FP8 baseline in Figure 1a of the revised manuscript.

W2. Novelty over TetraJet-v2 OutControl

We appreciate this insightful comparison. While both frameworks isolate outliers into higher precision, they differ fundamentally in how and where that precision is allocated. TetraJet-v2's OutControl routes large-magnitude outliers using static rules tied to the tensor type. In contrast, AdaHOP identifies the geometric reason why Hadamard rotation fails, which we formalize as the scaling-axis blind spot, and adaptively decides which operand to extract from for each individual GEMM. This per-GEMM directional choice is essential for recovering accuracy. As shown in Table 4 of our manuscript, extracting along the wrong axis is ineffective even within the same pattern pair: right-side outlier extraction (OE-R) outperforms left-side extraction (OE-L) on RC and CC pairs, whereas only OE-L is effective on RN pairs. Because a static tensor-type rule cannot adapt to these per-GEMM directional changes, it leaves substantial blind-spot error unaddressed.

However, a direct numerical comparison with OutControl would not be like-for-like. OutControl is built on TetraJet-v2's NVFP4-specific double-block quantizer, which pairs 16-element blocks and E4M3 scales with an additional FP32 outer scale, whereas AdaHOP targets MXFP4 with 32-element blocks and power-of-two E8M0 scales, and the scaling-axis blind spot depends directly on this block geometry. OutControl also keeps 10% of activation channels in FP8, a far larger high-precision share than the 1.8% BF16 FLOPs of AdaHOP-Lv1, so matching budgets would require changing its own configuration. We therefore test the principle behind it, a static rule tied to the tensor or path type, within our format and budget through Path rule A, which uniformly applies OE-R to every GEMM on the blind-spot-dominated weight-gradient path (∇W\nabla W) at a comparable BF16 compute budget. Across three random seeds, AdaHOP achieves a lower training loss gap than Path rule A (detailed in our responses to Reviewers teQ5 and sFYX), and we will highlight this result in the revision while citing and discussing TetraJet-v2 in Section 2 and Table 14.

From a systems perspective, our structural extraction also enables a more hardware-friendly execution path. Rather than producing irregular sparse tensors, extracting a fixed number of rows or columns yields dense, thin matrices. This regular structure allows our fused kernel to interleave BF16 outlier tiles directly with MXFP4 residual tiles across CDNA4 Compute Units in a single kernel launch (Table 12).

Q2. Per-path IHT ablation

We thank the reviewer for this constructive question. Our experiments confirm that applying IHT selectively is indeed superior to applying it uniformly, and they further show that the optimal unit of selection is the GEMM operand pair rather than the computation path. In the ablation study of Table 10, the Pattern-aware IHT baseline applies IHT only to non-blind-spot operand pairs while leaving blind-spot pairs in plain MXFP4 without any BF16 extraction. By isolating the effect of removing rotation where it fails, this baseline nearly halves the training loss gap of Uniform IHT without using a single BF16 operation. Skipping IHT on blind-spot pairs improves training because rotation along the scaling axis cannot reduce their scale-driven error and can even amplify it (Figure 6).

The path-level sensitivity reported in prior work follows naturally from how these operand pairs distribute across the three computation paths (Table 15). The forward pass (FPROP) contains no blind-spot GEMMs and is dominated by CN pairs, where IHT substantially reduces quantization error (Table 4). Consequently, we observe no degradation from applying IHT on FPROP in MXFP4, although formats like NVFP4 with smaller 16-element blocks and E4M3 scales may behave differently. Conversely, on the ∇W\nabla W path, tensor transposition turns nearly every GEMM into a blind-spot pair, meaning rotation alone is largely ineffective there. Crucially, the input-gradient path (∇X\nabla X) mixes non-blind-spot CN and NN pairs with blind-spot RN pairs depending on layer depth. Because both regimes coexist within the same backward pass, no uniform path-level rule can separate them, whereas operand-pair selection can treat each pair according to its pattern. We will expand this discussion in the final manuscript.

Thank you for the detailed response.

The reported 1.12× end-to-end speedup over FP8 on MI350X addresses my efficiency concern, and the IHT ablation supports selecting rotations by operand pair. My remaining concern is the incremental benefit of adaptive outlier extraction. The comparison with a fixed WGRAD rule shows that the extraction choice matters, but it leaves open how much AdaHOP gains over an OutControl-style policy adapted to the same MXFP4 format and compute budget. I therefore maintain my rating.

Comments on the paper

1 comment

General Response to All Reviewers

We sincerely thank all reviewers (TB4h, teQ5, and sFYX) for their careful reading and constructive feedback. Your comments helped us complete the efficiency comparison, strengthen the controls behind our main claim, and make the theoretical analysis more precise. We have uploaded a revised version of the paper, in which all changes are marked in blue. Below we summarize the main revisions by topic and indicate the reviewers whose comments motivated each change.

Summary of Revisions

1. End-to-end and kernel comparison with FP8 (TB4h)

We add an FP8 baseline to the end-to-end efficiency comparison on Llama3.1-8B, measured with Torchtitan's torchao float8 configuration under the same setting as Figure 1a (Appendix L, Table 15). AdaHOP-Lv1 trains 1.12× faster than FP8 while storing 2.8× less activation memory. We also add hipBLASLt FP8 and plain MXFP4 GEMMs to the kernel latency breakdown (Table 14), showing that the fused AdaHOP GEMM takes only 0.03 ms longer than a plain MXFP4 GEMM and that the full AdaHOP pipeline is 1.28× faster than the FP8 GEMM on the MLP projection. We also describe the operation count of the auxiliary stages and the measurement setting, including selective activation recomputation (Appendix H).

2. Relation to TetraJet-v2 and the NVFP4 recipe (TB4h, teQ5)

We cite and discuss TetraJet-v2 and the NVFP4 pretraining recipe in Section 2 and add both methods to the comparison of decision units (Table 17). We also describe how OutControl relates to AdaHOP and measure a static rule tied to the tensor or path type under MXFP4 as Path rule A (see item 3).

3. Selective high-precision baselines compared by BF16 FLOPs over three seeds (teQ5, sFYX)

We add a comparison of selective high-precision policies by their actual BF16 FLOPs share over three seeds (Section 6.3, Table 12). Random index OE keeps AdaHOP-Lv1's GEMMs, operands, and BF16 FLOPs and draws only the extracted indices at random, which gives a 2.8× larger loss gap, and Path rule A, which applies OE-R to every weight-gradient GEMM, gives a 1.7× larger gap at a comparable budget. We also report the BF16 FLOPs of the count-matched Random OE in Table 10 and define the Pattern-aware IHT and Tseng et al. + PA arms (Appendix J).

4. Comparison with Quartet on a larger model (sFYX)

We add a comparison with Quartet on Llama3.2-1B under a 2B-token precision simulation that shares the optimizer, schedule, and data across methods (Table 11). AdaHOP-Lv1 reduces Quartet's loss gap to BF16 by 2.6×.

5. Precision of the theoretical analysis (teQ5)

We correct the statement and proof of Proposition 1 so that the random sign diagonal precedes the Hadamard mixing (ADHbADH_b), and state that the deployed kernel executes the fixed transform A↦AHbA \mapsto AH_b, whose bound is the fixed-HbH_b case under conditions C1 and C2 (Appendix K). We revise Proposition 4(c) to state that ϵq∥A∥F\epsilon_q\Vert A\Vert_F is a floor on the upper bound that no orthogonal transform can lower, and clarify in Table 13 why the untransformed patterns share the same normalized bound.

6. Clarification of memory reporting and per-path behavior (teQ5, TB4h)

We clarify throughout the paper that the 3.6× memory reduction refers to the activation memory stored for the backward pass, and decompose this memory for AdaHOP-Lv1 and Lv2 (Appendix L). We also point from the precision composition of Lv2 to the per-model CC counts in Table 1 (Appendix N). Finally, we expand the discussion of how the operand pairs distribute across the forward, weight-gradient, and input-gradient paths, which explains the path-level sensitivity reported in prior work (Appendix O).

Closing

We are grateful again to all reviewers for the time and effort they devoted to our submission. Your feedback substantially improved the clarity and completeness of the paper, and we would be happy to address any further questions during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.