- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies MXFP4 LLM training and identifies outlier patterns that Hadamard transforms cannot adequately address. AdaHOP calibrates these patterns for individual GEMM operand pairs, combines Hadamard transforms with selective BF16 outlier correction, and updates the selected outlier indices during training. Fused kernels reduce the correction overhead on AMD GPUs. Experiments evaluate training quality, downstream accuracy, and end-to-end training efficiency.
Critical strengths
-
The analysis provides a concrete explanation for why Hadamard transforms are insufficient for certain outlier patterns. Selecting treatments at the GEMM operand-pair level is a reasonable way to address these differences during training.
-
The paper demonstrates practical quality and efficiency benefits. AdaHOP-Lv1 trails BF16 by only 0.23 percentage points in average zero-shot accuracy across five 1B–8B models and achieves a 1.45× end-to-end training speedup on Llama3.1-8B.
Critical weaknesses
-
The quality comparison with Quartet is limited relative to the scope of the main results. Quartet is a closely related FP4 training method, but it is evaluated only on Qwen3-0.6B, while the main quality results cover models from 1B to 8B. Different hardware backends complicate direct speed comparisons, but do not prevent larger-model quality comparisons using the numerical simulation already adopted in this paper. It therefore remains unclear how AdaHOP compares with this strong baseline at larger scales.
-
The ablations provide evidence for pattern-aware assignment, but do not fully establish its advantage over simpler policies with comparable high-precision computation. Table 10 matches the number of GEMMs receiving outlier correction, which does not by itself guarantee equal BF16 FLOPs or runtime when GEMM shapes and extraction sides differ. Since selective high-precision correction is an established approach, comparisons against simple fixed policies under a matched computation budget are needed to determine whether the proposed assignment rule offers a better quality–efficiency trade-off.
I would consider raising my score if these concerns are addressed.
Questions for authors
-
How does AdaHOP compare with Quartet in training quality on a representative larger model under matched settings?
-
Does the proposed assignment rule outperform simpler fixed policies at a comparable BF16 computation budget or measured overhead?
3 comments
Thank you for the additional experiments and clarifications. The larger-model comparison with Quartet and the three-seed comparison against a fixed path-level policy at a comparable BF16 FLOPs budget substantially address my original concerns.
My remaining concern is the generality of the findings. AdaHOP relies on identifying stable outlier patterns and assigning suitable treatments, but it remains unclear whether these patterns and the effectiveness of the assignment policy extend to MoE or hybrid architectures. It would also be valuable to understand whether the observed benefits persist at larger scales and over longer training horizons, where quantization effects may differ from those captured by the current controlled comparisons. I view these as remaining limitations of the evidence rather than demonstrated failures of the method.
We thank the reviewer for reviewing our additional experiments and clarifications and for confirming that they address the original concerns. We also agree that the remaining points reflect limitations in our empirical evidence rather than failures of the method itself. Below we summarize the supporting evidence for each point.
MoE and hybrid architectures. We ran a preliminary check on the public DeepSeek-V2-Lite checkpoint (64 routed experts per layer) to verify that AdaHOP's calibration remains valid for MoE. Applying our 30-step calibration to two disjoint sets of C4 batches yielded 97.2% agreement in detected patterns across all 9,984 routed-expert GEMMs. This high consistency confirms that expert patterns remain stable under the same data distribution despite batch-to-batch token routing variations.
We further trained a small MoE model (Qwen2-MoE architecture, 494M parameters, 32 routed experts per layer) from scratch on C4, and the detected patterns of its routed-expert GEMMs agreed at 96.6% during 3,000 step training. We will include full MoE training results in the camera-ready version since the rebuttal timeframe does not permit full training runs.
In hybrid architectures, state-space components consume only a small fraction of compute and are known to be sensitive to reduced precision. Keeping these components in BF16 is therefore a natural choice. This approach aligns naturally with AdaHOP because our method already retains a small share of sensitive GEMMs in BF16. AdaHOP can then focus on accelerating the attention and MLP GEMMs that dominate the overall compute.
Scale and training length. We agree it is important to examine whether our observed benefits persist at larger scales and over longer training horizons. Across our Chinchilla-optimal training runs from 1B to 8B, AdaHOP delivers consistent gains throughout the entire tested compute range. Even on our largest Llama3.1-8B model trained on 160B tokens, AdaHOP-Lv1 stays within 0.05 points of BF16 in zero-shot accuracy whereas HALO falls 1.31 points behind (Section 6.1, Table 3). Beyond preserving accuracy at scale, AdaHOP becomes more efficient as models grow because the Lv1 BF16 share drops from 0.73% at 1B to 0.45% at 8B (Appendix N, Table 16).
Extending these benefits to longer training horizons relies on the stability of the detected outlier patterns over time. When we re-detect patterns 26% into the Llama3.2-3B run at step 30,000, the results still show a 95.8% agreement with our initial 30-step calibration (Appendix F). This long-term stability suggests that model architecture largely determines these patterns rather than training history. Indeed, our from-scratch weight and gradient patterns almost exactly match those of the public Llama3.2-1B checkpoint despite different training data, optimizers, learning-rate schedules, and token counts (Appendix G, Table 8). Even if the patterns were to drift during an exceptionally long run, periodic re-calibration easily acts as a safeguard at a negligible cost of 0.026% of training FLOPs per invocation (Table 5).
Training beyond 8B exceeded our compute budget, so we have stated validation at these scales as future work in the Limitations of the revision. We thank the reviewer again for the careful review and constructive feedback.
W1, Q1. Comparison with Quartet on a larger model
We sincerely thank the reviewer for this constructive feedback and for the willingness to raise the score if our responses address the concerns. To evaluate Quartet beyond the 0.6B scale within the rebuttal window, we conducted a controlled 2B-token numerical simulation on Llama3.2-1B using the same optimizer, learning-rate schedule, and C4 dataset across BF16, Quartet, and AdaHOP-Lv1. As shown in Table R1 below, AdaHOP-Lv1 reduces Quartet's training loss gap to BF16 by more than two-and-a-half times. As the reviewer noted, while hardware backend differences prevent a direct wall-clock speed comparison, our precision simulation environment provides a like-for-like algorithmic comparison under identical numerical conditions. These new results confirm that the optimization advantage AdaHOP demonstrated at the 0.6B scale transfers consistently to larger models.
[Table R1] Comparison with Quartet on Llama3.2-1B (C4 2B tokens, precision simulation, same optimizer and schedule, loss gap to BF16)
Furthermore, our extended 10B-token evaluation on Qwen3-0.6B in Table 3, which trains ten times longer than the ablation runs in Table 10, demonstrates that a smaller training loss gap translates directly into stronger downstream task performance. Across all four zero-shot benchmarks, AdaHOP-Lv1 consistently outperforms Quartet, reducing the average accuracy drop relative to BF16 to less than a third of Quartet's. We will incorporate the new Llama3.2-1B simulation results into the revised manuscript to provide a thorough comparison across model scales.
W2, Q2. Advantage over simple fixed policies at a comparable BF16 compute budget
We appreciate the reviewer's precise observation that selective high-precision policies must be compared under their actual BF16 FLOPs share rather than merely by matching the number of treated GEMMs. When we compute the BF16 FLOPs on Qwen3-0.6B using the true matrix dimensions of each layer, the count-matched Random OE baseline from Table 10 actually consumes more BF16 FLOPs than AdaHOP-Lv1 because the randomly selected GEMMs happen to have larger average dimensions. Despite using a larger high-precision compute budget, Random OE suffers roughly three times the three-seed loss gap of AdaHOP-Lv1. This confirms that the quality recovery stems from our pattern-aware placement rather than the sheer volume of BF16 computation.
To further test AdaHOP against a strong fixed path-level policy, we evaluated a heuristic called Path rule A. Because blind-spot operands concentrate heavily in the weight-gradient (∇W) pass, Path rule A uniformly applies right-side outlier extraction (OE-R) to every GEMM on the ∇W path while applying standard IHT everywhere else. Although Path rule A consumes nearly the same BF16 FLOPs as AdaHOP-Lv1, it misses the blind-spot GEMMs that appear on the input-gradient (∇X) path and extracts along the suboptimal axis for certain ∇W layers. When evaluated across three random seeds under the identical setup of Table 10, AdaHOP-Lv1 outperforms Path rule A by a margin that exceeds the across-seed standard deviation, as shown in Table R2 below. Adapting at the operand-pair level therefore achieves a superior quality-efficiency trade-off compared to fixed path-level heuristics, and we will add this FLOPs analysis and multi-seed comparison to the final manuscript.
[Table R2] Training comparison under the same BF16 budget (Qwen3-0.6B, C4 1B tokens, 3 seeds, gap to BF16 averaged over the last 100 steps, ± is the 3-seed std of each arm)