acceptodds
Under review as a conference paper at ICLR 2027

Hadamizer: Spatio-Temporally Adaptive Hadamard Rotation for Low-Precision Distributed Optimization

Abstract

Large Transformer models have become increasingly vital across numerous applications, yet training them is fundamentally bottlenecked on high-bandwidth memory (HBM) with 32-bit optimizer states occupying a dominant share. Quantization is a key technique to alleviate this pressure, but standard coordinate-wise quantization suffers from severe clipping errors caused by heavy-tailed outliers that widen the quantization range. While orthogonal rotations effectively disperse outliers, mixing coordinates across distributed systems makes collective communication prohibitively expensive. Here, we propose HADAMIZER, an optimizer-native orthogonal rotation framework designed for distributed training on accelerator meshes. Spatially, it aligns rotation blocks with local shard boundaries to avoid rotation-induced collectives. Temporally, HADAMIZER adapts the randomization of Hadamard transformations to tensor dynamics: target-adaptive keying employs static keys for persistent optimizer states and dynamic keys for stochastic gradients, preserving preconditioning with minimal quantization errors. Evaluated on TPUv5e meshes, HADAMIZER achieves near-full-precision-equivalent results on various pre-training and fine-tuning tasks, while delivering up to 9.58% higher end-to-end training throughput over the current rotation-based training frameworks.

Primary area:
optimization
Keywords:
Large Language Models, Quantization, Distributed Training, Hadamard Rotation, Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n7P7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Hadamizer, which applies a block-diagonal randomized Hadamard rotation before INT8 quantization in FSDP training, keeping each block inside a device's shard so that the rotation needs no extra communication. It comes in two separate variants. Hadamizer-Grad uses a fresh rotation key every step and quantizes gradients for the reduce-scatter. Hadamizer-State uses one fixed key and stores the Adam states in the rotated basis at low precision. Experiments with Gemma3 and Qwen2.5 models (270M–3B) on a TPU v5e mesh, covering C4 pre-training and MTNT fine-tuning, are used to support the claim of near-full-precision quality together with up to 9.58% higher throughput than HALO.

Critical strengths

The problem is practically relevant, and the method is lightweight. Rotation blocks stay within the local shard and the sign vectors are regenerated locally from a key, so the rotation itself requires no extra communication. Tables 4 and 10 show lower kernel latency than global rotation in every configuration, which makes the method easy to integrate into existing FSDP training. Designing the key policy around the different lifetimes of gradients (restored within a step) and optimizer states (accumulated across steps) is also a reasonable starting point. I consider this the most valuable part of the paper.

Critical weaknesses

The main claims are not supported by the results. The abstract and conclusion state that Hadamizer delivers near-full-precision quality together with up to 9.58% higher throughput, but these two results come from different variants. The throughput gain comes from Hadamizer-Grad, whose pre-training CE is 6.8–11.3% higher than FP32 Adam (Table 8) and whose fine-tuning CE is 3.2–6.0% higher (Table 9). Hadamizer-State, the variant with good quality, has no throughput advantage over FP32 Adam, and a version combining the two is not implemented (Sec. 4.1). In addition, the numbers in Figure 2 and Sec. 4.2 match the τ=2M rows of Table 5, whereas Sec. 4.1, Table 3, and the Gemma3-270M entries of Table 8 correspond to τ=1M. At τ=1M, both variants are slower than FP32 Adam: 168.79 ms for State and 159.97 ms for Grad, vs. about 155 ms for Adam in Figure 2. The central claim of the paper, efficiency gains without quality loss, is therefore not demonstrated.

Key comparisons are missing. Hadamizer-State is not compared with the existing optimizer-state quantization methods listed in Table 1, such as the block-wise 8-bit optimizer of Dettmers et al. (2021), which uses no rotation and reports 32-bit-level performance, or the 4-bit optimizer of Li et al. (2023). The gradient path is not compared with block-wise INT8 quantization without rotation either, so it is unclear how much rotation adds on top of existing block-wise quantization. On the temporal side, the advantage of the fixed key is shown only against a configuration that does not transform the accumulated states into the new basis. The paper itself calls this configuration a faulty mapping, so this design choice lacks a meaningful baseline. These two points concern the paper's core premise and one of its main contributions, and neither is currently validated.

Questions for authors
  1. The numbers in Figure 2 and Sec. 4.2 match the τ=2M results in Table 5. However, Sec. 4.1 and Table 3 state that Gemma3-270M uses τ=1M, and Table 8 also reports the τ=1M results. Which τ is used for the main results? And which tensors are actually rotated at τ=2M?
  2. The quantizer uses a per-block absmax scale, so each device has its own scales. How are the quantized values from different devices aggregated in the INT8 reduce-scatter? Also, Sec. 3.1 says that a scale table is never exchanged or stored. How does that fit with per-block scaling?
  3. According to Eq. (3), Gemma3-1B (ℓ=432) and Qwen2.5-0.5B (ℓ=304) should use b=256, but Table 4 highlights b=128 for both models. How was the block size actually chosen in the experiments?
  4. Table 1 lists the 8-bit optimizer (Dettmers et al., 2021) and the 4-bit optimizer (Li et al., 2023), but neither is included in the experiments. Could you add both under the same setting, together with results for the gradient path using block-wise INT8 quantization without rotation?
  5. The throughput advantage over FP32 Adam only appears at τ=2M, and Grad and State are evaluated separately. Could you fix one τ per model, report CE and step time for all models, and include results with Grad and State used together?
Reviewer_tGRhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Orthogonal rotations enable quantization of gradients and first- and second-moment optimizer states when outliers exist. But on distributed accelerator meshes, this increases communication cost. The paper addresses this by sharing keys for rotations across accelerators. Also, it suggests that due to the different characteristics of gradients and optimizer states, we should use different key selection strategies for each (dynamic keys for gradients, static keys for states). The experiments across diverse model families and tasks illustrate the advantages of Hadamizer in terms of speed and performance.

Critical strengths
  • identification of the challenge (communication cost) in applying orthogonal rotations on distributed accelerator meshes
  • comprehensive justification (both theoretical and empirical) for the choice of dynamic keys for gradients and static keys for optimizer states
Critical weaknesses
  • Limited hyperparameter search: Section 4.2 demonstrates that experiments were performed with lr=4×10−44\times10^{-4}. However, the best learning rates for AdamW and Muon are often different, and this explains why Muon, known to perform better than AdamW, underperforms.
  • Limited baseline: The work's main results do not provide the performance of related works in communication-efficient distributed training. For instance, Dion provides communication-efficient orthogonalization for distributed matrices with the performance of Muon. Together with the limitation above, it seems the benefit of Hadamizer is uncertain.
  • Performance of Hadamizer-Grad-State: While the paper demonstrates that the use of dynamic keys for gradients and static keys for states is optimal, it does not guarantee that the combination of the two techniques will be optimal. In fact, using dynamic keys for gradients and static keys for the first moment can increase quantization error, as the gradient quantized with dynamic keys has to be quantized with a static key to compute the first moment.
Questions for authors

My questions are related to the weaknesses above, but I believe addressing them would substantially change the current version of the paper. I recommend revising the paper and resubmitting it in the next cycle.

  • Would you justify why Muon underperformed AdamW?
  • Would you provide a comparison with Dion3 [1] (in Figure 2)?
  • Would you provide a comparison of the choices {static keys, dynamic keys} for each of the following: gradients, the first moment, and the second moment (8 options in total)?

[1] Dion3: Full-Stack Orthogonal Updates, arxiv 2608.11612

Reviewer_k11qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HADAMIZER, an Adam variant that applies randomized Hadamard rotation to gradients and optimizer states for low-precision distributed training. The spatial component aligns rotation blocks with device-local shard boundaries to eliminate rotation-induced collectives. The temporal component assigns static PRNG keys to persistent optimizer states and per-step dynamic keys to transient gradients. Evaluated on TPUv5e with GEMMA3 and QWEN2.5 models, HADAMIZER-STATE reduces the optimizer memory footprint by 62.5% while slightly improving loss over FP32 Adam; HADAMIZER-GRAD halves gradient communication volume but degrades cross-entropy by 6–11%.

Critical strengths
  1. The engineering motivation is real. Optimizer states dominate HBM usage in large-scale training, and gradient communication is a known bottleneck. Targeting both with a unified rotation framework addresses a practical pain point.
  2. The static-vs-dynamic key ablation (Table 6) is clean and informative. The cross-entropy degradation when keys are mismapped (static key for gradients, dynamic key for states) provides direct evidence for the key schedule design, and the effect sizes are large enough to be convincing.
  3. The rotation kernel latency benchmarks (Table 4, Table 10) across multiple mesh configurations are thorough. The absence of all-to-all collectives in the shard-aligned setting is well documented.
Critical weaknesses
  1. The "spatio-temporal" framing is misleading. "Spatial" here means aligning block boundaries with shard boundaries — an engineering constraint, not a spatial modeling contribution. "Temporal" means choosing different PRNG key refresh rates for different tensors. These are that the paper does not deliver. The method would be more accurately described as "shard-aligned block rotation with per-target key scheduling." As written, the framing inflates what are essentially two pragmatic engineering choices into a conceptual framework.

  2. The core ideas are thin. That rotation blocks should not span device boundaries is a natural consequence of wanting to avoid cross-device communication — not a discovery but a boundary condition. That persistent states need a fixed basis while transient gradients can tolerate per-step re-randomization follows directly from the definition of "persistent" and "transient." The paper formalizes these observations with roofline analysis and concentration bounds, but the underlying insights are close to self-evident upon stating the problem.

  3. HADAMIZER-GRAD's quality degradation is buried in the narrative. Table 8 shows cross-entropy increases of +6.84% to +11.28% across all five models versus FP32 Adam. The abstract claims "near-full-precision-equivalent results," which is true for HADAMIZER-STATE but outright false for HADAMIZER-GRAD. The 9.58% throughput gain over HALO (the headline number) comes from HADAMIZER-GRAD, which simultaneously has 11.20% worse cross-entropy than HALO on the same model. Reporting throughput gains without normalizing for the accompanying quality loss is misleading.

  4. No combined evaluation exists. The paper evaluates HADAMIZER-GRAD and HADAMIZER-STATE in isolation, arguing they target orthogonal resources (communication bandwidth and HBM, respectively). But a production deployment would use both — quantized gradients and quantized states simultaneously. The paper states this limitation without explaining why the combination was not implemented or what technical obstacles prevent it. Without a combined result, the paper describes two halves of a system that has not been demonstrated to work as a whole.

  5. The preconditioning interpretation in Appendix A.3 is suggestive but empirically unvalidated. The paper argues that maintaining second moments in a fixed Hadamard basis produces a block-diagonal preconditioner that avoids outlier-driven over-dampening, and this is offered as the explanation for why HADAMIZER-STATE slightly outperforms FP32 Adam. However, no experiment isolates the preconditioning effect from the quantization effect. A comparison against FP32 Adam with the same block-diagonal preconditioner (without quantization) would disentangle these, but is absent.

  6. All training experiments use INT8. INT4 results are limited to SQNR measurements in Appendix B.3 with no training runs. Given that memory reduction is the paper's central motivation, the absence of INT4 training results leaves unclear how far the approach can be pushed before quality collapses.

Questions for authors
  1. The term "spatio-temporal" carries significant baggage. Could the authors clarify what specific analogy or mechanism from spatio-temporal modeling (in the scientific computing or video understanding sense) applies here? If the term is being used loosely to mean "across space (shards) and time (steps)," would a more precise descriptor be warranted?

  2. What prevents HADAMIZER-GRAD and HADAMIZER-STATE from being combined? If the two modes compose without interference (as Section 3.2 states), what is the technical obstacle to a joint implementation and evaluation?

  3. For the 9.58% throughput claim, could the authors report quality-normalized throughput — i.e., at what throughput does HADAMIZER-GRAD match HALO's cross-entropy? This would give a fairer picture of the efficiency-quality tradeoff.

  4. What happens to HADAMIZER-STATE at INT4 for a full training run? The SQNR numbers suggest a ~24 dB drop from INT8, which is substantial. Training results at INT4 would establish the practical compression ceiling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.