- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Hadamizer, which applies a block-diagonal randomized Hadamard rotation before INT8 quantization in FSDP training, keeping each block inside a device's shard so that the rotation needs no extra communication. It comes in two separate variants. Hadamizer-Grad uses a fresh rotation key every step and quantizes gradients for the reduce-scatter. Hadamizer-State uses one fixed key and stores the Adam states in the rotated basis at low precision. Experiments with Gemma3 and Qwen2.5 models (270M–3B) on a TPU v5e mesh, covering C4 pre-training and MTNT fine-tuning, are used to support the claim of near-full-precision quality together with up to 9.58% higher throughput than HALO.
Critical strengths
The problem is practically relevant, and the method is lightweight. Rotation blocks stay within the local shard and the sign vectors are regenerated locally from a key, so the rotation itself requires no extra communication. Tables 4 and 10 show lower kernel latency than global rotation in every configuration, which makes the method easy to integrate into existing FSDP training. Designing the key policy around the different lifetimes of gradients (restored within a step) and optimizer states (accumulated across steps) is also a reasonable starting point. I consider this the most valuable part of the paper.
Critical weaknesses
The main claims are not supported by the results. The abstract and conclusion state that Hadamizer delivers near-full-precision quality together with up to 9.58% higher throughput, but these two results come from different variants. The throughput gain comes from Hadamizer-Grad, whose pre-training CE is 6.8–11.3% higher than FP32 Adam (Table 8) and whose fine-tuning CE is 3.2–6.0% higher (Table 9). Hadamizer-State, the variant with good quality, has no throughput advantage over FP32 Adam, and a version combining the two is not implemented (Sec. 4.1). In addition, the numbers in Figure 2 and Sec. 4.2 match the τ=2M rows of Table 5, whereas Sec. 4.1, Table 3, and the Gemma3-270M entries of Table 8 correspond to τ=1M. At τ=1M, both variants are slower than FP32 Adam: 168.79 ms for State and 159.97 ms for Grad, vs. about 155 ms for Adam in Figure 2. The central claim of the paper, efficiency gains without quality loss, is therefore not demonstrated.
Key comparisons are missing. Hadamizer-State is not compared with the existing optimizer-state quantization methods listed in Table 1, such as the block-wise 8-bit optimizer of Dettmers et al. (2021), which uses no rotation and reports 32-bit-level performance, or the 4-bit optimizer of Li et al. (2023). The gradient path is not compared with block-wise INT8 quantization without rotation either, so it is unclear how much rotation adds on top of existing block-wise quantization. On the temporal side, the advantage of the fixed key is shown only against a configuration that does not transform the accumulated states into the new basis. The paper itself calls this configuration a faulty mapping, so this design choice lacks a meaningful baseline. These two points concern the paper's core premise and one of its main contributions, and neither is currently validated.
Questions for authors
- The numbers in Figure 2 and Sec. 4.2 match the τ=2M results in Table 5. However, Sec. 4.1 and Table 3 state that Gemma3-270M uses τ=1M, and Table 8 also reports the τ=1M results. Which τ is used for the main results? And which tensors are actually rotated at τ=2M?
- The quantizer uses a per-block absmax scale, so each device has its own scales. How are the quantized values from different devices aggregated in the INT8 reduce-scatter? Also, Sec. 3.1 says that a scale table is never exchanged or stored. How does that fit with per-block scaling?
- According to Eq. (3), Gemma3-1B (ℓ=432) and Qwen2.5-0.5B (ℓ=304) should use b=256, but Table 4 highlights b=128 for both models. How was the block size actually chosen in the experiments?
- Table 1 lists the 8-bit optimizer (Dettmers et al., 2021) and the 4-bit optimizer (Li et al., 2023), but neither is included in the experiments. Could you add both under the same setting, together with results for the gradient path using block-wise INT8 quantization without rotation?
- The throughput advantage over FP32 Adam only appears at τ=2M, and Grad and State are evaluated separately. Could you fix one τ per model, report CE and step time for all models, and include results with Grad and State used together?