- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AdasignSGD, which uses the number of positive worker votes to determine the magnitude of a coordinate-wise update instead of keeping only the majority sign. A precomputed lookup table maps vote counts to adaptive scales under a Gaussian worker model. The paper also introduces VoteReduce to aggregate exact counts efficiently. Experiments compare training quality on C4 and measure optimizer memory, communication, and complete-step latency on P100 and H200 systems.
Critical strengths
- The method makes useful use of vote-margin information that majority voting discards, while keeping only one model-sized momentum state.
- The lookup table is small and precomputed, and VoteReduce provides a concrete implementation for collecting exact vote counts without communicating full gradients.
- The paper evaluates a broad model-budget grid with method-specific tuning and reports complete-step timing, including the smaller-model H200 cases where fused AdamW remains faster.
Critical weaknesses
-
The quality and speed results are measured under substantially different configurations. The quality experiments use eight workers, 64 sequences per worker, and sequence length 256, with BF16 forward/backward computation. The systems experiments instead use one sequence per worker per optimizer update and FP32 computation. These are valid measurements separately, but combining them does not establish that the method reaches the same quality faster in one training setup. I would like to see a time-to-target comparison with the same model, batch, precision, worker count, and training schedule.
-
The practical speedup is sensitive to the workload. Table 44 reports a 1.46x complete-step speedup for the uint8 path at 1B on 32 H200 GPUs, but this uses only 256 tokens per GPU per update. In Table 46, increasing sequence length to 1024 reduces the speedup to 1.14x. Fused AdamW is also faster for the 60M–350M models on H200. The paper acknowledges these results, but larger effective batches and gradient accumulation are needed to establish the useful operating range. The communication reduction is quoted against FP32 gradient traffic; at N=32, the uint8 payload would be about 28% of a BF16 dense-gradient payload, rather than 14%. Mixed-precision computation does not automatically imply BF16 gradient communication, so the actual communicated dtype and the measured performance of a BF16-communication baseline should be stated explicitly.
-
The theory does not fully characterize the implemented method under the measured worker heterogeneity. The diagnostics show that workers are not identically distributed, while the lookup table is derived from a homogeneous Gaussian model. The Poisson-binomial discussion and the more general perturbation framework address part of this issue, but the practical size of the resulting bias and perturbation floor remains unclear. Good sign-probability prediction alone does not establish that the inferred update magnitudes are accurate. Can the authors quantify the effect of heterogeneity on the lookup-table scales and final training quality?
Questions for authors
- Can you report end-to-end time to a common validation target under the same configuration used for the quality comparison, including mixed-precision computation and larger effective batches?
- Does the 1B comparison with AdamW hold over multiple seeds? Please report the paired differences as well as the mean and spread.
- What are the communication volume and complete-step speedup against a dense baseline communicating BF16 gradients? How do these change with gradient accumulation and worker count?
- Can the observed worker heterogeneity be used to estimate the lookup-table bias or instantiate the general perturbation bound? Would a calibration that accounts for worker heterogeneity improve the result?
3 comments
(3): BF16 Gradient Communication and Workload Sensitivity
We added H200 BF16 and P100 FP16 benchmarks against AdamW with native bucketed DDP overlap. The original abstract and Section 5.3 explicitly define traffic reductions relative to FP32 dense DDP.
H200 BF16. Table 7 reports the 32-GPU results, with speedups of 1.006–1.437× for uint8 and 1.007–1.303× for q-bit.
Table 7. H200 1B BF16 complete-step latency (ms; parentheses: speedup).
| GPUs | Microbatch | AdamW | AdaSignSGD uint8 | AdaSignSGD q-bit |
|---|---|---|---|---|
| 32 | 1 | 77.33 | 53.83 (1.437×) | 59.37 (1.303×) |
| 32 | 2 | 83.65 | 66.08 (1.266×) | 66.25 (1.263×) |
| 32 | 4 | 106.91 | 84.28 (1.269×) | 84.99 (1.258×) |
| 32 | 8 | 116.77 | 111.69 (1.045×) | 115.99 (1.007×) |
| 32 | 16 | 204.45 | 185.56 (1.102×) | 180.69 (1.131×) |
| 32 | 32 | 349.43 | 341.14 (1.024×) | 336.53 (1.038×) |
| 32 | 64 | 1367.48 | 1359.64 (1.006×) | 1271.49 (1.075×) |
P100 FP16. The added control uses FP16 compute/communication with FP32 parameters and optimizer states. Table 8 reports both timing metrics; the full sweep is provided in FP16 full results and logs.
Table 8. P100 FP16 latency on 128 GPUs (ms; parentheses: speedup).
| Timing metric | Model | FP16 AdamW | uint8 | q-bit |
|---|---|---|---|---|
| Complete step | 350M | 3002.50 | 1126.62 (2.67×) | 1360.40 (2.21×) |
| Communication path | 130M | 1090.3 | 327.2 (3.33×) | 415.1 (2.63×) |
Communication volume. BF16 reduces dense traffic, and VoteReduce reduces it further: at N=32, uint8 and q-bit retain only 28.125% and 21.875% of the dense BF16 payload, respectively. Table 9 gives reductions across worker counts. Gradient accumulation changes how often workers communicate, while our method reduces how much they communicate each time. Gradient-accumulation sweeps are outside the scope of this evaluation.
Table 9. Analytical payload reduction versus dense FP16/BF16 DDP (%).
| Encoding | N=4 | N=8 | N=16 | N=32 | N=64 | N=128 |
|---|---|---|---|---|---|---|
| uint8 | 71.88 | 71.88 | 71.88 | 71.88 | 71.88 | 71.88 |
| q-bit | 87.50 | 84.38 | 81.25 | 78.13 | 75.00 | 71.88 |
(4): LUT Scale Accuracy under Worker Heterogeneity
Different workers can have different means and variances. When we put their momentum values together, the overall distribution is Gaussian. Given the training history, the next-step momentum samples are independent across workers.
The original submission evaluates this setting on actual training trajectories. Section 5.1 and Appendix D test Gaussian predictions separately for each worker, retaining its own momentum history. The observed vote-count variance agrees with the prediction for conditionally independent, non-identical workers to within 0.211%.
The original submission also tests update scales directly. Table 27 compares the original LUT with full-value momenta from the same heterogeneous workers: scale MAE is 0.157–0.161 at N=8 and 0.045–0.046 at N=128. These measurements directly quantify the implemented LUT’s scale accuracy under the heterogeneity observed during training.
The analysis also allows different worker sign probabilities (Proposition 2), while Eq. 19 separates count-estimation error from RMS model mismatch [K6].
References
[K6] AdasignSGD: Turning 1-Bit Worker Votes into Adaptive Updates. ICLR 2027 submission 20809. Proposition 2, Eqs. 19 and 67, Tables 2–4, 13, 27, 44, and 46, and Appendices A, B.1, D, H, and I.
Dear Reviewer k8bQ,
We added paired-seed runs (1,680 H200 GPU-hours) and expanded systems benchmarks (2,000). Could you please review our responses before the discussion closes and let us know whether any concerns remain? Thank you!
(1): Time to Target under Matched Training Settings
AdaSignSGD reaches tuned AdamW’s final validation loss after 97,000 updates, reducing estimated training time from 37.12 to 31.16 hours: 1.191× speedup and 16.05% less time. We use validation losses from tuned AdamW at every 10,000 steps as targets. For each target, we find the first AdaSignSGD evaluation, recorded every 1,000 updates, whose loss is at or below it. The corresponding records are available as the AdamW training log and AdaSignSGD training log.
Table 5. Time to AdamW validation-loss targets.
(2): Paired Seeds at 1B Pretraining
AdaSignSGD achieves lower endpoint perplexity in all three completed 100k-update pairs, with a mean paired difference of -0.210 PPL. In response to the reviewer’s request, we now provide anonymized W&B curve exports for all seed pairs in Table 6.
Table 6. Paired-seed results for the original 1B/C4 comparison.
Full-budget multi-seed replication is not the default in LLM pretraining benchmarks. Compute-efficient evaluation instead emphasizes small-model tuning and scaling-law extrapolation of hyperparameters to larger models and longer training budgets, as in Wen et al. [K1, Section 3.4]. LoQT, Fira, and Distributed Lion likewise limit repeated runs due to training costs [K2, K3, K4, K5].
References
[K1] Wen et al., Fantastic Pretraining Optimizers and Where to Find Them, ICLR 2026. Section 1.
[K2] Zhao et al., GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024. Paper.
[K3] Loeschcke et al., LoQT: Low-Rank Adapters for Quantized Pretraining, NeurIPS 2024. Table 1.
[K4] Chen et al., Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint?, NeurIPS 2025. Checklist, item 7 (p. 29).
[K5] B. Liu et al., Distributed Lion for Communication Efficient Distributed Training, NeurIPS 2024. Checklist, item 7 (p. 27).