acceptodds
Under review as a conference paper at ICLR 2027

AdasignSGD: Turning 1-Bit Worker Votes into Adaptive Updates

Abstract

Momentum-sign methods reduce synchronization traffic in distributed AI training by transmitting each worker’s local momentum with one bit per coordinate. However, these methods discard local momentum magnitudes and often fail to match the optimization quality of full-precision methods. We introduce AdasignSGD, which uses the majority sign to determine the update direction and the vote margin to set the update magnitude, taking larger steps on coordinates with stronger worker agreement. In C4 language-model pretraining, AdasignSGD outperforms all evaluated 1-bit worker-sign baselines in validation perplexity across 13 model and training-budget settings from 60M to 1B parameters. Its gap to dense AdamW narrows with model scale. At 1B, AdasignSGD closely matches AdamW in validation perplexity while delivering a complete-step speedup over fused AdamW on 32 H200 GPUs. AdasignSGD uses one model-sized momentum state while incurring only 14.1% of FP32 dense-DDP model-sized traffic at 128 workers. On 128 P100 GPUs, AdasignSGD delivers a complete-step speedup over AdamW for the 350M model. Together, these results show that one-bit worker votes can be turned into adaptive updates that combine strong optimization quality with a smaller memory footprint and faster distributed training steps at scale.

Primary area:
optimization
Keywords:
Distributed optimization, 1-bit optimization, LLM pretraining, Communication-efficient training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k8bQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AdasignSGD, which uses the number of positive worker votes to determine the magnitude of a coordinate-wise update instead of keeping only the majority sign. A precomputed lookup table maps vote counts to adaptive scales under a Gaussian worker model. The paper also introduces VoteReduce to aggregate exact counts efficiently. Experiments compare training quality on C4 and measure optimizer memory, communication, and complete-step latency on P100 and H200 systems.

Critical strengths
  1. The method makes useful use of vote-margin information that majority voting discards, while keeping only one model-sized momentum state.
  2. The lookup table is small and precomputed, and VoteReduce provides a concrete implementation for collecting exact vote counts without communicating full gradients.
  3. The paper evaluates a broad model-budget grid with method-specific tuning and reports complete-step timing, including the smaller-model H200 cases where fused AdamW remains faster.
Critical weaknesses
  1. The quality and speed results are measured under substantially different configurations. The quality experiments use eight workers, 64 sequences per worker, and sequence length 256, with BF16 forward/backward computation. The systems experiments instead use one sequence per worker per optimizer update and FP32 computation. These are valid measurements separately, but combining them does not establish that the method reaches the same quality faster in one training setup. I would like to see a time-to-target comparison with the same model, batch, precision, worker count, and training schedule.

  2. The practical speedup is sensitive to the workload. Table 44 reports a 1.46x complete-step speedup for the uint8 path at 1B on 32 H200 GPUs, but this uses only 256 tokens per GPU per update. In Table 46, increasing sequence length to 1024 reduces the speedup to 1.14x. Fused AdamW is also faster for the 60M–350M models on H200. The paper acknowledges these results, but larger effective batches and gradient accumulation are needed to establish the useful operating range. The communication reduction is quoted against FP32 gradient traffic; at N=32, the uint8 payload would be about 28% of a BF16 dense-gradient payload, rather than 14%. Mixed-precision computation does not automatically imply BF16 gradient communication, so the actual communicated dtype and the measured performance of a BF16-communication baseline should be stated explicitly.

  3. The theory does not fully characterize the implemented method under the measured worker heterogeneity. The diagnostics show that workers are not identically distributed, while the lookup table is derived from a homogeneous Gaussian model. The Poisson-binomial discussion and the more general perturbation framework address part of this issue, but the practical size of the resulting bias and perturbation floor remains unclear. Good sign-probability prediction alone does not establish that the inferred update magnitudes are accurate. Can the authors quantify the effect of heterogeneity on the lookup-table scales and final training quality?

Questions for authors
  1. Can you report end-to-end time to a common validation target under the same configuration used for the quality comparison, including mixed-precision computation and larger effective batches?
  2. Does the 1B comparison with AdamW hold over multiple seeds? Please report the paired differences as well as the mean and spread.
  3. What are the communication volume and complete-step speedup against a dense baseline communicating BF16 gradients? How do these change with gradient accumulation and worker count?
  4. Can the observed worker heterogeneity be used to estimate the lookup-table bias or instantiate the general perturbation bound? Would a calibration that accounts for worker heterogeneity improve the result?
3 comments

(1): Time to Target under Matched Training Settings

AdaSignSGD reaches tuned AdamW’s final validation loss after 97,000 updates, reducing estimated training time from 37.12 to 31.16 hours: 1.191× speedup and 16.05% less time. We use validation losses from tuned AdamW at every 10,000 steps as targets. For each target, we find the first AdaSignSGD evaluation, recorded every 1,000 updates, whose loss is at or below it. The corresponding records are available as the AdamW training log and AdaSignSGD training log.

Table 5. Time to AdamW validation-loss targets.

AdamW stepTarget lossAdaSignSGD first stepAdaSignSGD lossAdamW time (h)AdaSignSGD time (h)Speedup
60,0002.93334869,0002.92853322.2722.171.005×
70,0002.90474374,0002.90178925.9923.771.093×
80,0002.87768380,0002.87534129.7025.701.155×
90,0002.85262486,0002.85197533.4127.631.209×
100,0002.83134597,0002.83104937.1231.161.191×

(2): Paired Seeds at 1B Pretraining

AdaSignSGD achieves lower endpoint perplexity in all three completed 100k-update pairs, with a mean paired difference of -0.210 PPL. In response to the reviewer’s request, we now provide anonymized W&B curve exports for all seed pairs in Table 6.

Table 6. Paired-seed results for the original 1B/C4 comparison.

SeedAdaSignSGD PPLAdamW PPLΔPPLAdaSignSGD record linkAdamW record link
016.95317.079-0.126[log][log]
116.96917.171-0.202[log][log]
216.94217.244-0.302[log][log]
3—16.994—[log]
Mean ± SD16.955 ± 0.01417.165 ± 0.083-0.210 ± 0.088——

Full-budget multi-seed replication is not the default in LLM pretraining benchmarks. Compute-efficient evaluation instead emphasizes small-model tuning and scaling-law extrapolation of hyperparameters to larger models and longer training budgets, as in Wen et al. [K1, Section 3.4]. LoQT, Fira, and Distributed Lion likewise limit repeated runs due to training costs [K2, K3, K4, K5].

References

[K1] Wen et al., Fantastic Pretraining Optimizers and Where to Find Them, ICLR 2026. Section 1.

[K2] Zhao et al., GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024. Paper.

[K3] Loeschcke et al., LoQT: Low-Rank Adapters for Quantized Pretraining, NeurIPS 2024. Table 1.

[K4] Chen et al., Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint?, NeurIPS 2025. Checklist, item 7 (p. 29).

[K5] B. Liu et al., Distributed Lion for Communication Efficient Distributed Training, NeurIPS 2024. Checklist, item 7 (p. 27).

(3): BF16 Gradient Communication and Workload Sensitivity

We added H200 BF16 and P100 FP16 benchmarks against AdamW with native bucketed DDP overlap. The original abstract and Section 5.3 explicitly define traffic reductions relative to FP32 dense DDP.

H200 BF16. Table 7 reports the 32-GPU results, with speedups of 1.006–1.437× for uint8 and 1.007–1.303× for q-bit.

Table 7. H200 1B BF16 complete-step latency (ms; parentheses: speedup).

GPUsMicrobatchAdamWAdaSignSGD uint8AdaSignSGD q-bit
32177.3353.83 (1.437×)59.37 (1.303×)
32283.6566.08 (1.266×)66.25 (1.263×)
324106.9184.28 (1.269×)84.99 (1.258×)
328116.77111.69 (1.045×)115.99 (1.007×)
3216204.45185.56 (1.102×)180.69 (1.131×)
3232349.43341.14 (1.024×)336.53 (1.038×)
32641367.481359.64 (1.006×)1271.49 (1.075×)

P100 FP16. The added control uses FP16 compute/communication with FP32 parameters and optimizer states. Table 8 reports both timing metrics; the full sweep is provided in FP16 full results and logs.

Table 8. P100 FP16 latency on 128 GPUs (ms; parentheses: speedup).

Timing metricModelFP16 AdamWuint8q-bit
Complete step350M3002.501126.62 (2.67×)1360.40 (2.21×)
Communication path130M1090.3327.2 (3.33×)415.1 (2.63×)

Communication volume. BF16 reduces dense traffic, and VoteReduce reduces it further: at N=32, uint8 and q-bit retain only 28.125% and 21.875% of the dense BF16 payload, respectively. Table 9 gives reductions across worker counts. Gradient accumulation changes how often workers communicate, while our method reduces how much they communicate each time. Gradient-accumulation sweeps are outside the scope of this evaluation.

Table 9. Analytical payload reduction versus dense FP16/BF16 DDP (%).

EncodingN=4N=8N=16N=32N=64N=128
uint871.8871.8871.8871.8871.8871.88
q-bit87.5084.3881.2578.1375.0071.88

(4): LUT Scale Accuracy under Worker Heterogeneity

Different workers can have different means and variances. When we put their momentum values together, the overall distribution is Gaussian. Given the training history, the next-step momentum samples are independent across workers.

The original submission evaluates this setting on actual training trajectories. Section 5.1 and Appendix D test Gaussian predictions separately for each worker, retaining its own momentum history. The observed vote-count variance agrees with the prediction for conditionally independent, non-identical workers to within 0.211%.

The original submission also tests update scales directly. Table 27 compares the original LUT with full-value momenta from the same heterogeneous workers: scale MAE is 0.157–0.161 at N=8 and 0.045–0.046 at N=128. These measurements directly quantify the implemented LUT’s scale accuracy under the heterogeneity observed during training.

The analysis also allows different worker sign probabilities (Proposition 2), while Eq. 19 separates count-estimation error from RMS model mismatch [K6].

References

[K6] AdasignSGD: Turning 1-Bit Worker Votes into Adaptive Updates. ICLR 2027 submission 20809. Proposition 2, Eqs. 19 and 67, Tables 2–4, 13, 27, 44, and 46, and Appendices A, B.1, D, H, and I.

Dear Reviewer k8bQ,

We added paired-seed runs (1,680 H200 GPU-hours) and expanded systems benchmarks (2,000). Could you please review our responses before the discussion closes and let us know whether any concerns remain? Thank you!

Reviewer_einQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaSignSGD transmits only the signs of the momentum terms, while using the count of "votes" to estimate the SNR and adaptively scale the step size; VoteReduce provides an inexpensive way to tally these votes. When applied to LLaMA models (up to 1B parameters), this method achieves performance comparable to AdamW, while maintaining memory and communication costs on par with 1-bit schemes.

Critical strengths
  1. It is an extensive piece that addresses some rather specific issues. For instance, I was curious why the authors assume a Gaussian distribution; the prevailing view in the community is that the noise in stochastic gradients for deep learning tasks actually follows a heavy-tailed distribution. The authors argue that momentum behaves more favorably given the specific characteristics of this noise. This is an interesting hypothesis—or perhaps a fact—that warrants further, separate investigation.

  2. I really liked the explanation of why you can’t just use the sum of signs from clients, and why LUT is necessary instead.

  3. Extensive experiments and ablations. The detailed presentation of the experiments and the specifics of the experimental protocol are also a pro.

Critical weaknesses
  1. The paper is quite difficult to read without the help of AI tools—it’s very long.

  2. The theoretical contribution of the paper is supplementary—both the theory presented in the main body and the convergence proof provided in the appendix. From a theoretical standpoint, the paper does not offer a significant contribution. While this is not a flaw in itself, it implies that the paper should be evaluated more heavily based on its practical value.

  3. I have reservations about experiments conducted solely on language models. What could be the issue here? The authors run these experiments on a very limited budget; it is quite possible that, in the later stages of training, a sign-based method might degrade significantly compared to Adam. Here is what I would recommend:

  • include other domains where training can be carried out to completion.

  • attempt to complete the full training process within the current setup (though I realize this is resource-intensive).

  1. I would recommend that the authors include the total training time in the paper. Currently, it can be reconstructed by knowing the time required for a single communication event, the time for a single computation step, and the total number of steps. This could be explicitly highlighted to better illustrate the scale of the gains achieved.

  2. I would recommend adding a brief experimental comparison explaining why the authors chose LUT specifically. Other activation functions or clipping operators could have been used to achieve a roughly similar effect. It would be interesting to see experimental results for approaches similar to LUT

Questions for authors

See Weaknesses

4 comments

(1): Improving Readability and Navigation

We appreciate your careful reading of the supplementary material. The main paper is self-contained and provides the method and key evidence without consulting the appendix. We have added a clickable table of contents for you to make these details easier to navigate.

Following Wen et al.’s emphasis on method-specific tuning [E1], we present the principal findings in the main text and retain the complete search grids, endpoint losses for 1,108 runs, and ablations of individual design choices in the appendix. The added navigation makes this experimental record easier to consult while preserving the detail needed for reproducibility.

(2): Practical Value and the Role of the Theory

Our analysis quantifies how accurately exact vote counts recover update direction and adaptive magnitude, building on the line of previous work on signSGD (ICML 2018) and majority voting (ICLR 2019) [E2, E3]. Section 4.2 quantifies the finite-worker estimation error, and Appendix A connects this error to descent and stationarity bounds under explicit assumptions [E4].

We evaluate the method through estimation accuracy, training quality, and systems performance. At normalized vote margin 0.25, agreement with the full-value update direction rises from about 83% at 8 workers to at least 99.9% at 128 workers (Table 1). Complementing these diagnostics, independently tuned 16-worker sweeps improve endpoint perplexity over the paper’s eight-worker results in general, including a reduction from 21.11 to approximately 19.83 on 350M/1×, supporting the practical relevance of our finite-worker analysis

[∗∗16−workerresultslink∗∗](https://anonymous.4open.science/r/rebuttal−runs−2F4F/AdasignSGD16workerresults.pdf)[**16-worker results link**](https://anonymous.4open.science/r/rebuttal-runs-2F4F/AdasignSGD_16_worker_results.pdf)

. AdaSignSGD achieves the lowest endpoint perplexity among the evaluated one-bit worker-sign methods in all 13 C4 model–budget settings (Table 2). At 1B on 32 H200 GPUs, uint8 achieves a 1.46× complete-step speedup over fused AdamW; the separate 1B memory benchmark shows a 24.4% reduction in peak allocated memory (Section 5.3) [E4].

Gaussian Approximation for Worker Momentum

We appreciate your recognition of the potentially different statistical behavior of raw gradients and momentum, and agree that understanding this distinction more deeply is a valuable research direction in its own right. Our Gaussian approximation is made for the EMA momentum rather than the raw stochastic gradients, and we test it empirically: momentum reduces excess kurtosis by 94.9–96.7% relative to raw gradients (Figure 4a; Appendix C.2), and the sign-based SNR estimate closely tracks the full-value SNR (Figure 4c). In the revision, we will point to this evidence more clearly in the main text and mention heavy-tailed gradient noise as a direction for future work.

References

[E1] Wen et al., Fantastic Pretraining Optimizers and Where to Find Them, ICLR 2026. Abstract and Section 3.

[E2] Bernstein et al., signSGD: Compressed Optimisation for Non-Convex Problems, ICML 2018. Theorems 1–3.

[E3] Bernstein et al., signSGD with Majority Vote Is Communication Efficient and Fault Tolerant, ICLR 2019. Theorems 1–2.

[E4] AdasignSGD: Turning 1-Bit Worker Votes into Adaptive Updates. ICLR 2027 submission 20809. Sections 3–5, Tables 1–2, Appendices A, B, D, and J.

(3): Full Training on a Non-Language Task

AdaSignSGD primarily targets communication-efficient training of large models across large GPU clusters. Following the reviewer’s suggestion, we added a full 200-epoch ResNet-18 comparison on CIFAR-100. AdaSignSGD achieves 72.44% validation accuracy, compared with 73.00% for AdamW. The results, configurations, and training curves are available in CIFAR-100 results and training logs.

(4): Longer-Budget Training and Total Training Time

Table 2 of the original submission reports 1×, 2×, 4×, and 8× Chinchilla-budget evaluations for the 60M, 130M, and 350M models [E4]. Each budget uses its own complete training schedule and final-checkpoint comparison.

In response to your request, we doubled the training budget for the largest model in our quality evaluation (1B), from 100k updates (13.1B tokens) to 200k updates (26.2B tokens). Both additional runs start from scratch with full-horizon schedules and retain the original peak learning rates. AdaSignSGD reaches 16.41 perplexity versus 16.48 for AdamW, retaining lower endpoint perplexity in this comparison after doubling the budget (Table 4).

For the matched 1B configuration, estimated training-loop time over 100k updates is 32.13 hours for AdaSignSGD versus 37.12 hours for AdamW, a 13.5% reduction.

Table 4. 1B/C4 results at the original and doubled budgets.

BudgetAdamW PPLAdamW RunAdaSignSGD PPLAdaSignSGD Run
1× (100k updates)17.08Original paper16.95Original paper
2× (200k updates)16.48[log]16.41[log]

(5): Choice of the Count-to-Scale Mapping

We appreciate the reviewer’s insightful question about our central contribution: mapping vote counts to adaptive update scales. Our Gaussian-derived mapping is supported by the motivation experiments and trajectory diagnostics (Sections 3 and 5.1).

D-Lion-Avg uses a linear count scale, but its authors report worse language-modeling results than D-Lion-MaVo (Majority Vote) [E5, Section 5.2]. This comparison suggests that using vote counts through a linear scale does not necessarily improve language-modeling performance, highlighting the importance of our count-to-scale mapping.

Separately tuned 16-worker runs improve perplexity over eight workers in general, including 350M/1× (21.11 to approximately 19.83)

[∗∗16−workersweepresults∗∗](https://anonymous.4open.science/r/rebuttal−runs−2F4F/AdasignSGD16workerresults.pdf)[**16-worker sweep results**](https://anonymous.4open.science/r/rebuttal-runs-2F4F/AdasignSGD_16_worker_results.pdf)

. These improvements are consistent with our finite-worker analysis, which predicts more accurate vote-based scale estimation as the worker count increases.

References

[E4] AdasignSGD: Turning 1-Bit Worker Votes into Adaptive Updates. ICLR 2027 submission 20809. Sections 3–5, Tables 1–2, Appendices A, B, D, and J.

[E5] B. Liu et al., Distributed Lion for Communication Efficient Distributed Training, NeurIPS 2024. Sections 3 and 5.2 and Appendices A and B.

Thank you to the authors for the response!

I have no further questions.

This paper provides a small, atomic plot that is of interest to part of the community. The paper is well‑structured, although it is somewhat excessive in length. The experiments are more than sufficient compared to other paper that are usually accepted at top‑tier conferences. I am raising my rating.

Dear Reviewer einQ,

Thank you very much for the time and care you have devoted to reviewing our paper and considering our responses. We are delighted that our clarifications and additional experiments addressed your concerns. Your recognition of our contribution and the experimental effort is especially encouraging and means a great deal to us. We sincerely appreciate your thoughtful consideration of the new evidence and your constructive suggestions, which helped us make the paper stronger.

Reviewer_VQoBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AdasignSGD, which uses agreement among workers’ momentum signs to construct adaptive updates while retaining low-bit communication. It provides theoretical analysis, optimization experiments, and systems evaluations of the proposed method.

Critical strengths
  • We find the algorithmic idea elegant. The authors use a clever mathematical connection to recover useful information discarded by conventional low-bit communication methods, supported by substantial theoretical discussion.

  • The analysis suggesting improved estimation with increasing data-parallel worker count is an encouraging signal for scalability.

  • The authors conduct extensive ablations and hyperparameter sweeps. The detailed discussions of finite-worker effects, statistical assumptions, and communication–computation overlap also represent a substantial amount of work.

Critical weaknesses
  1. Dependence on the training-token budget. We wonder whether the method might behave similarly to some sign-based optimizers, whose early advantage over AdamW may diminish or reverse over longer training. The current results show a gap to AdamW for smaller models at longer budgets, while the favorable 1B result uses a relatively shorter budget. Below 1B, the reported 1× settings use approximately 20 tokens per parameter, whereas the 1B setting uses fewer than 10. Calling both “1×” could therefore be confusing. We would appreciate a small-scale experiment examining whether the shorter relative training budget contributes to the favorable 1B result, helping clarify the method’s applicability.

  2. Microbatch size in the timing experiments. The H200 memory capacity would appear to allow microbatch sizes larger than 1, especially at sequence length 256. The quality experiments use a per-GPU microbatch size of 64, whereas the timing experiments use 1. Larger microbatches and longer sequences may provide more opportunities to overlap communication with computation. Although longer-sequence controls are included, they still use microbatch size 1. Additional timing measurements with larger microbatches would therefore help establish how broadly the reported complete-step speedups apply. We understand that smaller clusters may not reproduce the communication bottlenecks of large-scale training; recent external measurements showing substantial exposed all-reduce costs could also provide useful supporting evidence.

  3. Learning-rate search for AdamW. In the 1B setting, AdamW achieves its best result at the lowest tested learning rate, 5e-4, while the reported learning rate for the proposed method is 2.5e-4. We recognize that the update scales are not necessarily comparable. Nevertheless, testing a few smaller AdamW learning rates would strengthen confidence that the comparison is not affected by an optimum at the search boundary.

  4. Relationship to AdamW’s historical second-moment estimate. The method maintains an EMA of local gradients as momentum, then derives adaptive scaling from the current distribution of momentum signs across workers. It does not apply an additional temporal EMA to that scale, whereas AdamW explicitly maintains a historical second-moment estimate. This is not necessarily a disadvantage, but the distinction is interesting and would benefit from further discussion and targeted ablations.

We recognize that resource constraints may limit the experimental configurations. Smaller-scale comparisons or timing-only measurements, without complete training curves, would already help address several of these questions.

Questions for authors

The method combines information across workers within a step with historical information through momentum. How does adaptation based on the current cross-worker statistics differ in practice from adaptation based on explicitly accumulated historical second-moment information?

Roulet, V., & Agarwala, A. Per-example Gradients: a New Frontier for Understanding and Improving Optimizers. ICML, 2026.

3 comments

(1), (3): Longer 1B Pretraining and AdamW Learning-Rate Tuning

We doubled the 1B training budget from 100k updates (13.1072B tokens) to 200k updates (26.2144B tokens). Here, 1× denotes the model-specific reference budget. Our 1× setting follows the established LLaMA/C4 benchmark used by GaLore, LoQT, APOLLO, Fira, and CoLA [1, 2, 3, 4, 5]. Comparable studies also evaluate 1.3B models at 10B tokens [6, 7]. Moreover, Qiu et al. estimate compute-optimal ratios of 9.6 tokens per parameter for Adam and 7.4 for Muon on FineWeb [8], showing that budgets near ten tokens per parameter can already be compute-optimal for a well-tuned Adam baseline.

At the original learning rates (5×10⁻⁴ for AdamW and 2.5×10⁻⁴ for AdaSignSGD), doubling the budget improves AdamW from 17.08 to 16.48 perplexity and AdaSignSGD from 16.95 to 16.41. AdaSignSGD thus retains lower endpoint perplexity in this comparison after doubling the budget.

We also expanded AdamW’s learning-rate search to {1,2,3,4,5}×10⁻⁴ at 2× budgets. These results are consistent with our original claim that AdaSignSGD achieves training quality on par with AdamW.

Table 2. 1B/C4 endpoint perplexity at 100k and 200k updates.

OptimizerPeak LR100k updates: Val. PPL100k updates: Run200k updates: Val. PPL200k updates: Run
AdamW5×10⁻⁴17.08Original paper16.48[log]
AdamW4×10⁻⁴17.01[log]16.45[log]
AdamW3×10⁻⁴16.97[log]16.38[log]
AdamW2×10⁻⁴17.11[log]16.79[log]
AdamW1×10⁻⁴17.85[log]17.24[log]
AdaSignSGD2.5×10⁻⁴16.95Original paper16.41[log]

References

[1] Zhao et al., GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024. Table 2 and Appendix C.1.

[2] Loeschcke et al., LoQT: Low-Rank Adapters for Quantized Pretraining, NeurIPS 2024. Table 1 and Appendix B.1.

[3] Zhu et al., APOLLO: SGD-like Memory, AdamW-level Performance, MLSys 2025. Appendix A.6, Table 11.

[4] Chen et al., Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint?, NeurIPS 2025. Table 3 and Appendix B.1, Table 6.

[5] Liu et al., CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation, EMNLP 2025. Table 4.

[6] Marek et al., Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful, NeurIPS 2025. Figure 6 and Appendix A.1.

[7] Torroba Hennigen et al., On the Duality between Gradient Transformations and Adapters, ICML 2025. Section 4.

[8] Qiu et al., Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales, NeurIPS 2025. Appendix H.5 and Figure 13.

(2): Microbatch Size in the H200 Timing Experiments

We added complete-step benchmarks with larger microbatches, including the 1B configuration used for our training-quality evaluation: eight H200 GPUs, BF16, microbatch 64, and sequence length 256. Under this matched configuration, the uint8 and q-bit paths achieve 1.133× and 1.155× speedups over AdamW, reducing complete-step latency by 11.7% and 13.5%, respectively. We also expanded the timing study to 4–32 H200 GPUs, with microbatches 1–64 for 1B and 1–32 for 2B, 3B, 5B, and 7B, subject to memory limits . Speedups generally narrow as microbatch size increases (see Figure 1 Here). At 32 GPUs and microbatch 1, the 7B speedups reach 1.75–1.78× under BF16 and 1.53–1.56× under FP8-hybrid. Full benchmark results are available Here.

Our systems design targets exposed data-parallel communication in large-model training, when activation-memory or fixed-global-batch limits restrict local work [9, 10]. Single-sequence microbatches appear in OLMo 3 32B pretraining, BLOOM 176B’s batch-ramp phase, DeepSpeed’s 530B throughput benchmark, and Frontier’s 175B/1T scaling benchmarks [11, 12, 13, 14]. ZeRO++ reports greater communication-compression benefits with smaller per-GPU token workloads [15]. Our GPU-count sweep follows weak-scaling practice in MegaScale [16] as we mentioned in the original paper.

Table 3. 1B complete-step latency on H200 GPUs. dense AdamW communicates BF16 gradients. Parentheses report speedup over AdamW.

GPUsMicrobatchAdamW (ms)AdaSignSGD uint8 (ms)AdaSignSGD q-bit (ms)
8641336.421179.96 (1.133×)1156.61 (1.155×)
32177.3353.83 (1.437×)59.37 (1.303×)
32641367.481359.64 (1.006×)1271.49 (1.075×)

Additionally, in Appendix H of the original paper, Tables 39–40 separately benchmark encoding, communication, and decoding; their payload sizes are independent of microbatch size at fixed model and communication settings.

(4): Comparison with AdamW’s Historical Second-Moment Adaptation

AdaSignSGD incorporates gradient history through local momentum EMAs, while adapting to current cross-worker disagreement. For example, worker gradients (1,1,1,1) and (2,2,2,-2) at every step produce identical averaged-gradient histories, so AdamW makes identical updates. AdaSignSGD instead observes 4:0 versus 3:1 momentum votes and reduces the second update.

Uncompressed AdaSignSGD in original submission and AdamW share the same first moment before bias correction, but use different denominators: the cross-worker RMS of local momenta versus a EMA of squared averaged gradients. The sign-only variant estimates the former normalized update from momentum votes. Table 2 compares AdamW and uncompressed AdaSignSGD covering the cross-worker adaptive rule.

References

[9] Hagemann et al., Efficient Parallelization Layouts for Large-Scale Distributed Model Training, COLM 2024. Section 4.3 and Figure 3.

[10] Grattafiori et al., The Llama 3 Herd of Models, 2024. Section 3.3.2 and Table 4.

[11] Allen Institute for AI, OLMo 3 32B Stage 1 Pretraining Configuration. Training run summary and launch configuration.

[12] BigScience Workshop, BLOOM: A 176B-Parameter Open-Access Multilingual Language Model, JMLR 2024. Model report and pretraining chronicle.

[13] DeepSpeed Team, Azure Empowers Easy-to-Use, High-Performance, and Hyperscale Model Training Using DeepSpeed, 2022. Figure 6 (530B).

[14] Dash et al., Optimizing Distributed Training on Frontier for Large Language Models, ISC High Performance 2024. Table V and Section V.C.

[15] Wang et al., ZeRO++: Extremely Efficient Collective Communication for Large Model Training, ICLR 2024. Section 4.2 and the High-BW rows of Table 1(a).

[16] Jiang et al., MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs, NSDI 2024. Figure 9 and Section 6.1.

Dear Reviewer VQoB,

We added longer-budget runs (3,100 H200 GPU-hours), expanded AdamW tuning (1,800), and systems benchmarks (2,000). Could you please review our responses before the discussion closes and let us know whether any concerns remain? Thank you!

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank all three reviewers for the positive feedback on the central idea of using worker-vote counts for adaptive updates, the concrete VoteReduce implementation, and the extensive experiments, ablations, and method-specific tuning in our original submission.

To address the reviewers' concerns, we added longer-budget (2×Chinchilla) 1B pretraining, expanded AdamW tuning, matched-workload time-to-target evaluations, broader systems benchmarks, and CIFAR-100 experiments. We summarize the main new findings below. At 200k updates of 1B pretraining, AdaSignSGD reaches 16.41 perplexity versus 16.48 for AdamW at their original peak learning rates. Expanded AdamW tuning at 100k updates yields a best tested perplexity of 16.97, compared with 16.95 for AdaSignSGD. Under matched 1B/BF16 settings, AdaSignSGD reaches tuned AdamW’s final validation loss with a 1.191× estimated training-loop speedup. A standard-budget 1B pretraining run takes approximately 35 hours on eight H200 GPUs (≈ 280 GPU-hours).

Table 1. Summary of compute budgets.

ExperimentReviewer(s)H200 GPU-hours
1B model pretraining at 2× budgetVQoB, einQ≈ 3,100
Expanded AdamW tuning for 1B pretrainingVQoB≈ 1,800
Paired-seed 1B model pretrainingk8bQ≈ 1,680
BF16/FP8 and 1B-7B model+microbatch benchmarksVQoB, k8bQ≈ 2,000
16-worker pretraining sweepseinQ≈ 1,500
Full CIFAR-100 trainingeinQ≈ 72

The listed experiments represent an estimated budget of approximately 10,000 H200 GPU-hours. Several of these runs, in particular the 2×-budget pretraining and the paired-seed evaluations, each take several days of wall-clock time even on dedicated hardware. This is why our responses arrive later in the discussion period than we would have liked: we chose to post them once the evidence was complete rather than reply earlier with only partial results. We understand this leaves less time for discussion. We would be grateful if the reviewers and the AC could take this into account, and we will gladly answer any follow-up questions promptly in the remaining time.

In the revised manuscript, additions are highlighted in yellow. The new time-to-target results appear in Section 5 and Appendix K (Table 166), and the expanded systems benchmarks in Figure 8 and Appendix K (Table 165).

The responses below address each reviewer’s concerns individually and link the relevant evidence directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.