acceptodds
Under review as a conference paper at ICLR 2027

Robustifying Asynchronous SGD via Soft Throttling

Abstract

Asynchronous SGD is a popular algorithm for distributed learning where each client's gradient update is applied on arrival. This leads to a speed-up, but also an increased vulnerability to attacks, as fast clients can dominate the total update. We introduce Throttle, a Byzantine-robust generalization of asynchronous SGD where the key idea is to exponentially down-weight updates from faster clients by a factor . Both asynchronous SGD () and synchronous Byzantine-robust SGD () correspond to specific settings of Throttle. We provide a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. Remarkably, our experiments show that this down-weighting mechanism can also improve performance over standard asynchronous SGD even in the non-Byzantine setting.

Primary area:
optimization
Keywords:
Distributed Learning, Asynchronous SGD, Optimization, Byzantine-robust Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_RM1bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors present a new algorithm that aims to make asynchronous SGD robust to Byzantine clients. The algorithm bridge asynchronous SGD and synchronous Byzantine-robust SGD via a parameter qq. This parameter qq, central to their method, serves to downweight exponentially a client's step size by 1/qc1/q^c for repeated updates within the same round. It then synchronize all clients at the end of a round. They claim robustness by providing a convergence rate upper bound and evaluate the method empirically against known Byzantine attacks and other algorithms tackling the same problem. Interestingly, they also provide empirical evaluation of the proposed method without Byzantine clients.

Critical strengths

The problem tackled by the authors is very interesting. The soft throttling mechanism is an intuitive idea, and the two-stage virtual sequence is a creative analytical approach

Critical weaknesses

The paper is clearly presented until Section 3, where the contribution is detailed. There, there are presentation flaws:

  • Validity of Theorem 1 (see questions). Indeed, the authors use a random step-size that they falsely treat as constant in the theory part of the paper. This is compounded by the fact that some of the important quantities (specifically the random one: η^t\hat{\eta}_t, r(t)r(t)) are not defined in the main text.
  • In experiments, qq is set quite low when the theory requires high qq. Moreover, the claim "THROTTLE is robust and outperforms existing algorithms in almost all type of attacks." does not match the experiments. Indeed, in every experiment, at least one existing algorithm achieve similar or better accuracy. We acknowledge the fact that momentum might be one of the explanation as important in Byzantine optimization, but then why not directly doing the analysis with momentum as proven necessary for noisy Byzantine optimization? Moreover, the graph is very difficult to read due to the color choices (which change between different graph...): I cannot be sure which is BSGD and which is μ2\mu^2-SGD reading the graph. I would advise the authors to use dotted or dashed lines. Additionally, the Figure 2 would probably gain to stop prior 15 000 simulated wall-clock time as it shadow the interesting first steps. In Figure 3, the authors "averaged over three seeds" but does not report variance.
Questions for authors

Theoretical section.

I have one major question which if not resolved is a major flaw invalidating Theorem 1.

  • η\eta and η^t\hat\eta_t are not defined in Theorem 1, while they are important quantities to analyze the main terms of the bound through θT\theta_T. θT\theta_T is not analyzed or explained. Upon deeper inspection (Appendix C.2, lines 888, 896, and 941), θT\theta_T, NTN_T, TeffT_{\text{eff}} are a random variables while they appear in an expected bound, which is a fundamental flaw. To prove a valid bound, the authors need to prove (1) an almost sure bound that holds for every single possible realization; or (2) show how qq constraints the worst-case θT\theta_T and TeffT_{eff} when a Byzantine adversary tries to flood the server. As stated, the learning rate η\eta of the Theorem 1 depends on the random variable θT\theta_T and TeffT_{\text{eff}}, and hence using variables that depend on the future, which is a flaw invalidating Theorem 1. Here, some instances of this error:
    • In the proof of theorem 1, the line 1192 is invalid because η^t\hat\eta_t is random and correlated with xtx_t.
    • In the appendix, the filtration Ft\mathcal{F}_t, used throughout the proofs, is only defined fuzzily in an optional Lemma 1. More importantly, in Lemma 6 (Line 1076) and Theorem 2 (Line 1152), the authors write the conditional expectation assuming η^t∈Ft\hat\eta_t \in \mathcal{F}_t where this is not true because for t>r(t)t > r(t), η^t=ηnext(t+1,jt)\hat\eta_t = \eta_{next(t+1, j_t)}. Similarly, Line 1188 (Proof of Theorem 1) if false.

Upon answering the validity or invalidity of the main theorem (Theorem 1), I have the following other questions.

  • Why do you choose a diameter-based aggregation compared to (f,κ)(f,\kappa)-robustness which prove better in their problem parameters dependence?
  • The authors rely on assuming the partition between Byzantine and good client is fixed (line 067). I wonder whether Throttle algorithm would hold without this assumption.
  • In Algorithm 1 between processing first arrivals (Line 5) and repeated arrivals (Line 7) requires jumping to the appendix to fully decipher. Why after first arrival the authors only clip the gradient?

Experimental section.

  • Why experiments use different setting than in theory (lines 321-323)? Indeed, the author used λt=1\lambda_t = 1 instead of GG. How the authors expect the result change with this parameter? Why does throttle fail under the Empire attack compared to BASGDm, and why were baselines only tuned on non-Byzantine data with such a coarse grid? Does this not risk confounding algorithmic failure with poor hyperparameter selection?
  • The authors acknowledge the necessity of using momentum to make the σ\sigma term in the convergence rate of synchronous Byzantine optimization vanish. What are the main difficulties to use momentum in their framework?
  • How do you justify the practical utility of the convergence bound in Theorem 1 (if made true) when the optimal empirical setting (q=1.1q=1.1) causes the O(1/(q−1)2)\mathcal{O}(1/(q-1)^2) error term to be huge, thereby nullifying the theoretical robustness guarantee?
  • Why does throttle fail under the Empire attack compared to BASGDm? Why were baselines only tuned on non-Byzantine data with such a coarse grid? Does this not risk confounding algorithmic failure with poor hyperparameter selection?

If clearly addressed, I am willing to increase my score for acceptance.

Minor comments.

  • Line 026, the claim "Distributed learning trains a shared model faster by spreading the training workload across many clients." requires a citation or a proof.
  • Line 066, "dataset" instead of "data distribution"? Is the sample in Equation (1) a fresh sample every time or drawn from a dataset?
  • As stated, Assumption 1 contain a superfluous conditioning on a constant vector xx, or does not encompass the definition the authors would want to assume.
  • Line 416, "drawn with probability proportional to its index" instead of "inversely proportional"?
2 comments

RM1b

We thank the reviewer for the constructive comments.

Theory

Clarification on TeffT_{eff}

We appreciate the careful check of our proof. We agree the proof is incorrect: the decomposition at line 1192 does not give the conditional unbiasedness used in the descent step. We have fixed this by assuming the schedule S\mathcal S is independent of the gradient samples. Given S\mathcal S and the past, η^t\widehat\eta_t, x(t)x^{(t)} and z(t)z^{(t)} are fixed and gtg_t is still unbiased, so the updated Theorem holds for E[⋅∣S]\mathbb E[\cdot\mid\mathcal S] with the same bound as before. The assumption may look restrictive but the fixed computation model (Mishchenko et al., 2022; Tyurin & Richtárik, 2023) satisfies the assumption. We have made new detailed remarks on this matter after the proof of the Theorem 1 in the paper.

Aggregator Choice

We thank the reviewer for this insightful comment. We agree that (f,κ)(f,\kappa)-robustness (Allouah et al., 2023, Proposition 9) provides strictly better dependence on problem parameters such as the dimension dd, compared to the diameter-based (c,δ)(c,\delta)-robustness criterion used in our current analysis. We appreciate the reviewer bringing this comparison to our attention.

Byzantine Partition

We are grateful for this question. There are potential three cases to consider:

  1. Fixed Byzantine set, time-varying attacks: This setting is covered by our analysis.
  2. Byzantine set changing across restart rounds: Our analysis does not rely on historical Byzantine identities outside the local aggregator, so we expect our guarantees to extend to this case with minor modifications, provided the aggregator's guarantee and the sampling assumptions still hold within each round.
  3. Byzantine set changing within a restart round: Here, the non-vanishing error depends on all clients that act Byzantine at any point during a restart interval IrI_r. If ∣⋃t∈IrBt∣≤M|\bigcup_{t\in I_r}\mathcal B_t| \leq M uniformly across rounds (and other assumptions hold), the non-vanishing contribution from repeated updates scales as O ⁣(θTG2M2(q−1)2)\mathcal{O}\!\left(\theta_T \frac{G^2 M^2}{(q-1)^2}\right).

Necessity of Gradient Clipping

We thank the reviewer for pointing this out. We chose clipping for repeated arrivals to simplify the analysis. With (λt=G\lambda_t=G), clipping leaves honest gradients unchanged under Assumption 3, while every clipped vector has norm at most (GG). Soft throttling then bounds the total contribution of repeated Byzantine updates within each round. These properties are sufficient for our convergence guarantee.

Experiments

Choice of Clipping Parameter λt\lambda_t

We welcome this point. As in above section, for theory, we set λt=G\lambda_t=G in the theorem for analytical tractability. In practice, the clipping radius is a tunable parameter, and we used λt=1\lambda_t=1 in our paper's experiments (see Table below for more values of λ\lambda).

This type of argument also arises in differentially private optimization. There, theoretical analyses (Feldman et al., 2018; Feldman et al., 2020) commonly assume uniformly bounded gradients (equivalent to Lipschitz assumption under the regularity of the loss in our paper), whereas empirical large-scale language model training uses thresholds that actively clip most gradients (Li et al., 2022).

Additional Hyperparameter tuning for Empire Attack (Improved!)

This suggestion was very helpful. After this comment, we re-ran the experiments for Empire case with other hyperparameters centered clipping parameter τ\tau and gradient clipping parameter λ\lambda (other than this, we made other factors fixed).

The following table shows the final accuracy for Throttle with three different seeds.

qλτ=0.1τ=1τ=10τ=100
1.1100.216 ± 0.0690.149 ± 0.0510.086 ± 0.0210.142 ± 0.031
1.110.750 ± 0.0290.713 ± 0.0820.680 ± 0.0830.596 ± 0.129 †
1.10.10.604 ± 0.0610.643 ± 0.0380.027 ± 0.0370.105 ± 0.064
1.5100.061 ± 0.0360.056 ± 0.0150.114 ± 0.0320.065 ± 0.033
1.510.742 ± 0.0450.741 ± 0.0560.613 ± 0.0570.615 ± 0.111 †
1.50.10.721 ± 0.0270.784 ± 0.0350.155 ± 0.1010.164 ± 0.022

Throttle's performance significantly improved (if not best among all methods) with careful tuning of hyperparameters even without momentum.

Minor comments

We thank the reviewer for catching this typo. “Proportional” is correct (clients are sampled pi∝i,p_i \propto i,). We have updated the manuscript accordingly for other points.

References

  • Feldman, V., Mironov, I., Talwar, K., and Thakurta, A. (2018). Privacy Amplification by Iteration. FOCS 2018.
  • Feldman, V., Koren, T., and Talwar, K. (2020). Private Stochastic Convex Optimization: Optimal Rates in Linear Time. STOC 2020.
  • Li, X., Tramèr, F., Liang, P., and Hashimoto, T. (2022). Large Language Models Can Be Strong Differentially Private Learners. ICLR 2022.

Thanks for the answers. Given the limitation of the current manuscript, I will keep my score but I hope this reviewing round helps the authors to carefully revise their manuscript.

Reviewer_zVNpOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies Byzantine-robust asynchronous stochastic gradient descent and proposes THROTTLE. The algorithm geometrically downweights repeated updates from each client within a round and performs a hard restart once every client has responded at least once. First arrivals are processed using a local robust aggregator, while repeated arrivals are clipped. The authors provide a nonconvex stationarity analysis under bounded-gradient and robust-aggregation assumptions, together with MNIST attack simulations and a Byzantine-free least-squares experiment. Overall, the topic is interesting, but the results are not strong enough, so the reviewer cannot support acceptance at this time.

Critical strengths
  • THROTTLE provides a simple and interpretable mechanism for limiting the cumulative influence of fast clients. The summable geometric weights directly address the possibility that a Byzantine client dominates the update stream by sending messages more frequently.
  • Combining Byzantine robustness with asynchronous optimization is technically challenging. Asynchrony introduces stale gradients and separates the fraction of malicious messages from the fraction of malicious clients. Developing principled algorithms for this setting is therefore a worthwhile contribution.
Critical weaknesses
  1. THROTTLE relies on unavailable parameters.
  2. Part of analysis and empirical results remains problematic.
  3. The benefit of asynchrouns SGD remains unclear.
  4. The empirical evaluation is limited in scale.

See Questions for details.

Questions for authors
  1. The practical availability of the gradient bound GG is unclear.

    The theoretical configuration requires a uniform upper bound GG on the norms of honest stochastic gradients. This quantity is used to set the clipping threshold, constrain the anchor, and establish the bounded-contribution conditions required by the analysis. However, the manuscript does not explain how a practitioner can obtain or reliably estimate such a bound.

    This matters because an empirical maximum over previously observed gradients does not certify a uniform bound over future samples and parameter values. Underestimating GG can cause honest gradients to be clipped, introducing bias that is not covered by the current analysis. A loose upper bound permits larger malicious contributions and weakens the resulting error bound.

    The experimental configuration does not resolve this issue: MNIST uses λ=1\lambda=1 and τ=100\tau=100, and the page 6 footnote explicitly acknowledges that clipping of honest gradients is typically active and remains outside the analysis. Please explain how the required bound is obtained, quantify sensitivity to its misspecification, or extend the guarantee to an implementable threshold-selection procedure.

  2. The single-step centered-clipping implementation is not shown to satisfy the robustness requirement inherited by Definition 2.

    Definition 2 requires both local decomposability and the robustness property of Definition 1. The decomposition

    A(v;a,τ)=a+clip⁡τ(v−a)A(v;a,\tau)=a+\operatorname{clip}_{\tau}(v-a)

    establishes the first property, but does not establish the second.

    A simple counterexample illustrates the distinction. Consider one-dimensional inputs with nine honest clients reporting 0.50.5, one Byzantine client reporting −1-1, and

    G=1,a=0,τ=2G=2.G=1,\qquad a=0,\qquad \tau=2G=2.

    No input is clipped, so the centered-clipping output is

    g^=9(0.5)−110=0.35.\widehat g=\frac{9(0.5)-1}{10}=0.35.

    The honest mean is 0.50.5, giving

    ∣g^−gˉG∣2=0.0225.|\widehat g-\bar g_{\mathcal G}|^2=0.0225.

    However, all honest inputs are identical, so their pairwise dispersion is zero. Definition 1 therefore requires zero aggregation error for any finite robustness constant cc. Here I interpret its cardinality condition using the intended non-strict inequality.

    These honest inputs can arise from deterministic gradients of f(x)=log⁡cosh⁡xf(x)=\log\cosh x at x=arctanh⁡(0.5)x=\operatorname{arctanh}(0.5), satisfying smoothness, lower boundedness, and bounded gradients. Thus, the issue concerns the applicability of the theorem to the proposed implementation.

    The original centered-clipping guarantee requires suitable initialization and radius selection; in particular, its single-step guarantee assumes an anchor sufficiently close to the honest mean. Projecting the anchor onto a ball only controls its norm. Please establish the required anchor-tracking guarantee and radius conditions, and revise Lemma 3 and the main bound accordingly. Karimireddy et al., 2021, Corollary V

  3. The practical benefit of retaining asynchrony is not sufficiently established.

    The general motivation for asynchronous SGD—reducing idle time caused by slow workers—is clear. What remains uncertain is how much of this advantage THROTTLE retains after introducing geometric throttling and all-client hard restarts.

    Theorem 1 characterizes convergence using effective optimization steps, rather than elapsed time. This does not quantify the effects of heterogeneous worker speeds, communication, server-side processing, or restart costs. Figure 3 on page 9 compares methods using the number of computed gradients, while the MNIST experiments use simulated time with zero communication latency.

    A runtime analysis under an explicit computation model would strengthen the contribution. At minimum, please report measured time-to-target performance against a synchronous Byzantine-robust baseline using comparable aggregation, hardware resources, and tuning budgets. Such a comparison is particularly relevant because completing a THROTTLE round still requires every client to respond.

  4. Figure 3 does not isolate the contribution of asynchrony from update frequency and effective batch size.

    With 40 workers, synchronous minibatch SGD normally averages 40 worker gradients before one parameter update, whereas ordinary asynchronous SGD applies one arriving worker gradient at a time. Consequently, under the same gradient-computation budget, asynchronous SGD can perform approximately 40 times as many parameter updates. Each worker gradient itself uses a minibatch of 256 samples.

    Equalizing the number of computed gradients is a meaningful measure of computational efficiency. However, the resulting comparison also changes the effective batch size and the frequency of parameter updates. The observed improvement may therefore reflect the optimization behavior of smaller-batch, more frequent updates, rather than the system benefit of avoiding synchronization.

    Appendix E already reports separate learning-rate tuning for each method, so the baseline cannot simply be assumed to be untuned. Please report actual update counts and effective batch sizes, provide learning-rate sensitivity results, and consider a sequential small-batch SGD control to help isolate the update-frequency effect. These controls should complement the matched-resource wall-clock comparison requested above.

  5. The empirical evaluation is limited in scale and representativeness.

    The Byzantine experiments primarily use a small CNN on i.i.d. MNIST within an event-driven simulator. The additional least-squares experiment uses 40 Ray actors on a single workstation and does not include Byzantine clients. These experiments provide preliminary evidence, but offer limited support for the broader claims about robust distributed learning.

    Please include at least one more demanding learning task and evaluate robustness under a real distributed setup with heterogeneous computation or communication. Varying the client count, Byzantine fraction, and attack frequency would also help establish scalability and clarify how the choice of qq changes with system size. Reporting restart overhead, discarded computation, and server load would make the practical tradeoff easier to assess.

1 comment

We thank the reviewer for the detailed review.

The practical availability of the gradient bound.

We thank the reviewer for pointing out that past gradient norms cannot certify a bound on future ones. We agree that knowing a uniform bound GG is a strong assumption, and we did not have a certified bound for MNIST. As a first step, we set λt=G\lambda_t=G in Theorem 1 to leave honest gradients unchanged while studying how soft throttling limits the cumulative influence of a client that sends repeated updates.

In practice, even though more aggressive clipping (smaller clipping radius) can bias honest updates, it places a tighter limit on each repeated Byzantine update and can improve accuracy. Under the Empire attack, for example, lowering λ\lambda from 11 to 0.10.1 at fixed q=1.5q=1.5 and τ=1\tau=1 increased mean MNIST accuracy from 74.1% to 78.4% across three seeds (see our response to Reviewer RM1b for more).

The single-step centered-clipping implementation is not shown to satisfy the robustness requirement inherited by Definition 2.

We thank the reviewer for the counterexample. We were wrong to claim that our single-step centered clipping rule satisfies Definition 2.

The proof of Theorem 1 goes through for a local aggregator that satisfies the robustness condition at each restart. Our mistake was to use that condition in Lemma 3 to obtain a variance-dependent bound for our single-step rule.

We now derive the rate for this rule directly in Appendix C, in the new paragraph “Convergence rate with single-step centered clipping without robust aggregator assumption.” With τ=2G\tau = 2G, the O(cδσ2)O(c\delta\sigma^2) aggregation term in Theorem 1 becomes O(δ2G2)O(\delta^2 G^2). This gives a weaker guarantee at low noise variance.

Response to Questions 3 and 4

We appreciate these questions. Following Mishchenko et al. (2022), we show that asynchronous SGD matches the convergence rate of synchronous minibatch SGD at the same gradient budget, despite using stale gradients. As in their analysis, our guarantee translates into a wall-clock comparison (see §1.2 of their paper for why such rates are meaningful in wall-clock time)

In experiments, measuring wall-clock training time depends on the hardware, communication latency, and server load. In Figure 3, to show pure asynchronous performance of Throttle against existing methods, we therefore use real workers, including a setting with a straggler, and compare methods at the same number of computed gradients, as in Mishchenko et al. (2022, Fig. 1). For MNIST, to show Byzantine robustness under asynchrony, we follow an experimental protocol similar to that of Dahan and Levy (2024), simulating heterogenous client updates and using the same schedule across methods. We agree that batch size and update frequency can affect the comparison with synchronous SGD. We were unable to run the additional controls or the matched wall-clock comparison during rebuttal.

The empirical evaluation is limited in scale and representativeness.

We thank the reviewer for raising this limitation. Our MNIST simulations test how Throttle handles fast Byzantine clients. The 40-actor Ray experiment uses only honest clients, so robustness across heterogeneous machines remains an open empirical question. With the time and compute available during rebuttal, we could not complete a multi-node evaluation on a more demanding task.

Reviewer_Q4pCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studied the problem of robust asynchronous distributed learning. The authors proposed a technique called Throttle, where the key idea is to exponentially down-weight updates from faster clients. The authors provided a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. The experiments showed that this down-weighting mechanism can also improve performance over standard asynchronous SGD in both Byzantine and non-Byzantine settings.

Critical strengths
  1. The authors provided in-depth theoretical analysis for the proposed method.

  2. The presentation of this paper is clear and easy to follow.

Critical weaknesses
  1. The novelty of this paper is very limited. Byzantine-robust distributed/federated learning has been a heavily studied area for many years. The design of asynchronous distributed/federated learning methods is also a well-investigated area. The same is also true for the combination of these two areas, i.e., Byzantine-robust asynchronous distributed learning. This can been seen in the references cited in this paper, most of which are old papers in the sense of rapidly developing machine learning research.

  2. Even in the areas of Byzantine-robust and asynchronous distributed learning, the techniques and assumptions used in this paper are also old and restricted. For example, it is well-known that the bounded gradient assumption (Assumption 3) can be relaxed without affecting the final convergence rate guarantee. Also, the clipping idea is also very standard. Also, the authors claimed that the literature assumed finite maximum delay in asynchronous distributed learning, which is clearly not true. For example, the following works assumed unbounded delays:

[R1] Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Peter W. Glynn, Yinyu Ye, Li-Jia Li, and Li Fei-Fei., Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go? ICML 2018

[R2] Xin Zhang, Jia Liu, and Zhengyuan Zhu, Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learning, CDC 2020.

  1. The experiments in this paper is inadequate. The authors only conducted small and limited scale experiments with MNIST, which is widely considered too simple and outdated. In fact, in the recent distributed and federated learning literature, CIFAR is also considered outdated and many works have switched to training LLMs with large datasets.
Questions for authors
  1. Could the authors highlight the major differences and challenges in this work compared to other Byzantine-robust asynchronous distributed/federated learning methods?

  2. Could the authors relax the assumptions mentioned above?

  3. Could the authors provide larger scale experiments?

1 comment

We thank the reviewer for taking the time to review our paper.

Novelty of this paper

We appreciate the opportunity to clarify our novelty. In synchronous setting (conventional Byzantine-robust methods), each client contributes once per round. In asynchronous setting, a single Byzantine worker can send most of the updates even when nearly all workers are honest. This makes naively incorporating existing method from synchronous method to asynchronous method difficult because of the very structure of asynchronous SGD.

Our method exactly fixes this issue. As said in our paper, the existing methods (BASGD and mu2sgd) lacks either sound convergence guarantee or limits this asynchronous structure by having strong assumption that potentially undermines the whole benefit of asynchronous SGD.

Relax on gradient clipping and clarification on maximum delay

We thank the reviewer for asking this. We agree that gradient clipping is standard. In Throttle, clipping bounds the size of each repeated update, while soft throttling bounds the total weight a client can accumulate within a round. Assumption 3 lets us study this effect while leaving honest gradients unchanged. Without it, a smaller clipping radius can bias honest updates, while a larger radius permits larger Byzantine updates. Whether Throttle retains the same rate under this tradeoff remains open. Empirically, tighter clipping helped the performance. Under Empire attack, lowering λ\lambda from 11 to 0.10.1 at fixed q=1.5q=1.5 and τ=1\tau=1 increased mean MNIST accuracy from 74.1% to 78.4% across three seeds (see our response to RM1b for more).

We are grateful for the pointer to these earlier results. Also, we have revised Section 5 to acknowledge these earlier convergence results for asynchronous SGD under unbounded delays.

Scale of experiment

We are thankful for the suggestion to evaluate at a larger scale. Our main contribution is a convergence analysis of asynchronous SGD in the presence of fast Byzantine clients. The MNIST experiments complement that analysis by testing how the choice of qq affects training under attack. We agree that these results leave open how Throttle performs on larger tasks, including LLM training. With the time and compute available during rebuttal, we could not complete a larger-scale experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.