acceptodds
Under review as a conference paper at ICLR 2027

LASA: Byzantine-Robust Decentralized Optimization via Local-Loss Aware Weighting

Abstract

A key challenge in Byzantine-robust decentralized learning is to mitigate the adverse effects of Byzantine updates while preserving beneficial information exchange. To this end, we introduce aggregation budgets that provide loss-based control of Byzantine influence by bounding the weighted excess local loss relative to the self-candidate. We then propose a novel Local-Loss Aware Softmax Aggregator (LASA) with explicit aggregation budgets. Based on local-loss evaluations, LASA assigns positive weights to all candidates without prior Byzantine information, balancing robustness with opportunities for useful cooperation. For LASA and any other aggregator admitting such budgets, we establish global stationarity bounds for smooth convex objectives under a relative gradient heterogeneity condition without assuming an honest majority, and obtain sharper guarantees under an additional quadratic growth condition. We further provide sufficient conditions under which LASA admits a strictly tighter stationarity bound than self-only training. Experiments on MNIST and CIFAR-10 demonstrate the effectiveness of LASA across several communication graphs and Byzantine attacks.

Primary area:
optimization
Keywords:
distributed learning, Byzantine-robust optimization, decentralized optimization, local-loss weighting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8G9uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper tackles the timely and important problem of Byzantine robustness in decentralized learning.

The main contributions can be summarized as follows:

  1. Introducing a new Byzantine-robustness aggregation based on local loss at the nodes.
  2. Deriving a sufficient coordination certificate showing when employing collaborative learning improves over learning on the local data.
  3. Theoretical guarantees on the convergence of the algorithm for convex loss functions.
  4. Numerical validation on three different graphs for logistic regression on MNIST and ResNet20 on CIFAR10 data set with data heterogeneity.
Critical strengths

The paper is very well written and easy to read.

The concept of aggregation budgets defined in the paper is interesting and can encompass the algorithms of other papers. The paper connects the aggregation budgets to convergence guarantees through the convexity of the loss function.

The idea of comparing the optimization bound to self-learning is insightful and shows when collaboration under Byzantine attacks outperforms local learning.

Critical weaknesses

The theoretical guarantees are restricted to convex loss functions. A few years ago, this would have been fine, but now the literature on Byzantine robustness in decentralized learning has matured significantly. Convergence guarantees for non-convex loss functions have been established in many other works.

The comparison to the literature is not complete. Works such as Balance and Resist are not compared against. Such comparisons will help position the robustness of the introduced method relative to existing defenses.

Balance: Fang et al. Byzantine-Robust Decentralized Federated Learning, ACM SIGSAC Conference on Computer and Communications Security, 2024 Resist: Fang et al. RESIST: Resilient Decentralized Learning Using Consensus Gradient Descent, TMLR 2026

The numerical experiments are rather limited in the following sense:

  1. Logistic regression on MNIST and ResNet20 on CIFAR-10 do not represent current learning tasks.
  2. The considered communication graphs are not exhaustive. In particular, the regular graph has only two malicious nodes, and in the interesting graph of Figure 3.c, results for the dissensus attack are shown. It would be better to show the results for all the attacks considered.
  3. Data heterogeneity is modeled through the Dirichlet distribution, which is great. However, intrinsically heterogeneous datasets such as FEMNIST would be more suitable, especially in the graph of Figure 3.c.
Questions for authors
  1. Would your methods provide convergence for non-convex loss functions?

  2. Could you please provide a comprehensive comparison to existing defenses in the literature?

  3. Could you provide additional numerical experiments validating the robustness of LASA in graphs with more malicious nodes, say 40% of the nodes are malicious, with more data heterogeneity, and different architectures?

  4. Are there additional attacks that should be considered in your study? For example, ALIE, min-sum, and min-max attacks?

ALIE: Baruch et al. A little is enough: Circumventing defenses for distributed learning, NeurIPS, 2019. Min-sum and min-max: Manipulating the Byzantine: Optimizing model poisoning attacks and defenses for federated learning. NDSS, 2021.

  1. Can you discuss what would happen if, for example, in the graph of Figure 3.c, node 0 were to be malicious and node 8 honest?
4 comments

We are grateful for your valuable feedback.

Q1. The theoretical guarantees are restricted to convex loss functions. Would your methods provide convergence for non-convex loss functions?

Our current guarantees rely on Jensen's inequality to transfer the aggregation budget on candidate losses to a bound on the aggregated loss. Thus, these guarantees do not directly extend to general nonconvex objectives under the remaining assumptions. A possible extension replaces Δijt\Delta_{ij}^t in the LASA score with

-\lambda\\|\tilde{x}_{ij}^t-\tilde{x}_{ii}^t\\|^2.$$ The above regularized local loss may allow the stationarity analysis to extend to smooth nonconvex objectives without additional assumptions. This modification can be a direction for future work. > Q2. The comparison to the literature is not complete. Could you please provide a comprehensive comparison to existing defenses in the literature? As per your suggestion, we have added BALANCE to our comparisons on MNIST and CIFAR-10 (Tables 1, 2, and 4). On MNIST, LASA achieves higher MTA and WTA than BALANCE under all five tested attacks on the regular graph and four attacks on the Erdős–Rényi graph (Tables 1 and 4). On CIFAR-10 under Dissensus, LASA achieves MTA/WTA of $70.21\\%/63.85\\%$, compared with $55.01\\%/42.22\\%$ for BALANCE. These highlight the excellent performance of LASA. As for RESIST, on our regular graph with $b=2$, the CWTM steps in RESIST discard all four neighbor candidates, which degrades to SELF. Thus, a separate RESIST experiment would duplicate the SELF baseline. > The considered communication graphs are not exhaustive. Aggregators requiring an estimate of the maximum number of Byzantine neighbors (e.g., NNA, IOS, and RESIST) cannot be applied to graphs where the prescribed filtering bound exceeds the minimum local degree. We therefore exclude these infeasible combinations of methods and graphs from our comparisons. LASA requires no such estimate and remains well-defined at every node, making it applicable to a broader range of sparse and irregular graphs. > In the interesting graph of Figure 3.c, results for the dissensus attack are shown. It would be better to show the results for all the attacks considered. The experiment on the graph in Figure 3(c) is designed as a targeted stress test, where an honest node has three Byzantine neighbors and only one honest neighbor. We select Dissensus because it produces the lowest LASA accuracy among all attacks in our main MNIST experiments. > Q3. Could you provide additional numerical experiments validating the robustness of LASA in graphs with more malicious nodes, say 40% of the nodes are malicious, with more data heterogeneity, and different architectures? We have added FEMNIST experiments that directly address these aspects. We use a new 50-node Watts–Strogatz graph with 40% Byzantine nodes. We train a CNN on 164,289 training examples from 800 writers using writer-based partitions, capturing the intrinsic heterogeneity. The following table compares LASA with BALANCE on FEMNIST. The first and second rows report mean test accuracy (MTA) and worst-node test accuracy (WTA) in percentages, with entries showing mean $\pm$ standard deviation over three seeds. | Method | Metric | Benign | Gaussian | Sign Flip | Label Flip | FOE | Dissensus | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | BALANCE | MTA | $84.53\pm0.20$ | $84.01\pm0.18$ | $82.99\pm0.21$ | $83.54\pm0.37$ | $84.01\pm0.18$ | $82.47\pm0.27$ | | BALANCE | WTA | $82.63\pm0.28$ | $82.03\pm0.52$ | $81.36\pm0.28$ | $78.33\pm5.70$ | $82.03\pm0.52$ | $75.41\pm2.53$ | | **LASA** | MTA | $85.25\pm0.20$ | $84.30\pm0.05$ | $84.19\pm0.14$ | $84.39\pm0.16$ | $84.32\pm0.04$ | $84.19\pm0.14$ | | **LASA** | WTA | $84.03\pm0.39$ | $81.66\pm0.94$ | $82.09\pm0.94$ | $82.14\pm0.88$ | $81.66\pm1.70$ | $81.95\pm1.45$ | Full results are provided in Table 5 of Section D.6. Among all compared methods, LASA achieves the highest mean MTA in all six settings and the highest mean WTA in four. These results support LASA's effectiveness on a larger, naturally heterogeneous dataset with a different graph and a higher Byzantine fraction.

Q4. Are there additional attacks that should be considered in your study? For example, ALIE, min-sum, and min-max attacks?

Our original evaluation has covered a broad range of Byzantine attacks common in decentralized optimization: random failures, adversarial model manipulation, label poisoning, and disruption of decentralized cooperation. As per your suggestion, we consider ALIE and Min-Sum/Min-Max-inspired variants. Min-Sum and Min-Max were originally developed for server-based federated learning (Shejwalkar and Houmansadr, 2021). We adapt their distance-constrained principles to the candidate models exchanged in our decentralized setting. Results below report mean ±\pm standard deviation over three seeds.

Mean test accuracy (MTA, %) ↑\uparrow

MethodALIEMin-SumMin-Max
DSGD4.88±0.004.88_{\pm 0.00}40.92±2.3040.92_{\pm 2.30}53.50±2.1853.50_{\pm 2.18}
IOS4.88±0.004.88_{\pm 0.00}4.88±0.004.88_{\pm 0.00}29.16±3.6029.16_{\pm 3.60}
NNA82.47±0.5282.47_{\pm 0.52}79.49±0.45\mathbf{79.49_{\pm 0.45}}78.87±1.59\mathbf{78.87_{\pm 1.59}}
CG66.26±7.0266.26_{\pm 7.02}51.08±6.3551.08_{\pm 6.35}58.03±0.5358.03_{\pm 0.53}
IL79.26±0.9979.26_{\pm 0.99}73.15±2.4073.15_{\pm 2.40}73.36±3.0673.36_{\pm 3.06}
BALANCE70.25±6.8570.25_{\pm 6.85}63.58±3.0963.58_{\pm 3.09}64.04±2.1664.04_{\pm 2.16}
SELF70.36±0.1670.36_{\pm 0.16}70.36±0.1670.36_{\pm 0.16}70.36±0.1670.36_{\pm 0.16}
LASA83.42±0.38\mathbf{83.42_{\pm 0.38}}78.32±0.6978.32_{\pm 0.69}77.34±1.7077.34_{\pm 1.70}

Worst-node test accuracy (WTA, %) ↑\uparrow

MethodALIEMin-SumMin-Max
DSGD4.88±0.004.88_{\pm 0.00}36.74±3.2936.74_{\pm 3.29}50.12±1.5850.12_{\pm 1.58}
IOS4.88±0.004.88_{\pm 0.00}4.88±0.004.88_{\pm 0.00}22.61±2.4622.61_{\pm 2.46}
NNA80.23±1.77\mathbf{80.23_{\pm 1.77}}77.33±1.20\mathbf{77.33_{\pm 1.20}}73.98±3.4073.98_{\pm 3.40}
CG4.01±0.764.01_{\pm 0.76}42.47±5.7942.47_{\pm 5.79}49.60±4.3949.60_{\pm 4.39}
IL68.94±3.4568.94_{\pm 3.45}60.86±2.8660.86_{\pm 2.86}66.30±3.2866.30_{\pm 3.28}
BALANCE4.68±0.974.68_{\pm 0.97}57.26±5.2657.26_{\pm 5.26}56.03±6.4456.03_{\pm 6.44}
SELF66.34±0.9766.34_{\pm 0.97}66.34±0.9766.34_{\pm 0.97}66.34±0.9766.34_{\pm 0.97}
LASA79.82±1.6379.82_{\pm 1.63}74.59±1.5774.59_{\pm 1.57}74.18±1.26\mathbf{74.18_{\pm 1.26}}

Consensus error (CE) ↓\downarrow

MethodALIEMin-SumMin-Max
DSGD$ > 10^4$0.115±0.0210.115_{\pm 0.021}0.041±0.000\mathbf{0.041_{\pm 0.000}}
IOS$ > 10^4$$ > 10^4$0.200±0.0300.200_{\pm 0.030}
NNA0.030±0.002\mathbf{0.030_{\pm 0.002}}0.046±0.004\mathbf{0.046_{\pm 0.004}}2.316±3.9562.316_{\pm 3.956}
CG$ > 10^4$0.223±0.0460.223_{\pm 0.046}0.207±0.0310.207_{\pm 0.031}
IL0.109±0.0090.109_{\pm 0.009}0.133±0.0230.133_{\pm 0.023}0.128±0.0120.128_{\pm 0.012}
BALANCE104.3±25.73104.3_{\pm 25.73}0.100±0.0120.100_{\pm 0.012}0.104±0.0200.104_{\pm 0.020}
SELF100.8±2.464100.8_{\pm 2.464}100.8±2.464100.8_{\pm 2.464}100.8±2.464100.8_{\pm 2.464}
LASA0.143±0.0150.143_{\pm 0.015}1.203±0.4071.203_{\pm 0.407}1.454±0.3941.454_{\pm 0.394}

LASA and NNA rank among the top two methods in both accuracy metrics across all three attacks. Unlike NNA, LASA requires no estimate of the maximum number of Byzantine neighbors. These results demonstrate the robustness of LASA under additional model poisoning attacks.

Q5. Can you discuss what would happen if, for example, in the graph of Figure 3.c, node 0 were to be malicious and node 8 honest?

In this case, honest nodes 6 and 8 would be disconnected from the other honest nodes in the honest subgraph. They could no longer reliably benefit from the data held by the other honest nodes. This restriction would make it difficult for learning methods to achieve good performance, particularly under heterogeneous data.

We hope that our responses have adequately addressed your concerns.

Reviewer_8G9uOpenReview ↗

Thank you, I will raise my score to 3

Dear Authors,

Thank you for the detailed discussion. The provided information addresses most of my concerns and therefore, I will raise my score to 3.

Providing a formal proof of convergence for non-convex loss function is still important. Having more experimental results on complex datasets and comparing to Balance and Resist in the paper is important.

Good luck and best regards, Reviewer 8G9u

AuthorsOpenReview ↗

Thank you for your support

Dear Reviewer 8G9u,

We are grateful for your positive feedback and thoughtful follow-up. Your suggestions have been very helpful in improving our manuscript.

Reviewer_in1bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces aggregation budgets that bound the weighted excess local loss relative to the self-candidate in Byzantine-robust decentralized optimization. It proposes a Local-Loss Aware Softmax Aggregator (LASA), which assigns positive weights to all candidates without requiring prior Byzantine information. The authors establish global stationarity bounds for smooth convex objectives under relative gradient heterogeneity, obtain sharper guarantees under quadratic growth, and provide sufficient conditions for a strictly tighter stationarity bound than self-only training. Experiments evaluate LASA on MNIST and CIFAR-10 across several communication graphs and Byzantine attacks.

Critical strengths

LASA provides a simple aggregation rule that does not require prior knowledge of Byzantine participation. Its empirical results show competitive average accuracy and strong worst-node accuracy across the tested attacks, particularly under Dissensus. Improving performance at vulnerable honest nodes is consequential because average accuracy alone can conceal severe failures in decentralized learning.

Critical weaknesses

The empirical contribution of retaining higher-loss candidates is not clearly established. LASA and the inverse-loss baseline differ in both their weighting functions and their candidate-selection rules, so the comparison does not isolate the benefit of retaining these candidates. Although Appendix C.3 demonstrates this benefit in a constructed example, the experiments on MNIST and CIFAR-10 lack a controlled ablation of this design choice. Consequently, the evidence explaining how this central design choice contributes to the reported improvements remains limited.

Questions for authors

None

1 comment

We sincerely thank you for your recognition of our work.

The empirical contribution of retaining higher-loss candidates is not clearly established. The experiments on MNIST and CIFAR-10 lack a controlled ablation of this design choice. Consequently, the evidence explaining how this central design choice contributes to the reported improvements remains limited.

We have added a matched ablation on MNIST (regular and Erdős–Rényi graphs) and CIFAR-10 (regular graph). The ablation variant Loss-Filtered LASA (LASA-LF) removes candidates with strictly higher local loss than the self-candidate before applying the softmax rule. specifically, LASA-LF applies the LASA softmax rule only to Si+,t:=j∈Si:ℓijt≤ℓiit\mathcal{S}_i^{+,t} :=\\{j\in\mathcal{S}_i:\ell_{ij}^t\leq\ell_{ii}^t\\}, with weights

\begin{cases} \displaystyle \frac{\exp(-\tau_t\ell_{ij}^t)} {\sum_{k\in\mathcal{S}_i^{+,t}}\exp(-\tau_t\ell_{ik}^t)}, & \text{if }j\in\mathcal{S}_i^{+,t},\\\\[6pt] 0, & \text{otherwise}. \end{cases}$$ All other experimental settings remain unchanged, isolating the contribution of candidate retention. The CIFAR-10 results below show mean test accuracy / worst-node test accuracy (%), averaged over five seeds. Full results are provided in Section 5.3 and Tables 1, 2, and 4 of the revised manuscript. | Setting | SELF | IL | LASA-LF | LASA | | --- | ---: | ---: | ---: | ---: | | Benign | 46.34 / 34.35 | 64.17 / 54.47 | 63.89 / 54.54 | **84.45 / 84.44** | | Gaussian | 46.34 / 34.35 | 60.85 / 48.11 | 61.51 / 53.88 | **82.89 / 82.88** | | Sign Flip | 46.34 / 34.35 | 59.01 / 46.87 | 57.27 / 41.58 | **79.05 / 79.04** | | Label Flip | 46.34 / 34.35 | 63.37 / 54.54 | 59.45 / 41.40 | **76.68 / 76.68** | | FOE | 46.34 / 34.35 | 54.28 / 40.75 | 54.00 / 41.31 | **77.00 / 76.99** | | Dissensus | 46.34 / 34.35 | 57.57 / 38.16 | 61.39 / 53.24 | **70.21 / 63.85** | Both IL and LASA-LF improve over SELF, demonstrating benefits from communication even after filtering. LASA delivers substantial improvements over both methods in every reported setting. The MNIST experiments likewise show better performance for LASA than IL and LASA-LF throughout. Since the matched ablation changes only candidate retention, these results directly support the value of retaining higher-loss candidates in LASA. We hope that our responses and the revised manuscript have addressed your concerns.
Reviewer_iTSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Byzantine-robust decentralized optimization through local-loss-based aggregation. It introduces aggregation budgets that limit the weighted excess loss relative to a node's own SGD candidate. LASA uses softmax weights over candidate losses and retains every candidate. The analysis gives global stationarity bounds for smooth convex objectives under relative gradient heterogeneity, sharper bounds under quadratic growth, and sufficient conditions for a tighter bound than self-only training. Experiments use MNIST logistic regression and CIFAR-10 ResNet-20 under several attacks and static graphs.

Critical strengths
  1. The paper is well motivated. It clearly explains the tension between robust defense and useful cooperation. Under heterogeneous data, a neighbor's model can have higher local loss while still helping the global objective.

  2. The aggregation-budget framework makes the cost of aggregation explicit. It connects local-loss control to global stationarity without requiring an honest majority.

Critical weaknesses
  1. Novelty and parameter selection. The reviewer is not yet convinced that the aggregation rule itself is a substantial contribution. Softmax mixing is already used in decentralized learning, including L2C and meta-L2C [1]. LASA uses ωijt∝exp⁡(−τtℓijt)\omega_{ij}^t\propto\exp(-\tau_t\ell_{ij}^t). Those methods learn weights for personalization, but LASA weights candidate losses for Byzantine-robust optimization. This is a meaningful distinction, but the paper should separate what is new in the weighting rule from what is new in the budget framework and its analysis.

    The inverse temperature also requires tuning. Proposition 4.2 gives the sufficient budget βi,t=log⁡(mi)/τt\beta_{i,t}=\log(m_i)/\tau_t, but it does not identify a useful operating point. Figure 1 studies sensitivity on MNIST, and Appendix D reports validation-based tuning separately for each setting, including a small CIFAR-10 grid. The remaining concern is how to choose a temperature before knowing the attack or network conditions. Large temperatures tighten the budget but suppress the higher-loss candidates that motivate the method.

  2. Scope of the cooperation guarantee and experimental support. Equation (2.4) can be written as

    \leq \ell_{ii}^t+\beta_{i,t}.$$ For convex local objectives, this controls how much aggregation can worsen local loss relative to the self-candidate. It does not directly certify a global benefit from a locally worse candidate. The gain used in Theorem 4.3 is $$\Delta^{\mathrm{gain}}_{i,t} =\ell_{ii}^t-\min_{j\in S_i\cap\mathcal H}\ell_{ij}^t.$$ It therefore measures improvement from the best locally evaluated honest candidate. The theorem does provide a tighter stationarity bound when cumulative expected gains exceed the entropy penalties. Appendix C.3 also gives a useful quadratic example where retaining a higher-local-loss honest candidate improves the global objective. Still, this is a particular construction, and the main certificate does not give a general guarantee for the cooperation benefit emphasized in the motivation. The mechanism behind the improvement over SELF also remains unclear. SELF receives no Byzantine messages. A tighter LASA upper bound than the SELF bound does not generally prove better realized performance or separate the value of cooperation from resistance to attacks. The experiments include both benign and attacked settings, but do not isolate the contribution of retaining higher-loss candidates. The validation is limited to 10–20 nodes, MNIST logistic regression, and CIFAR-10 ResNet-20. Only SGD without momentum is tested. The graphs are fixed; there is no test with changing topology. These results are promising, but the small scale and narrow training settings leave the practical reach of the method uncertain.

References

[1] Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Learning to Collaborate in Decentralized Learning of Personalized Models. CVPR, 2022. Paper.

Questions for authors
  1. What is the main contribution beyond existing softmax mixing: the weighting rule, the aggregation-budget abstraction, or the resulting guarantees? Please contrast the weighting scores, objectives, and guarantees with [1].
  2. Beyond the example in Appendix C.3, what evidence shows that retaining higher-local-loss candidates helps the global objective under attacks? A matched ablation that removes those candidates may clarify the source of the gains over SELF and IL.
  3. Does a temperature selected in a benign setting remain effective under the tested attacks? This would clarify how much the results depend on tuning separately for each setting.
  4. Can the authors provide a larger-dataset result or evidence with momentum/adaptive optimization? Changing topology is also an important untested setting, though extending the theory to it can remain future work.
2 comments

We appreciate your insightful questions and valuable suggestions.

W1 & Q1. The reviewer is not yet convinced that the aggregation rule itself is a substantial contribution. What is the main contribution beyond existing softmax mixing? Please contrast the weighting scores, objectives, and guarantees with [1].

We exploit two roles of softmax in Byzantine-robust optimization: (i) limiting the influence of candidates with abnormally high losses under attacks and (ii) retaining potentially useful candidates that may be discarded by filtering-based methods. Based on this insight, we propose LASA, which explicitly balances Byzantine robustness and beneficial cooperation. We further develop a general framework covering LASA and other loss-based aggregators, whose convergence rates and error floors match existing guarantees. The differences from [1] are:

Objectives. [1] targets personalization without attacks. Ours optimizes a shared global objective under Byzantine attacks.

Scores. L2C learns softmax logits, while meta-L2C uses dot products between encoded model changes. Both learn weighting parameters using post-aggregation validation loss. Our proposed LASA directly computes weights from local losses via a closed-form softmax rule, without learning mixing parameters.

Guarantees. [1] does not provide convergence guarantees. We establish convergence bounds for a class of aggregators. For LASA, we further provide an entropy-regularized characterization to explain the robustness–cooperation trade-off by softmax.

W2 & Q2. The main certificate does not give a general guarantee for the cooperation benefit emphasized in the motivation. Beyond the example in Appendix C.3, what evidence shows that retaining higher-local-loss candidates helps the global objective under attacks?

Thanks for your insightful suggestion. We have added a controlled ablation on MNIST and CIFAR-10 in Section 5.3. The ablation variant, called LASA-LF, removes candidates whose locally evaluated losses exceed the self-candidate loss before applying the same softmax rule. All other settings remain unchanged, isolating candidate retention from the choice of weighting function.

The CIFAR-10 results below report mean test accuracy (MTA) / worst-node test accuracy (WTA) (%), averaged over five seeds. Full results are provided in Tables 1, 2, and 4.

SettingSELFILLASA-LFLASA
Benign46.34 / 34.3564.17 / 54.4763.89 / 54.5484.45 / 84.44
Gaussian46.34 / 34.3560.85 / 48.1161.51 / 53.8882.89 / 82.88
Sign Flip46.34 / 34.3559.01 / 46.8757.27 / 41.5879.05 / 79.04
Label Flip46.34 / 34.3563.37 / 54.5459.45 / 41.4076.68 / 76.68
FOE46.34 / 34.3554.28 / 40.7554.00 / 41.3177.00 / 76.99
Dissensus46.34 / 34.3557.57 / 38.1661.39 / 53.2470.21 / 63.85

Both IL and LASA-LF outperform SELF, showing that communication remains beneficial after filtering. This provides empirical support for the benefit of lower-loss candidates in Theorem 4.3. LASA further outperforms both methods in every setting. This further shows the benefit of retaining candidates with higher losses and complement the concrete improvement in Appendix C.3.

W1 & Q3. The inverse temperature also requires tuning. Does a temperature selected in a benign setting remain effective under the tested attacks?

Figure 1 shows that attack-specific temperature tuning is not sensitive in the tested settings. MNIST exhibits a broad common operating range across benign conditions and attacks, as a fixed τ=3\tau=3 performs well throughout. On CIFAR-10, fixing τ=10\tau=10 yields MTA/WTA (%) of 83.61/83.59 (Benign), 82.60/82.60 (Gaussian), and 77.97/77.96 (Sign Flip), with other results in Table 2 unchanged. This preserves our main advantages over the baselines. The small grid for Table 2 is used to select the best validation configuration.

The remaining concern is how to choose a temperature before knowing the attack or network conditions.

The broad range of effective inverse temperatures in Figure 1 suggests a practical selection rule. Under benign settings, select a sufficiently large τ\tau whose validation performance remains within a predefined tolerance of the best observed benign result. A larger τ\tau tightens the certified aggregation budget log⁡(mi)/τ\log(m_i)/\tau, while the validation criterion maintains good benign performance. This procedure requires neither prior knowledge of the attack nor global network parameters.

W2 & Q4. Can the authors provide a larger-dataset result or evidence with momentum/adaptive optimization?

We have added FEMNIST experiments using a CNN on 164,289 training examples. We expand the communication network to 50 nodes, including 20 Byzantine nodes, and fix τ=3\tau=3 for LASA across all settings.

The following table compares LASA with NNA on FEMNIST. The first and second rows report mean test accuracy (MTA) and worst-node test accuracy (WTA) in percentages, with entries showing mean ±\pm standard deviation over three seeds.

MethodMetricBenignGaussianSign FlipLabel FlipFOEDissensus
NNAMTA81.63±1.2781.63\pm1.2783.56±0.2883.56\pm0.2882.52±0.5482.52\pm0.5482.51±0.5382.51\pm0.5383.56±0.2883.56\pm0.2882.36±0.9582.36\pm0.95
NNAWTA76.05±1.8276.05\pm1.8280.75±1.9180.75\pm1.9180.35±1.5880.35\pm1.5880.36±1.5980.36\pm1.5980.75±1.9180.75\pm1.9179.53±2.7279.53\pm2.72
LASAMTA85.25±0.2085.25\pm0.2084.30±0.0584.30\pm0.0584.19±0.1484.19\pm0.1484.39±0.1684.39\pm0.1684.32±0.0484.32\pm0.0484.19±0.1484.19\pm0.14
LASAWTA84.03±0.3984.03\pm0.3981.66±0.9481.66\pm0.9482.09±0.9482.09\pm0.9482.14±0.8882.14\pm0.8881.66±1.7081.66\pm1.7081.95±1.4581.95\pm1.45

Full results are in Table 5 in Appendix D.6. Compared to other aggregators, LASA achieves the highest mean MTA in all settings and the highest mean WTA in four. These results show robustness of LASA on a larger dataset.

We agree that momentum/adaptive optimization are important. Our current convergence guarantees are established for SGD-based updates, and extending them to these settings requires substantial changes. We therefore focus our present experiments on SGD.

W2 & Q4. Changing topology is also an important untested setting, though extending the theory to it can remain future work.

Our original experiments follow a common fixed-topology setting in Byzantine-robust decentralized optimization. As per your suggestion, we additionally conduct MNIST experiments with 20 nodes (with 4 Byzantine nodes), independently resampling an Erdős–Rényi communication graph with p=0.35p=0.35 at each round. Results after 2,000 rounds are reported as mean ±\pm SD over five seeds.

SettingMTA (%)WTA (%)CE (×10−3\times 10^{-3})
Benign92.02±0.0692.02 \pm 0.0691.84±0.0391.84 \pm 0.030.960±0.1340.960 \pm 0.134
Gaussian91.67±0.0391.67 \pm 0.0391.44±0.0691.44 \pm 0.061.260±0.3651.260 \pm 0.365
Sign Flip91.67±0.0391.67 \pm 0.0391.45±0.0791.45 \pm 0.071.260±0.3651.260 \pm 0.365
Label Flip91.21±0.2091.21 \pm 0.2090.28±0.4190.28 \pm 0.415.060±1.6505.060 \pm 1.650
FOE91.67±0.0391.67 \pm 0.0391.45±0.0791.45 \pm 0.071.260±0.3651.260 \pm 0.365

We do not test Dissensus since it is designed originally for fixed weighted graphs. These results demonstrate that LASA can effectively handle changing topologies without any modification. Our aggregation-budget framework can also accommodate time-varying graphs under the same assumptions. A more refined analysis explicitly capturing topology dynamics requires further development.

We hope that our reply addresses your questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We sincerely thank the reviewers for their constructive feedback and the AC for the time and consideration. We summarize the main concerns and our responses below.

  • Clarification of novelty and sensitivity. The novel combination of local loss evaluation and softmax weighting limits the influence of Byzantine nodes while retaining potentially useful candidates. Compared to previous filtering-based methods, our method LASA balances Byzantine robustness and beneficial cooperation without any prior information on Byzantine nodes. We propose an aggregation-budget framework to provide theoretical guarantees for LASA and other loss-based aggregators. We also identify conditions under which LASA achieves a tighter bound than self-only training. Empirically, LASA performs well across a broad range of inverse temperatures, and a fixed inverse temperature remains effective across all tested attacks.

  • Controlled ablation. We have added an ablation variant called LASA-LF, which removes candidates with strictly higher local losses than the self-candidate while keeping other settings unchanged. LASA-LF improves over SELF, providing empirical support for the benefit of lower-loss candidates in Theorem 4.3. LASA further outperforms LASA-LF, supporting the benefit of retaining candidates with higher losses.

  • Larger dataset, different graphs and more defenses. We have added BALANCE as a baseline and evaluated LASA on FEMNIST on a 50-node Watts–Strogatz graph with 40% Byzantine nodes. Across all tested settings, LASA achieves robust performance compared to other defenses.

  • More attacks and changing topology. On FEMNIST, LASA ranks among the top two methods under three additional attacks, namely ALIE and decentralized variants of Min-Sum and Min-Max. With the communication graph resampled at every round, LASA maintains over 90% worst-node test accuracy under every tested attack on MNIST.

We believe that these clarifications and additional results have addressed the reviewers' main concerns. We would be pleased to provide any further clarification if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.