- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces History-Guided Directional Aggregation (HGDA), a server-side defense for Byzantine-robust federated learning. Each client is assigned a score derived from how well its recent update history aligns with two reference checkpoints obtained by training on a small trusted dataset. Clients whose current update directions are identical are grouped together and share a single weight determined by the highest score in the group, and the resulting aggregated direction is applied through an Adam update. The authors give a convergence analysis under stated conditions and evaluate robustness on five datasets against multiple attacks and participation settings.
Critical strengths
-
The problem is relevant: filtering malicious updates without discarding the useful contributions of heterogeneous benign clients. Pairing a trusted reference with per-client alignment histories is a sensible design choice.
-
The experiments span multiple datasets, participation regimes, and attack schedules, and the appendix contains ablations and phase-specific breakdowns that make the robustness claims easier to assess.
Critical weaknesses
-
The direction-grouping step depends on normalized updates being exactly equal. This creates an easy evasion: an attacker can take a copied benign update and add a small, distinct perturbation per malicious client, so that each copy is treated as a separate direction and receives its own weight. Each client could then keep a favorable historical score while the group as a whole regains its full aggregate influence. The Adaptive Mimic experiment does not cover this case, since all attackers still submit exactly identical updates. The paper should evaluate perturbed-copy attacks and report whether the scoring mechanism alone can bound the attackers' influence once exact grouping no longer triggers.
-
The choice of best test accuracy as the primary metric is not justified and appears to overstate the defense. In Table A.51, best accuracy reaches 85.29%, whereas the mean accuracy over attack-on rounds is only 38.00%. A gap of this size indicates that peak accuracy can hide severe degradation while the attack is active, and therefore does not faithfully reflect how well the defense protects the model.
Questions for authors
-
How does HGDA behave when each attacker submits a copied benign update with a small, client-specific perturbation? Please sweep the perturbation magnitude and report both the attackers' combined aggregation weight and the resulting model accuracy.
-
Do the reported gains over the baselines hold when the metric is final accuracy or average accuracy over the late training rounds, rather than best test accuracy?
9 comments
Rebuttal by Authors (Part 2)
Q1 & W1: Robustness Across Perturbation Scales and Scope
The tables below report three-seed CIFAR-10 test accuracy (%) within the shared communication budget as Last-100 mean ± SD (Best mean), where Last-100 averages the final 100 rounds, Best is each run's maximum and bold marks each column's best (Part 3 then reports the complete 20%–80% sweep pairing combined aggregation weight with model accuracy):
% Byzantine clients
| Method | |||||
|---|---|---|---|---|---|
| HGDA () | 80.85±1.30 (82.19) | 75.93±1.04 (79.83) | 76.61±0.90 (80.31) | 78.97±0.86 (81.62) | 80.94±1.33 (82.35) |
| HGDA () | 80.49±0.93 (82.14) | 73.60±1.56 (78.95) | 74.77±0.15 (78.84) | 78.13±1.60 (81.03) | 80.19±0.93 (81.56) |
| FLTrust-Early | 62.58±1.99 (72.59) | 63.35±2.51 (73.19) | 64.14±2.02 (73.16) | 68.91±1.87 (75.96) | 76.57±2.21 (79.84) |
| Fed-NGA | 70.82±2.10 (77.73) | 69.86±3.65 (77.32) | 70.75±3.19 (78.29) | 73.08±2.62 (78.37) | 81.27±1.17 (81.84) |
| BOBA | 59.44±1.97 (61.77) | 60.13±2.68 (62.00) | 60.27±1.98 (62.37) | 61.41±1.75 (63.12) | 59.56±2.27 (61.22) |
% Byzantine clients
| Method | |||||
|---|---|---|---|---|---|
| HGDA () | 71.26±0.30 (73.29) | 55.42±0.52 (65.81) | 55.15±2.03 (65.97) | 56.56±1.05 (66.60) | 70.22±0.98 (72.79) |
| HGDA () | 70.86±0.89 (73.06) | 57.35±2.15 (64.52) | 55.10±2.68 (63.86) | 57.13±1.47 (65.94) | 70.45±1.13 (72.71) |
| FLTrust-Early | 50.05±0.42 (61.56) | 50.79±1.72 (61.43) | 50.93±2.00 (61.61) | 53.42±1.62 (63.04) | 66.27±1.83 (70.31) |
| Fed-NGA | 47.40±6.48 (56.61) | 47.39±7.88 (57.43) | 47.91±3.04 (56.00) | 51.00±0.97 (58.96) | 62.83±0.69 (65.96) |
| BOBA | 20.02±4.31 (25.89) | 20.90±5.39 (28.33) | 22.86±3.34 (28.20) | 20.41±7.54 (28.21) | 21.40±5.93 (26.78) |
Across all five scales at % and 80%, HGDA () wins 29 of 30 external-baseline comparisons on Last-100 (save Fed-NGA's 81.27±1.17% vs. 80.94±1.33% at 40%, , within one SD) and all 30 on Best accuracy, with tighter SDs (0.30%–2.03% vs. up to 7.88%):
- When tiny perturbations () bypass exact grouping, HGDA degrades gracefully at %, stays bounded without collapse at % and leads every baseline. At , exact copies share one group with the copied benign client. At , near-copies split into distinct directions and receive separate weights, returning HGDA () to its client-level behavior (matching client-level HGDA under exact copies at %: 55.42±0.52% vs. 55.16±1.73% Last-100, and 65.81% vs. 66.05% Best). Crucially, accuracy never collapses: at %, HGDA () dips gracefully by only 4.93 Last-100 and 2.36 Best points from , beating Fed-NGA by 6.06 Last-100 (with under one-third of its SD, 1.04% vs. 3.65%) and 2.51 Best points, while even at 4-to-1 attacker majority (%) degradation remains bounded, leading FLTrust-Early by 4.63 Last-100 and 4.38 Best points.
- How dual-reference historical scoring (), anchored in , bounds near-copy bias (). Because both references are recomputed from on each round, repeatedly amplifying one skewed benign client shifts toward its local classes and weakens subsequent alignment with . A longer window ( vs. ) further stabilizes scores against short-term fluctuations: across the full five-fraction (%) by five-scale grid (see Part 3), beats in 21 of 25 Last-100 and 24 of 25 Best cells (winning 9 of 10 near-copy settings on Last-100 and all 10 of 10 on Best across 20%–80%).
- Larger perturbations () weaken reference alignment and restore accuracy. As grows, the shared copied component shrinks by while independent orthogonal components average down across attackers, returning HGDA () at to its unperturbed level at both 40% and 80%.
Scope. Even when near-copies bypass Eq. (5) (), HGDA's -anchored historical scoring keeps degradation bounded and outperforms every external baseline, supporting our choice to keep Eq. (5) exact here (preserving benign non-IID diversity, Section 1) while leaving dedicated near-copy weight control for future work (Section 5).
Rebuttal by Authors (Part 3)
Q1 & W1: Combined Weight and Accuracy Across All 25 Settings
Following Parts 1–2, the table below pairs combined aggregation weight with model accuracy across all 25 settings (three seeds each). Unlike opposing attacks where HGDA drives attacker weight near zero, near-copy attackers submit nearly collinear copies of the highest-scoring benign update , testing how Eqs. 1–4 (-anchored historical scoring) and Eq. 5 work together:
- Wt (%): three-seed mean over the shared budget of total weight on copy-containing groups (the shared benign group at , and the sum of distinct copy weights at ).
- Test Acc (%): three-seed CIFAR-10 accuracy as Last-100 mean ± SD (Best mean), bolding the higher mean.
| Wt (%) | Test Acc (%) | Wt (%) | Test Acc (%) | ||
|---|---|---|---|---|---|
| 20% | 3.85 | 83.00±0.54 (84.46) | 4.95 | 82.46±0.53 (83.96) | |
| 20% | 30.15 | 81.68±0.23 (83.83) | 33.33 | 80.75±0.50 (83.18) | |
| 20% | 30.51 | 81.62±0.55 (83.94) | 33.40 | 80.75±0.42 (83.25) | |
| 20% | 28.90 | 82.68±0.11 (84.53) | 30.98 | 81.82±0.71 (84.09) | |
| 20% | 10.80 | 82.63±0.62 (84.19) | 9.20 | 82.00±0.12 (83.51) | |
| 40% | 4.56 | 80.85±1.30 (82.19) | 5.66 | 80.49±0.93 (82.14) | |
| 40% | 48.01 | 75.93±1.04 (79.83) | 51.60 | 73.60±1.56 (78.95) | |
| 40% | 48.22 | 76.61±0.90 (80.31) | 51.51 | 74.77±0.15 (78.84) | |
| 40% | 45.77 | 78.97±0.86 (81.62) | 47.03 | 78.13±1.60 (81.03) | |
| 40% | 21.97 | 80.94±1.33 (82.35) | 19.98 | 80.19±0.93 (81.56) | |
| 50% | 4.97 | 79.44±1.00 (81.03) | 6.36 | 78.72±0.89 (80.40) | |
| 50% | 54.53 | 71.62±0.64 (77.64) | 55.78 | 68.54±3.54 (75.80) | |
| 50% | 55.00 | 71.82±1.95 (77.59) | 56.32 | 67.73±1.93 (76.00) | |
| 50% | 53.29 | 76.21±0.79 (79.63) | 53.66 | 74.39±0.17 (78.62) | |
| 50% | 27.97 | 79.23±1.23 (80.84) | 25.02 | 78.82±1.29 (80.54) | |
| 60% | 6.14 | 77.75±0.89 (79.43) | 7.52 | 77.53±1.18 (79.54) | |
| 60% | 61.78 | 66.37±1.82 (73.30) | 65.44 | 63.84±2.03 (71.43) | |
| 60% | 60.91 | 66.69±2.66 (74.40) | 65.19 | 64.11±2.03 (71.80) | |
| 60% | 60.36 | 71.72±1.06 (76.94) | 61.58 | 71.88±0.61 (76.82) | |
| 60% | 34.60 | 77.85±0.83 (79.29) | 31.42 | 77.35±0.95 (79.04) | |
| 80% | 9.31 | 71.26±0.30 (73.29) | 11.73 | 70.86±0.89 (73.06) | |
| 80% | 79.06 | 55.42±0.52 (65.81) | 83.44 | 57.35±2.15 (64.52) | |
| 80% | 78.69 | 55.15±2.03 (65.97) | 83.27 | 55.10±2.68 (63.86) | |
| 80% | 76.90 | 56.56±1.05 (66.60) | 77.89 | 57.13±1.47 (65.94) | |
| 80% | 51.60 | 70.22±0.98 (72.79) | 46.69 | 70.45±1.13 (72.71) |
- Exact copies (, 20%–80%): Although copies hold
63.14%–87.69%(40%–80%) of pre-grouping client scores, Eq. (5) merges them into the copied benign client's group (3.85%–9.31% weight across 20%–80%, zero duplication gain). - Near-copies (): graceful degradation at 20%–40% and bounded influence without collapse at 50%–80%. Once perturbations bypass
torch.equal, near-copies receive client-level weights as valid benign replicas, yet lowers combined weight below in all 10/10 near-copy settings across 20%–80% (including−4.38and−4.57pp at 80%, and well belowFLTrust-Early's 61.16%–61.22% at 40% and 87.28%–87.56% at 80%). At 20%–40%, accuracy degrades gracefully by only1.32–4.93Last-100 points. Even under majority corruption (50%–80%), attacker influence and accuracy remain bounded without collapse, winning9/10Last-100 and10/10Best pairs over while outperforming every baseline in Part 2 (even at 80%, leadingFLTrust-Early,Fed-NGAandBOBAby+4.22–4.63,+7.24–8.03and+32.29–34.52Last-100 pp), because near-copies keep positive scores only while staying aligned with both references (Part 1). - Larger perturbations (, 20%–80%): Stronger deviations lose alignment with , cutting weight (
10.80%–51.60%across 20%–80% at ) and restoring accuracy (70.22% Last-100 / 72.79% Best at 80%). Together with Parts 1–2, this answers W1 directly: HGDA does not hinge on Eq. (5) alone, as -anchored historical scoring degrades gracefully at 20%–40%, keeps near-copy influence bounded without collapse up to 80% and outperforms all baselines throughout.
Rebuttal by Authors (Part 1)
Q1 & W1: Exact Direction Matching and Adaptive Near-Copy Evaluation
Thank you for raising this question. In our implementation, direction equality is checked by exact elementwise equality (
torch.equal) on unit-normalized updates, with no angular tolerance. As motivated in Section 1, HGDA targets broad Byzantine robustness across diverse tasks, poisoning attacks, Byzantine fractions and participation conditions under non-IID data: it combines dual-checkpoint alignment histories (Eqs. 1–4, anchored in Ds) with unit-direction weighting (Eq. 5) that removes identical copies while preserving benign directional diversity, and we candidly noted the near-copy boundary in Section 5. To test this setting directly, we extended our adaptive Mimic attack (Section 4.4) with orthogonal perturbations across five scales: even when tiny perturbations bypass exact grouping, HGDA degrades gracefully without collapsing and outperforms the strongest Mimic baselines across the sweep.1. Implementation and why exact matching is a reasonable design choice
Each positive-score client's nonzero update Δit is normalized to a unit vector uit=Δit/∥Δit∥2, and updates share a group in Eq. (5) only when all coordinates match (
torch.equal). Keeping Eq. (5) exact is well-motivated for three reasons:2. Adaptive near-copy Mimic across perturbation scales
Under our Section 4.1 CIFAR-10 setup (three seeds within the shared communication budget), the adaptive attacker tailors target selection to each defense's own mechanism. For HGDA, the attacker identifies the highest-scoring (most trusted) benign update Δ⋆t (u⋆t=Δ⋆t/∥Δ⋆t∥2) each round and perturbs it along independent orthogonal unit vectors zit while preserving ∥Δ⋆t∥2:
Δit=∥Δ⋆t∥2uit,uit=1+ε2u⋆t+εzit,∥zit∥2=1,⟨zit,u⋆t⟩=0.
Sweeping ε∈{0,10−3,0.1,1,10} across m∈{20,40,50,60,80}% spans exact copies (ε=0), nearly collinear copies (ε=10−3,0.1 with cosines 0.9999995,0.995 to u⋆t) that bypass
torch.equaland wider perturbations (ε∈{1,10}).In our CIFAR-10 Mimic matrix (Table A.20), classical aggregators (Multi-Krum, RFA, Coordinate Median, Trimmed Mean and Bulyan) drop to 35.47%–57.94% or N/A at m=40% and below 40% or N/A at 80%. We therefore compare against the three external baseline families competitive at 40% under Mimic, evaluating all three at 40% and 80%: FLTrust-Early (best external baseline at 77.74% and 67.27%, beating FLTrust-Late at all fractions), Fed-NGA (best server-data-free baseline at 77.32% and 59.04%) and BOBA (68.68% at 40%). Against each baseline, the attacker likewise adapts target selection to its rule: copying the highest-trust benign update each round for FLTrust-Early, the benign client with the largest sample-count weight for Fed-NGA and a benign client near BOBA's server PCA subspace selected to minimize aggregate alignment with the server update (perturbed in that subspace).