- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The present paper explores Byzantine robustness in q-fair federated learning, whereby each client contributes its update scaled by its loss. Since both of these are combined, the server is unable to distinguish honest clients that have high losses from adversaries scaling their updates. The authors demonstrate that under the assumption of label skew with shared class-conditionals, the honest clients' updates continue being nonnegative combinations of class directions, but this time with weights summing up to the fairness parameter, such that a cone is formed rather than a simplex. The FUSE aggregator makes use of a small labeled dataset to construct class reference directions, recover the coefficients of the updates via nonnegative least squares, filter based on the mass of the coefficients and the cone residual, and normalize the step sizes using estimated losses. Both recovery and exact-screening guarantees are established. On four benchmark datasets and seven attacks, FUSE achieves an increase in accuracy on worst-attack performance of 3.9-30.1 points while retaining most high-loss honest clients.
Critical strengths
-
The main point of the author's work is that when you scale for fairness, you are able to keep honest updates as non-negative class mixtures. However, you change the coefficient sum of those honest updates. When you do this, you turn what would have been a simplex into a cone. This is not simply stated by the authors. The authors are able to show that when using the checkpoint, client, and trusted basis as fixed components, the cone residual remains at 0.178 on MNIST while the simplex residual increases from 0.185 to 1.077. Additionally, when the authors swapped out only the cone score with a simplex score within the same pipeline, the Byzantine recall for Min-Max and Min-Sum fell from 1.00 down to approximately 0.6. This demonstrates that the improvements seen were due to the proposed idea itself, and is why I feel that the author's contribution is valid.
-
Robustness without compromising on fairness, backed up with thorough evaluation. On each of the four datasets, FUSE has the highest worst attack accuracy (3.9 to 30.1 percent higher than the most robust benchmark) when evaluated with a conservative worst attack metric based upon seed-wise worst attacks across five different seeds and seven different types of attacks. In addition to being the most robust approach to Byzantine attacks, the authors also show which clients were rejected by the system. While FUSE rejected only 0-9.7 percent of those clients that had the largest loss during training, and were not malicious, the norm-based filtering approaches rejected 22-36% of those same clients. However, FUSE was able to reject 85.7-99.8 percent of all submissions from malicious clients. These findings provide strong evidence in support of the author's primary claim, and the transparency of the authors' reporting (including showing failed runs, etc.) helps build confidence in the authors' results.
Critical weaknesses
-
The practical significance of the fairness being preserved is not argued for, and the comparison could be in favor of FUSE. Fairness is evaluated by rejections rather than by improvement of q-fair training on the weak clients. The authors' own figures cast doubt on that. For MNIST, accuracy for the worst-10% group decreases as q increases (86.21 when q = 0.5 and 84.85 when q = 2). Also, the q=0 simplex screening gives 96.55/92.13 clean and 93.95/89.26 when q = 1 and robustness against tested attacks. There is no baseline that removes fairness scaling and applies an existing robust defense, so it remains unclear whether the defense of q-FFL is worth it.
-
Second, all approaches seem to employ a single learning rate for each dataset when applying loss-scaling updates. Some baselines are unstable even without attack (q-FFL on EMNIST: 63.79 ± 34.5), and FUSE comes with step-size control out-of-the-box. It is significant since it concerns the motivation and the main comparison.
-
The evidence backing the central arguments is less convincing than claimed. Exact recovery is only guaranteed by the theory when the number of Byzantines is known, and each attack is far apart, using metrics on the server side that are not available. No experiment has ever verified these preconditions on their benchmark data sets. Moreover, there is no proof of convergence and no bound when the screening step fails, which is where the experiments take place.
-
The analysis of the mechanism, aimed at verifying the cone geometry benefits, also contains incongruities between the text and the tables. In particular, the AGNews ablation study mentions 53.34/38.33 for the full FUSE mechanism and characterizes the loss normalization as a trade-off, but Table 16 gives 79.45/62.86 for the normalized version and 35.92/25.22 for the non-normalized one. According to Table 16, the version without scale screening outperforms full FUSE despite the claim that this stage is never harmful.
-
IPM recall of the cone score 0.69 in Section 6.2, 0.62 in Table 20 and 0.59 in Table 17. Given that this analysis involves just one seed, the mentioned inaccuracies undermine my confidence in the conclusions about the components' performance.
Questions for authors
-
As shown in Table 19, for MNIST, the q = 0 run gets 96.55/92.13 compared to 93.95/89.26 for q = 1, while Table 23 shows that worst-10% accuracy decreases with q. Is it the difference in the step size because the losses are less than one, or do q-fair optimization methods not favor the least well-performing clients here? Do you have any prior results where q > 0 helps improve worst-10% accuracy or reduces client accuracies' variance compared to q = 0?
-
Have you used different learning rates for each method, or did you set one in Table 4 for everyone? Some baselines appear quite unstable without an attack (for instance, q-FFL on EMNIST with 63.79 ± 34.5). Did they get some step size control like Lipschitz normalization in q-FedAvg?
-
The numbers mentioned in E.1 are 53.34/38.33 for full FUSE and 55.45/40.77 when not using loss normalization, whereas Table 16 gives 79.45/62.86 and 35.92/25.22. Also, the text mentions 1000 rounds, whereas the caption states 2000. Which are the right numbers? If Table 16 is correct, how to explain the fact that the variant without the scale screen (80.44/66.99) performs better than the full FUSE given the claim that this phase never decreases the worst-attack accuracy?
-
The recall of the cone-score on the MNIST dataset under IPM in Section 6.2 is 0.69, 0.62 in Table 20, and 59.4% in Table 17. Are these separate runs or different pooling windows? Please specify which run demonstrates the 2.44 gain.
-
From your logs, give the measured honest radial spread r_H and cone variation, as well as the frequency at which log κ > 2r_H and the other separation conditions hold for each of the attacks?
Thanks for the careful review. We answer the five questions below.
1. Does q>0 actually improve fairness here?
The original q-FFL paper does show that positive q can improve low-performing clients, but q is tuned per dataset rather than fixed to 1. Li et al. (2020, Table 2/App. F.1) report:
Thus q-FFL motivates giving high-loss clients more relative influence, but it does not imply that q=1 must improve W10 on every dataset.
We also reran q=0/1/2 with the same tuning budget for each q: 3 LRs x 2 clean tuning seeds, then 3 fresh confirmation seeds.
So in our setting, q=1 does not give a clear W10 gain. On CIFAR it slightly reduces variance, while on MNIST it does not. We will therefore narrow the claim: FUSE preserves the intended high-loss weighting when q>0 is used; we do not claim that q=1 itself always improves fairness.
The reviewer is also right that step size changes with q. Our q-FedAvg diagnostic gives effective α=0.090/0.00524/0.00294 for CIFAR q=0/1/2 and 0.300/0.140/0.083 on MNIST. This suggests under-optimization may contribute, but it is not a causal proof.
2. Were learning rates and step-size control compared fairly?
The original main table used one LR per dataset, so we added a clean-tuned comparison. Every method/q gets 3 LR candidates x 2 validation seeds; the selected setting is frozen before 3 fresh test seeds. BOBA/FLTrust use native q=0 and H-nobs uses q=1.
We also separated screening from normalization with a fixed-LR 2×2 test:
Thus normalization helps once bad updates are screened, but normalization alone does not provide robustness. We will add the independently tuned native-baseline results and make the LR protocol explicit.
3. Why do the AGNews numbers differ?
The manuscript mixes two diagnostic protocols and did not label them clearly enough.
The shorter 1000-round factorial was used to keep the full component grid affordable; AGNews's main horizon is 2000 rounds. The sentence in E.1 saying Tables 13–16 all use 1000 rounds is therefore misleading and will be corrected. These numbers should not have been compared as if they came from one protocol.
This also fixes the wording about the scale stage: we will not say it “never hurts.” Its effect is attack- and horizon-dependent; the ablation only supports its role against radial attacks.
4. Why are the IPM recalls 0.69, 0.62, and 59.4%?
These numbers come from different analyses, and one summary number was not synchronized:
The 2.44-point gain is from the matched q=1 training control in Tables 19–20: global accuracy is 89.91 with simplex screening and 92.35 with scale+cone, i.e. +2.44 points. Table 20's 62% recall is the corresponding screening statistic. Therefore the 0.69 in Sec. 6.2 is a stale summary value and will be corrected to 0.62. Table 17 remains separate because it reports the production FUSE run, not the matched simplex-control run.
5. Do the theorem conditions hold in training?
We agree this should be shown directly. We audited the round-level separation conditions over 3 seeds:
This also explains where the theorem is strong and where it is not: Magnitude/Min-Max/Min-Sum satisfy the separation conditions almost always, while SignFlip and IPM overlap more with honest updates. The theorem is therefore a conditional certificate, not a claim that its sufficient conditions hold every round. We will add this audit and report the corresponding honest radial spread and cone variation with it.