- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a new federated learning mechanism by combining Byzantine robustness, energy efficiency, and incentive alignment. They introduce AFL (Autonomous Federated Learning), where the server relies on an external validation set to validate client updates and flag Byzantine clients. They also propose Proof-of-Useful-Learning, a new reward mechanism to compensate clients for verified utility (accuracy) gain per unit energy spent.
Critical strengths
The proposed approach aims to provide an integrated treatment of Byzantine robustness, client incentives, and energy efficiency in federated learning settings, which can be useful in practical energy-constrained settings.
Critical weaknesses
-
I do not think I agree with the theoretical guarantees. The convergence analysis seems to rely on an assumption that the global loss is L-smooth? In fact, Appendix A starts with a descent lemma on the global gradient using the smoothness assumption. And then convergence is demonstrated as a stationarity result for the global gradient. This doesn't make much sense to me. The global loss will include the local loss functions of the Byzantine clients, which do not need to satisfy the smoothness condition. They are by definition Byzantine and can define their local loss function to be any function. If there is no notational mix-up here the convergence analysis seems ill-defined.
-
Both Byzantine resilience and/or energy efficiency, along with various client incentivizes have been studied extensively in the context of federated learning. While combining these with a unified treatment is useful, without rigorous new technical insights the novelty is limited. The theoretical gaps mentioned above also amplifies the need for a more rigorous treatment of the problem.
-
I do not think it is realistic to assume D_{val} is private. The algorithm is interactive, hence I think Byzantine adversaries can adapt to it, even though they cannot observe D_{val} directly they can observe the server actions. Accordingly, I think this should also be incorporated in the Byzantine robustness analysis. On another note, how does the server ensures it has a representative validation set, since the paper particularly emphasizes the non-iid federated setting?
-
The selection strategy in equation (1) appears as several heuristics combined together, and it is not clear the theoretical justification behind these arguments. I will list some of these below:
-
Where did equation 4 come from, can the authors provide more intuition to this specific selections? How is \mu_trust selected in practice? Similarly, how is \gamma_1, \gamma_2, \gamma_3 tuned in practice, and how can we make sure we select good parameters for different applications?
-
How is \delta_sel selected (equation 2) in practice? This seems like a critical choice, since a bad selection could turn the Byzantine clients into a majority.
-
How do you select "K" in Section 2? This will also affect Byzantine robustness.
- Similarly, the experiments also do not provide ablations to validate the specific choices of the heuristic parameters discussed above. Since the paper relies on many heuristic design parameters, this is critical to validate the numerical performance.
Questions for authors
Please see the points raised above in the weaknesses. Other than those, some minor comments include:
-
tau_i in equation 1 is only defined in equation 4.
-
In equation 3, only selected clients will train on local data. What if the selected clients are not available? As a result Byzantine clients may become majority, in which case Byzantine resilience may not hold?
-
"MAD" is used several times in the intro, but only defined later in Section 3 in the paragraph before equation 7.