- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the client contribution evaluation problem in an asynchronous federated learning setup. It claims that existing evaluators rank clients in conflicting ways. To address such an issue, this work introduces FedACE, which estimates the value of asynchronous updates in separate cooperative games for each arriving update.
Critical strengths
- The paper targets an important question and finds interesting conflicts between existing evaluators.
Critical weaknesses
- The motivation of this work is based on an overstated claim, and the proposed framework is not clearly motivated. Specifically, the paper states “it is unclear what quantity their scores estimate.” In fact, several prior evaluators (e.g., TMC-Shapley, GTG-Shapley, SPACE) are designed to approximate the real Shapley value, which requires retraining a new model for each coalition. This is a solid value that prior works aim to approximate, and evaluator performance is usually estimated by correlation. Therefore, the statement made by this paper is overstated. In addition, the ground truth definition of training a new model for each coalition actually holds even under the asynchronous setup. This paper simply ignores such a fact and suggests evaluating each method on proposed games based on the dependency interval.
- The evaluation setup proposed in this work is not well motivated. The applications of contribution evaluation generally include client selection, free rider detection, and malicious client detection. However, none of these is supported by the proposed setup. Whether fitting in separate cooperative games leads to ability in these downstream tasks is unknown.
Questions for authors
- Some figures are not well presented. For example, Fig. 1b is confusing and hard to follow without an appropriate description to guide the reader on how to read it.
- Multiple uncommon terms are used without/before introduction, making the paper poorly readable. For example, “dependency interval”, “frozen-update deletion protocol”, “candidate state”, “retained events”, “anchor”, “score-blind ring”, “anchor-only auditable ledger”, etc. These terms do not follow convention description in FL context and greatly harm the readability.
- The experiment setup is not well designed. The exact estimation method, of course, gets zero error in estimation. This is too trivial to report.
- There’s no preliminary or problem formulation section. For example, the definition of the Shapley value is well-known prior work and should be introduced in the preliminary section instead of the method.