- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes FLAIRE, an incentive framework for federated learning with heterogeneous client label distributions. It estimates label distributions from uploaded model updates using final-layer gradient inversion. The framework combines inverse-frequency label-rarity weights with a cosine-similarity-based approximation of Shapley contributions, normalizes contributions according to label coverage, and allocates rewards under a budget. Experiments on MNIST, Fashion-MNIST, EMNIST, and FEMNIST evaluate five mappings from cumulative contribution scores to client participation probabilities under non-overlapping label settings. Several configurations improve accuracy over fixed participation probabilities. However, the gains depend on the dataset and mapping, and some reported results contradict the claim of improvement across all settings.
Critical strengths
S1. The framework addresses a relevant limitation of contribution-based incentives. Clients holding rare but useful labels can be undervalued when reward allocation does not account for label scarcity or coverage. Combining contribution quality with rarity is a promising approach to encouraging their continued participation. This motivation connects client incentives to the global model’s coverage of underrepresented classes.
S2. The design reduces certain server-side data requirements. FLAIRE estimates client label distributions from existing model updates and approximates contributions without requiring a separate balanced server-side validation set. This reduces dependence on additional data and self-reported label information. The improvements reported in several configurations provide preliminary support for adapting participation to client contributions under strong label heterogeneity.
Critical weaknesses
W1. Inconsistent performance reporting undermines the central empirical claims. The paper claims at least a 5% improvement across all settings and consistent superiority of the Bayesian configuration. Table 3, however, reports an increase from 58.06% to 58.80% for FEMNIST A, or 0.74 percentage points, and a decrease from 66.28% to 61.44% for FEMNIST B, or 4.84 percentage points. These results contradict the stated generality of the gains, regardless of whether “5%” refers to an absolute or relative improvement. Several improvement entries also appear inconsistent with the corresponding accuracies. For FMNIST A, the Logistic configuration improves over the baseline by 4.67 percentage points, whereas the reported −20.59 appears to use the Bayesian configuration as its reference. For EMNIST A, the Linear configuration improves by 1.79 percentage points, while the reported improvement is 16.70%. These discrepancies require an explicit and consistently applied calculation convention. The concern is not that every configuration must outperform the baseline. It is that the reported evidence does not support the paper’s principal performance claim, and it is unclear which numerical entries are authoritative. Establishing reliable conclusions requires reconciling the tables and curves with the underlying experimental records and reporting variability across runs.
W2. Unresolved configurations and missing controls prevent reproducibility and attribution. The client-group counts sum to 24 for EMNIST Setting B and 22 for FEMNIST Setting B, while the main text states that there are 20 clients. This is consequential because client composition affects label-frequency weights, participation, and potentially local data allocation. Without identifying the configuration actually executed, the corresponding results cannot be reproduced or interpreted reliably. The paper also omits complete formulas and parameters for the five participation-probability mappings. Since these mappings directly determine participation, they are necessary to reproduce the comparisons. Moreover, comparing mapping functions against fixed participation does not isolate the effects of rarity weighting, coverage normalization, or inferred label profiles. The observed gains could depend primarily on the participation rule rather than the proposed combination of components. Taken together, these gaps extend beyond incomplete exposition. The current evidence does not establish which configuration produced the reported gains or which components account for them. Addressing this concern requires verified experimental specifications and controlled component comparisons.
Questions for authors
Q1. Numerical consistency. What do the parenthesized values in Tables 2 and 3 represent, and which configuration serves as the reference? Please reconcile the FMNIST A Logistic, EMNIST A Linear, and FEMNIST A Bayesian entries with their reported accuracies. Can the original per-round records and repeated-run summaries establish consistency between the tables and Figure 2?
Q2. Scope of the claims. How is the claim of “at least a 5% improvement across all settings” supported by the FEMNIST results? If the accuracies are correct, please revise the claim and discuss the Bayesian configuration’s degradation on FEMNIST B.
Q3. Experimental configuration. How many clients, and how many clients per group, were actually used in EMNIST and FEMNIST Setting B? Please provide the complete participation-probability update equations, parameter values, and configurations corresponding to each table and curve.
Q4. Attribution and profiling errors. With the training setup and participation mapping fixed, what happens when rarity weighting and coverage normalization are removed separately? How do reward allocations and training outcomes change when inferred label profiles are replaced with ground-truth profiles?