acceptodds
Withdrawn from ICLR 2027

Valuing Updates, Not Rounds, in Fully Asynchronous Federated Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contribution evaluation assigns credit to clients in federated learning and underpins incentives and participation. Existing methods rely on synchronous rounds, which supply fixed participants, a common reference model, and a well-defined aggregate for every subset of updates. Fully asynchronous training removes all three. Replayed on the same training traces, eight published evaluators rank clients in conflicting ways, so it is unclear what quantity their scores estimate. We introduce FedACE, which values each asynchronous update through its own cooperative game. The players are the update itself and the updates applied between its dispatch and its arrival, the reference is the global model from which it was computed, and every coalition is composed in the recorded arrival order. All three are read from the execution record rather than chosen by the evaluator. We prove that this game is well-posed, that the recorded sequence of global models identifies its composition order under an explicit non-degeneracy condition, and that it coincides with the classical round game when coalitions are composed by a synchronous batch rule. FedACE solves small games exactly by enumeration, estimates larger ones with score-blind random permutation rings, credits each update exactly once, and never alters the training trajectory. Because the game admits exact evaluation, we score every estimator against the same exact Shapley value rather than a proxy. Across 27,000 updates on CIFAR-10, CIFAR-100, and Tiny-ImageNet, FedACE solves 73.4% of the games exactly. On all 681 nine-update games, it uses 23% of the utility queries required by exhaustive enumeration and is closer to the exact value than every compared estimator in each of the nine training runs. At the same query budget, Monte Carlo sampling incurs 1.7 to 1.9 the error of FedACE.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nmKcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the client contribution evaluation problem in an asynchronous federated learning setup. It claims that existing evaluators rank clients in conflicting ways. To address such an issue, this work introduces FedACE, which estimates the value of asynchronous updates in separate cooperative games for each arriving update.

Critical strengths
  • The paper targets an important question and finds interesting conflicts between existing evaluators.
Critical weaknesses
  • The motivation of this work is based on an overstated claim, and the proposed framework is not clearly motivated. Specifically, the paper states “it is unclear what quantity their scores estimate.” In fact, several prior evaluators (e.g., TMC-Shapley, GTG-Shapley, SPACE) are designed to approximate the real Shapley value, which requires retraining a new model for each coalition. This is a solid value that prior works aim to approximate, and evaluator performance is usually estimated by correlation. Therefore, the statement made by this paper is overstated. In addition, the ground truth definition of training a new model for each coalition actually holds even under the asynchronous setup. This paper simply ignores such a fact and suggests evaluating each method on proposed games based on the dependency interval.
  • The evaluation setup proposed in this work is not well motivated. The applications of contribution evaluation generally include client selection, free rider detection, and malicious client detection. However, none of these is supported by the proposed setup. Whether fitting in separate cooperative games leads to ability in these downstream tasks is unknown.
Questions for authors
  • Some figures are not well presented. For example, Fig. 1b is confusing and hard to follow without an appropriate description to guide the reader on how to read it.
  • Multiple uncommon terms are used without/before introduction, making the paper poorly readable. For example, “dependency interval”, “frozen-update deletion protocol”, “candidate state”, “retained events”, “anchor”, “score-blind ring”, “anchor-only auditable ledger”, etc. These terms do not follow convention description in FL context and greatly harm the readability.
  • The experiment setup is not well designed. The exact estimation method, of course, gets zero error in estimation. This is too trivial to report.
  • There’s no preliminary or problem formulation section. For example, the definition of the Shapley value is well-known prior work and should be introduced in the preliminary section instead of the method.
Reviewer_TnssOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FedACE for contribution evaluation in fully asynchronous federated learning. Each arriving update defines a cooperative game containing itself and the updates applied since its dispatch. Coalition values use frozen uploads and mixing coefficients, composed in factual arrival order. FedACE computes small games exactly, estimates larger games using randomized permutation rings, and credits only the arriving update. The paper establishes properties of this construction and evaluates it on 27,000 updates across three image datasets. On the audited nine-update games, the ring estimator improves accuracy over the compared estimators while using substantially fewer queries than exhaustive enumeration.

Critical strengths
  • The paper gives asynchronous contribution evaluation a precise target. The distinction between factual composition order and the permutations used to compute Shapley values is useful, and the construction makes its counterfactual assumptions explicit. This provides a sound basis for discussing what an asynchronous contribution score measures.

  • Evaluating approximation error against exact Shapley values of the same game is a strength. The matched-budget comparisons and anchor-value errors provide concrete evidence that the proposed estimator works well on the audited games. The reported unbiasedness and per-game efficiency also have clear justifications.

Critical weaknesses
  • The experiments do not adequately establish the solver’s contribution relative to closely related sampling methods. The ring estimator combines position stratification and antithetic pairing, but the main randomized comparisons are ordinary MC/TMC. The cited Mitchell et al. (2022) already study antithetic and more structured permutation sampling. Comparisons with these methods at matched unique-query budgets are needed to determine how much the ring construction adds. This matters because query efficiency is one of the paper’s main contributions.

  • Table 12 mixes differences in valuation rules with differences in the events being evaluated. The methods’ client scores use the full stream, while the exact reference is available only for events with complete coalition tables. Since this coverage depends on interval size and staleness, it can vary systematically across clients. The caption acknowledges the issue, but calibrating it for FedACE does not resolve it for the other methods. The resulting correlations therefore provide limited evidence about how their valuation targets differ. This concern does not affect the matched-game comparisons in Table 1.

Questions for authors
  • How does FedACE compare with antithetic MC and a stronger permutation sampling method, such as orthogonal sampling, at the same unique-query budget on the existing exact tables?

  • Do the conclusions of Table 12 hold when every method uses exactly the same reference-covered events and per-client denominators? Please also report how coverage varies across clients or staleness groups.

  • How does the proposed game differ from In-Run Data Shapley (ICLR 2025) and Ripple Shapley (AAAI 2026)? A comparison of the valued unit, coalition intervention, and temporal scope would help establish which aspects of the formulation are new.

Reviewer_cRnGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedACE for contribution evaluation in fully asynchronous federated learning. It constructs an event-level cooperative game based on the dependency interval of each update, evaluates coalitions in their recorded arrival order, and estimates Shapley values through exact enumeration or randomized permutation rings. An anchor-only ledger avoids duplicate credit assignment. Experiments on three datasets demonstrate accurate Shapley estimation with reduced evaluation cost.

Critical strengths
  1. The paper addresses an interesting problem of contribution evaluation in fully asynchronous federated learning.

  2. The event-level cooperative game provides a clear formulation of contribution evaluation without relying on synchronous rounds.

  3. The theoretical analysis establishes several properties of the proposed formulation, and the experimental evaluation is relatively comprehensive.

  4. The comparisons with exact Shapley values and budget-matched estimators provide convincing evidence of the proposed solver's estimation efficiency.

Critical weaknesses
  1. The main concern is whether the proposed game accurately reflects actual client contributions. The frozen-update replay retains historical local models and aggregation coefficients, which may differ substantially from the training trajectory after removing a client. The justification for this contribution definition remains insufficient.

  2. Several theoretical results, particularly Theorem 1, follow directly from the standard properties of Shapley values once the cooperative game is defined. Moreover, the anchor-only accounting does not guarantee a globally consistent allocation of the overall training utility.

  3. The experiments mainly evaluate approximation accuracy against the exact Shapley values of the proposed game. However, this does not directly establish the effectiveness of the resulting client contributions for incentive allocation or participation decisions.

  4. The experimental settings are relatively limited. The evaluation considers a specific FedAsync protocol and simulated staleness patterns, while the sensitivity to other asynchronous aggregation mechanisms and public utility datasets remains insufficiently explored.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.