- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript investigates the federated optimization problem under the conditions of heterogeneous client objective functions and uneven client availability, as well as non-stationary client participation. It proposes a unified state-based approach for communication reduction, named TRUSS, which simultaneously addresses these two issues by maintaining only one persistent state memory of equal size to the model on each client. In terms of experiments, the authors tested five client availability patterns on CIFAR-10, CIFAR-100, and SVHN to verify the communication efficiency and convergence robustness of TRUSS.
Critical strengths
The proposed method achieves high performance without significantly increasing overhead. Furthermore, the manuscript is comprehensive in terms of workload, with rich experimental and theoretical explanations.
Critical weaknesses
The research question is relatively conventional, and there have already been many excellent works proposed in this area. Furthermore, although the manuscript contains numerous experiments, it lacks some necessary discussions, such as those related to federated learning.
Questions for authors
(1) Regarding the experimental aspect of federated learning, the experiments in the full paper are only fixed at N=10, K=50. Different training steps and client numbers should be set, especially for large-scale client verification.
(2) When the client comes back online after experiencing a long period of complete offline, will the value of be too large, potentially causing the local model to diverge?
(3) Why do FedAWE and SGFocus experience catastrophic failures on CIFAR-100 and SVHN in Table 2, even under near-IID conditions? Furthermore, it is suggested to supplement Table 1 with experimental comparisons of the closest competing method, FedSUM-CR, under the same non-stationary scenario?
(4) In Figure 1, SCAFFOLD and SCALLION have almost the same accuracy as TRUSS during training. Why are SCAFFOLD and SCALLION only achieving 59.2% and 67.1% respectively under the same settings in Table 2?
6 comments
Response to Reviewer tJDL (2/3)
Comment 1: Client-count and local-step scaling
Appendix C.2 (Figure 5) extends the original setting to SVHN/CNN experiments with 10–100 clients and ranging from 10 to 50 under exponential client participation. Increasing the client count at a fixed participation fraction generally reduces the rounds needed to reach test accuracy. More local steps also reduce rounds, with diminishing gains as grows.
Appendix C.2 (Figure 6) distinguishes this from holding the expected active-client count fixed. Its fixed-count slices are derived from an interpolated surface fitted to measured points and suggest increasing round counts as the total client count grows. The evidence establishes these trends on SVHN/CNN up to 100 clients; larger-scale behavior remains outside the tested range.
Comment 2: Stability after prolonged client inactivity
For the SGD updates analyzed in Algorithm 1, a returning client restarts at the current server model. A long absence can enlarge the memory–model discrepancy; the analysis does not assume it is uniformly small. Since , the correction contributes
to each local step. Summing the steps yields , where averages the current trajectory's raw stochastic gradients. This refresh replaces the old memory, while its influence on the trajectory remains accounted for through . Intuitively, a larger refresh upon return can compensate for changes accumulated during inactivity and preserve the influence of rarely participating clients.
Appendix A.5 jointly bounds local drift and memory-tracking error. Under Assumptions 1–3 and the step-size conditions of Theorem 1, these bounds establish convergence in expectation for independent participation with , without imposing a maximum inactivity duration. The guarantee therefore accommodates long realized absences, but does not assert that every returning client's update is small.
Response to Reviewer tJDL (3/3)
Comment 3: Baseline performance and FedSUM-CR comparison
Table 2 uses strongly non-i.i.d. data partitions with Dirichlet . “Uniform” refers to client participation probabilities, not the data distribution. Learning rates are selected separately for each method and setting under the same reported communication budget (Appendix B.3).
We hypothesize that SGFocus's poor performance at partly reflects limited benefit from additional local computation under this budget. Its stochastic non-convex theorem does not establish linear speedup in and includes a step-size restriction that tightens with (Theorem 6).
FedAWE multiplies a client's update by the number of rounds since its last participation, while its local SGD lacks an explicit drift-correction control variate (Section 5). This can amplify local drift under severe heterogeneity. Consistent with sensitivity to heterogeneity, Appendix C.1 shows CIFAR-10 mean test accuracy increasing from to as increases from to .
We have added FedSUM-CR to Table 2 across all three datasets and five participation patterns under the reported fixed communication budget. Each method is tuned separately using the same grid in Appendix B.3, with selection based on average test accuracy over the final 25 rounds. Under this protocol, TRUSS achieves higher mean test accuracy in all 15 settings. On SVHN/CNN under sine participation, TRUSS achieves , compared with for FedSUM-CR (mean standard deviation over three seeds).
We believe FedSUM-CR's retained gradients and temporally averaged corrections may slow adaptation under heterogeneous data and intermittent participation, while TRUSS's correction using accumulated model movement may better support progress within the evaluated budget.
As a sanity check, we also evaluated FedSUM-CR on SVHN/CNN at Dirichlet , as used in the FedSUM paper. Accuracy improves across all five participation patterns. Agreement with the published curves is a qualitative check because the experimental code for reproducibility is not available.
SVHN/CNN: test accuracy (%, mean SD)
| Client participation | TRUSS () | FedSUM-CR () | FedSUM-CR () |
|---|---|---|---|
| Uniform | |||
| Stationary | |||
| Staircase | |||
| Sine | |||
| Exponential |
TRUSS requires one persistent algorithmic vector per client and one at the server, compared with two at each for FedSUM-CR (Table 1). The comparison supports higher reported accuracy within the evaluated tuning grid with lower persistent algorithmic storage.
Comment 4: Figure 1 versus Table 2
Table 2 reports accuracy under an aggregate model-payload communication budget of , where is model size, whereas Figure 1 (left) compares accuracy over communication rounds. For method , the budget counts all active clients' uplink and downlink model payloads:
SCAFFOLD and SCALLION therefore consume the budget more quickly for a given participation sequence.
The and values are averages over the final 25 rounds of the budget-limited runs, with the tuned learning rates listed in Table 3. Similar accuracy at a given round can therefore coexist with different accuracy at a given communication budget. This explains the distinction between the reported metrics.
Response to Reviewer tJDL (1/3)
Thank you for reading our paper and providing constructive feedback. Our responses are below.
Practical significance and contribution
Thank you for recognizing the efficiency of TRUSS and the breadth of its theoretical and experimental evaluation.
Novelty of the research question.
We agree that client drift and partial participation have each been studied extensively, and Section 2 and Table 1 compare TRUSS with many of these works. However, data heterogeneity, irregular client participation, and communication and memory overhead remain major barriers between federated learning methods and their practical, large-scale deployment. A recent account of production federated learning systems reports that coordinating training across devices that differ in compute, bandwidth, availability, and data hinders broader adoption, and that modern large models are orders of magnitude beyond what classical cross-device federated learning has handled (Daly et al., 2024). Each barrier has a direct optimization cost. Heterogeneous data cause local drift, which limits the benefit of local computation. Irregular, time-varying availability biases the model toward frequently available clients and can change the objective that is actually optimized. Communication overhead is widely identified as the main bottleneck for real-world deployment, because large model updates must be exchanged repeatedly (Le et al., 2024). The same issues now arise beyond classical federated learning: federated-averaging variants with many local steps are used to train large language models across poorly connected, geographically distributed clusters whose available resources change during training (Douillard et al., 2023). Our experiments are run on such a geographically distributed deployment, with nodes in different U.S. states (Section 7).
These barriers also interact. Methods that correct drift and participation bias typically pay for these corrections with exactly the overhead that limits deployment: extra control vectors in communication, additional client state, or auxiliary server memory. For large models, parameter transmission can dominate the per-round cost. Per-client memory is limited on resource-constrained devices. Server storage that grows with the number of clients, as in MIFA, becomes prohibitive at scale. Addressing all three barriers at once, rather than trading one for another, is therefore essential for deploying federated learning in practice.
Concretely, what remains open is achieving all of the following at once:
Each existing method gives up at least one of these properties (Table 1):
To our knowledge, TRUSS is the first method to satisfy all four properties. The enabling idea is that a single memory per client performs both corrections: the gap between a client's memory and the current server model supplies the drift correction, and it also records the server's movement while the client was absent (Section 4). On the analysis side, analysis-only backward participation weights establish descent at the actual server model without requiring unbiased server updates. This is what allows the guarantee to cover unequal, time-varying participation probabilities (Sections 5.2–5.3).
References.
Daly et al. (2024). Federated Learning in Practice: Reflections and Projections. IEEE TPS-ISA. arXiv:2410.08892
Le et al. (2024). Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspective. arXiv:2405.20431
Douillard et al. (2023). DiLoCo: Distributed Low-Communication Training of Language Models. arXiv:2311.08105