- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper theoretically investigates the benefits of assigning client-specific memory window sizes, rather than using a shared window size across all clients, in federated learning with continuously arriving data. Specifically, the authors propose a client-specific data refresh with reset stochastic solve algorithm. They first establish an upper bound on the performance gain over a common memory window and then derive a minimax lower bound that matches the upper bound up to a constant factor. Furthermore, by comparing the risks of the two approaches within a specified estimator class, they demonstrate that the risk ratio can be unbounded. When drift bounds are unavailable, the authors address this issue through validation samples and expert aggregation. Experiments in two settings, involving temperature stations and clinics, show that client-specific memory windows outperform a common memory window, although the performance gains diminish as drift heterogeneity decreases.
Critical strengths
- Interesting problem formulation and theoretical analysis. The scenario in which clients with different drift rates maintain client-specific memory window sizes is interesting and well-formalized theoretically. The proposed method addresses the issue of reusing retained samples through a reset accelerated solve procedure and disjoint batches.
- Tight theoretical bounds. The paper establishes not only an upper bound but also a minimax lower bound that matches the upper bound up to a constant factor, providing a tight characterization of the theoretical guarantees.
- Additional theoretical and methodological support. The paper further supports the proposed approach by analyzing the actual risk ratio within a specified estimator class and introducing an expert aggregation strategy for settings where drift bounds are unknown.
Critical weaknesses
- Limited empirical evidence. Although client-specific memory windows appear to offer performance improvements over a common memory window, the results of the recorded-order experiment on temperature stations and the comparison with the data-floored common-window baseline in the clinic experiments raise questions about the robustness and significance of these gains.
- Limited characterization of client drift. It is unclear whether the bound on the client’s population gradient in Eq. (2) is sufficient to adequately characterize client drift, particularly in settings with more complex temporal patterns, such as the seasonal drift observed in the clinic experiments.
- Unrealistic initialization assumption. The theoretical analysis assumes that each optimization problem is initialized from a fixed point. This assumption differs from typical continual federated learning settings, where optimization proceeds from previously learned models as new data arrive, potentially limiting the applicability of the theoretical results to practical scenarios.
Questions for authors
- Practical implications of the theoretical analysis. While the proposed method appears theoretically advantageous over using a common memory window, the experimental results suggest that these benefits may be limited in more realistic settings.
- Characterization of client drift. Different types of client drift, such as seasonal or periodic drift, may require different strategies. However, the current drift bound does not appear to account for such temporal patterns. Could this limitation reduce the practical effectiveness of the proposed method?
- Discrepancy from standard federated learning settings. Several assumptions in the proposed framework appear to differ substantially from typical continual federated learning settings, including initialization from a fixed point for each optimization problem, communication with all clients at every refresh, and the ability to directly transmit data to the server. What motivates these assumptions, and to what extent can the proposed method be applied when these assumptions do not hold?
3 comments
Thank you for your detailed response and the additional experiments.
While these revisions help clarify the scope of the theoretical guarantees and provide additional empirical evidence, I remain concerned that the proposed method and evaluation settings are primarily aimed at validating the theoretical findings rather than demonstrating practical effectiveness in realistic continual federated learning scenarios.
Therefore, I will maintain my positive assessment, but do not see sufficient grounds to raise my score further.
Dear reviewer pXKV, thank you for your work and positive evaluation!
pXKV , thank you for review!
We added experiments that choose windows from earlier observations and vary the minimum durations and timing of drift (Appendix I.6, Table 4). With the same drift timing and minimum window durations corresponding to 16 records, client-specific windows reduce the mean gap by 7.39% relative to the common rule. Client-specific windows still improve on the common rule after the stationary clinics receive identical buffers. Other settings favor the common rule or have a difference whose interval includes zero. Appendix O adds tests using only earlier weather observations and clinic outcomes available after 90 days. The December weather interval excludes zero, the July interval includes zero, and the common rule has the lower mean gap in the regenerated clinic study.
Section 8 now explains when a common window with a minimum duration for each client is a simpler choice. Selecting windows from earlier observations gives nearly identical windows for the two methods in the outbreak and seasonal tests. Varying the minimum durations and drift timing produces both improvements and cases where client-specific windows perform worse. The abstract limits the empirical claim to the tested settings.
We added a discussion in Section 8 and an example with sinusoidal drift in Appendix M.3. Smooth periodic changes can satisfy Eq. (2), but the bound limits only how quickly the gradient changes. It does not use the timing or repetition of a seasonal cycle. Appendix I.6 adds four recurring clinic patterns with different cycle lengths and an additional oscillation.
Section 8 now explains that the drift bound does not take advantage of recurrence and recommends trying long windows and full history. Appendix I.8 clarifies that the bound on bias increases with the age of observations, although averaging over earlier cycles can reduce the actual bias. In the original clinic experiment, separate tuning for each clinic selects full history for pediatrics and primary care. Appendix I.6 adds the comparison under more complex recurring patterns.
We added this limitation in Section 8, final paragraph, and Appendix M.3, under “Reset point.” Resetting makes the first gradient query independent of the retained data. Each later query uses a batch that has not yet been used in that solve. Starting from a model trained on records still in the buffer can violate this independence, so the present proof does not cover that case.
Section 8 and Appendix M.3 now explain why these assumptions are used. Resetting allows the proof to justify reusing retained observations. Regular arrivals and participation by every client provide the batches needed at each round. Algorithm 1 can keep records local and send gradients. The present guarantee does not cover starting from a model trained on retained records, missing client rounds, or limiting message sizes.