- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In federated learning, most existing methods initialize local training from the current global model. FedEcho proposes a novel approach that instead leverages clients' historical responses to determine the starting points for local training. Motivated by the observation that clients' relative responses exhibit a degree of persistence across participation rounds, the server estimates longitudinal relative responses and uses them to construct client-specific starting points. Specifically, FedEcho determines these starting points by minimizing a surrogate objective that approximates local path divergence. Experiments on vision tasks with CNN-based models demonstrate that FedEcho generally outperforms the baselines.
Critical strengths
- Identifying better initialization points for local training, rather than simply relying on the current global model, is an important yet relatively underexplored research direction. The proposed approach for determining consensus-preserving starting points is intuitive and novel.
- FedEcho consistently outperforms the baselines under standard experimental settings. Furthermore, the ablation studies provide empirical evidence supporting the contribution of each component.
- Since the optimization of local starting points is performed entirely on the server side, FedEcho introduces no additional computational overhead on the client side, unlike methods that rely on client-side regularization.
Critical weaknesses
-
Limited experimental scope. All experiments are conducted on vision tasks using CNN-based models. Evaluations on more diverse tasks and model architectures would help establish the generalizability of the proposed method.
-
Reliance on heuristic design choices. Much of the proposed method relies on heuristic design choices. While the objective in Eq. (4) is intuitive, the computation of response coordinates in Eq. (2) and the incorporation of participation gaps in Eq. (3) also appear to be heuristic, with their effectiveness supported primarily by empirical results rather than theoretical justification.
-
Limited gains in the early and middle stages of training. Based on the reported results, most of the performance gains over the baselines appear to emerge in the later stages of training, with relatively limited improvements in the early and middle stages. This raises concerns about whether the observed gains would remain substantial under more constrained training budgets or alternative training schedules, such as those involving learning rate decay, which does not appear to have been considered in the experiments.
-
Clarity and presentation. The paper is somewhat difficult to follow. In particular, Figure 1 introduces several terms and concepts that are not formally defined until Section 3, making it difficult to interpret the figure without first reading the subsequent methodological details.
-
Sensitivity to client participation. As acknowledged in the Limitations section, the performance gains diminish when the client participation ratio decreases or when participation becomes sparser due to an increase in the total number of clients.
Questions for authors
-
Justification for Eq. (2). It would be helpful to provide further justification for the formulation in Eq. (2). For example, what motivates the use of weighted centering rather than directly using the full local training displacement?
-
Performance gains across training stages. Why do most of the performance gains of FedEcho over the baselines emerge in the later stages of training, rather than in the early or middle stages?
-
Compatibility with regularization-based methods. Since FedEcho takes a different approach from regularization-based methods, could it be combined with such methods in a complementary manner? It would be interesting to investigate whether their benefits are additive.
-
Performance discrepancies in Table 1. Table 1 shows a substantial performance gap between FedPhonix and FedEcho on the one hand, and the other baseline methods on the other. Could the authors provide some insight into the reasons behind this discrepancy or share the relevant implementation code to help clarify the results?
4 comments
Q4: Performance gaps between baselines.
The large performance gap between FedPhoenix and FedAvg is also reported in the original FedPhoenix paper. On ResNet-18/CIFAR-10/IID, our reproduced FedAvg and FedPhoenix results are 65.03% and 85.54%, respectively, close to the 64.12% and 84.85% reported in Table 2 of that paper.
In its preliminary non-IID study, the original FedPhoenix paper reports feature-masking results showing FedAvg's strong reliance on a small number of features and its vulnerability to masking, providing mechanistic context for parameter resetting. In addition, the newly added SCAFFOLD baseline achieves 80.99±0.50% in the same configuration, exceeding FedAvg by 15.96 percentage points. This shows that substantial improvements can also be obtained with other types of federated optimization methods. FedEcho achieves 89.69%, a further gain of 4.15 percentage points over FedPhoenix. We will detail the configurations of the added baselines in the revised appendix and plan to release our implementation upon acceptance.
W1/W4: Experimental scope and presentation.
Thank you for suggesting broader model architectures and tasks. Our current empirical validation covers CNN-based image classification, as explicitly stated in §4.5. Within this scope, we evaluate 27 configurations using VGG-16, ResNet-18, and MobileNet-V1 on CIFAR-10, CIFAR-100, and Tiny-ImageNet under IID and two non-IID partitions. FedEcho outperforms the methods compared in the original paper in all these configurations. We will also briefly explain relative responses, cross-participation persistence, participation gaps, and matched controls before Figure 1; correct the Figure 3 caption regarding how the strength of decay varies with participation gaps; and consistently name the corresponding ablation in Table 2 “w/o Dispatch Centering” to distinguish it from historical response centering in Eq. (2).
Thank you for the clarifications. While some of my concerns remain, I appreciate the authors' responses and will maintain my original positive rating.
Thank you for your thoughtful engagement with our work. We appreciate your positive rating and your feedback, which has helped us improve the manuscript. Once again, we are sincerely grateful for the time and care you devoted to reviewing our paper.
We appreciate your positive assessment of FedEcho's novelty and experimental results, as well as your specific suggestions. We address your questions below.
Q1/W2: Historical response centering and method design.
Eq. (2) uses same-round weighted centering to extract each client's response relative to the participating cohort, along the contemporaneous global trajectory. Different clients' historical records come from different rounds, so directly using full local updates would mix in the effects of common optimization motion from those rounds. We subtract the same-round mean local displacement, computed using the same weights as server aggregation. Historical response centering defines the information stored, while dispatch centering coordinates the starting points of current participants. If full historical updates were used directly, the common components from different observation rounds would undergo different amounts of decay; current dispatch centering generally cannot remove these components individually. In Table 2, using absolute historical responses reduces accuracy from 86.85% to 83.91%, supporting the value of the relative response representation.
Eq. (3) adopts a simple exponential decay model consistent with the short-term persistence observed in Figure 1 and its weakening as participation gaps increase. Its coefficient is estimated by least squares from valid historical response pairs available before the current round (Appendix B.1, Eq. (8)). Removing temporal decay reduces accuracy by 0.89 percentage points, supporting the practical value of accounting for participation gaps.
Eq. (4) uses a low-dimensional linear-path surrogate. Eq. (18) in Appendix D.1 shows that, under the weighted zero-sum constraint, this objective differs from the disagreement of the predicted components around their weighted consensus path only by a term independent of the dispatch variables. The objective thus has a clear interpretation in terms of the predicted consensus path; given this surrogate, the offsets are jointly determined through constrained optimization. When the trust-region constraint is inactive, the factor 1/2 in the closed-form solution follows from uniform integration over normalized local training progress (Appendix C).
Q2/W3/W5: Training stages, training budgets, and client participation.
The timing of performance gains varies with architecture and participation conditions. In Figure 6, some ResNet-18 and MobileNet-V1 configurations show an advantage relatively early, whereas gains emerge later in some VGG-16 configurations. Figure 4 also shows early separation from the baselines at participation rates of 0.2 and 0.5.
Sparser participation slows the accumulation and updating of client response histories, which may delay the emergence of gains and reduce their magnitude. For example, with N=500 and 10 participating clients per round, FedEcho trails most baselines during early and middle training and overtakes them only later. Under this sparse participation setting, a shorter training budget may not capture its eventual advantage. The current experiments follow FedPhoenix in using a fixed learning rate; relative gains under learning-rate decay have not been directly compared.
Q3: Compatibility with regularization-based methods.
Thank you for suggesting combinations with regularization-based methods. FedEcho adjusts client training starting points on the server without requiring changes to the original local objective, so it can, in principle, be combined with methods such as FedProx. Since the current global model differs from the actual dispatched starting point, such a combination must specify the reference model for the proximal regularizer. We have not directly evaluated whether the benefits of the two approaches are additive and leave this to future work.