acceptodds
Under review as a conference paper at ICLR 2027

FedEcho: Echoes of the Past Guide Future Starts in Federated Learning

Abstract

Federated learning typically uses the current global model both as the shared state for continual aggregation and as the common local training starting point for all participating clients, although the latter itself constitutes an independently designable degree of freedom in optimization. Prior work adjusts local initialization using historical model differences or random perturbations, but how to use the historical responses of repeatedly participating clients relative to the concurrent collaborative process to predict their relative optimization trends at the next participation, and accordingly coordinate starting-point dispatch, remains underexplored. We observe that client relative responses, measured relative to the same-round weighted-average displacement and along the concurrent global trajectory, exhibit short-term cross-participation persistence that decays as the participation gap increases. This suggests that what transfers across participations is not a stale gradient or a fixed parameter direction, but a client's response tendency relative to the collaborative process. Building on this observation, we propose FedEcho, which recasts training starting-point dispatch as a jointly coordinated, low-dimensional optimization control among clients participating in the same round. The server predicts each client's relative optimization trend from its historical relative response and jointly determines client-specific training starting points to pre-compensate for the predictable relative component of local-path divergence. The dispatch guarantees that the weighted-average starting point across participating clients equals the current global model, while leaving the original local training and server aggregation rules unchanged. Under smoothness conditions, local-path divergence upper-bounds the corresponding aggregated gradient mismatch, providing an optimization interpretation of this design. Experimental results across three datasets, three network architectures, and three data distributions show that FedEcho achieves the highest test accuracy in all 27 settings, outperforming the strongest baseline by 2.80 percentage points on average.

Primary area:
optimization
Keywords:
federated learning, heterogeneous optimization, model-state dispatch, longitudinal client participation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LBxbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In federated learning, most existing methods initialize local training from the current global model. FedEcho proposes a novel approach that instead leverages clients' historical responses to determine the starting points for local training. Motivated by the observation that clients' relative responses exhibit a degree of persistence across participation rounds, the server estimates longitudinal relative responses and uses them to construct client-specific starting points. Specifically, FedEcho determines these starting points by minimizing a surrogate objective that approximates local path divergence. Experiments on vision tasks with CNN-based models demonstrate that FedEcho generally outperforms the baselines.

Critical strengths
  1. Identifying better initialization points for local training, rather than simply relying on the current global model, is an important yet relatively underexplored research direction. The proposed approach for determining consensus-preserving starting points is intuitive and novel.
  2. FedEcho consistently outperforms the baselines under standard experimental settings. Furthermore, the ablation studies provide empirical evidence supporting the contribution of each component.
  3. Since the optimization of local starting points is performed entirely on the server side, FedEcho introduces no additional computational overhead on the client side, unlike methods that rely on client-side regularization.
Critical weaknesses
  1. Limited experimental scope. All experiments are conducted on vision tasks using CNN-based models. Evaluations on more diverse tasks and model architectures would help establish the generalizability of the proposed method.

  2. Reliance on heuristic design choices. Much of the proposed method relies on heuristic design choices. While the objective in Eq. (4) is intuitive, the computation of response coordinates in Eq. (2) and the incorporation of participation gaps in Eq. (3) also appear to be heuristic, with their effectiveness supported primarily by empirical results rather than theoretical justification.

  3. Limited gains in the early and middle stages of training. Based on the reported results, most of the performance gains over the baselines appear to emerge in the later stages of training, with relatively limited improvements in the early and middle stages. This raises concerns about whether the observed gains would remain substantial under more constrained training budgets or alternative training schedules, such as those involving learning rate decay, which does not appear to have been considered in the experiments.

  4. Clarity and presentation. The paper is somewhat difficult to follow. In particular, Figure 1 introduces several terms and concepts that are not formally defined until Section 3, making it difficult to interpret the figure without first reading the subsequent methodological details.

  5. Sensitivity to client participation. As acknowledged in the Limitations section, the performance gains diminish when the client participation ratio decreases or when participation becomes sparser due to an increase in the total number of clients.

Questions for authors
  1. Justification for Eq. (2). It would be helpful to provide further justification for the formulation in Eq. (2). For example, what motivates the use of weighted centering rather than directly using the full local training displacement?

  2. Performance gains across training stages. Why do most of the performance gains of FedEcho over the baselines emerge in the later stages of training, rather than in the early or middle stages?

  3. Compatibility with regularization-based methods. Since FedEcho takes a different approach from regularization-based methods, could it be combined with such methods in a complementary manner? It would be interesting to investigate whether their benefits are additive.

  4. Performance discrepancies in Table 1. Table 1 shows a substantial performance gap between FedPhonix and FedEcho on the one hand, and the other baseline methods on the other. Could the authors provide some insight into the reasons behind this discrepancy or share the relevant implementation code to help clarify the results?

4 comments

We appreciate your positive assessment of FedEcho's novelty and experimental results, as well as your specific suggestions. We address your questions below.

Q1/W2: Historical response centering and method design.

Eq. (2) uses same-round weighted centering to extract each client's response relative to the participating cohort, along the contemporaneous global trajectory. Different clients' historical records come from different rounds, so directly using full local updates would mix in the effects of common optimization motion from those rounds. We subtract the same-round mean local displacement, computed using the same weights as server aggregation. Historical response centering defines the information stored, while dispatch centering coordinates the starting points of current participants. If full historical updates were used directly, the common components from different observation rounds would undergo different amounts of decay; current dispatch centering generally cannot remove these components individually. In Table 2, using absolute historical responses reduces accuracy from 86.85% to 83.91%, supporting the value of the relative response representation.

Eq. (3) adopts a simple exponential decay model consistent with the short-term persistence observed in Figure 1 and its weakening as participation gaps increase. Its coefficient is estimated by least squares from valid historical response pairs available before the current round (Appendix B.1, Eq. (8)). Removing temporal decay reduces accuracy by 0.89 percentage points, supporting the practical value of accounting for participation gaps.

Eq. (4) uses a low-dimensional linear-path surrogate. Eq. (18) in Appendix D.1 shows that, under the weighted zero-sum constraint, this objective differs from the disagreement of the predicted components around their weighted consensus path only by a term independent of the dispatch variables. The objective thus has a clear interpretation in terms of the predicted consensus path; given this surrogate, the offsets are jointly determined through constrained optimization. When the trust-region constraint is inactive, the factor 1/21/2 in the closed-form solution follows from uniform integration over normalized local training progress (Appendix C).

Q2/W3/W5: Training stages, training budgets, and client participation.

The timing of performance gains varies with architecture and participation conditions. In Figure 6, some ResNet-18 and MobileNet-V1 configurations show an advantage relatively early, whereas gains emerge later in some VGG-16 configurations. Figure 4 also shows early separation from the baselines at participation rates of 0.2 and 0.5.

Sparser participation slows the accumulation and updating of client response histories, which may delay the emergence of gains and reduce their magnitude. For example, with N=500N=500 and 10 participating clients per round, FedEcho trails most baselines during early and middle training and overtakes them only later. Under this sparse participation setting, a shorter training budget may not capture its eventual advantage. The current experiments follow FedPhoenix in using a fixed learning rate; relative gains under learning-rate decay have not been directly compared.

Q3: Compatibility with regularization-based methods.

Thank you for suggesting combinations with regularization-based methods. FedEcho adjusts client training starting points on the server without requiring changes to the original local objective, so it can, in principle, be combined with methods such as FedProx. Since the current global model differs from the actual dispatched starting point, such a combination must specify the reference model for the proximal regularizer. We have not directly evaluated whether the benefits of the two approaches are additive and leave this to future work.

Q4: Performance gaps between baselines.

The large performance gap between FedPhoenix and FedAvg is also reported in the original FedPhoenix paper. On ResNet-18/CIFAR-10/IID, our reproduced FedAvg and FedPhoenix results are 65.03% and 85.54%, respectively, close to the 64.12% and 84.85% reported in Table 2 of that paper.

In its preliminary non-IID study, the original FedPhoenix paper reports feature-masking results showing FedAvg's strong reliance on a small number of features and its vulnerability to masking, providing mechanistic context for parameter resetting. In addition, the newly added SCAFFOLD baseline achieves 80.99±0.50% in the same configuration, exceeding FedAvg by 15.96 percentage points. This shows that substantial improvements can also be obtained with other types of federated optimization methods. FedEcho achieves 89.69%, a further gain of 4.15 percentage points over FedPhoenix. We will detail the configurations of the added baselines in the revised appendix and plan to release our implementation upon acceptance.

W1/W4: Experimental scope and presentation.

Thank you for suggesting broader model architectures and tasks. Our current empirical validation covers CNN-based image classification, as explicitly stated in §4.5. Within this scope, we evaluate 27 configurations using VGG-16, ResNet-18, and MobileNet-V1 on CIFAR-10, CIFAR-100, and Tiny-ImageNet under IID and two non-IID partitions. FedEcho outperforms the methods compared in the original paper in all these configurations. We will also briefly explain relative responses, cross-participation persistence, participation gaps, and matched controls before Figure 1; correct the Figure 3 caption regarding how the strength of decay varies with participation gaps; and consistently name the corresponding ablation in Table 2 “w/o Dispatch Centering” to distinguish it from historical response centering in Eq. (2).

Thank you for the clarifications. While some of my concerns remain, I appreciate the authors' responses and will maintain my original positive rating.

Thank you for your thoughtful engagement with our work. We appreciate your positive rating and your feedback, which has helped us improve the manuscript. Once again, we are sincerely grateful for the time and care you devoted to reviewing our paper.

Reviewer_A79EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedEcho, a server-side federated learning method that treats each client’s local training initialization as a controllable variable. It measures a client’s historical response relative to other participants along the global optimization trajectory and models the response’s short-term persistence using temporal decay. For each round, FedEcho assigns participating clients differentiated starting points along the current global update direction while keeping their weighted mean anchored at the global model. Across 27 experimental settings, it consistently outperforms strong baselines. The paper also relates local-path divergence to aggregated gradient mismatch through an upper bound.

Critical strengths
  1. Technically, FedEcho recasts initialization dispatch as a coordinated, low-dimensional control problem using longitudinal, cohort-relative response signals. Also, FedEcho offers a server-only, plug-in design that could be widely applicable and combined with other FL advances.
  2. The authors conduct a comprehensive evaluation across multiple settings (CIFAR-10/100, Tiny-ImageNet; VGG-16, ResNet-18, MobileNet-V1; IID and two Dirichlet non-IID splits), with consistent improvements over strong baselines (e.g., FedPhoenix, FedMut).
  3. The motivation is clear, and the authors provide empirical evidence that relative client responses persist across participation rounds.
Critical weaknesses
  1. The theoretical result bounds aggregated gradient mismatch in terms of local-path divergence, but does not establish convergence guarantees or a formal improvement over FedAvg.
  2. The method assumes consistent local training protocols and persistent client responses. Its performance under sparse participation or changing local settings remains unclear.
  3. Some widely used optimization baselines, such as SCAFFOLD, FedDyn, and FedDC, are discussed but omitted from the main comparison. If possible, the authors could consider including at least one to strengthen the empirical evaluation.
  4. Improvements reported in IID settings are very large. This calls for a deeper diagnostic (e.g., why large gains when client heterogeneity is minimal).
Questions for authors
  1. In rounds where ∥∣mt∥∣\||m_t\|| is small and dispatch reverts to zero offsets, do you observe performance regressions or delayed progress? Roughly what fraction of rounds are “inactive” in your runs?
  2. Does per-client decay (rather than pooled) help?
4 comments

We appreciate your positive assessment of FedEcho's design and experimental results. We address your questions below.

Q1/W2: Inactive rounds and sparse participation

Figures 4–5 examine how the total number of clients and the participation rate affect FedEcho. FedEcho achieves the highest test accuracy in all tested settings, but its advantage generally emerges later as participation becomes sparser. In particular, with N=500N=500 and 10 participating clients per round, FedEcho trails most baselines during early and middle training and gains an advantage later. As clients participate less frequently, their response histories accumulate and are updated more slowly, which may partly explain why the advantage emerges later.

We define a fully inactive round as one in which the actual model dispatch offsets are zero for all participating clients. Such rounds account for 0.5%, 0.4%, and 0.4% of all communication rounds under Dir⁡(0.3)\operatorname{Dir}(0.3), Dir⁡(0.6)\operatorname{Dir}(0.6), and IID partitions, respectively. In these rounds, clients still receive the global model and perform local SGD, and the server performs FedAvg aggregation. The current observations do not quantify whether zero-offset handling causes a relative delay in training progress. Moreover, a client without valid history uses a zero prediction, but its final offset need not be zero after centering across the participating cohort.

Q2: Pooled decay

Thank you for suggesting client-specific decay. FedEcho uses client-specific historical responses for prediction, while sharing the temporal decay coefficient. Under partial participation, individual clients have few historical response pairs available, particularly early in training. Pooling valid response pairs across clients provides more samples for estimating the decay coefficient. This is the statistical motivation for using a shared coefficient (Appendices B.1 and D.2). Whether client-specific decay further improves performance has not been directly evaluated, and we leave this question to future work.

W1: Theoretical analysis

FedEcho solves for client training starting points under a consensus-preserving constraint, strictly ensuring that the weighted-average starting point of the current participants equals the global model wtw_t (Appendix C). When each local objective is LL-smooth, Eq. (7) bounds the squared norm of aggregated gradient mismatch by L2L^2 times the weighted local-path divergence. This provides theoretical motivation for the path-divergence surrogate objective, but does not by itself guarantee improved convergence over FedAvg.

W3: Additional baseline comparisons

We added comparisons with FedInit, FedDyn, and SCAFFOLD across three network architectures on CIFAR-10/100 under IID and two non-IID partitions, covering 18 configurations. The results below report the mean and sample standard deviation of test accuracy (%) over three random seeds.

ModelDataSettingFedInitSCAFFOLDFedDyn
VGG-1610β=0.3\beta=0.378.53±0.3279.32±0.4978.59±0.28
β=0.6\beta=0.680.77±0.0281.98±0.5980.84±0.19
IID81.03±0.2482.88±0.0881.31±0.63
100β=0.3\beta=0.356.11±0.8359.59±0.9154.18±0.98
β=0.6\beta=0.656.94±0.5461.08±1.5555.16±0.66
IID58.10±0.8662.54±0.1656.41±0.96
ResNet-1810β=0.3\beta=0.375.35±0.8775.44±1.3576.53±1.56
β=0.6\beta=0.678.97±0.1679.98±0.3578.67±0.06
IID79.30±0.1080.99±0.5080.09±0.07
100β=0.3\beta=0.347.88±0.1446.83±0.1348.16±0.57
β=0.6\beta=0.650.29±0.4148.81±0.4950.06±0.89
IID50.00±0.3749.34±0.3451.30±0.44
MobileNet-V110β=0.3\beta=0.361.13±2.3564.20±0.1660.68±0.23
β=0.6\beta=0.664.68±1.1168.28±0.0964.70±0.85
IID70.27±0.3175.33±0.9869.82±0.68
100β=0.3\beta=0.335.76±0.2539.21±1.0635.25±1.44
β=0.6\beta=0.638.52±1.4842.61±1.2237.28±0.72
IID40.92±2.3244.83±1.1140.11±1.85

Data: 10 = CIFAR-10; 100 = CIFAR-100.

Together with the corresponding FedPhoenix and FedEcho results in the original Table 1, these results show that FedEcho outperforms all three added baselines in all 18 CIFAR configurations. FedPhoenix remains the strongest baseline in these configurations, and FedEcho exceeds it by 3.33 percentage points on average. We will detail the configurations of the added methods in the revised appendix.

W4: Performance gains under IID

On ResNet-18/CIFAR-10/IID, FedAvg, FedPhoenix, and FedEcho achieve 65.03%, 85.54%, and 89.69%, respectively. The original FedPhoenix paper reports 64.12% for FedAvg and 84.85% for FedPhoenix in the corresponding setting, showing that large gains over FedAvg are not unique to FedEcho. FedEcho further improves upon FedPhoenix by 4.15 percentage points in this setting. Grouping all 27 configurations in the original paper by partition, FedEcho's mean gains over FedPhoenix are 2.60, 2.75, and 3.07 percentage points under IID, β=0.6\beta=0.6, and β=0.3\beta=0.3, respectively. Its incremental gain is therefore not unusually large under IID.

An IID partition does not imply identical actual local optimization responses across clients. Each client trains on its own fixed, finite set of training samples, which can give rise to different empirical objectives. To quantify the predictive value of historical information under IID, we generate predictions from historical responses available before dispatch in the current round on VGG-16/CIFAR-10/IID. We compare them with the component-wise relative responses computed using Eq. (20) in the appendix after actual dispatch and local training. Overall Pearson correlation is 0.2705, and weighted NMSE is 0.8948. NMSE is weighted by client aggregation weights and the squared norms of the control directions, and is normalized by the zero-prediction error on the same samples with the same weights. The zero-prediction baseline is therefore 1.

Analysis by participation gap shows positive Pearson correlations in every group (0.2573–0.2971), while NMSE increases with the gap (0.8125–0.9411). Thus, historical responses retain some predictive signal under IID, while prediction error depends on the participation gap.

Participation GapComponent samplesPearsonNMSE
11,9880.25730.8125
2–33,4300.28830.8645
4–107,5240.27990.9222
≥11\geq 116,8380.29710.9411

The table includes only samples with a previous valid observation, for which the participation gap is defined. Another 200 component samples without history are included in the overall statistics. These results show that historical relative responses retain predictive information in this IID setting, with overall weighted prediction error below the zero-prediction baseline.

Thank you so much for your additional experiments and efforts. They have addressed my concerns. I will keep the positive score.

Thank you very much for your positive feedback and for taking the time to consider our additional experiments and responses. We sincerely appreciate your thoughtful assessment and encouragement. Your feedback has been very helpful in improving our work.

Reviewer_K5qyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedEcho, a federated learning framework that improves global model training by assigning client-specific starting points rather than initializing all participating clients with the same global model. The key motivation is the observation that clients' relative optimization responses exhibit short-term persistence across participation rounds. Specifically, FedEcho predicts clients' future relative optimization tendencies from their historical responses and adjusts their local training starting points within a low-dimensional space defined by the global optimization trajectory. A consensus-preserving constraint ensures that the weighted average of these starting points remains equal to the current global model. The authors also provide an optimization interpretation based on local-path divergence. Experiments on three datasets, three architectures, and three data distributions demonstrate improvements over the compared methods.

Critical strengths
  1. The paper explores an interesting perspective on federated optimization by decoupling the shared global model from the starting points of local training. Unlike conventional approaches that modify local objectives or gradients, FedEcho leverages historical client responses to guide client-specific initialization while preserving standard local training and aggregation. This provides a potentially useful approach to mitigating client drift.

  2. The experimental results are promising, with FedEcho outperforming the compared methods across all 27 reported settings. The ablation studies further demonstrate the contributions of several key components, including historical response modeling, temporal decay, and consensus-preserving dispatch. The evaluations under different client participation rates and client population sizes also provide useful insights into the behavior of the proposed method.

Critical weaknesses
  1. The effectiveness of FedEcho fundamentally relies on the assumption that historical relative responses can predict clients' future optimization tendencies. Although Figure 1 provides some empirical evidence of cross-participation persistence, the paper does not directly evaluate the accuracy of these predictions or demonstrate that the proposed dispatch actually reduces local-path divergence. Moreover, the theoretical analysis in Eq. (7) only establishes a general upper bound on aggregated gradient mismatch, without showing that FedEcho reduces this mismatch or guarantees improved convergence. Consequently, the connection between the proposed prediction mechanism and the observed performance improvements remains insufficiently validated.

  2. The experimental comparisons omit several closely related federated optimization methods. In particular, FedInit, which directly investigates local model initialization and is discussed in Section 2.2, is not included in the experiments. Other representative client-drift correction methods, such as SCAFFOLD, FedDyn, and FedDC, are also missing. These omissions make it difficult to fully assess the advantages of FedEcho over existing optimization strategies. Furthermore, the experiments are limited to image classification with CNN architectures, leaving the generalizability of the proposed approach insufficiently explored.

Questions for authors
  1. Could the authors quantitatively evaluate the prediction accuracy of historical relative responses, particularly under different participation gaps and degrees of data heterogeneity? More importantly, could they demonstrate that FedEcho actually reduces local-path divergence or aggregated gradient mismatch compared with existing methods?

  2. Could the authors provide additional comparisons with FedInit and representative client-drift correction methods such as SCAFFOLD, FedDyn, and FedDC? In particular, comparison with FedInit would help clarify the advantages of using historical relative responses rather than historical parameter differences for local initialization.

  3. The main formulation introduces a rank-one control space, while the experiments employ a rank-two construction for CNNs. Could the authors provide an ablation study comparing these two configurations to clarify how much of the improvement comes from the historical response prediction mechanism versus the structured control-space design?

4 comments

We appreciate your positive assessment of the motivation and experimental results of FedEcho. To address your questions about mechanism validation, comparisons with related methods, and the contributions of the Rank-1/Rank-2 constructions, we have added analyses of historical response prediction quality and actual local-path divergence, comparisons with representative federated optimization baselines, and a control-space comparison.

1. Historical response prediction and actual optimization effects (W1/Q1)

FedEcho uses historical relative responses to predict clients' subsequent optimization trends. This is motivated by the cross-participation persistence shown in Figure 1. In the relevant VGG-16 and ResNet-18 experiments, FedAvg, FedProx, ClusteredSampling, and FedPhoenix all exhibit greater cross-participation response similarity for the same client than for different-client controls, with persistence generally weakening as the participation gap increases. None of these runs uses historical-response-driven model dispatch. Thus, this phenomenon is not produced by FedEcho's own feedback mechanism and provides independent empirical support for using client histories to predict subsequent responses.

Table 2 provides further ablation evidence on performance. On VGG-16/CIFAR-10 with β=0.3\beta=0.3, shuffling the correspondence between clients and their historical responses (Shuffled History) reduces accuracy from 86.85% to 81.42%, a drop of 5.43 percentage points. Shuffled History still outperforms FedAvg (76.71%). This difference should therefore be interpreted as the incremental benefit of correct client-history correspondence over shuffled history, rather than as an additive decomposition of FedEcho's overall performance gain.

To further examine historical response prediction and its effects on actual optimization paths, we conducted the following two analyses.

Prediction accuracy. On VGG-16/CIFAR-10 under IID, β=0.6\beta=0.6, and β=0.3\beta=0.3 partitions, we generate predictions from historical responses available before dispatch in the current round. We compare them with the actual relative responses computed after local training using the structured form of Eq. (2), namely Eq. (20) in the appendix. We evaluate prediction quality using Pearson correlation and weighted normalized mean squared error (NMSE), examining differences across data heterogeneity, participation gaps, and Rank-2 response components.

Statistical protocol. For each partition, we include all valid pairs of predictions made before the round and responses observed after the round, yielding 19,980 component samples, with 9,990 each for Base and Spatial. Round 1 is excluded because its control direction is invalid; samples without a previous valid observation and samples with extreme responses are retained. Pearson correlation assigns equal weight to valid samples; Overall Pearson is recomputed after pooling Base and Spatial samples. NMSE is weighted by the current-round client aggregation weights, based on sample size, and the squared norms of the control directions. It is normalized by the zero-prediction error on the same samples with the same weights. The zero-prediction baseline is therefore 1. Overall NMSE is computed by pooling the error numerators and normalization denominators of both components before taking their ratio. All groups follow the same protocol.

Prediction quality across data heterogeneity levels

PartitionComponentPearsonNMSE
β=0.3\beta=0.3Overall0.43730.7103
Base0.49400.6026
Spatial0.24350.8589
β=0.6\beta=0.6Overall0.35090.7373
Base0.38150.6433
Spatial0.23080.8623
IIDOverall0.27050.8948
Base0.28940.8665
Spatial0.20410.9296

Overall prediction quality across participation gaps

GapN (component samples)β=0.3\beta=0.3 (Pearson/NMSE)β=0.6\beta=0.6 (Pearson/NMSE)IID (Pearson/NMSE)
11,9880.3531/0.75840.3760/0.75720.2573/0.8125
2–33,4300.4993/0.65620.4752/0.66780.2883/0.8645
4–107,5240.5125/0.71180.3264/0.74310.2799/0.9222
≥11\geq 116,8380.4558/0.72040.4427/0.76990.2971/0.9411
No previous valid observation200N/A/1.0000N/A/1.0000N/A/1.0000

For samples without a previous valid observation, the predicted response is identically zero. Pearson correlation is therefore undefined within this group because the predictions have zero variance, while NMSE equals the zero-prediction baseline of 1. These samples remain included in the overall statistics.

1. Historical response prediction and actual optimization effects (W1/Q1, continued)

Predictions correlate positively with observed responses under all three partitions, and overall weighted NMSE is below the zero-prediction baseline, although prediction quality varies considerably across conditions. Base has higher Pearson correlation and lower NMSE than Spatial under all three partitions; the predictive signal is weaker under IID. Pearson correlation does not decrease monotonically across participation gaps. Pearson measures linear association between predicted and observed responses, whereas NMSE measures prediction error, including magnitude error. Eq. (3) adjusts the magnitude of predictions based on historical responses according to the participation gap. Removing temporal decay reduces accuracy by 0.89 percentage points, supporting its contribution to final performance.

Actual path divergence. On VGG-16/CIFAR-10 under Dir⁡(0.3)\operatorname{Dir}(0.3), Dir⁡(0.6)\operatorname{Dir}(0.6), and IID partitions, we compare the actual local paths of FedEcho, FedAvg, FedProx, FedMut, ClusteredSampling, and FedPhoenix under their respective full training protocols. At communication rounds 100, 500, and 900, we record trajectories over all five local epochs. We sample normalized training progress at 21 points, including the start and end points, and use linear interpolation to align update progress across clients. At each point, we compute the squared L2 distance of each client model from the sample-size-weighted consensus and aggregate these distances using the same weights. Trapezoidal integration over the full five-epoch local training interval yields that round's local-path divergence. This analysis uses one random seed and covers all trainable parameters, excluding BN running buffers. The tables report the results at each checkpoint and the equally weighted mean across the three checkpoints.

Dir(0.3)

RoundFedAvgFedProxFedMutClusteredSamplingFedPhoenixFedEcho
1001.2385301.2753923.5651341.4518732.2832291.440370
5001.4309691.8397793.3306930.8008551.2825670.715917
9000.7194710.2021191.7576990.1008600.3947450.537041
Mean1.1296571.1057632.8845090.7845291.3201810.897776

Dir(0.6)

RoundFedAvgFedProxFedMutClusteredSamplingFedPhoenixFedEcho
1002.0517052.4177494.9997721.6332483.1333371.900308
5000.2198610.4548580.8126550.3860822.2001720.804766
9000.1506170.1179012.3847320.0566812.3080370.398476
Mean0.8073940.9968362.7323860.6920042.5471821.034517

IID

RoundFedAvgFedProxFedMutClusteredSamplingFedPhoenixFedEcho
1001.1750900.9728513.5084821.1291012.1820380.851045
5003.4909e-54.6662e-48.3634e-40.0023061.3186153.0554e-4
9004.3350e-68.0844e-60.0018672.1069e-40.0912733.2373e-6
Mean0.3917100.3244421.1703950.3772061.1973090.283784

Averaged equally over the three checkpoints, FedEcho has lower full-path divergence than FedPhoenix and FedMut under all three partitions, while the comparison with the other methods varies by partition. At the starting point κ=0\kappa=0, instantaneous path divergence is

D(0)=∑ipi∥Δi∥2.D(0)=\sum_i p_i\lVert\Delta_i\rVert^2.

This starting point is included in the full-path measurement. Eq. (7) provides an upper bound on gradient mismatch, so the ordering of full-path divergence does not directly determine the ordering of actual aggregated gradient mismatch.

As described in §3.3 and Appendix D.1, FedEcho uses predictions based on historical relative responses to pre-compensate for controllable low-dimensional relative path components, while ensuring that the weighted average of participating clients' training starting points is exactly the current global model. Full-path divergence also includes differences introduced by client-specific initial dispatch and cannot, by itself, serve as a measure of optimization effectiveness or final performance. The cross-method comparison provides descriptive evidence of actual paths under different full training protocols; it does not isolate the causal effect of the dispatch mechanism.

Theoretical interpretation. Under LL-smooth local objectives, Eq. (7) bounds aggregated gradient mismatch in terms of actual local-path divergence, providing theoretical motivation for FedEcho's objective. The inequality itself does not guarantee that FedEcho reduces actual path divergence or aggregated gradient mismatch, nor does it guarantee faster convergence or higher final test accuracy.

2. Comparison with related federated optimization methods (W2/Q2)

FedEcho's contribution lies in combining prediction from clients' historical relative responses with consensus-preserving joint starting-point coordination, rather than client-specific initialization alone. FedInit adjusts client starting points using the parameter difference between a historical local model and the current global model. FedEcho instead uses weighted centering over the clients participating in each observation round to extract response coordinates relative to the concurrent collaborative process in that round. It predicts current relative responses using these historical relative responses recorded along the contemporaneous global trajectories. This centering reduces interference from common optimization motion across historical rounds when comparing responses across rounds. The server then jointly determines the starting points of current participants, ensuring that their weighted average is exactly the current global model wtw_t.

Table 2 further supports this design. Replacing relative responses with absolute historical responses reduces accuracy from 86.85% to 83.91%, supporting the value of the relative response representation across rounds. Removing dispatch centering in Eq. (5) reduces accuracy to 85.47%, supporting the combined role of dispatch centering and its consensus-preserving constraint. This ablation changes both dispatch centering and the corresponding zero-sum constraint, so the performance change cannot be attributed to either in isolation.

To further assess the complete method's competitiveness, we added comparisons with FedInit, SCAFFOLD, and FedDyn. FedInit represents initialization based on historical models, while SCAFFOLD and FedDyn represent client-drift correction methods.

ModelDatasetSettingFedInitSCAFFOLDFedDyn
VGG-16CIFAR-10β=0.3\beta=0.378.53±0.3279.32±0.4978.59±0.28
β=0.6\beta=0.680.77±0.0281.98±0.5980.84±0.19
IID81.03±0.2482.88±0.0881.31±0.63
CIFAR-100β=0.3\beta=0.356.11±0.8359.59±0.9154.18±0.98
β=0.6\beta=0.656.94±0.5461.08±1.5555.16±0.66
IID58.10±0.8662.54±0.1656.41±0.96
ResNet-18CIFAR-10β=0.3\beta=0.375.35±0.8775.44±1.3576.53±1.56
β=0.6\beta=0.678.97±0.1679.98±0.3578.67±0.06
IID79.30±0.1080.99±0.5080.09±0.07
CIFAR-100β=0.3\beta=0.347.88±0.1446.83±0.1348.16±0.57
β=0.6\beta=0.650.29±0.4148.81±0.4950.06±0.89
IID50.00±0.3749.34±0.3451.30±0.44
MobileNet-V1CIFAR-10β=0.3\beta=0.361.13±2.3564.20±0.1660.68±0.23
β=0.6\beta=0.664.68±1.1168.28±0.0964.70±0.85
IID70.27±0.3175.33±0.9869.82±0.68
CIFAR-100β=0.3\beta=0.335.76±0.2539.21±1.0635.25±1.44
β=0.6\beta=0.638.52±1.4842.61±1.2237.28±0.72
IID40.92±2.3244.83±1.1140.11±1.85

Results are means and sample standard deviations over three random seeds. Together with the corresponding FedPhoenix and FedEcho results in the original Table 1, these comparisons show that FedEcho outperforms the added FedInit, SCAFFOLD, and FedDyn baselines in all 18 CIFAR configurations. FedPhoenix remains the strongest baseline in these configurations, and FedEcho exceeds it by 3.33 percentage points on average. These results further support FedEcho's competitiveness across model architectures and data partitions.

The current empirical evidence covers image classification with CNNs, as stated in §4.5. These additions strengthen the mechanism analysis and comparative evidence within that scope.

3. Historical responses and the structured control space (Q3)

Historical response prediction and the structured control space serve different roles in FedEcho. With the Rank-2 control structure fixed, shuffling the client-history correspondence reduces accuracy by 5.43 percentage points (Table 2), showing that correct historical correspondence remains valuable within the same control structure.

To assess the structured extension, we compare Rank-1 and Rank-2 on VGG-16 across CIFAR-10, CIFAR-100, and three data partitions. Rank-1 controls starting points along the current global displacement direction, while Rank-2 uses the two orthogonal structured components in Appendix D.3. Both retain historical prediction and consensus preservation; the main CNN results in the original paper use Rank-2. Test accuracy (%) is reported below.

SettingCIFAR-10 Rank-1CIFAR-10 Rank-2CIFAR-100 Rank-1CIFAR-100 Rank-2
β=0.3\beta=0.386.13±0.5686.85±0.4963.34±0.3464.86±0.28
β=0.6\beta=0.687.45±0.2288.04±0.2464.57±0.2265.19±0.21
IID89.91±0.0990.34±0.0766.22±0.1866.59±0.15

Results are means and sample standard deviations over three random seeds. Rank-1 already outperforms all comparison baselines in all six configurations, showing that FedEcho's performance advantage does not require the Rank-2 extension. Rank-2 adds 0.37–1.52 percentage points, demonstrating the incremental value of the complete Rank-2 configuration. Rank-2 changes both the control basis and the corresponding response coordinates; these results therefore measure the gain of the complete Rank-2 configuration over Rank-1, rather than the effect of increasing control dimensionality alone. Together with history shuffling under fixed Rank-2 control, these comparisons provide experimental support for correct historical correspondence and the structured control configuration, respectively.

We will add the statistical protocols and results of the prediction and path-divergence analyses, the new baseline comparisons and their implementation details and hyperparameter settings, and the Rank-1/Rank-2 comparison to the revised appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for the time you devoted to reviewing our paper and for your constructive feedback. In response, we conducted additional experiments and analyses to strengthen the empirical evidence for FedEcho and revised the manuscript to clarify the method’s design and theoretical motivation. The main additions are summarized below.

1. Predicting relative responses from client history. On VGG-16/CIFAR-10, we compared predictions made before model dispatch with relative responses observed after local training under one IID and two non-IID data partitions (Appendix E.1, Tables 4 and 5). Overall Pearson correlations ranged from 0.2705 to 0.4373, and overall weighted NMSE was below the zero-prediction baseline of 1 in all three partitions. We also examined how prediction quality varied across participation gaps and response components.

2. Actual local training paths and theoretical motivation. We measured full local-path divergence for six methods under their respective training protocols at communication rounds 100, 500, and 900 for each of the three data partitions (Appendix E.2, Tables 6 to 8). Averaged over these checkpoints, FedEcho had lower path divergence than FedPhoenix and FedMut in all three partitions. Its ordering relative to the other methods varied by partition. When each local objective is LL-smooth, Eq. (7) shows that the squared norm of the aggregated gradient mismatch is at most L2L^2 times the weighted local-path divergence. This bound motivates the surrogate objective in Eq. (4), which targets the predictable low-dimensional component of the relative local path (§3.3 and Appendix D.1).

3. Additional baseline comparisons. We added comparisons with FedInit, SCAFFOLD, and FedDyn across 18 configurations covering three network architectures, two CIFAR datasets, and three data partitions (Appendix E.3, Table 9). In every configuration, FedEcho achieved higher mean test accuracy over three random seeds than each of the three added baselines. Its average gain over FedPhoenix across the 18 configurations was 3.33 percentage points.

4. Client histories and the structured control space. In the original Table 2 ablation on VGG-16/CIFAR-10 with β=0.3\beta=0.3, we held the Rank-2 control structure fixed. Shuffling the correspondence between clients and their historical responses reduced accuracy by 5.43 percentage points. The new comparison of Rank-1 and Rank-2 covers six VGG-16/CIFAR configurations. Rank-1 outperformed all comparison baselines in all six, and the full Rank-2 configuration improved accuracy by a further 0.37 to 1.52 percentage points over Rank-1 (Appendix E.4, Table 10).

Before Figure 1, we clarified the meaning of relative response and participation gap and noted that the observations came from runs that did not use client response histories for model dispatch. We also clarified the distinction between historical response centering and dispatch centering. Appendix E.5 reports the fraction of rounds in which all participating clients received zero dispatch offsets.

Thank you again for your thoughtful feedback.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.