- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GeoSF, a decentralized federated learning framework that dynamically separates parameters into slow and fast groups with different learning rates and synchronization frequencies. It further adjusts neighbor aggregation weights based on model similarity. Experiments across heterogeneous data distributions, model architectures, and communication topologies demonstrate the effectiveness of the proposed approach.
Critical strengths
-GeoSF considers both differences in parameter stability and compatibility between clients. Combining adaptive parameter partitioning with different update and synchronization frequencies provides a flexible way to balance shared knowledge and local personalization.
-The experiments cover feature skew and label skew, image and text classification, different model architectures, multiple communication topologies, and settings with up to 100 clients. The results show strong improvements across diverse configurations.
-The paper evaluates different combinations of the update, synchronization, and weighting mechanisms, compares score-based parameter selection against random selection, and examines sensitivity to major hyperparameters.
Critical weaknesses
-The paper would benefit from a more thorough discussion of what these forms of heterogeneity represent, how they arise, and how they affect optimization and collaboration in federated learning.
-Datasets such as CIFAR-10, CIFAR-100, and Office10 provide limited evidence of performance under more complex data heterogeneity. Including a more challenging benchmark, such as DomainNet, would strengthen the empirical validation.
-The tasks considered may not be sufficiently challenging. The high accuracy achieved in some classification settings raises the question of whether these tasks adequately test the capabilities of the proposed method. I would encourage the authors to explore more demanding vision or embodied tasks to demonstrate its broader applicability.
Questions for authors
Could the authors clarify how model parameters are selected for computing distances between client models? How do the computational and communication costs scale as the number of parameters increases, and does the approach remain practical for large models?
4 comments
Questions:
Q1: Parameter selection for distance computation.
A1: Thank you for raising this insightful question. In GeoSF, distances between client models are computed using a selected subset of shared parameters. Specifically, each client evaluates directional consistency and magnitude variance across parameter coordinates in its neighbors’ models, then selects the top-ranked coordinates according to a predefined shared-parameter ratio. The resulting binary mask identifies the coordinates used for distance computation. When comparing client ’s model with a neighbor’s model, the mask constructed locally by client is applied to both models to ensure that the distance is computed over corresponding parameter coordinates.
Q2: Scalability of computational and communication costs.
A2: Thank you for raising this important question. To evaluate how GeoSF’s computational and communication costs scale with model size, we tested ResNet18, ResNet50, ViT-Base, ViT-Large, and ViT-Huge, as reported below. Total computation time increases from 15.40 s to 76.96 s, with local training consistently accounting for most of the computation. For ViT-Huge, computation beyond local training takes only 3.68 s, approximately 5.0% of the local training time. This indicates that GeoSF’s additional computational overhead remains modest even for large models under the tested settings.
Furthermore, GeoSF’s communication cost scales linearly with the number of transmitted parameters, ranging from 89.45 MB to 5.05 GB per node per round. Since GeoSF transmits no additional statistical information, its communication cost matches that of standard decentralized methods, such as D-PSGD and DFedAvgM, under the same model and communication settings. These results support GeoSF’s computational practicality for large models, provided that device resources are sufficient to support model training.
| Metric | ResNet18 | ResNet50 | ViT-B | ViT-L | ViT-H |
|---|---|---|---|---|---|
| Local training | 12.94 s | 23.65 s | 28.72 s | 47.29 s | 73.28 s |
| Total Computation | 15.40 s | 25.82 s | 32.32 s | 50.29 s | 76.96 s |
| Communication | 89.45 MB | 188.65 MB | 687.68 MB | 2.43 GB | 5.05 GB |
Thank you for the author's response. Most of my concerns has been solved. I especially like the extra experiments of extra application. I will maintain my original score. Good luck.
Thank you again for your time and valuable suggestions, which have further helped us improve the completeness of our work.
We sincerely thank Reviewer aFHc for carefully evaluating our work and providing constructive feedback that helped us improve the manuscript. Below, we provide point-by-point responses to these comments.
Weaknesses:
A1: Thank you for your valuable suggestions. Heterogeneity refers to the divergence in the direction (sign consistency) and magnitude (variance) of model parameters arising from differences in data distributions across clients, as illustrated in Fig. 1(b) of the manuscript. This heterogeneity complicates both optimization and collaboration. Specifically, direct model aggregation may combine conflicting parameters, thereby degrading model performance on individual clients. Based on the degree of parameter divergence, we divide model parameters into “slow” and “fast” components: the former support federated collaboration through aggregation, while the latter facilitate local adaptation. We have expanded the discussion in the revised manuscript to clarify the impact of heterogeneity.
A2: Thank you very much for this helpful suggestion. We agree that more challenging benchmarks would provide a more thorough evaluation of GeoSF under complex data heterogeneity. Accordingly, we have conducted additional experiments on DomainNet. As shown below, GeoSF achieves the best average performance, with a C.avg of 23.31±0.43 and a D.avg of 29.05±0.44, further supporting its effectiveness under more challenging domain heterogeneity. We have included these experiments and the corresponding discussion in the revised manuscript.
A3: Thank you for your valuable suggestion. We agree that evaluation on more challenging tasks would better demonstrate GeoSF’s broader applicability. To this end, we have extended our experiments beyond image classification to multi-center prostate MRI segmentation on the ProstateMRI dataset, which contains MRI scans and segmentation annotations from six medical centers. This task requires precise pixel-level delineation of anatomical structures while handling data heterogeneity across centers, providing a complementary and demanding evaluation of GeoSF. The results are summarized in the table below.
GeoSF achieves a Dice score of 85.04±0.87 and an IoU of 75.53±1.09, outperforming all evaluated baselines. These results demonstrate that GeoSF’s benefits extend beyond classification to dense prediction under client data heterogeneity. We have added these experiments and discussions to the appendix of the revised manuscript.