- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes FedJigsaw, a novel framework that reformulates model personalization as bottom-up model reassembly. It introduces several components to enable effective module selection and collaborative model training.
I am generally positive about this paper, but I have concerns about the effectiveness of the proposed approach. I would also like to consider the feedback from other reviewers.
Critical strengths
- The idea of constructing personalized models through module reassembly is interesting.
- The paper is well written and clearly presented.
Critical weaknesses
My main concern is the empirical effectiveness of the proposed framework.
- Hyperparameter tuning and training curves. Please provide the hyperparameter settings and tuning procedures for all experiments, at least those in the main paper, including the ablation studies. For example, please specify the learning-rate search space and how other algorithm-specific hyperparameters were selected. Please also provide training curves showing training loss and test accuracy across communication rounds. These results should be relatively straightforward to provide if already recorded.
- Results in Table 1. Among the peer-to-peer heterogeneous FL methods, which I consider the most directly comparable baselines, FedJigsaw underperforms ProxyFL on CIFAR-10 but outperforms the decentralized baselines on CIFAR-100, EMNIST, and SVHN. This inconsistency across datasets raises concerns about the sensitivity of the results to hyperparameter tuning. Could the authors explain why FedJigsaw performs differently across these datasets?
- Inconsistent experimental settings. The main experiments (Section 4.2) use 20 clients and up to 1,000 communication rounds, whereas the ablation studies (Section 4.3) use eight clients and 300 rounds with (\alpha=0.3). Please explain the rationale for these different settings and whether they affect the conclusions drawn from the ablation studies.
Minor issues:
- According to the ICLR formatting guidelines, table captions should appear above the tables.
- Line 237: Remove either "the" or "its."
- Line 262: Correct the typo "rwae" to "raw."
Questions for authors
None.
4 comments
Thanks for the detailed responses.
-
My weakness 1. The authors report the details of the hyperparameters tuning. I appreciate that the authors acknowledge their limitations that the hyperparameters are not tuned for best performance for each task. Although I believe that well-tuned hyperparameters could improve the results of this paper, I also recognize the challenges involved in hyperparameter tuning.
-
My weakness 2. I appreciate the explanation of the performance differences across datasets.
-
My weakness 3. I appreciate the new-added ablation study.
Remaining concern: The training curves of FedJigsaw appear to exhibit a sudden jump at round 150 on the vision tasks (Figure 5). Does this mean the performance of FedJigsaw is worse in the early stage of the training? Could the authors explain this phenomenon?
Thank you for this insightful observation. During policy learning, clients’ assembly policies and shared modules evolve simultaneously, which can lead to more variable early-stage accuracy. This phase is intended to explore suitable module combinations for sustained collaboration. Particularly in practical deployments, our objective extends beyond rapid initial convergence to long-term collaboration stability and effective knowledge sharing. Rapid early convergence alone does not establish the quality of continued collaboration or the best attainable performance. Once policy learning ends, the policies are fixed while model training continues. This removes one source of non-stationarity and allows models to be refined under more stable assembly choices. AG News differs because its models start from pretrained RoBERTa representations, providing a strong foundation for downstream adaptation. This may reduce sensitivity to evolving assembly choices and help explain its smoother trajectory compared with the vision tasks.
We appreciate your constructive feedback and hope these clarifications address your remaining concern. We would be grateful if you could reconsider your assessment in light of the additional experiments and revised evidence. Thank you so much!
Thanks for the response. I will make the final decision after reading the comments of other reviewers.
Thank you for recognizing the motivation and clarity of model reassembly. We have expanded uncertainty reporting, repeated the ablations, and clarified the experimental settings and remaining tuning limitations.
W1. Hyperparameter settings and training curves Appendix A.4.1 now provides reference implementations, learning rates, batch sizes, optimizers, and method-specific parameters. Appendix A.4.3 and Fig. 5 add accuracy trajectories and cumulative/per-round communication.
We designed the evaluation with deployments in mind where extensive task-specific tuning may be impractical. We did not fine-tune every scenario for its best attainable performance; instead, we reused settings such as learning rates where applicable and followed original papers or reference implementations for method-specific parameters, documenting adaptations. This supports a consistent and reproducible comparison, without claiming equally optimal tuning for every method.
We agree that sensitivity analysis is important. Our resources do not support a systematic grid search across multiple seeds sufficient to establish reliable task-dependent tuning guidelines. Appendix A.5 acknowledges this limitation and identifies broader sensitivity analysis as future work. The current figures provide accuracy and communication curves, not training-loss curves or a systematic sensitivity study.
W2. Dataset-dependent performance ProxyFL slightly outperforms FedJigsaw on CIFAR-10. For this relatively small-scale classification task, its compact proxy may have sufficient capacity for effective knowledge alignment through distillation. We acknowledge this strength. With more complex models and larger workloads, a compact proxy may constrain knowledge sharing, while local–proxy distillation and DeSA's anchor synthesis introduce additional computation.
FedJigsaw directly reuses learned functional modules without these auxiliary operations. Its computational advantage is evident in the reported larger-workload benchmarks: on EMNIST, round time is 51.19 seconds versus 99.90 for ProxyFL and 64.79 for DeSA; on AG News, it is 431.13 seconds versus ProxyFL's 610.21. Table 1 also shows higher local accuracy than the decentralized baselines on CIFAR-100, EMNIST, and SVHN. Proxy capacity is a plausible explanation for the accuracy differences, rather than an independently isolated cause.
Hyperparameters may also influence the comparison. Our tests indicated that ProxyFL was sensitive to batch size, which affects both local training and local–proxy distillation. As explained above, we did not extensively tune individual scenarios. The settings are documented in Appendix A.4.1, and the sensitivity question is acknowledged in Appendix A.5.
W3. Consistency of experimental protocols The original experiments used separately reserved computational resources, leading to different budgets across studies. We apologize for not explaining this clearly and agree that the eight-client, 300-round ablation did not directly establish component effects under the main protocol.
We have rerun the ablations with 20 clients and 1,000 rounds on CIFAR-10, EMNIST, and SVHN at both α=0.3 and α=1.0. Full uses CoRe-Tune for the first 500 rounds. Table 2 reports across-seed variability and supports the component contributions under this expanded setting.
The shorter system-efficiency studies require dedicated resources and Mininet network emulation to limit interference in timing and communication measurements. Their budgets are a practical compromise imposed by limited resources, and their final accuracies are not treated as equivalent to those of the longer performance studies. Section 4.1 explicitly distinguishes these two groups; Secs. 4.3 and 4.5 and Appendix A.4.1 detail their protocols.
M1–M3. Presentation corrections Table captions are now above the tables. The corrected wording is “Each client broadcasts its local repository metadata to its neighbors” and “raw samples.” Thank you for identifying these issues.