acceptodds
Under review as a conference paper at ICLR 2027

FedJigsaw: Multi-Agent Collaborative Model Reassembly for Decentralized Heterogeneous Federated Learning

Abstract

Model-Heterogeneous Federated Learning (MHFL) enables clients with diverse resource constraints to collaboratively train models with heterogeneous architectures. Existing approaches typically construct client models by extracting sub-models from a predefined global backbone. However, this top-down paradigm confines personalization to a prescribed architectural space. Meanwhile, other architecture-search-based methods impose significant computational and memory overhead to the server. We present FedJigsaw, a novel framework that reformulates model personalization as bottom-up model reassembly. Our key insight is that cohesive functional modules can serve as reusable units of both model structure and learned knowledge. Analogous to assembling a Jigsaw puzzle, each client dynamically constructs its personalized architecture by selecting compatible modules learned locally or obtained from its neighbors. We formulate the assembly process as a sequential decision problem, enabling each client to adapt its architecture to its resource constraints and learning context. To keep this process lightweight, FedJigsaw employs an attention-based pointer network to directly score a variable set of feasible modules, rather than enumerating the complete candidate architectures. Our experimental results demonstrate that, compared with the state-of-the-art MHFL schemes, FedJigsaw improves local accuracy by up to 8.56%, reduces per-client communication by 63.8%, and shortens round time by 17.3%, demonstrating the effectiveness and scalability of module-level reassembly.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Modular Model Reassembly, Neural Architecture Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FBHYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FedJigsaw, a novel framework that reformulates model personalization as bottom-up model reassembly. It introduces several components to enable effective module selection and collaborative model training.

I am generally positive about this paper, but I have concerns about the effectiveness of the proposed approach. I would also like to consider the feedback from other reviewers.

Critical strengths
  1. The idea of constructing personalized models through module reassembly is interesting.
  2. The paper is well written and clearly presented.
Critical weaknesses

My main concern is the empirical effectiveness of the proposed framework.

  1. Hyperparameter tuning and training curves. Please provide the hyperparameter settings and tuning procedures for all experiments, at least those in the main paper, including the ablation studies. For example, please specify the learning-rate search space and how other algorithm-specific hyperparameters were selected. Please also provide training curves showing training loss and test accuracy across communication rounds. These results should be relatively straightforward to provide if already recorded.
  2. Results in Table 1. Among the peer-to-peer heterogeneous FL methods, which I consider the most directly comparable baselines, FedJigsaw underperforms ProxyFL on CIFAR-10 but outperforms the decentralized baselines on CIFAR-100, EMNIST, and SVHN. This inconsistency across datasets raises concerns about the sensitivity of the results to hyperparameter tuning. Could the authors explain why FedJigsaw performs differently across these datasets?
  3. Inconsistent experimental settings. The main experiments (Section 4.2) use 20 clients and up to 1,000 communication rounds, whereas the ablation studies (Section 4.3) use eight clients and 300 rounds with (\alpha=0.3). Please explain the rationale for these different settings and whether they affect the conclusions drawn from the ablation studies.

Minor issues:

  1. According to the ICLR formatting guidelines, table captions should appear above the tables.
  2. Line 237: Remove either "the" or "its."
  3. Line 262: Correct the typo "rwae" to "raw."
Questions for authors

None.

4 comments

Thank you for recognizing the motivation and clarity of model reassembly. We have expanded uncertainty reporting, repeated the ablations, and clarified the experimental settings and remaining tuning limitations.

W1. Hyperparameter settings and training curves Appendix A.4.1 now provides reference implementations, learning rates, batch sizes, optimizers, and method-specific parameters. Appendix A.4.3 and Fig. 5 add accuracy trajectories and cumulative/per-round communication.

We designed the evaluation with deployments in mind where extensive task-specific tuning may be impractical. We did not fine-tune every scenario for its best attainable performance; instead, we reused settings such as learning rates where applicable and followed original papers or reference implementations for method-specific parameters, documenting adaptations. This supports a consistent and reproducible comparison, without claiming equally optimal tuning for every method.

We agree that sensitivity analysis is important. Our resources do not support a systematic grid search across multiple seeds sufficient to establish reliable task-dependent tuning guidelines. Appendix A.5 acknowledges this limitation and identifies broader sensitivity analysis as future work. The current figures provide accuracy and communication curves, not training-loss curves or a systematic sensitivity study.

W2. Dataset-dependent performance ProxyFL slightly outperforms FedJigsaw on CIFAR-10. For this relatively small-scale classification task, its compact proxy may have sufficient capacity for effective knowledge alignment through distillation. We acknowledge this strength. With more complex models and larger workloads, a compact proxy may constrain knowledge sharing, while local–proxy distillation and DeSA's anchor synthesis introduce additional computation.

FedJigsaw directly reuses learned functional modules without these auxiliary operations. Its computational advantage is evident in the reported larger-workload benchmarks: on EMNIST, round time is 51.19 seconds versus 99.90 for ProxyFL and 64.79 for DeSA; on AG News, it is 431.13 seconds versus ProxyFL's 610.21. Table 1 also shows higher local accuracy than the decentralized baselines on CIFAR-100, EMNIST, and SVHN. Proxy capacity is a plausible explanation for the accuracy differences, rather than an independently isolated cause.

Hyperparameters may also influence the comparison. Our tests indicated that ProxyFL was sensitive to batch size, which affects both local training and local–proxy distillation. As explained above, we did not extensively tune individual scenarios. The settings are documented in Appendix A.4.1, and the sensitivity question is acknowledged in Appendix A.5.

W3. Consistency of experimental protocols The original experiments used separately reserved computational resources, leading to different budgets across studies. We apologize for not explaining this clearly and agree that the eight-client, 300-round ablation did not directly establish component effects under the main protocol.

We have rerun the ablations with 20 clients and 1,000 rounds on CIFAR-10, EMNIST, and SVHN at both α=0.3\alpha=0.3 and α=1.0\alpha=1.0. Full uses CoRe-Tune for the first 500 rounds. Table 2 reports across-seed variability and supports the component contributions under this expanded setting.

The shorter system-efficiency studies require dedicated resources and Mininet network emulation to limit interference in timing and communication measurements. Their budgets are a practical compromise imposed by limited resources, and their final accuracies are not treated as equivalent to those of the longer performance studies. Section 4.1 explicitly distinguishes these two groups; Secs. 4.3 and 4.5 and Appendix A.4.1 detail their protocols.

M1–M3. Presentation corrections Table captions are now above the tables. The corrected wording is “Each client broadcasts its local repository metadata to its neighbors” and “raw samples.” Thank you for identifying these issues.

Thanks for the detailed responses.

  • My weakness 1. The authors report the details of the hyperparameters tuning. I appreciate that the authors acknowledge their limitations that the hyperparameters are not tuned for best performance for each task. Although I believe that well-tuned hyperparameters could improve the results of this paper, I also recognize the challenges involved in hyperparameter tuning.

  • My weakness 2. I appreciate the explanation of the performance differences across datasets.

  • My weakness 3. I appreciate the new-added ablation study.

Remaining concern: The training curves of FedJigsaw appear to exhibit a sudden jump at round 150 on the vision tasks (Figure 5). Does this mean the performance of FedJigsaw is worse in the early stage of the training? Could the authors explain this phenomenon?

Thank you for this insightful observation. During policy learning, clients’ assembly policies and shared modules evolve simultaneously, which can lead to more variable early-stage accuracy. This phase is intended to explore suitable module combinations for sustained collaboration. Particularly in practical deployments, our objective extends beyond rapid initial convergence to long-term collaboration stability and effective knowledge sharing. Rapid early convergence alone does not establish the quality of continued collaboration or the best attainable performance. Once policy learning ends, the policies are fixed while model training continues. This removes one source of non-stationarity and allows models to be refined under more stable assembly choices. AG News differs because its models start from pretrained RoBERTa representations, providing a strong foundation for downstream adaptation. This may reduce sensitivity to evolving assembly choices and help explain its smoother trajectory compared with the vision tasks.

We appreciate your constructive feedback and hope these clarifications address your remaining concern. We would be grateful if you could reconsider your assessment in light of the additional experiments and revised evidence. Thank you so much!

Thanks for the response. I will make the final decision after reading the comments of other reviewers.

Reviewer_JtR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedJigsaw allows clients with different resource budgets to build personalized models from compatible neural-network modules available locally or from neighboring clients. A learned assembly policy selects the modules, and each client trains its assembled model before returning updated modules to their original owners for aggregation. An optional centralized critic helps train the assembly policies, while model assembly and module exchange can operate using local and neighborhood information. Experiments on image and text classification report competitive accuracy and strong efficiency across several settings.

Critical strengths
  1. The paper addresses a meaningful problem with an intuitive approach. Reusing functional modules provides a concrete way for clients with different model capacities to collaborate without requiring every personalized architecture to be extracted from one global backbone.
  2. The main components form a coherent design. The attention-based policy accommodates changing candidate sets, while metadata discovery, selective parameter retrieval, and aggregation at module owners provide a clear mechanism for sharing learned components.
  3. The experimental evaluation covers several datasets and model families. The paper also includes component ablations, analyses of learned assembly behavior, and measurements of communication, computation, and network sensitivity, providing useful evidence beyond classification accuracy alone.
Critical weaknesses
  1. The main comparisons do not isolate the accuracy and cost of fully decentralized training: Table 1 does not separately report FedJigsaw without CoRe-Tune, and the available ablation uses eight clients and 300 rounds rather than the main 20-client protocol. Moreover, disabling CoRe-Tune removes both the centralized critic and the cross-client exploration reward, so the ablation cannot isolate the critic’s contribution (Appendix A.2). The system-efficiency benchmark also does not clearly establish whether centralized policy-training computation and communication are included.
  2. The evidence for scalability and topology robustness remains limited: Appendix A.4.4 provides useful neighborhood-degree experiments, but the graph construction procedure and sensitivity to different graph structures at comparable degrees are insufficiently specified. Federation-size experiments stop at 20 clients, while the main system-efficiency benchmark uses only four clients, leaving performance and resource costs in a larger federated learning setting unverified.
Questions for authors
  1. Which main accuracy and efficiency results use CoRe-Tune, and do the reported costs include its centralized training and communication? Please compare the full and fully decentralized variants under the same main experimental protocol, and separately ablate the centralized critic and the cross-client exploration reward.
  2. How are the communication graphs constructed, and are their edges directed or undirected? Could the authors evaluate a larger federation (e.g., 100 clients) and compare different sparse graph structures at a matched average degree, reporting accuracy, round time, communication, and per-client memory requirements?
2 comments

Thank you for recognizing the functional-module design. Your comments motivated clearer accounting of optional coordination and new evaluations of larger federations and matched-degree graphs.

W1 / Q1. Decentralized operation, CoRe-Tune, and cost accounting Tables 1–2 now compare Full FedJigsaw with w/o CoRe-Tune under the 20-client setting and include comparisons against other baselines. Our focus is how much an optional coordination server improves performance and whether FedJigsaw remains effective without it. The results support effective collaboration without centralized policy coordination and additional improvements with CoRe-Tune. The variant removes the centralized critic and cross-client exploration reward together, measuring their combined contribution. Limited computational resources prevented separate component ablations, which remain valuable for understanding their individual roles.

In the 1,000-round ablation, Full uses CoRe-Tune for the first 500 rounds. The system benchmark uses it for the first 150 of 300 rounds on CIFAR-10/EMNIST and the first 50 of 100 rounds on AG News. Figure 5 and Appendix A.4.3 show per-round and cumulative communication across both phases, including the pronounced reduction after CoRe-Tune ends on CIFAR-10 and EMNIST. The discussion therefore distinguishes the initial policy-learning overhead from recurring execution costs; lower execution-phase traffic does not erase earlier cumulative costs. These phase comparisons do not constitute separate critic/reward ablations.

W2 / Q2. Larger federations, graph structure, and memory We expanded the system benchmark from four to 20 clients and added 40-, 60-, 80-, and 100-client experiments. Exactly 20 clients train per round, while the others remain reachable as module providers. This accommodates our computational and network-emulation budget while evaluating collaboration across a larger federation. It does not represent 100 clients training simultaneously.

Table 7 reports accuracy, round time, and endpoint communication per registered client. At 100 clients, local accuracy is 62.48%, versus 75.53% at 20 clients, while average round time increases from 6.68 to 10.97 seconds. FedJigsaw thus remains operational and maintains useful predictive performance at larger scales with a moderate increase in round time. The accuracy difference should also be interpreted in light of reduced per-client participation under the fixed round budget, rather than attributed to federation size alone.

Links are bidirectional. The regular construction connects clients symmetrically around a ring. Table 9 compares regular, degree-preserving rewired, and random-regular graphs, all with 20 clients and five neighbors per client. Local accuracy ranges from 76.69% to 78.21%, and round time from 2.61 to 2.76 seconds, suggesting limited sensitivity to the tested graph constructions. Appendix A.4.4 reports their communication costs and additional bandwidth comparisons. These are fixed-topology experiments, rather than tests of peers disconnecting during training.

Because clients share a physical server, dividing aggregate GPU usage among clients would not reliably measure individual-device memory requirements. We report aggregate GPU memory in the system benchmark rather than present such an estimate as a per-client measurement. Direct per-device measurements on heterogeneous hardware are planned as future work.

Thanks for your response. After reading the revised manuscript and the authors' response, I decided to keep my score unchanged because, according to Table 1, FedJigsaw's superiority mainly comes from the CoRe-Tune module. However, this module requires the centralized critic and the cross-client exploration reward, so FedJigsaw should be compared with those server-coordinated baselines. In this sense, the proposed FedJigsaw cannot demonstrate improved performance.

Moreover, after revisiting the paper, I feel the literature review is a bit weak, particularly because it does not discuss the most classic paper [1].

[1] Hybrid Local SGD for Federated Learning with Heterogeneous Communications

Reviewer_o2iEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the FedJigsaw framework. FedJigsaw is an approach for Model-Heterogeneous FL. This approach introduces a decentralized P2P method in which attention-based methods are used to probabilistically select suitable modules from the available modules of participating clients during policy training, based on the requirements and resources of the respective client. In this way, an individual model is assembled from the selected and subsequently retrieved modules, meeting the given requirements of the client. This model is then trained locally. Afterward, the trained modules originating from other clients are sent back to them. Aggregation takes place at the respective source clients using the returned updates to their own modules. Optionally, FedJigsaw can be combined with CoRe-Tune, which adds centralized training coordination. Compared with various baseline methods, FedJigsaw achieves good accuracy across multiple datasets and comparable accuracy to centralized approaches.

Critical strengths
  • A decentralized MHFL framework that outperforms other baseline methods in most cases and also achieves comparable accuracy to centralized approaches. Particularly noteworthy are the significantly higher accuracies on AG News with RoBERTa-base compared to the centralized methods evaluated there (Table 1).

  • A well-considered and structured method using attention-based and RL techniques to select suitable modules, so that the module with the highest probability is not simply always selected, but the context of each client is also taken into account during selection (Section 3).

  • Extensive evaluation on image and text classification tasks, including communication, runtime, and memory requirements (Section 4).

  • A well-explained approach and a well-written, easy-to-understand paper.

Critical weaknesses
  • The state-of-the-art section lacks a discussion of decentralized MHFL methods, which are later also used in the evaluation (Section 2).

  • The main experiments, ablation studies, and supplementary analyses use different numbers of clients, communication rounds, and, in some cases, different levels of data heterogeneity (Sections 4.2–4.4). In particular, the assembly policy is optimized during the first 500 rounds in the main comparison, whereas the ablation studies already end after 300 rounds. Although the individual comparisons are internally consistent, it remains unclear whether the observed ablation effects also hold under the main experimental configuration.

  • Important details of the experimental evaluation are missing, particularly the construction of client-specific test data distributions and the variance across independent experiments. Table 1 reports standard deviations across clients but does not provide uncertainty across different seeds. Furthermore, decentralized baseline comparisons are missing for the remarkably large improvements on AG News.

  • Although early stopping is used (lines 310–313), the paper does not report after how many communication rounds the individual methods actually converge or terminate. Since Table 3 only reports communication overhead per round, the total communication and time required until convergence remain unclear. Accuracy-over-Rounds and Accuracy-versus-Cumulative-Communication plots would significantly improve the assessment of the claimed efficiency advantages.

Minor Issues:

  • An explicit mathematical objective function, including performance and resource constraints, should already be presented in Section 3.1. The reward definition is also too central to the method to be explained exclusively in the appendix. I would like to see at least the final reward function and the essential optimization conditions of your RL approach included in the main paper, as I consider these to be a core component of the method.

  • Further non-IID evaluation with a lower alpha would be great i.e. $\alpha = 0.1 $.

  • Heterogeneous devices are only simulated. How realistic are your simulations, and how robust will the method be on real devices?

  • The architecture search space is limited to manually defined, sequentially composable modules, as the authors rightly acknowledge in the limitations. Nevertheless, the scientific novelty compared to existing methods could be highlighted more clearly.

  • Table captions should be placed above the tables rather than below them.

  • A space is missing in "CoRe-Tuneas" (line 262). There is also a typo in line 263: "rwae samples" instead of "raw samples"?

Questions for authors
  • How are training, validation, and test data distributed across clients, and are the improvements reproducible across multiple independent seeds?

  • In Section 4.5, the communication reduction compared to FedMRL is highlighted for CIFAR-100 (lines 457–459). However, the decentralized baselines ProxyFL and DeSA require significantly less communication than FedJigsaw. How do you assess this accuracy-communication trade-off, and why is the comparison with the decentralized methods not discussed in more detail here?

  • Why does the method not reduce communication costs as much on CIFAR-100 or with VGG models as on other datasets or models?

  • In what realistic scenario is collaboration between heterogeneous devices beneficial, and what benefits do resource-rich clients, in particular, gain from collaborative training? This should be used to strengthen the motivation.

  • What did your communication topology look like? How did you select the neighbors? Randomly?

  • How do you determine global accuracy?

  • Is it possible to ablate the critic and exploration reward separately?

  • Why does CoRe-Tune require centralized coordination, and why is it not also implemented in a fully decentralized manner?

  • How are the neighborhoods constructed and selected? How robust is FedJigsaw to dynamic changes in the network structure, for example, when existing neighborhood connections are lost during training or new connections are established?

2 comments

Thank you for your constructive review. References refer to the revision.

W1. Related work. Section 2 adds ProxyFL, DeSA, multi-headed distillation (CVPR 2023), and query-based transfer (ICLR 2025), distinguishing their mechanisms from module reuse.

W2/W4. Protocols and learning costs. Resource reservations led to different study budgets; we apologize for unclear reporting. Section 4.1 and Appendix A.4.1 distinguish performance studies from dedicated system tests. We reran ablations with 20 clients, 1,000 rounds, and both α=0.3\alpha=0.3 and 1.01.0 on CIFAR-10, EMNIST, and SVHN; Full uses CoRe-Tune for 500 rounds. Table 2 reports across-seed variability.

Early stopping was considered but not applied, to keep metrics comparable; we corrected the description and recomputed seed variability from original logs. Longer runs lacked exclusive resources for controlled timing. Dedicated Mininet tests use 300 image-task/100 AG News rounds due to resource costs. Figure 5 (Appendix A.4.3) reports accuracy and cumulative/per-round traffic, including the CTDE transition. These do not measure cost to convergence or full-budget accuracy, or directly plot accuracy against cumulative traffic.

W3/Q1/Q6. Evaluation and reproducibility. Appendix A.4.1 specifies class-wise Dirichlet allocation of official training data, followed by seeded 80/10/10 train/validation/local-test splits per client. Policy feedback uses validation data, not local test data. Global accuracy averages personalized models' accuracies on the same official test set. Table 1 reports three seeds: mean local accuracy and population client SD, each with its across-seed sample SD in brackets. AG News ProxyFL reaches 92.59%/94.38% (low/high), versus FedJigsaw's 94.44%/96.20%. The text proxy is documented; DeSA requires a separate text-anchor adaptation.

M1. Objective and reward. Section 3.1, Eq. (1), maximizes mean validation accuracy subject to neighborhood, compatibility, and depth constraints. Section 3.2.3 adds the accuracy/exploration reward and policy update, Eqs. (5)-(6); Appendix A.2 details optimization. Depth bounds architecture, not runtime memory or latency.

M2. Stronger heterogeneity. Extreme label skew can hinder shared-model learning while narrowing local tasks. We use α=0.3/1.0\alpha=0.3/1.0 and also report shared-distribution accuracy to distinguish local specialization from collaboration. Testing α=0.1\alpha=0.1 remains future work.

M3/Q9. Deployment and dynamics. Training uses physical GPUs and Mininet TCP links in separate namespaces; bandwidth is tested in Appendix A.4.4. Fixed graphs and inactive-but-reachable clients do not simulate disconnection. Appendix A.5 acknowledges dynamic-resource limitations. We plan Mininet-WiFi and neighbor-prediction studies. Dynamic support requires peer discovery, interrupted-transfer handling, and update consistency, beyond module-availability checks.

M4. Novelty. Unlike pFedHR's server-side layer reassembly and PerFedRLNAS's architecture search, FedJigsaw composes learned functional modules from neighbors, preserving internal structure and returning updates to their owners. This enables contextual, bidirectional sharing without a global supernet. Section 4.4 tests initialization; Appendix A.5 discusses predefined boundaries, sequential assembly, and future graph construction.

Q2/Q3. Communication trade-offs. We reran dedicated system tests with explicit endpoint accounting, clarifying the Docker-network scope: CIFAR-10, EMNIST, and AG News, not CIFAR-100. Small proxies/anchors can cost fewer bytes than VGG modules; FLOPs do not determine payload size. Table 1 shows higher CIFAR-100 accuracy than ProxyFL/DeSA at both heterogeneity levels. FedJigsaw is faster than ProxyFL/DeSA/FedMRL on EMNIST and ProxyFL/FedMRL on AG News, but ProxyFL communicates less on AG News. Sections 4.2/4.5 discuss these trade-offs; individual causes are not isolated.

Q4. Motivation. Small cameras may observe conditions absent at powerful peers; returned updates let stronger owners learn from borrowers' data (Appendix A.4.2).

Q5. Graphs. The 20-client regular graph has bidirectional links to offsets ±1\pm1, ±2\pm2, and the opposite client. Neighbors remain fixed; module selection is learned. Table 9 tests degree-five rewired/random-regular alternatives: accuracy is 76.69%-78.21%.

Q7/Q8. Optional coordination. Evolving actors/modules make learning non-stationary. The critic uses cross-client context to evaluate local actions, not choose them or aggregate a model. Table 1 includes fully decentralized w/o CoRe-Tune. We test the server's overall benefit and effectiveness without it. Removing critic and exploration reward together measures their joint contribution; separate ablations remain future work.

M5/M6. Presentation. Table captions are above tables; “CoRe-Tune as” and “raw samples” have been corrected.

Thank you for the detailed rebuttal and the substantial revisions. I appreciate the improved experimental consistency, additional seed results, and clearer documentation, which address several of my original concerns.

However, some concerns remain. While the clarification regarding early stopping is sufficient, the total communication required to reach comparable accuracy remains unclear. In particular, the substantial communication overhead compared to ProxyFL on AG News deserves further consideration. Furthermore, the individual contributions of the centralized critic and exploration reward remain unverified.

I also noticed that the conclusion incorrectly attributes the reported communication and runtime reductions of 36.3% and 48.8% to text classification, although these values appear to correspond to EMNIST (Table 4).

Overall, the revisions strengthen the paper, but the remaining concerns prevent me from increasing my rating.

Comments on the paper

1 comment
Area_Chair_mgdqOpenReview ↗

Reminder: Reviewer–Author Discussion

As the discussion deadline is approaching, I would kindly remind the reviewers to review the authors' rebuttal and provide their follow-up comments. In particular, it would be helpful to clarify whether your original concerns have been addressed and whether your assessment has changed. Even if your assessment remains unchanged, a brief response would be greatly appreciated.

Authors are also encouraged to remain engaged and respond to any follow-up questions. Thank you all for your time and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.