acceptodds
Under review as a conference paper at ICLR 2027

FedSSP: Shared-Branch Stitching into Private Models for Heterogeneous Federated Learning

Abstract

Heterogeneous Federated Learning (HtFL) has attracted increasing attention for its ability to support collaboration under both model and data heterogeneity. However, existing methods require visibility into private architectures, rely on public datasets, or introduce additional asymmetric knowledge transfer. To overcome these limitations, We propose FedSSP, which represents transferable global knowledge as a shared branch stitched into individual private models. Through a lightweight stitching layer, FedSSP jointly optimizes shared knowledge injection and local personalization in single-stage dual-path training, without a standalone proxy or separate distillation. FedSSP selects client-specific stitch positions by complementing intra-model KL-based block capacity with cross-entropy improvement to account for local task utility. For aggregation, shared-feature shifts and relative private-model updates serve as complementary contribution proxies, capturing representation changes and local adaptation. Extensive experiments on various datasets with eleven model architectures show that FedSSP consistently outperforms existing HtFL approaches, achieving an average accuracy improvement of 7.24%, without sharing any private model parameters or relying on public datasets.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Model Stitching, Heterogeneous Model Architectures, Heterogeneous Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JjnwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on the challenge of heterogeneous federated learning (HFL), where differences in parameter dimensionality across local models prevent direct parameter aggregation. Specifically, this method targets the local training oscillation problem that may occur when bidirectional knowledge transfer occurs between large and small local models in proxy-model-based approaches. The paper proposes FEDSSP, a model-stitching-based HFL training framework that uses a predefined shared branch for knowledge aggregation and transfer. The experiments demonstrate the feasibility of the proposed approach in the evaluated settings.

Critical strengths

An alternative perspective on heterogeneous model aggregation. This paper explores solutions to the heterogeneous model aggregation problem through model stitching only.

Critical weaknesses
  1. Inconsistency between motivation, design and evaluation. The research is actually motivated by local training oscillations arising during knowledge transfer between large and small models with substantial architectural differences (lines 58-63). However, FEDSSP primarily aggregates and transfers knowledge through a predefined shared architecture (feature-extraction layers). The feasibility of this approach rests on the assumption that the heterogeneous local models do not differ substantially in architecture, as reflected in the model zoo configurations used in the paper, including the ResNet-family models in Table 1 and the first three of the four columns for each dataset in Table 2. The proposed approach and its evaluation are therefore inconsistent with the stated research motivation.
  2. Limited additional contributions. FEDSSP builds on the bottom–top model-stitching principle for transferring knowledge [1]. Relative to prior approaches, its main differences appear to be twofold: (i) rather than replacing the original generic feature-extraction layers, it retains them and introduces shared features as an auxiliary, additive representation to augment the input to subsequent layers; and (ii) it evaluates and selects personalized stitching positions, introducing additional overhead. The additional contributions are limited, and the necessity of these design choices is not sufficiently established. Prior studies [2–4] have already demonstrated that stitching a feature extractor to a personalized classifier can achieve good performance without the additional warm-up, position-evaluation, and selection procedures introduced here. Furthermore, the finding that shallow stitching yields the worst performance has already been reported in [5]. Reproducing this observation may provide corroborating evidence, but offers limited support for the novelty or necessity of the proposed position-selection mechanism.

[1].Yang X, Zhou D, Liu S, et al. Deep model reassembly[J]. Advances in neural information processing systems, 2022, 35: 25739-25753.

[2]. Steiner, A.P., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2021). How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. Trans. Mach. Learn. Res., 2022.

[3]. Bansal Y, Nakkiran P, Barak B. Revisiting model stitching to compare neural representations[J]. Advances in neural information processing systems, 2021, 34: 225-236.

[4]. Leng Z, Zhang C, Long G, et al. LEGO-FL: Learning Heterogeneous Federated Models as a LEGO Assembly Games[C]//Forty-third International Conference on Machine Learning.

[5].Lenc K, Vedaldi A. Understanding image representations by measuring their equivariance and equivalence[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 991-999.

Questions for authors

Some key details remain insufficiently specified, including the layers corresponding to the personalized stitching positions across heterogeneous architectures, performance when the architectures differ substantially, and the criterion used to describe the shared branch as “lightweight.” However, clarification of these details alone would not resolve the two main weakness outlined above. I therefore do not raise additional questions in this section.

Reviewer_sS5kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedSSP, which integrates a shared branch into private client models and combines personalized stitch-position selection with contribution-aware aggregation to enable knowledge sharing in heterogeneous federated learning.

Critical strengths
  1. The joint training of the shared branch and private models enables global knowledge sharing and local personalization.

  2. Experiments are conducted on multiple datasets and heterogeneous model architectures, with additional analyses of communication and computational costs.

Critical weaknesses
  1. The method mainly combines existing techniques, including model stitching, position selection, and weighted aggregation, resulting in limited overall novelty.

  2. Feature changes and parameter update magnitudes are used to measure client contributions, but the rationale behind these two metrics is not sufficiently analyzed.

  3. The main comparison experiments use only a single random seed, without statistical results from multiple independent runs, leaving the stability of the experimental results insufficiently validated.

Questions for authors

See above.

Reviewer_QFQhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedSSP for personalized federated learning under both model and data heterogeneity. A homogeneous shared branch is connected to each client’s private model through a lightweight stitching layer. Shared and private features are fused and jointly optimized through a unified task loss. Client-specific stitch positions are selected using inter-block KL divergence and cross-entropy improvement from probes attached to warmed-up private models. Aggregation weights combine shared-feature shifts with relative private-model updates. Experiments cover several vision datasets, model families, and client participation settings, with additional component ablations, architectural extensions, and communication and computation analyses.

Critical strengths

1.The problem is practically relevant, and the design is clear. The paper provides an implementable approach for heterogeneous private models to collaborate through a lightweight shared component while retaining local personalization, making the framework potentially useful in practical deployments.

2.The empirical evaluation has broad coverage and useful supplementary analyses. The study examines data and model heterogeneity, scalability, resource costs, and the roles of major components. The supplementary architectural extensions and contribution diagnostics further improve understanding of the method’s applicability.

Critical weaknesses

1.The incremental contribution is not adequately established against the closest direct feature-fusion approaches.

FedMRL (NeurIPS 2024) already combines shared and heterogeneous private representations through feature fusion and joint training. Its without-MRL ablation removes the shared prediction head and computes the loss through the private head. This directly relevant approach is neither discussed nor evaluated.

FedSSP’s client-specific internal insertion and aggregation rule may provide meaningful differences. However, the current experiments do not establish how much these new choices improve upon existing direct-fusion frameworks. Comparisons matched for private models, shared-component capacity, and training budget are needed to identify the incremental contribution, beyond the advantages over methods that use separate knowledge-transfer stages.

2.The main evaluation protocol does not sufficiently establish stable and comparable performance improvements.

Section 5.1 and Appendix A.2 state that the main comparisons use a fixed random seed and report the highest average test accuracy over 100 communication rounds. FedSSP’s hyperparameters are selected through sensitivity analyses, whereas the baselines use their original defaults. Although Appendices C.7–C.9 include three-seed sensitivity experiments, an independent validation procedure for hyperparameter selection is not clearly described, and uncertainty across seeds is not reported for the final method comparisons.

Selecting the best test round introduces optimistic selection bias. A common seed helps control the data partition for one comparison, but does not establish robustness to randomness. This matters particularly for small improvements: for example, the gain over the strongest baseline in the CIFAR-10 Mixed setting of Table 2 is only 0.21 percentage points. Without a clearly separated validation procedure, comparable tuning budgets, and repeated comparisons, the stability of the reported advantages remains uncertain.

3.The stitch-selection experiments do not isolate the value of the added cross-entropy utility signal.

The addition of cross-entropy improvement to the KL-based inter-block transition score is presented as an important design choice for identifying useful stitch positions. The comparison with three fixed positions in Figure 5(a) supports the importance of placement, but does not directly establish whether the added utility term improves upon KL-only selection.

Appendix C.8 evaluates mixing coefficients from 0.1 to 0.9, omitting the KL-only and utility-only endpoints. Furthermore, agreement with the most frequently selected position across coefficient settings measures selection stability rather than downstream benefit. The necessity and added value of the joint criterion over its individual components therefore remain insufficiently isolated.

Questions for authors

1.Closest comparisons and incremental contribution: How does FedSSP compare with FedMRL and its without-MRL feature-fusion variant under matched private models, comparable shared-component capacity, identical data partitions, and local training budgets? Can the gains from client-specific internal insertion and the aggregation mechanism be isolated?

2.Evaluation protocol and stability: Were independent validation data used to select hyperparameters and communication rounds? Please provide repeated matched-seed comparisons with comparable baseline tuning budgets, validation-selected checkpoints, and uncertainty estimates. Please also specify how the headline average improvement of 7.24% is calculated, including whether it denotes percentage points or relative improvement, the comparison baselines, and the aggregation across settings.

3.Stitch-selection criterion: How do KL-only, utility-only, and joint selection perform under identical warm-up and probing budgets? For a representative subset of clients, does the ranking of candidate positions by the proposed score agree with validation performance after training at those positions?

4.Interpretation of contribution weights: Appendix C.10 provides a useful leave-one-out diagnostic. Could the authors additionally quantify within-round cross-client rank agreement between aggregation weights and marginal utility, including stability across seeds and training stages? How does the rule affect aggregation when a client exhibits a large update magnitude but negative marginal utility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.