- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper focuses on the challenge of heterogeneous federated learning (HFL), where differences in parameter dimensionality across local models prevent direct parameter aggregation. Specifically, this method targets the local training oscillation problem that may occur when bidirectional knowledge transfer occurs between large and small local models in proxy-model-based approaches. The paper proposes FEDSSP, a model-stitching-based HFL training framework that uses a predefined shared branch for knowledge aggregation and transfer. The experiments demonstrate the feasibility of the proposed approach in the evaluated settings.
Critical strengths
An alternative perspective on heterogeneous model aggregation. This paper explores solutions to the heterogeneous model aggregation problem through model stitching only.
Critical weaknesses
- Inconsistency between motivation, design and evaluation. The research is actually motivated by local training oscillations arising during knowledge transfer between large and small models with substantial architectural differences (lines 58-63). However, FEDSSP primarily aggregates and transfers knowledge through a predefined shared architecture (feature-extraction layers). The feasibility of this approach rests on the assumption that the heterogeneous local models do not differ substantially in architecture, as reflected in the model zoo configurations used in the paper, including the ResNet-family models in Table 1 and the first three of the four columns for each dataset in Table 2. The proposed approach and its evaluation are therefore inconsistent with the stated research motivation.
- Limited additional contributions. FEDSSP builds on the bottom–top model-stitching principle for transferring knowledge [1]. Relative to prior approaches, its main differences appear to be twofold: (i) rather than replacing the original generic feature-extraction layers, it retains them and introduces shared features as an auxiliary, additive representation to augment the input to subsequent layers; and (ii) it evaluates and selects personalized stitching positions, introducing additional overhead. The additional contributions are limited, and the necessity of these design choices is not sufficiently established. Prior studies [2–4] have already demonstrated that stitching a feature extractor to a personalized classifier can achieve good performance without the additional warm-up, position-evaluation, and selection procedures introduced here. Furthermore, the finding that shallow stitching yields the worst performance has already been reported in [5]. Reproducing this observation may provide corroborating evidence, but offers limited support for the novelty or necessity of the proposed position-selection mechanism.
[1].Yang X, Zhou D, Liu S, et al. Deep model reassembly[J]. Advances in neural information processing systems, 2022, 35: 25739-25753.
[2]. Steiner, A.P., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2021). How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. Trans. Mach. Learn. Res., 2022.
[3]. Bansal Y, Nakkiran P, Barak B. Revisiting model stitching to compare neural representations[J]. Advances in neural information processing systems, 2021, 34: 225-236.
[4]. Leng Z, Zhang C, Long G, et al. LEGO-FL: Learning Heterogeneous Federated Models as a LEGO Assembly Games[C]//Forty-third International Conference on Machine Learning.
[5].Lenc K, Vedaldi A. Understanding image representations by measuring their equivariance and equivalence[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 991-999.
Questions for authors
Some key details remain insufficiently specified, including the layers corresponding to the personalized stitching positions across heterogeneous architectures, performance when the architectures differ substantially, and the criterion used to describe the shared branch as “lightweight.” However, clarification of these details alone would not resolve the two main weakness outlined above. I therefore do not raise additional questions in this section.