- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a heterogeneous submodel based federated learning (FL) method, where the submodels can differ in the number of connections between neurons and can also skip entire layer computations. It then goes on to show how use of smaller submodels can help mitigate membership inference attacks (MIA), though that link is quite tenuous.
Critical strengths
The paper provides a very nice description of related works on training of submodels in FL. It provides an informative summary table along with limitations of each method. In the main body of the paper, it compares the performance of StitchFL against several submodel training methods, such as FedGSE, HaloFL, and SubFLOT.
Critical weaknesses
Although the heterogeneous submodel based training method presented in this paper is intended for mitigating MIA, the focus of the paper seems to be on heterogeneous submodel aggregation with very little on how it effectively mitigates MIA. The only link between heterogeneous submodel aggregation and MIA mitigation seems to be that smaller models help improve privacy. The related work does not describe existing MIA mitigation methods; the results section does not describe the performance of the proposed method against other well-established MIA methods for FL. So the link to MIA is very weak!
One of the key contributions claimed in this paper is its ability to support model aggregation when different submodels have skipped layers. However, based on the setup in Appendix C, it seems that none of the experiments presented in the main results section actually evaluate this capability. So it is not clear whether the submodel aggregation actually works when layers are skipped. Further, Section 3.2.1 proposes a stitching mechanism that reconstructs skipped layers under the assumption that the intermediate transformations can be approximated as linear. However, this assumption is questionable for deep neural networks with nonlinear activation functions, such as ReLU.
The effectiveness of this method for larger network models and more number of clients is also questionable. The paper presents results for 3 or 5 clients, which is very small for FL settings. The network models used in the evaluation are diverse (MLP, CNN and LSTM), they are all very small.
In Figure 3, StitchFL is compared with FedGSE, HaloFL, and SubFLOT under different submodel capacities. It is stated that methods with the same capacity share the same privacy level. However, this assumption is never justified. For a privacy vs accuracy analysis, I would have assumed that a MIA study similar to what is shown in Fig 1 for StitchFL would be done for each of the competing methods. Also, in Figure 3, the performance of FedGSE and SubFLOT are very close to StitchFL for both [24,24,24] and [36,36,36]. So is StitchFL effective only when the submodel size is very small?
StitchFL introduces an output-layer repair mechanism based on synthetic anchors and ridge regression in Section 3.1.2. However, a highly similar classifier calibration method has already been proposed by Dong et al. in SphereFed: Hyperspherical Federated Learning (ECCV 2022), which is not cited.
Questions for authors
How important is layer skipping to the submodel aggregation method and consequently, MIA mitigation? It was described in great details in Section 3 but missing in all the evaluations. In fact, in Figure 3, all clients seem to have the same submodel configuration.
How does StitchFL compare against existing MIA defense techniques?
For very small models, StitchFL has slightly better performance compared to FedGSE and SubFLOT. However there is almost no difference when the model capacity increases. Why is that so? Then what is the advantage of using StitchFL?
In Figure 3, privacy scores seem to be calculated solely based on the submodel capacity. Is this a correct assumption? Shouldn’t an evaluation, similar to that shown in Fig 1, be done for all the methods?
How reliable is the linear approximation used in stitching when applied to deeper networks with nonlinear activation functions?
How does the output-layer repair mechanism differ from existing classifier calibration methods? What is the specific novelty and advantage of the proposed calibration strategy?