acceptodds
Withdrawn from ICLR 2027

StitchFL: Heterogeneous Submodel Training in Federated Learning for Membership Inference Attacks Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated Learning (FL) enables organizations to collaboratively train models without sharing raw data, yet exchanged models can remain vulnerable to membership inference attacks (MIAs). This vulnerability often increases with model size, creating a privacy–utility trade-off, since bigger models tend to perform better. Submodel training in FL offers a natural way to limit model exposure by allowing each client to train only a fraction of the global architecture. However, many existing approaches impose structural constraints such as nested submodels, shared bases, full-model clients, coverage of every layer, or auxiliary data at the server. We present StitchFL, a heterogeneous submodel training method that allows clients to train arbitrary subsets of neurons across layers (including skipping layers entirely) and reconstructs the global model without requiring private or auxiliary data at the aggregator. StitchFL combines count-normalised averaging of overlapping slices, pseudo-inverse stitching of skipped layers, and anchor-based output recalibration using synthetic inputs. We instantiate the method for dense, convolutional, and recurrent architectures. Using a dense neural network, StitchFL reaches 0.89 accuracy compared to 0.83 achieved by the second best submodel training baseline. It also outperforms competing submodel methods on the convolutional case and forecasting task. Across six MIAs, restricting clients to 40% of each hidden layer reduces the strongest MIA effect by nearly 68.3%. These results show that flexible submodel allocation can substantially reduce client-side model exposure while preserving most of the utility of full-model federated training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Fedetader Learning, Machine Learning, Submodel Training, Membership Inference Attack

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pnRpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a heterogeneous submodel based federated learning (FL) method, where the submodels can differ in the number of connections between neurons and can also skip entire layer computations. It then goes on to show how use of smaller submodels can help mitigate membership inference attacks (MIA), though that link is quite tenuous.

Critical strengths

The paper provides a very nice description of related works on training of submodels in FL. It provides an informative summary table along with limitations of each method. In the main body of the paper, it compares the performance of StitchFL against several submodel training methods, such as FedGSE, HaloFL, and SubFLOT.

Critical weaknesses

Although the heterogeneous submodel based training method presented in this paper is intended for mitigating MIA, the focus of the paper seems to be on heterogeneous submodel aggregation with very little on how it effectively mitigates MIA. The only link between heterogeneous submodel aggregation and MIA mitigation seems to be that smaller models help improve privacy. The related work does not describe existing MIA mitigation methods; the results section does not describe the performance of the proposed method against other well-established MIA methods for FL. So the link to MIA is very weak!

One of the key contributions claimed in this paper is its ability to support model aggregation when different submodels have skipped layers. However, based on the setup in Appendix C, it seems that none of the experiments presented in the main results section actually evaluate this capability. So it is not clear whether the submodel aggregation actually works when layers are skipped. Further, Section 3.2.1 proposes a stitching mechanism that reconstructs skipped layers under the assumption that the intermediate transformations can be approximated as linear. However, this assumption is questionable for deep neural networks with nonlinear activation functions, such as ReLU.

The effectiveness of this method for larger network models and more number of clients is also questionable. The paper presents results for 3 or 5 clients, which is very small for FL settings. The network models used in the evaluation are diverse (MLP, CNN and LSTM), they are all very small.

In Figure 3, StitchFL is compared with FedGSE, HaloFL, and SubFLOT under different submodel capacities. It is stated that methods with the same capacity share the same privacy level. However, this assumption is never justified. For a privacy vs accuracy analysis, I would have assumed that a MIA study similar to what is shown in Fig 1 for StitchFL would be done for each of the competing methods. Also, in Figure 3, the performance of FedGSE and SubFLOT are very close to StitchFL for both [24,24,24] and [36,36,36]. So is StitchFL effective only when the submodel size is very small?

StitchFL introduces an output-layer repair mechanism based on synthetic anchors and ridge regression in Section 3.1.2. However, a highly similar classifier calibration method has already been proposed by Dong et al. in SphereFed: Hyperspherical Federated Learning (ECCV 2022), which is not cited.

Questions for authors

How important is layer skipping to the submodel aggregation method and consequently, MIA mitigation? It was described in great details in Section 3 but missing in all the evaluations. In fact, in Figure 3, all clients seem to have the same submodel configuration.

How does StitchFL compare against existing MIA defense techniques?

For very small models, StitchFL has slightly better performance compared to FedGSE and SubFLOT. However there is almost no difference when the model capacity increases. Why is that so? Then what is the advantage of using StitchFL?

In Figure 3, privacy scores seem to be calculated solely based on the submodel capacity. Is this a correct assumption? Shouldn’t an evaluation, similar to that shown in Fig 1, be done for all the methods?

How reliable is the linear approximation used in stitching when applied to deeper networks with nonlinear activation functions?

How does the output-layer repair mechanism differ from existing classifier calibration methods? What is the specific novelty and advantage of the proposed calibration strategy?

Reviewer_RQLSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents STITCHFL, a heterogeneous submodel training method that allows clients to train arbitrary subsets of neurons across layers and reconstructs the global model without requiring private or auxiliary data at the aggregator. STITCHFL combines count-normalised averaging of overlapping slices, pseudo-inverse stitching of skipped layers, and anchor-based output recalibration using synthetic inputs. We instantiate the method for dense, convolutional, and recurrent architectures. Using a dense neural network, STITCHFL reaches 0.89 accuracy compared to 0.83 achieved by the second-best submodel training baseline. It also outperforms competing submodel methods on the convolutional case and forecasting task. Across six MIAs, restricting clients to 40% of each hidden layer reduces the strongest MIA effect by nearly 68.3%.

Critical strengths
  • The problem is important when clients have different computational capabilities.
  • Exposing smaller submodels may reduce the effectiveness of membership inference attacks.
Critical weaknesses
  • The stitching method uses a pseudo-inverse and assumes that nonlinear neural network activations are approximately linear. The paper does not provide sufficient evidence that this assumption holds or that the reconstructed model remains accurate.
  • The paper does not adequately show that StitchFL provides additional privacy benefits beyond simply reducing model capacity.
Questions for authors
  • How is the approximate linearity assumption behind pseudo-inverse stitching justified, and how does stitching error affect end-to-end accuracy as more layers are skipped?
  • How do the authors isolate the privacy benefit of StitchFL from the effect of simply reducing the exposed model's parameter count and capacity?
  • How do synthetic Gaussian anchors produce reliable output-layer calibration on real data under non-IID distributions?
  • How do the authors explain StitchFL's outstanding accuracy degradation under non-IID data, and what evidence demonstrates its robustness across different degrees of heterogeneity?
  • What formal basis supports the differential privacy discussion, and are the baseline comparisons fair in terms of model capacity and evaluation protocol?
Reviewer_wcEDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

StitchFL combines client networks of different sizes through weight averaging, pseudo-inverse stitching and output-layer repair using synthetic inputs. The paper evaluates classification and forecasting, and tests membership inference on an MLP. The aim is to recover useful global predictions while exposing smaller models to clients.

Critical strengths

1. Combining overlapping submodels, including ones that skip layers, is a useful problem. Repairing the output layer with synthetic inputs is a practical way to improve the fused model without giving the server training samples.

2. The ablations help explain what each component contributes. Table 2 reports gains from 0.7380 to 0.7844 with output-layer repair and from 0.7459 to 0.7651 with stitching. The paper also reports non-IID settings where the method struggles.

Critical weaknesses
  1. The privacy comparison assumes that equally sized models leak equally. Sections 4.2.1 and 4.2.3 assign the same privacy level across methods without reporting attacks on each baseline’s submodels or the small-model FedAvg reference. Appendix D.1.2 shows a trend with model size, but does not establish equal leakage across training methods. Direct comparisons using the same budgets and data splits are needed.

  2. The theory behind fusion needs more justification. Appendix B.1.1 treats the intermediate activation as roughly linear, but gives no clear conditions for when this is accurate or a bound on the resulting error. Frankle et al. (2020) concerns mode connectivity, which does not establish this assumption. The ablations help, but a broader question remains: when partitioning leaves some connections untrained and set to zero, under what conditions can fusion and output-layer repair still approach the accuracy of the full model?

  3. The privacy–accuracy trade-off is not fully established. StitchFL improves on small-model FedAvg, but also exposes some clients to larger networks. Without measuring leakage for both, it is unclear how much privacy accompanies this accuracy gain. The headline 68.3% reduction also needs a clear definition, including the metric and how results are combined across clients.

  4. The main accuracy result is inconsistent across sections. Section 4.2.2 reports 0.89 MLP accuracy, while Appendix D.2/Figure 5 indicates approximately 0.862. Appendix D.3.1 also describes 0.862 as essentially the ten-round result. Please explain the difference, since it affects the reported gains over FedGSE and small-model FedAvg.

Questions for authors
  1. Can you compare accuracy and membership leakage for StitchFL, small-model FedAvg, full-model FedAvg and the submodel baselines under matched attacker access? Please report attack checkpoints, sample and reference-model counts, and uncertainty.

  2. Appendix C considers access to one client submodel. What happens if an attacker combines information from several clients’ models, possibly across rounds? Does the privacy advantage over full-model FedAvg remain under matched access? This goes beyond the stated threat model, but would clarify the claim’s scope.

  3. What explains the 0.89 versus 0.862 accuracy values? How is the 68.3% privacy reduction calculated, and does it describe the best, average or most exposed client?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.