acceptodds
Under review as a conference paper at ICLR 2027

Convergence of Split Federated Learning via a Masked Compositional Optimization Lens

Abstract

Split federated learning (SFL) trains a split neural network across distributed clients by keeping initial layers on-device and offloading the remaining layers to a server. The split point is called the cut layer, where clients transmit intermediate activations to the server and receive backpropagated gradients. While SFL reduces client compute and memory, the resulting client–server coupling makes its optimization behavior poorly understood. This paper develops, to our knowledge, the first compositional convergence analysis of SFL. We show that SFL optimizes a single-expectation compositional objective, i.e., a client-side forward map produces cut-layer representations, and a server-side map completes the network and evaluates the loss. Unlike existing federated compositional optimization where the main challenge is tracking an inner expectation, SFL's inner map is a per-minibatch network transform, so the key challenge is controlling the split-induced coupling. We address the challenge and establish nonconvex convergence results for two major SFL variants. For the parallel variant SFL-V1, after communication rounds, we obtain an rate. For the sequential variant SFL-V2, we obtain an rate with a leading constant that depends on and split-dependent quantities, where is the number of clients. Enabled by the compositional formulation, our theory reveals insights on SFL design choices, including (i) how the cut layer affects convergence, and (ii) how SFL-V1 and SFL-V2 scale differently with . Experiments on regression and classification tasks corroborate our theory.

Primary area:
optimization
Keywords:
Split federated learning, distributed optimization, compositional optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mUJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript presents a convergence analysis of the two variants of Split Federated Learning (SFL), SFL-v1 and SFL-v2 as presented originally in Thapa et al. (2022), for non-convex smooth objectives. While the same algorithms have previously been analyzed in Han et al. (2024), this manuscript uses a compositional optimization problem with non-nested expectations, and a masked structure that separates client and server models. Analyses and experiments included seem to introduce and measure the dependence of convergence on the cut-layer depth.

Critical strengths

The paper is, in general, well-structured and the authors address a relevant problem in split learning, i.e., the impact of the cut-layer depth on convergence.

Critical weaknesses

There seem to be several major concerns, mainly with the contribution of the work as listed below:

  • The initial claim of SFL being a compositional optimization problem is not precise and needs clarification. The difference from conventional composite optimization is not just the nesting of expectations, as the text makes it appear:

    • The important difference is that the only random variable in eq. (1) is ξ\xi, which represents the mini-batch, whereas in compositional optimization, there are two r.v.'s, e.g. in (2), γ\gamma, that acts through the inner function and ϵ\epsilon, that acts through the outer function. Without the outer-function's r.v., and the inner function's expectation, the problem becomes a single-function optimization problem.
    • Although, the authors state it as a technical challenge compared to composite optimization in eq. (2), eq. (1) is more straightforward to solve, since there is no stochastic gradient bias in (1), which is usually the difficulty in solving (2).
    • Further down in Section 3.1, the r.v. ζ\zeta seems to be artificially introduced, and the equation (3) rewritten to include this variable in the outer function, but in SFL, there exists no such randomness on the server-side.
    • Accordingly, the assumptions and the lemmas in Appendix C also use an expectation over this artificial r.v. ζ\zeta, which is incorrect.
  • The main novel contribution to convergence analysis of SFL seems to be only its dependence on the cut-layer depth. However, the treatment of the quantity ω\omega is not rigorous:

    • The manuscript claims that "Eq. (12) shows how the compositional formulation connects cut-layer to convergence", which is again vague, since the compositional formulation does not really address anything about the impact of the cut-layer any differently than previous works.
    • The dependence on the cut-layer is introduced quite loosely through the constants in Assumption 4.2 and 4.3. Thus, ω\omega doesn't emerge from the theoretical analysis, but is merely a notational artifact introduced. For instance, the constants affected by ω\omega are also equally affected by the model architecture.
    • In Assumption 4.2, CfC_f should also equally depend on ω\omega since the cut-layer also affects the definition of ff and yy as much as it does gg. Similarly, in assumption 4.3, σf\sigma_f also depends on ω\omega. Perhaps the authors can clarify how the conclusions after Theorems 4.6 and 4.7 are updated accordigly
    • The remarks on "cut-layer impact" after the main theorems, and the experiments do not give a conclusive statement about the dependence of convergence rate on ω\omega. The authors hypothesize that "an earlier split may induce larger variability on the client-side mapping", but this is not rigorously backed by the theory. In my opinion, since this cut-layer dependence is the main contribution of this manuscript, the analysis should give a conclusive answer on this dependence.
Questions for authors
  • In the contribution bullets (line 85), it is important to clarify that a stronger rate of convergence ,O(1/T)O(1/T), already exists under similar assumptions in Han et al (2022), and the new weaker rate of O(1/T)O(1/\sqrt{T}) doesn't really add to that contribution. Given prior work, the client-drift analysis and sequential server updates are not novel either.

  • Although the authors claim in Table 1, that they don't need the bounded-heterogeneity assumption H\mathcal{H}, such an assumption is implied by the bounded-gradients Assumption 4.2 (second inequality). Thus, the assumptions in this work are essentially the same as those made in Han et al. (2024). This similarity should be clarified, or else the comparison in the table seems to oversell the mildness of the assumptions made by the authors.

Minor comments

  • Table 1 and its counterpart in Appendix G are not entirely true. In Table 2, for Han et al. (2024), the "variant-level distinction" does exist, and the authors do present convergence analyses for both variants.

  • It is perhaps useful to note that the quantity g(k)(⋅)g^{(k)}(\cdot) is a vector, and so the respective norms defined in the assumptions are Frobenius norms and not the conventional spectral norms.

  • In Eq. (6), ∇xs\nabla_{x_s} should be ∇x\nabla_x, or appropriately redefine the outer ⊙\odot as a concatenation.

References

  • Thapa, Chandra, et al. "Splitfed: When federated learning meets split learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 8. 2022.
  • Han, Pengchao, et al. "Convergence analysis of split federated learning on heterogeneous data." Advances in Neural Information Processing Systems 37 (2024): 103476-103544.
Reviewer_RwEaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a masked compositional formulation of split federated learning (SFL), in which a binary mask splits the model at the cut layer into a client-side forward map and a server-side map that completes the network and evaluates the loss. Under this formulation, they prove nonconvex O(1/T)O(1/\sqrt{T}) convergence for SFL-V1, with parallel updates and periodic averaging, and SFL-V2, with sequential server-side updates. The leading constant for SFL-V2 scales with K2K^2. From the mask-dependent constants, they derive guidance on cut-layer selection, synchronization period, and the choice between variants. Experiments on regression and image classification show that later cut layers perform better, that SFL-V2 is more sensitive to the cut layer, and that SFL-V1 can outperform SFL-V2 when the number of clients is large.

Critical strengths
  • A careful treatment of the sequential server handoff in SFL-V2. In SFL-V2, each client's stochastic gradient depends on samples drawn by earlier clients within the same iteration. The analysis handles this cleanly in three steps. It tracks the synchronized model x^t\hat{x}_t rather than the plain average. It derives an exact block-scaled recursion: client-side parameters move by the averaged gradient, and server-side parameters by the sum of the KK sequential gradients. And it conditions on a filtration that follows the client order. I consider this the paper's most substantive technical contribution. SFL-V2 is also the variant in which the cut layer genuinely affects the optimization trajectory, so this machinery could support a sound analysis of how the split point matters.

  • A clear and accessible formulation of SFL. The masked formulation and the pseudocode make the two SFL variants precise and easy to compare. That clarity could benefit future theoretical work on split learning. However, the formulation is a reparameterization of the standard full-network loss rather than a new problem class, so this strength carries limited weight in my overall assessment.

Critical weaknesses
  • The cut-layer insights for SFL-V1 reflect the bound, not the algorithm. The client and server maps are two halves of the same network, so the stochastic gradient in Eq. 6 is identical for every mask ω\omega. Algorithm 1 is local SGD with iterates independent of the cut layer. The ω\omega-dependent constants are therefore just different upper bounds on the same quantities. The experiments showing strong cut-layer effects for SFL-V1 cannot reflect Algorithm 1 as written. I consider this the most consequential weakness, because cut-layer guidance is a headline contribution.

  • The K2K^2 separation between SFL-V1 and SFL-V2 does not support the variant-choice guidance. At the same η\eta, SFL-V2's server step is KK times larger (Eq. 46). This produces the K2K^2 term, while Eq. 48 discards the matching gain in descent. The comparison also pits a variance-based bound against a second-moment bound. In addition, the paper's own Appendix G shows that Han et al. (2024) already give separate SFL-V1 and SFL-V2 bounds with a K2K^2 term. FEMNIST at K=100K=100 (Table 3) shows no consistent SFL-V1 advantage.

Questions for authors

Clarification

  1. Since Eq. 6 equals the full-network gradient for any mask ω\omega, the iterates of Algorithm 1 appear to be independent of the cut layer. Is this correct? If so, which property of SFL-V1's convergence, as opposed to the tightness of its bound, depends on ω\omega? Please also describe how SFL-V1 was implemented to produce the cut-layer effects in Figs. 3a, 4a and 7. In particular:

    • Do the client-side and server-side parts use the same synchronization schedule?
    • How are optimizer states handled?
    • Are random seeds fixed across cut layers?
    • In vanilla SL, are client-side weights passed between clients?
  2. Does the K2K^2 term remain if SFL-V2's server step is rescaled to η/K\eta/K, or if the KK-fold descent gain discarded in Eq. 48 is retained? Could you compare the two variants empirically at matched effective server step sizes, or with each variant's step size tuned separately?

  3. Relation to Han et al. (2024). Appendix G, Table 2 lists separate SFL-V1 and SFL-V2 bounds from Han et al., including a K2K^2 term for SFL-V2. How does your variant-level distinction go beyond theirs, and why does Table 1 mark it as absent?

  4. Bounded gradients imply bounded client dissimilarity (Lemma D.1), so your bounds are identical for IID and non-IID data. How do the "connection to theory" explanations of heterogeneity effects follow from the analysis?

Reviewer_BPkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes convergence of split federated learning (SFL), where clients train the first few layers of the network and the server trains the rest of them. They modeled the network as a client-side function feeding a server-side loss, which makes the cut layer appear in the bounds. THe analysis proves O(1/√T) convergence for the two main variants of SFL namely, SFL-V1 and SFL-V2, and concludes that later cut layers help and that V1 scales better with many clients. They provide experimental results on regression and image classification tasks and vary the cut layer and the number of clients.

Critical strengths
  • The proofs are complete and carefully written, specially the inclusion of sequential case (SFL-V2).
  • Portraying SFL as a client-side function feeding a server-side loss is a new lens. It brings the cut layer into the convergence analysis, which earlier SFL theory didn't do explicitly.
Critical weaknesses

My main concern is about the significance and impact of this work.

  • W1: The paper never says which practical question SFL practitioners cannot answer with existing analyses and that this formulation answers.
  • W2: The intro motivation (p.2, "Why does the compositional formulation matter?") is that it "makes the cut layer an explicit part of the analysis". Yet the paper never uses that to predict or choose a cut layer.
  • W3: The guidance it gives are already established and known e.g., prefer a later cut or V2 is more cut-sensitive and has already been reported by Han et al. (NeurIPS 2024) and other existing works.
  • W4: The "technical challenge" paragraph (p.2) contrasts SFL with nested compositional problems (Eq. 2), but nobody has proposed applying nested-compositional methods to SFL, so the challenge it sets up does not motivate the work.
  • W5: Importantly, choosing a cut layer in practice depends on client memory, activation traffic and privacy, none of which the analysis models.

Furthermore

  • W6: Under the paper's assumptions (Lemma D.2), SFL-V1 reduces to standard local SGD, whose convergence is already known. Can you compare against existing similar works on SGD [1]

  • W7: The "split-dependent constants" appear to be G, σ and L written out through the chain rule. They are never measured: Also, H.5 reports cross-client variance, which is not considered in any of the bounds.

  • W8: The claimed V1/V2 distinction does not seem new. The paper claims to be "the first theoretical characterization that distinguishes" SFL-V1 and SFL-V2. However, Han et al. (NeurIPS 2024) already give separate V1/V2 bounds as mentioned in the the paper's own Table 2.

[1] Yu, Hao, Sen Yang, and Shenghuo Zhu. "Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.

Questions for authors
  1. What practical decision does this analysis enable that Han et al. (2024) and standard local-SGD bounds [1] do not? For example, can the constants be estimated before or early in training to rank cut layers?
  2. Can you estimate the constants that actually appear in the bounds (δ_g, σ_g, C_g, L_F) for each cut layer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.