- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript presents a convergence analysis of the two variants of Split Federated Learning (SFL), SFL-v1 and SFL-v2 as presented originally in Thapa et al. (2022), for non-convex smooth objectives. While the same algorithms have previously been analyzed in Han et al. (2024), this manuscript uses a compositional optimization problem with non-nested expectations, and a masked structure that separates client and server models. Analyses and experiments included seem to introduce and measure the dependence of convergence on the cut-layer depth.
Critical strengths
The paper is, in general, well-structured and the authors address a relevant problem in split learning, i.e., the impact of the cut-layer depth on convergence.
Critical weaknesses
There seem to be several major concerns, mainly with the contribution of the work as listed below:
-
The initial claim of SFL being a compositional optimization problem is not precise and needs clarification. The difference from conventional composite optimization is not just the nesting of expectations, as the text makes it appear:
- The important difference is that the only random variable in eq. (1) is , which represents the mini-batch, whereas in compositional optimization, there are two r.v.'s, e.g. in (2), , that acts through the inner function and , that acts through the outer function. Without the outer-function's r.v., and the inner function's expectation, the problem becomes a single-function optimization problem.
- Although, the authors state it as a technical challenge compared to composite optimization in eq. (2), eq. (1) is more straightforward to solve, since there is no stochastic gradient bias in (1), which is usually the difficulty in solving (2).
- Further down in Section 3.1, the r.v. seems to be artificially introduced, and the equation (3) rewritten to include this variable in the outer function, but in SFL, there exists no such randomness on the server-side.
- Accordingly, the assumptions and the lemmas in Appendix C also use an expectation over this artificial r.v. , which is incorrect.
-
The main novel contribution to convergence analysis of SFL seems to be only its dependence on the cut-layer depth. However, the treatment of the quantity is not rigorous:
- The manuscript claims that "Eq. (12) shows how the compositional formulation connects cut-layer to convergence", which is again vague, since the compositional formulation does not really address anything about the impact of the cut-layer any differently than previous works.
- The dependence on the cut-layer is introduced quite loosely through the constants in Assumption 4.2 and 4.3. Thus, doesn't emerge from the theoretical analysis, but is merely a notational artifact introduced. For instance, the constants affected by are also equally affected by the model architecture.
- In Assumption 4.2, should also equally depend on since the cut-layer also affects the definition of and as much as it does . Similarly, in assumption 4.3, also depends on . Perhaps the authors can clarify how the conclusions after Theorems 4.6 and 4.7 are updated accordigly
- The remarks on "cut-layer impact" after the main theorems, and the experiments do not give a conclusive statement about the dependence of convergence rate on . The authors hypothesize that "an earlier split may induce larger variability on the client-side mapping", but this is not rigorously backed by the theory. In my opinion, since this cut-layer dependence is the main contribution of this manuscript, the analysis should give a conclusive answer on this dependence.
Questions for authors
-
In the contribution bullets (line 85), it is important to clarify that a stronger rate of convergence ,, already exists under similar assumptions in Han et al (2022), and the new weaker rate of doesn't really add to that contribution. Given prior work, the client-drift analysis and sequential server updates are not novel either.
-
Although the authors claim in Table 1, that they don't need the bounded-heterogeneity assumption , such an assumption is implied by the bounded-gradients Assumption 4.2 (second inequality). Thus, the assumptions in this work are essentially the same as those made in Han et al. (2024). This similarity should be clarified, or else the comparison in the table seems to oversell the mildness of the assumptions made by the authors.
Minor comments
-
Table 1 and its counterpart in Appendix G are not entirely true. In Table 2, for Han et al. (2024), the "variant-level distinction" does exist, and the authors do present convergence analyses for both variants.
-
It is perhaps useful to note that the quantity is a vector, and so the respective norms defined in the assumptions are Frobenius norms and not the conventional spectral norms.
-
In Eq. (6), should be , or appropriately redefine the outer as a concatenation.
References
- Thapa, Chandra, et al. "Splitfed: When federated learning meets split learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 8. 2022.
- Han, Pengchao, et al. "Convergence analysis of split federated learning on heterogeneous data." Advances in Neural Information Processing Systems 37 (2024): 103476-103544.