acceptodds
Under review as a conference paper at ICLR 2027

CoeF-SFL: Preserving Collaborative Server-Client Learning in Split Federated Learning with Enhanced Communication Efficiency

Abstract

Split Federated Learning (SFL) enables resource-constrained clients to participate in collaborative training, but vanilla SFL exchanges smashed data and gradients at every batch, which incurs significant communication overhead. Recent methods reduce this overhead with an auxiliary network at the client-side cut layer. However, we identify that this approach makes the client optimize a local objective that differs from the end-to-end objective, which fundamentally limits the collaborative training between the client and the server. We propose Compensated Feedback based SFL (CoeF-SFL), a communication-efficient framework that retains the end-to-end objective without any auxiliary network. In CoeF-SFL, the client and the server exchange the smashed data and the gradients once per round and reuse them during local training. Since this reuse makes the gradients stale on the client side, we compensate them with a curvature-based correction in the activation space and develop two variants. CoeF-D approximates the Hessian with a diagonal gradient outer product, while CoeF-J exploits the tractable Jacobian-based Hessian of a surrogate loss that upper-bounds the true loss. We provide the theoretical background of each method, characterizing its compensation. Across vision and language tasks, model capacities, cut layers, and data distributions, CoeF-SFL significantly outperforms auxiliary-network-based methods under the same communication frequency, and the improvement is most substantial on vision tasks. Code is available at https://anonymous.4open.science/r/CoeF-SFL-2686/README.md

Primary area:
optimization
Keywords:
Split learning, Federated learning, Split federated learning, Communication efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3tbxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Split Federated Learning (SFL) under low communication frequency. The authors propose CoeF-SFL, which exchanges smashed data and gradients at the beginning of each round and employs curvature-based correction to support multiple local client updates, thereby reducing communication exchanges without introducing auxiliary networks. CoeF-D uses a diagonal gradient outer-product approximation, while CoeF-J employs a surrogate model based on the server-side Jacobian. The paper provides theoretical analysis under specific conditions and demonstrates performance improvements over gradient reuse without compensation, achieving results close to vanilla SFL across vision and language tasks.

Critical strengths

1.The paper addresses an important trade-off between communication exchanges and training quality in SFL. Leveraging server-side feedback to support client updates under low communication frequency is practically meaningful. 2.The comparisons between gradient reuse without compensation and the perfect-compensation oracle provide valuable motivation. CoeF-D and CoeF-J recover substantial performance degradation across multiple settings, demonstrating the potential of the proposed approach. 3.The experiments cover multiple tasks, model capacities, cut layers, and data distributions. The appendix further investigates auxiliary-network capacity, curvature approximations, and resource costs.

Critical weaknesses

W1: The methodological novelty relative to existing gradient compensation techniques is insufficiently established. The paper applies gradient compensation to activation staleness caused by multiple local updates in SFL. However, the discussion of closely related work remains insufficient, and the methodological novelty is not clearly distinguished. DC-ASGD (Zheng et al., 2017) already uses a first-order Taylor expansion and a diagonal Hessian approximation based on the gradient outer product to compensate for stale gradients, which closely resembles the core design of CoeF-D. CELU-VFL (Fu et al., 2022) supports multiple local updates by caching and reusing forward activations and backward derivatives while accounting for the resulting staleness errors. Furthermore, FedLite (Wang et al., 2022) employs an activation-space Taylor expansion and Hessian approximation to correct gradient mismatch in split learning. Although these methods differ in their training settings and sources of error, they have already explored gradient staleness compensation, activation and derivative reuse, and activation-space gradient correction, respectively, all of which are closely related to the proposed approach. Therefore, the authors should clarify what technical challenges arising from activation staleness under multiple local updates remain unresolved by existing methods, and what substantive new algorithmic designs or theoretical analyses are introduced by CoeF-D and CoeF-J. Simply emphasizing that this is the first work to study activation-space gradient staleness is insufficient to clearly establish the technical contributions beyond prior work.

W2: The theoretical analysis of CoeF-J does not adequately characterize its practical compensation mechanism. CoeF-J constructs a surrogate loss to guide client updates, and Proposition 2 establishes an upper-bound property of this surrogate with respect to the server-side loss under specific conditions. However, this result relies on the exact sensitivity matrix and a fixed curvature upper-bound coefficient, whereas the practical algorithm approximates the sensitivity matrix using a single random projection and employs sample-dependent compensation coefficients. In particular, the practical coefficients may be smaller than the curvature upper bound required by the theoretical analysis, so the established upper-bound property is not necessarily preserved. Although the paper acknowledges this discrepancy, it does not further analyze whether these approximations undermine the key properties of the surrogate model or how they affect the discrepancy between the compensated gradients and the true server-side gradients.

W3: The communication efficiency advantage is not sufficiently demonstrated. The paper emphasizes that CoeF-SFL improves efficiency by reducing communication exchanges between clients and the server. However, Table 9 shows that CoeF-D and CoeF-J transmit more data and incur additional server-side computation compared with auxiliary-network-based methods such as AccSFL and FSL-SAGE. Therefore, the current experiments mainly demonstrate that CoeF-SFL can maintain high model accuracy under low communication frequency, but do not sufficiently establish whether it provides better overall communication efficiency than existing methods.

Questions for authors

Q1. The total number of training rounds is fixed at T=100. As training progresses, the client-side model parameters and the resulting activation representations continue to evolve. However, the paper does not analyze how activation displacement and gradient staleness vary across different stages of training. Can CoeF-D and CoeF-J continue to compensate for gradient errors effectively over longer training horizons and maintain their performance advantages over gradient reuse without compensation?

Q2. The effectiveness of the compensation mechanisms is mainly evaluated through final test accuracy, making it unclear whether the performance improvements actually result from correcting stale gradients. In particular, the compensation terms in CoeF-D and CoeF-J may improve training stability by modifying gradient directions or suppressing activation displacement, without necessarily making the feedback gradients closer to the true gradients. Could the authors provide direct analyses of gradient errors or directional alignment to validate the actual effects of the compensation mechanisms and distinguish the contributions of gradient correction from those of optimization stabilization?

Q3. How do the authors precisely define "preserving the end-to-end objective"? The proposed method uses server-side feedback computed at the beginning of each round, while the server-side model continues to be updated during local training. Therefore, even completely eliminating the gradient staleness caused by activation displacement does not necessarily make the client-side gradients consistent with the current server-side model. For example, on Non-IID Food-101, the perfect-compensation oracle achieves 55.6% accuracy, substantially below the 63.9% achieved by vanilla SFL. How do the authors explain this gap?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 3tbx (1/6)

Please also see our general response (Summary of Revisions).

[W.1] Novelty relative to existing gradient compensation techniques: DC-ASGD, CELU-VFL, and FedLite

We thank the reviewer for pointing out these related studies, which helped us sharpen the contribution of this work. We clarify below that (i) we distinguish the client-side staleness from model drift as the main cause of the degradation, (ii) existing methods only mitigate it heuristically, and (iii) the novelty of our compensation lies mainly in CoeF-J.

The problem and its characterization. As the reviewer pointed out, CELU-VFL (Fu et al., 2022) addresses a similar problem in vertical federated learning: it also reuses cached derivatives over multiple local updates, but treats the resulting staleness as a generic approximation error. In contrast, we distinguish the client-side staleness caused by gradient reuse from model drift, and characterize it both theoretically and empirically (see our response to Reviewer 4GpS (Q2) and Eq. (6) in Section 3). In particular, to first order, the staleness of the gradient equals the curvature of the server-side loss at the transmitted smashed data applied to the displacement of the activation caused by the client's own updates, and hence it accumulates over the local updates. DC-ASGD (Zheng et al., 2017) instead addresses model drift, where the gradient is exact for an outdated model that actually existed, and FedLite (Wang et al., 2022) addresses a one-shot quantization error that does not accumulate. Under gradient reuse, however, the client is trained against a fictitious server-side model that never existed, which deviates further as the smashed data move. Accordingly, in Table 1, gradient reuse without compensation drops to 37.137.1\\% on MNLI-m, whereas gradient reuse with perfect compensation, subject only to model drift, performs close to vanilla SFL.

Differences in addressing the staleness. CELU-VFL only down-weights stale instances by cosine similarity and does not correct the stale gradient. Correction is difficult in SFL because the required curvature belongs to the server-side model, which the client does not hold, and no further exchange is allowed within the round. FedLite (Wang et al., 2022) uses a scaled identity, which carries no curvature information of the server-side loss. Hence, only the DC-ASGD-type surrogate, built solely from the received gradient, can be applied directly.

CoeF-D and CoeF-J. In terms of the compensation itself, CoeF-D adopts the surrogate of DC-ASGD as it is. However, by analyzing it theoretically in our setting, we obtain interpretations that DC-ASGD does not provide: the diagonal outer product of the received gradient agrees in expectation with the GGN diagonal with respect to the activation, it reduces the feedback error of gradient reuse if and only if λ<2λi⋆\lambda<2\lambda_i^\star (Proposition 1), and the scale of λ\lambda required for the observed labels grows as the model fits them, which explains our choice of λ=3000\lambda=3000 (Proposition 3). In contrast, CoeF-J combines techniques that none of the above methods uses: the Jacobian of the server-side model, its random projection to reduce the communication, and the closed-form curvature bound of the loss with an adaptive coefficient, from which it builds an upper-bound surrogate of the loss instead of approximating the curvature (Proposition 2). We believe that this design has sufficient methodological novelty, and our theory supports it: under the conditions of Proposition 2, the client performs majorization–minimization of the end-to-end loss (Proposition 5), and with the adaptive coefficient and random projections, the bound holds in expectation with a violation decreasing as 1/R1/\sqrt{R} (Proposition 4).

We have added a paragraph titled “Gradient compensation and reuse” to Section 2, where we introduce DC-ASGD, CELU-VFL, and FedLite and explain how the client-side staleness and our compensation differ from theirs.

References

Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. Towards communication-efficient vertical federated learning training via cache-enabled local updates. Proceedings of the VLDB Endowment, 15(10):2111–2120, 2022.

Jianyu Wang, Hang Qi, Ankit Singh Rawat, Sashank Reddi, Sagar Waghmare, Felix X. Yu, and Gauri Joshi. FedLite: A scalable approach for federated learning on resource-constrained clients. arXiv preprint arXiv:2201.11865, 2022.

Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In International Conference on Machine Learning (ICML), pp. 4120–4129, 2017.

AuthorsOpenReview ↗

Response to Reviewer 3tbx (2/6)

[W.2] Theoretical analysis of CoeF-J: Guarantees under the practical compensation

We thank the reviewer for drawing attention to the distinction between the theoretical surrogate and the implemented correction. As the reviewer pointed out, the original analysis did not cover the compensation parameters used in practice, i.e., the compensation factor λ\lambda of CoeF-D, and the adaptive coefficient MiM_i and the random projections of CoeF-J. Through this revision, we have added the corresponding analyses to Appendix C of the revised manuscript. For CoeF-D, we show that the diagonal outer product for the observed labels becomes smaller than its predictive-label expectation as the model fits its labels, so that the scale of λ\lambda matching the curvature grows with the confidence of the model, which explains our choice of λ=3000\lambda=3000 (Proposition 3). For CoeF-J, we show that the loss upper bound of Proposition 2 remains valid with the adaptive coefficient, even though it can be smaller than wMwM, and holds in expectation over the random projections, with a violation whose expected amount decreases as 1/R1/\sqrt{R} (Proposition 4); in practice, violations are rare with R=1R=1 (Appendix F.2). Moreover, since Eq. (14) holds for any B^i\widehat B_i, the discrepancy between the compensated gradient and the true server-side gradient remains bounded by the curvature mismatch $\Vert M_i\widehat P_i-H_{0,i}\Vert $ for every realization of the projections. Finally, we have added a convergence analysis of CoeF-SFL in Appendix E, which requires only a finite bound on this mismatch and therefore applies equally to the implemented CoeF-J with R=1R=1 and the adaptive coefficient, which converges at the rate O(1/T)O(1/\sqrt{T}). For details, please refer to the propositions and appendices above, or to our responses to Reviewer V8MQ (Q2 on the guarantees of CoeF-J and Q3 on the sensitivity to λ\lambda and RR).

[W.3] Communication efficiency: Comparison with auxiliary-network-based methods

We appreciate the reviewer's distinction between fewer exchanges and overall communication efficiency. We agree that CoeF-SFL transmits more data and uses more server computation than auxiliary-network-based methods. However, these savings of auxiliary-network-based methods come at the cost of the end-to-end objective and a large loss in accuracy, whereas the additional cost of CoeF-SFL is small in wall-clock time. We therefore believe that the two should be compared in terms of the trade-off between accuracy and communication, in which CoeF-SFL offers a substantially better trade-off.

Trade-off between accuracy and cost. Auxiliary-network-based methods save communication and server computation by training the client with a local objective instead of the end-to-end objective, which limits their accuracy; e.g., in Table 2, AccSFL achieves 47.447.4\\% on Tiny-ImageNet, whereas CoeF-SFL achieves 78.378.3\\%, close to vanilla SFL. Enlarging the auxiliary network improves their accuracy, but it shifts the cost to the memory and computation of the clients, the scarcest resources in SFL, and the accuracy still remains below that of CoeF-SFL (Table 15). In contrast, the additional cost of CoeF-SFL lies only in the downlink and on the server: CoeF-SFL uploads the same activations as AccSFL and FSL-SAGE (Table 9 of the original manuscript, Table 14 in the revised manuscript), and the additional download takes only a small fraction of the round time, since the downlink is typically much faster than the uplink in cellular networks. Compared with vanilla SFL, CoeF-SFL exchanges data once per round instead of once per mini-batch, which shortens the round time, e.g., from 46.3646.36 s to 13.5313.53 s on a commercial 5G mmWave network with two local epochs (Table R2 (Reviewer V8MQ, Q1)). The round-time model and the comparison with vanilla SFL are detailed in our response to Reviewer V8MQ (Q1) and Appendix G.3.

AuthorsOpenReview ↗

Response to Reviewer 3tbx (3/6)

[Q.1] Longer training horizons: Displacement and staleness across training stages

We thank the reviewer for this thoughtful suggestion. To answer this question, we extended training to T=200T=200 rounds on ViT-Tiny/CIFAR-100 and examined the accuracy, activation displacement, and feedback alignment over the training horizon. Both CoeF-D and CoeF-J continue to compensate the staleness effectively and maintain their advantage over gradient reuse without compensation throughout the 200 rounds. We have added these results as Appendix F.3 of the revised manuscript.

Accuracy over training. As shown in Table R1, under the IID split, CoeF-D and CoeF-J reach 84.7984.79\\% and 83.9083.90\\% at round 200, respectively, whereas gradient reuse without compensation trains only with a ten times smaller learning rate, converges more slowly, and remains below CoeF-D throughout. Under α=0.1\alpha=0.1, gradient reuse diverges and stays at 11\\% accuracy, whereas CoeF-D and CoeF-J reach 69.2069.20\\% and 72.3872.38\\%, even exceeding the 66.5466.54\\% of the oracle with perfect compensation.

Displacement and feedback alignment. Table R2 reports the displacement $\Vert \delta\Vert $ of the activation and the cosine similarity of the received feedback g0g_0 and the applied feedback g~\widetilde g with g⋆g^\star, the gradient of the round-initial server at the current activation. With CoeF-SFL, the displacement does not grow as training progresses, e.g., it stays around 55 under the IID split for both variants, and the received feedback remains well aligned with g⋆g^\star at every stage, so the staleness does not accumulate over longer horizons. Under α=0.1\alpha=0.1, the displacement of gradient reuse grows explosively, whereas it remains bounded with CoeF-SFL, and CoeF-J keeps it the smallest (≈5\approx5) with cos⁡(g0,g⋆)≈0.99\cos(g_0,g^\star)\approx0.99. The low cosine similarity of the applied feedback of CoeF-J is expected: CoeF-J does not approximate g⋆g^\star but minimizes a majorizer of the end-to-end loss under the conditions of Proposition 5, and its correction penalizes the displacement in the directions to which the server output is sensitive, which keeps the received feedback accurate. CoeF-D instead prevents the feedback from deviating far from g⋆g^\star rather than raising the alignment at every step, as will be discussed in our response to Q2. Note that g⋆g^\star is evaluated along the trajectory of each method, so the cosine values indicate the staleness within each method and should not be compared across methods.

[Table R1. Test accuracy (%) on ViT-Tiny / CIFAR-100 (ν=1\nu=1, T=200T=200). Learning rate 10−310^{-3}, except IID reuse (10−410^{-4}).]

IIDα=0.1\alpha{=}0.1
Round5010015020050100150200
Gradient reuse (with perfect comp.)84.1984.6384.6284.5061.7965.9667.8566.54
Gradient reuse (w/o comp.)69.8278.6680.6782.071.001.001.001.00
CoeF-D84.0484.6885.0984.7961.1769.7369.6069.20
CoeF-J83.5484.1684.2083.9065.4971.4971.3072.38
Reviewer_V8MQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Compensated Feedback-based Split Federated Learning (CoeF-SFL), a communication-efficient framework designed to retain server-provided training feedback without requiring an auxiliary network at the client-side cut layer. The authors identify a limitation of auxiliary-network-based SFL: local auxiliary objectives can deviate from the end-to-end (E2E) objective and impair collaborative server-client optimization. To address this problem, CoeF-SFL reuses the server-provided activation gradients over multiple local updates and introduces curvature-based compensation to mitigate gradient staleness. Two variants are developed: CoeF-D employs a scaled diagonal gradient outer product, while CoeF-J uses a Jacobian-based sensitivity matrix derived from a surrogate loss. The paper provides theoretical characterizations of these corrections and evaluates the methods on vision and language tasks with multiple architectures, data distributions, and model split points. The experiments show that CoeF-SFL substantially outperforms auxiliary-network-based baselines at comparable communication frequency, often approaching vanilla SFL accuracy

Critical strengths
  1. Good motivation and a clearly identified problem: The paper identifies an important trade-off between communication frequency and optimization quality. Its observation that auxiliary-network-based SFL optimizes a different local objective is well motivated by prior work on locally supervised learning. The introduction of a perfect-compensation oracle helps distinguish the impact of gradient staleness from the limitations of auxiliary objectives.

  2. Consistent empirical improvements across diverse tasks: The reported results are strong, particularly on vision tasks. The experiments also cover different model capacities, cut layers, IID and non-IID partitions, and several language tasks. These results support the effectiveness of the proposed feedback mechanism beyond a single benchmark.

Critical weaknesses
  1. Limited validation of practical communication efficiency: Although CoeF-SFL substantially reduces synchronization frequency, Table 9 shows that CoeF-D transmits the same total communication volume as Vanilla SFL, while CoeF-J transmits more data. Therefore, it remains unclear whether the proposed methods provide meaningful end-to-end training speedups, particularly under bandwidth-constrained or low-latency network conditions. The lack of wall-clock comparisons across different network settings makes it difficult to assess the practical efficiency benefits over Vanilla SFL.

  2. Sensitivity to compensation parameters remains unclear: CoeF-D employs a fixed compensation factor of λ = 3000, substantially larger than the reference scale arising in Proposition 1. Although the authors acknowledge that λ is empirically chosen, the paper provides limited evidence about its sensitivity, tuning cost, or transferability across architectures and data distributions. Similarly, the effectiveness of CoeF-J with a single random projection deserves further investigation because the underlying theoretical argument assumes a more accurate sensitivity matrix. These issues raise questions about how reliably the methods can be deployed under configurations different from those evaluated.

Questions for authors
  1. Practical efficiency over Vanilla SFL: Table 9 indicates that CoeF-D transmits the same total communication volume as Vanilla SFL, while CoeF-J transmits more data. Although reducing synchronization frequency is beneficial, it is unclear whether these reductions translate into improved end-to-end training efficiency. Could the authors clarify the network conditions under which CoeF-SFL is expected to outperform Vanilla SFL in wall-clock training time? If available, a representative runtime comparison under different bandwidth and latency settings would help establish the practical efficiency benefits over Vanilla SFL. This clarification would materially affect my assessment of the method's practical significance.

  2. Theoretical justification of CoeF-J: The theoretical analysis assumes an exact sensitivity matrix and a global curvature bound, while the implementation uses a single random projection and an adaptive coefficient. Although these approximations are computationally practical, it is unclear whether the theoretical guarantees still apply. Could the authors clarify which guarantees remain valid? If available, an empirical analysis of how often the surrogate upper bound is violated under the practical approximations would help assess the applicability of the theoretical analysis.

  3. Sensitivity to compensation parameters: CoeF-D uses a fixed λ = 3000, while CoeF-J uses a single random projection (R = 1). Although these settings yield promising results, it is unclear whether the methods are sensitive to these choices. Could the authors clarify how performance varies with λ and R? If available, a limited sensitivity analysis would help establish the robustness and generalizability of the proposed methods.

11 comments
AuthorsOpenReview ↗

Response to Reviewer V8MQ (1/5)

Please also see our general response (Summary of Revisions).

[W.2] Sensitivity to compensation parameters: Experimental sensitivity to λ\lambda and RR

Thank you for this thoughtful comment. To answer this question, we conducted sensitivity experiments on the compensation factor λ\lambda of CoeF-D and the number of projections RR of CoeF-J, using ViT-Tiny/CIFAR-100 and DistilRoBERTa/20 Newsgroups under IID and non-IID splits. We swept five values of λ\lambda and three values of RR, each with three learning rates, and report the results in Table R3. Overall, both methods show low sensitivity to these parameters. For CoeF-D, once λ\lambda is sufficiently large (λ≥3000\lambda\ge3000), the accuracy changes by at most about 11 point over two orders of magnitude of λ\lambda, except under the most heterogeneous split (α=0.1\alpha=0.1), where an even larger λ\lambda improves the accuracy by about 2.52.5 points. For CoeF-J, increasing RR from 11 to 88 changes the accuracy by at most about 0.30.3 points in all settings, while the payload of the projections grows in proportion to RR, which supports the use of R=1R=1. On the vision task, a too small λ\lambda (3030 or 300300) fails to compensate the staleness even with learning-rate tuning, whereas the language task is insensitive to λ\lambda over the entire range; we detail this observation in Appendix F.1 of the revised manuscript, where we have added these results. The theoretical analysis of these two hyperparameters is given in our response to Q3.

[Table R3. Test accuracy (%) for the λ\lambda and RR sweeps. Entries use the main learning rate / the best of 10−4,5×10−4,10−3\\{10^{-4},5\times10^{-4},10^{-3}\\} ( ν=1\nu=1, T=100T=100). Main settings are in bold.]

ViT-TinyCIFAR-100DistilRoBERTa20 Newsgroups
IIDα=0.5\alpha{=}0.5α=0.1\alpha{=}0.1IIDα=0.5\alpha{=}0.5
λ=30\lambda=302.68 / 78.751.00 / 69.461.00 / 1.0069.32 / 69.3263.78 / 65.32
λ=300\lambda=3009.45 / 78.685.21 / 70.101.98 / 2.6569.61 / 69.6162.32 / 64.95
λ=3000\boldsymbol{\lambda=3000}84.68 / 84.6880.59 / 80.5969.73 / 69.7369.73 / 69.7367.02 / 67.02
λ=30000\lambda=3000084.38 / 84.3879.97 / 79.9772.19 / 72.1969.72 / 69.7268.12 / 68.12
λ=300000\lambda=30000084.34 / 84.3479.85 / 79.8572.02 / 72.0269.72 / 69.7267.78 / 67.78
R=1\boldsymbol{R=1}84.16 / 84.1679.60 / 79.6071.49 / 71.4969.94 / 69.9467.84 / 67.84
R=4R=484.38 / 84.3879.48 / 79.4871.49 / 71.4969.98 / 70.1068.06 / 68.06
R=8R=884.07 / 84.0779.53 / 79.5371.35 / 71.3570.09 / 70.1368.14 / 68.14
AuthorsOpenReview ↗

Response to Reviewer V8MQ (2/5)

[W.1 / Q.1] Practical efficiency over vanilla SFL: Wall-clock time under different network conditions

We thank the reviewer for this important point, and the reviewer is correct. The original manuscript discussed only the reduced communication frequency and did not show how it translates into lower latency. Since the latency benefit of multiple local updates over vanilla SFL had already been well established in AccSFL (Han et al., 2021), we did not address this aspect carefully enough. Following the reviewer's feedback, we model the time required by vanilla SFL and CoeF-SFL to perform the same updates (Modeling the round time of each method), and compare them under realistic network conditions to show the practical significance (Wall-clock time under real 5G networks).

Why fewer exchanges reduce latency. Before presenting the results, we explain why increasing local updates and reducing the exchange frequency lowers the latency. In vanilla SFL, training on each mini-batch requires the client-side forward pass, the server-side forward and backward passes, and the client-side backward pass to be executed sequentially, so that one side always waits for the other. In contrast, in CoeF-SFL, as in auxiliary-network-based methods, the forward and backward passes of the client and those of the server are executed concurrently after the initial exchange, which yields the effect of pipelining. Furthermore, another dominant factor in reducing the latency is the reduction in the communication load required to perform the same number of updates. The result of Table 9 (in original manuscript) assumes one local epoch, in which the communication load of CoeF-SFL is comparable to that of vanilla SFL. However, as the number of local updates increases, the relative communication load of CoeF-SFL decreases rapidly, since the activations and gradients of the training data are exchanged once and no further communication is needed during the subsequent local training.

Modeling the round time of each method. For each method, we model the time required to perform the updates of ν\nu local epochs in one round, using the notation in Table R1, which lists the meaning, value, and source of each symbol.

Vanilla SFL.

Tvanilla=ν(Ac+As+Bu+Bd+nBLfix)+C+D.T_{\mathrm{vanilla}}=\nu\bigl(A_c+A_s+B_u+B_d+n_BL_{\mathrm{fix}}\bigr)+C+D.

CoeF-D/CoeF-J.

T_{\mathrm{CoeF\text{-}D}} &=E_c+B_u+E_s+\max\\!\left\\{B_d+\nu A_{\mathrm{CoeF}}+C+D,\ \nu A_{s,\mathrm{CoeF}}\right\\}+L_{\mathrm{fix}},\\\\ T_{\mathrm{CoeF\text{-}J}} &=E_c+B_u+E_{s,J}+\max\\!\left\\{B_d+B_J+\nu A_{\mathrm{CoeF}}+C+D,\ \nu A_{s,\mathrm{CoeF}}\right\\}+L_{\mathrm{fix}}. \end{aligned}$$ **[Table R1. Notation of the round-time model. Computation times are measured on ViT-Tiny/CIFAR-100 on a single RTX 4090 GPU; for the initial preparation and local training of CoeF-SFL, they are measured with full-batch forward and backward passes. Communication times are the payloads in Table 14 of the revised manuscript divided by the throughputs measured by Ghoshal et al. (2025).]** | | | | | | :--- | :--- | :--- | :--- | | **Symbol** | **Meaning** | **Value** | **Source** | | $n_B,\nu$ | Batches per epoch and epochs per round | 32; 1 or 2 | Setting | | $A_c,A_s$ | Vanilla client and server training times per epoch, summed over the $n_B$ mini-batches | 0.3846 s, 0.7052 s | Measured | | $A_{s,\mathrm{CoeF}}$ | CoeF server training time per epoch | 0.4884 s | Measured | | $A_{\mathrm{CoeF}}$ | CoeF client training time per epoch, including compensation | 0.2419 s (D), 0.2701 s (J) | Measured | | $E_c$ | Initial client activation computation time | 0.0338 s (D), 0.0337 s (J) | Measured | | $E_s,E_{s,J}$ | Initial server feedback preparation times for CoeF-D and CoeF-J | 0.1344 s, 0.2636 s | Measured | | $D$ | Aggregation time | 0.02 s | Measured | | $P_u,P_d$ | Activation upload and feedback download payloads per client and epoch | 144.29 MiB each | Table 14 | | $r_u,r_d$ | Uplink and downlink throughputs | Table R2 | Ghoshal et al. (2025) | | $B_u,B_d$ | Activation upload and feedback download times per epoch | $P_u/r_u$, $P_d/r_d$ | – | | $B_J$ | Additional CoeF-J downlink transfer time for the probe payload | $B_d$ ($R=1$) | Table 14 | | $C$ | Client model upload time | 0.21 MiB\$/r_u+L_{\mathrm{fix}}/2\$ | – | | $L_{\mathrm{fix}}$ | Fixed delay per exchange, including RTT and serialization, RPC, and memory-copy overheads | 0.3 s | Ghoshal et al. (2025) |
AuthorsOpenReview ↗

Response to Reviewer V8MQ (3/5)

[W.1 / Q.1] Practical efficiency over vanilla SFL: Wall-clock time under different network conditions (continued)

Wall-clock time under real 5G networks. We evaluate the above round-time models with the uplink and downlink throughputs of two commercial 5G networks measured by Ghoshal et al. (2025), taking their 75th and 25th percentiles as the abundant and constrained conditions, respectively. Table R2 reports the results with one and two local epochs. With one local epoch, the communication load of CoeF-SFL is the same as that of vanilla SFL, so the latency reduction comes only from the efficiency of computation, i.e., pipelining and consecutive local updates, and from the fewer fixed delays. From two local epochs, the reduced communication load itself additionally lowers the latency, since vanilla SFL repeats its exchanges in every epoch, whereas CoeF-SFL exchanges once per round.

[Table R2. Estimated round time (s) per client of vanilla SFL, CoeF-D, and CoeF-J with one and two local epochs (ν=1,2\nu=1,2).]

ν=1\nu=1ν=2\nu=2
NetworkConditionUL/DL (Mbps)VanillaCoeF-DCoeF-JVanillaCoeF-DCoeF-J
AT&T 5G-mmWaveAbundant105/1,39523.2713.2914.3246.3613.5314.59
AT&T 5G-mmWaveConstrained7/87197.94187.96202.03395.45188.20202.30
T-Mobile 5G-midAbundant72/50230.1120.1322.7060.0220.3722.97
T-Mobile 5G-midConstrained16/14395.0885.1093.73189.8985.3594.00

Revision. We have added this analysis to Appendix G.3 of the revised manuscript.

References

Dong-Jun Han, Hasnain Irshad Bhatti, Jungmoon Lee, and Jaekyun Moon. Accelerating federated learning with split learning on locally generated losses. In ICML 2021 Workshop on Federated Learning for User Privacy and Data Confidentiality, 2021.

Moinak Ghoshal, Omar Basit, Imran Khan, Z. Jonny Kong, Sizhe Wang, Yufei Feng, Phuc Dinh, Y. Charlie Hu, and Dimitrios Koutsonikolas. Replication: Performance of cellular networks on the wheels. In Proceedings of the ACM Internet Measurement Conference (IMC), pp. 381–396, 2025.

Reviewer_4GpSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript presents a communication-efficient approximation of the traditional split federated learning (SFL) algorithm proposed by Thapa et al. (2022). The main motivation is that local auxiliary-model idea used by state-of-the-art communication-efficient approaches in SFL, can have large errors since such methods approximate the loss function via smaller server-model surrogates to circumvent the need to wait for server feedback. Instead the authors propose to use stale smashed gradient updates from the server, and approximate the true gradient from these stale updates via approximate second-order updates at the client.

Critical strengths
  • The proposed SFL algorithm is simple and mainly uses only firstorder derivatives to approximate gradient updates, which is useful from a memory and compute efficiency perspective.

  • The experimental results are detailed and shows a clear improvement of the proposed method in comparison to auxiliary based SFL methods.

Critical weaknesses
  • Overall, the proposed methods lack a convergence analysis under clearly stated assumptions. This is a common practice in federated learning and split learning, including many of the auxiliary-based methods (e.g., Nair et al., 2025). It is hard to determine if the proposed method is generally convergent under standard assumptions or requires additional specific assumptions, or what the exact rate of convergence is.

  • There is a discrepancy between the exact form of the gradient in eq. (6), and how it gets approximated in (8) and (9). Specifically, eqs. (8) and (9) only approximate the Hessian at the cut-layer values s0,is_{0,i}, while B^i\widehat{B}_i is supposed to be an approximation to the whole integral in (6). Also, after (7), the text says that the difference between B^i\widehat{B}_i and Hˉi\bar{H}_i is a curvature mismatch, whereas, it is actually a mismatch of the line integral of the curvature on the path between and sis_i and its initial value. This detail makes the approximations seem heuristic.

  • It is not clear why the proposed method is a better approximation compared to auxiliary-based methods. In both settings, the server model is updated based on the true loss gradient, while the client model updates based on an approximation of the true gradient.

Questions for authors

Minor Questions

  • Isn't the Gradient reuse (w/ perfect comp.) the same as the true gradient returned by the server? If this is true, shouldn't Vanilla SFL match with Gradient resue (w/ perfect comp.) in Table 1?

  • The server-side update in eq. (5) is still stale compared to original SFL since client model changes between iterations?

  • Is the diag(⋅)diag(\cdot) in (8) a typo?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4GpS (1/3)

Please also see our general response (Summary of Revisions).

[W.1] Convergence analysis: Assumptions and convergence rate

We thank the reviewer for this suggestion. In the original manuscript, we focused our theoretical analysis on the compensation itself rather than on the guarantee of convergence. Reflecting the reviewer's comment, we have now conducted a convergence analysis of CoeF-SFL. Both CoeF-D and CoeF-J attain an O(1T)O\left(\frac{1}{\sqrt T}\right) bound on the average expected squared gradient norm of the joint client and server objective. In particular, we derive a general convergence bound that holds for any feedback error, and then obtain the rates of CoeF-D and CoeF-J by bounding the feedback error of each compensation. We have added the formal assumptions, the theorem, and its proof as Appendix E of the revised manuscript.

[W.2] Eqs. (6)–(9): Path-averaged curvature and its approximation at the transmitted smashed data

We thank the reviewer for this careful reading. The reviewer is correct: Hˉi\bar H_i in Eq. (6) of the original manuscript is the line integral of the Hessian along the segment from s0,is_{0,i} to sis_i, i.e., the curvature averaged along the displacement. Hence B^i−Hˉi\widehat B_i-\bar H_i in Eq. (7) of the original manuscript is a mismatch with respect to this averaged curvature, rather than the “curvature mismatch” at the transmitted smashed data that we intended.

This issue is clearly resolved by the first-order expansion at the transmitted smashed data instead of the line integral. Let H0,i:=∇s2L(s0,i;θs)H_{0,i}:=\nabla_s^2\mathcal L(s_{0,i};\theta^s) denote the Hessian at the transmitted smashed data. Then

gi=g0,i+H0,iδi+ρi,g_i=g_{0,i}+H_{0,i}\delta_i+\rho_i,

where ρi\rho_i is the second-order remainder. The compensated gradient therefore satisfies

g~i−gi=(B^i−H0,i)δi−ρi.\widetilde g_i-g_i=(\widehat B_i-H_{0,i})\delta_i-\rho_i .

With this form, the “curvature mismatch” refers exactly to B^i−H0,i\widehat B_i-H_{0,i}, i.e., the mismatch with the Hessian at the transmitted smashed data. In addition, the remainder satisfies ∥ρi∥≤LH2∥δi∥2\Vert \rho_i\Vert \le\tfrac{L_H}{2}\Vert \delta_i\Vert ^2 under the LHL_H-Lipschitz continuity in Assumption 1, so that the approximation is not heuristic. We have revised all related expressions accordingly, including Eqs. (7)–(8) in Section 3.

AuthorsOpenReview ↗

Response to Reviewer 4GpS (2/3)

[W.3] Comparison with auxiliary-network-based methods: Why CoeF-SFL is a better approximation

We thank the reviewer for this question. As the reviewer notes, the server-side model is updated in the same way in both settings, using the true loss gradient. The two approaches differ fundamentally, however, in the objective that the client-side model optimizes, as described below.

CoeF-SFL optimizes the end-to-end loss on the client. Apart from the approximation error of the compensation, CoeF-SFL aims to update the client as in Eq. (3),

\qquad J_{i-1}:=\frac{\partial s_{i-1}}{\partial\theta^{c}} .$$ By the chain rule, multiplying the activation-space gradient by the client Jacobian gives $$J_{i-1}^{\top}\nabla_s\mathcal L\big(s_{i-1};\theta^{s}\big) =\nabla_{\theta^{c}}\mathcal L\big(f_s(f_c(x;\theta_{i-1}^{c});\theta^{s})\big),$$ which is the gradient of the same end-to-end loss $\mathcal L$ that the server minimizes. Hence the client and the server optimize a common objective, as in vanilla SFL. The compensated gradient $g_{0,i}+\widehat B_i\delta_i$ is an approximation of exactly this gradient, and its error is explicitly bounded in Eq. (14); it vanishes when the displacement is zero and grows with $\lVert\delta_i\rVert$. Moreover, multiplying the compensated gradient by the client Jacobian shows that the local objective of CoeF-SFL is a Gauss–Newton-type quadratic model of the end-to-end loss with respect to the client-side model, and for CoeF-J, an upper bound of it that is minimized by majorization–minimization. Related discussion can be found in Section 4 and Appendix D (Proposition 5) of the revised manuscript and in our response to Reviewer 3tbx, Q2. **Auxiliary-network-based methods optimize a different objective on the client.** In contrast, auxiliary-network-based methods update the client as in Eq. (4), $$\theta_{i}^{c}=\theta_{i-1}^{c}-\eta\\,\nabla_{\theta^{c}}\mathcal L_{\mathrm{aux}}\big(s_{i-1};\theta_{i-1}^{c}\big),$$ where $\mathcal L_{\mathrm{aux}}$ is computed by an auxiliary classifier attached to the end of the client-side model. The client is therefore trained to minimize its own local loss, separately from the server-side model, and $\nabla_s\mathcal L_{\mathrm{aux}}$ generally differs from $\nabla_s\mathcal L$ even at the transmitted smashed data. This discrepancy is not an approximation error that shrinks with a smaller displacement; it is a difference in the objective itself. As discussed in the Introduction, such a local objective tends to over-compress the features at the cut layer and discard task-relevant information that the server-side layers would otherwise exploit (Wang et al., 2021), and this information cannot be recovered by the server even with additional capacity. We have added this discussion after Eqs. (3)–(4) in Section 3. **Empirical evidence.** Consistent with this distinction, Gradient reuse (w/ perfect comp.) in Table 1, which corresponds to Eq. (3) without approximation error, substantially outperforms AccSFL, and a gap remains even when the capacity of the auxiliary network is increased. **References** Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: An alternative to end-to-end training. In *International Conference on Learning Representations (ICLR)*, 2021. > **[Q.1] Table 1: Gradient reuse (w/ perfect comp.) versus vanilla SFL** Thank you for the question. Gradient reuse (w/ perfect comp.) does not use the true gradient of the server that is being updated, but the true gradient of the server **at the beginning of the round,** i.e., $\nabla_s\mathcal L(s_{i-1};\theta^{s})$ with the round-initial $\theta^{s}$ in Eq. (3), rather than $\nabla_s\mathcal L(s_{i-1};\theta_{i-1}^{s})$. Likewise, the server is updated with the smashed data received at the beginning of the round, as in Eq. (5). Hence, both the client and the server are trained during the round with the model on the other side fixed at its round-initial state. In vanilla SFL, this staleness does not arise, since the client and the server communicate at every batch. We have clarified this distinction in the Introduction. In other words, Gradient reuse (w/ perfect comp.) is an oracle of CoeF-SFL in which the compensation is perfect, so that the gradient computed with the round-initial server is recovered exactly. However, the staleness that comes from the two sides training separately over multiple batches still remains.
AuthorsOpenReview ↗

Response to Reviewer 4GpS (3/3)

[Q.2] Eq. (5): Staleness of the server-side update

Yes. As explained in Q1, the server-side update in Eq. (5) is still stale compared with vanilla SFL. However, this staleness also arises in auxiliary-network-based methods, and it differs substantially, both in its nature and in its impact, from the client-side staleness that CoeF-SFL addresses. The analysis below appears in the Introduction and in the paragraph “Staleness of the activation-space gradient” in Section 3.

Server-side staleness targets a previous client. The server-side staleness arises because the client is also updated while the server is trained with the smashed data s0=fc(x;θc)s_0=f_c(x;\theta^{c}) that the client sent. Since the server is trained against a client model that actually existed just before the update, this staleness does not cause a large performance degradation, as confirmed in Table 1.

Client-side staleness targets a server that never existed. The client-side staleness arises for a different reason. For an input xx, let g0=∇sL(s0;θs)g_0=\nabla_s\mathcal L(s_0;\theta^{s}) be the gradient received at the beginning of the round. After local updates, the client becomes θ^c\hat\theta^{c} and the smashed data become s=fc(x;θ^c)=s0+δs=f_c(x;\hat\theta^{c})=s_0+\delta. Reusing g0g_0 at ss is equivalent to using the true gradient of some server θ~s\tilde\theta^{s} satisfying ∇sL(s;θ~s)=g0\nabla_s\mathcal L(s;\tilde\theta^{s})=g_0. Expanding to first order in both arguments around (s0,θs)(s_0,\theta^{s}),

\qquad H_0:=\nabla_s^2\mathcal L(s_0;\theta^{s}),\quad M:=\nabla_{\theta^{s}}\nabla_s\mathcal L(s_0;\theta^{s}),$$ so, to first order, the implied server must satisfy $M(\tilde\theta^{s}-\theta^{s})=-H_0\delta$. Since $\lVert M(\tilde\theta^{s}-\theta^{s})\rVert\le\lVert M\rVert\lVert\tilde\theta^{s}-\theta^{s}\rVert$, this gives $$\lVert\tilde\theta^{s}-\theta^{s}\rVert\ge\frac{\lVert H_0\delta\rVert}{\lVert M\rVert}.$$ That is, reusing the same gradient amounts to targeting a server that deviates from the round-initial server $\theta^{s}$ in proportion to the client displacement. Unlike the server-side case, $\tilde\theta^{s}$ never existed during training, which can severely degrade the collaborative learning, as confirmed throughout our experiments, including Table 1. CoeF-SFL aims to remove this mismatch by compensating $H_0\delta$, so that the client, like the server, is trained against the round-initial model on the other side. We are particularly grateful for this question. We regard the client-side staleness as a central contribution of our work, and answering this question allowed us to explain analytically how it differs from the server-side staleness, and why it degrades performance far more severely. Based on this analysis, we have substantially revised the Introduction to better convey the motivation of CoeF-SFL. > **[Q.3] Eq. (8): The $\operatorname{diag}(\cdot)$ notation** Thank you for pointing this out. The $\operatorname{diag}(\cdot)$ in Eq. (8) of the original manuscript, Eq. (9) in the revised manuscript, is intended: CoeF-D uses the diagonal matrix $\widehat B_{\mathrm D,i}=\lambda\operatorname{diag}(g_{0,i}\odot g_{0,i})$, whose entries are the squared elements of the received gradient. The typo was instead in Eq. (9) of the original manuscript, Eq. (10) in the revised manuscript, whose left-hand side should be the outer product $g_{0,i}g_{0,i}^{\top}$ rather than $g_{0,i}\odot g_{0,i}$, i.e., $$g_{0,i}g_{0,i}^{\top}=J_{s,i}^{\top}\big(r_ir_i^{\top}\big)J_{s,i}.$$ The diagonal of this outer product is $\operatorname{diag}(g_{0,i}\odot g_{0,i})$ in Eq. (9). We have corrected Eq. (10) accordingly and clarified that its diagonal gives the matrix used in Eq. (9). Both changes appear in the CoeF-D paragraph of Section 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank all reviewers for their constructive feedback. The main revisions are listed below, and all changes in the revised manuscript are marked in blue.

  1. Client-side staleness vs. model drift (Introduction, Section 3; 4GpS, 3tbx)
  2. Unified view of the compensation as a Gauss–Newton-type model (Section 4, Appendix D; 3tbx)
  3. Convergence analysis (Appendix E; 4GpS)
  4. Analysis of the compensation parameters and the implemented CoeF-J (Appendix C, F.1; V8MQ, 3tbx)
  5. Violation of the loss upper bound in practice (Appendix F.2; V8MQ, 3tbx)
  6. Longer training and feedback diagnostics (Appendix F.3; 3tbx)
  7. Round time under real networks (Appendix G.3; V8MQ, 3tbx)
  8. Related work on gradient compensation and reuse, and clarifications of Eqs. (7)–(10) (Sections 2–3; 3tbx, 4GpS)

Note on numbering. Tables R1–R7 are newly added in our responses. The new Eq. (6) shifts the subsequent equation numbers by one. Propositions 1 and 2 keep their original numbers, and Propositions 3–5 are new.

Dear all,

We're the half-way through the discussion phase. The authors have provided their response to your comments. Some of you have started involving discussion, but all of us can take action now.

If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible, we need to keep the conversation going!

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.