- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates Split Federated Learning (SFL) under low communication frequency. The authors propose CoeF-SFL, which exchanges smashed data and gradients at the beginning of each round and employs curvature-based correction to support multiple local client updates, thereby reducing communication exchanges without introducing auxiliary networks. CoeF-D uses a diagonal gradient outer-product approximation, while CoeF-J employs a surrogate model based on the server-side Jacobian. The paper provides theoretical analysis under specific conditions and demonstrates performance improvements over gradient reuse without compensation, achieving results close to vanilla SFL across vision and language tasks.
Critical strengths
1.The paper addresses an important trade-off between communication exchanges and training quality in SFL. Leveraging server-side feedback to support client updates under low communication frequency is practically meaningful. 2.The comparisons between gradient reuse without compensation and the perfect-compensation oracle provide valuable motivation. CoeF-D and CoeF-J recover substantial performance degradation across multiple settings, demonstrating the potential of the proposed approach. 3.The experiments cover multiple tasks, model capacities, cut layers, and data distributions. The appendix further investigates auxiliary-network capacity, curvature approximations, and resource costs.
Critical weaknesses
W1: The methodological novelty relative to existing gradient compensation techniques is insufficiently established. The paper applies gradient compensation to activation staleness caused by multiple local updates in SFL. However, the discussion of closely related work remains insufficient, and the methodological novelty is not clearly distinguished. DC-ASGD (Zheng et al., 2017) already uses a first-order Taylor expansion and a diagonal Hessian approximation based on the gradient outer product to compensate for stale gradients, which closely resembles the core design of CoeF-D. CELU-VFL (Fu et al., 2022) supports multiple local updates by caching and reusing forward activations and backward derivatives while accounting for the resulting staleness errors. Furthermore, FedLite (Wang et al., 2022) employs an activation-space Taylor expansion and Hessian approximation to correct gradient mismatch in split learning. Although these methods differ in their training settings and sources of error, they have already explored gradient staleness compensation, activation and derivative reuse, and activation-space gradient correction, respectively, all of which are closely related to the proposed approach. Therefore, the authors should clarify what technical challenges arising from activation staleness under multiple local updates remain unresolved by existing methods, and what substantive new algorithmic designs or theoretical analyses are introduced by CoeF-D and CoeF-J. Simply emphasizing that this is the first work to study activation-space gradient staleness is insufficient to clearly establish the technical contributions beyond prior work.
W2: The theoretical analysis of CoeF-J does not adequately characterize its practical compensation mechanism. CoeF-J constructs a surrogate loss to guide client updates, and Proposition 2 establishes an upper-bound property of this surrogate with respect to the server-side loss under specific conditions. However, this result relies on the exact sensitivity matrix and a fixed curvature upper-bound coefficient, whereas the practical algorithm approximates the sensitivity matrix using a single random projection and employs sample-dependent compensation coefficients. In particular, the practical coefficients may be smaller than the curvature upper bound required by the theoretical analysis, so the established upper-bound property is not necessarily preserved. Although the paper acknowledges this discrepancy, it does not further analyze whether these approximations undermine the key properties of the surrogate model or how they affect the discrepancy between the compensated gradients and the true server-side gradients.
W3: The communication efficiency advantage is not sufficiently demonstrated. The paper emphasizes that CoeF-SFL improves efficiency by reducing communication exchanges between clients and the server. However, Table 9 shows that CoeF-D and CoeF-J transmit more data and incur additional server-side computation compared with auxiliary-network-based methods such as AccSFL and FSL-SAGE. Therefore, the current experiments mainly demonstrate that CoeF-SFL can maintain high model accuracy under low communication frequency, but do not sufficiently establish whether it provides better overall communication efficiency than existing methods.
Questions for authors
Q1. The total number of training rounds is fixed at T=100. As training progresses, the client-side model parameters and the resulting activation representations continue to evolve. However, the paper does not analyze how activation displacement and gradient staleness vary across different stages of training. Can CoeF-D and CoeF-J continue to compensate for gradient errors effectively over longer training horizons and maintain their performance advantages over gradient reuse without compensation?
Q2. The effectiveness of the compensation mechanisms is mainly evaluated through final test accuracy, making it unclear whether the performance improvements actually result from correcting stale gradients. In particular, the compensation terms in CoeF-D and CoeF-J may improve training stability by modifying gradient directions or suppressing activation displacement, without necessarily making the feedback gradients closer to the true gradients. Could the authors provide direct analyses of gradient errors or directional alignment to validate the actual effects of the compensation mechanisms and distinguish the contributions of gradient correction from those of optimization stabilization?
Q3. How do the authors precisely define "preserving the end-to-end objective"? The proposed method uses server-side feedback computed at the beginning of each round, while the server-side model continues to be updated during local training. Therefore, even completely eliminating the gradient staleness caused by activation displacement does not necessarily make the client-side gradients consistent with the current server-side model. For example, on Non-IID Food-101, the perfect-compensation oracle achieves 55.6% accuracy, substantially below the 63.9% achieved by vanilla SFL. How do the authors explain this gap?
8 comments
Response to Reviewer 3tbx (2/6)
[W.2] Theoretical analysis of CoeF-J: Guarantees under the practical compensation
We thank the reviewer for drawing attention to the distinction between the theoretical surrogate and the implemented correction. As the reviewer pointed out, the original analysis did not cover the compensation parameters used in practice, i.e., the compensation factor of CoeF-D, and the adaptive coefficient and the random projections of CoeF-J. Through this revision, we have added the corresponding analyses to Appendix C of the revised manuscript. For CoeF-D, we show that the diagonal outer product for the observed labels becomes smaller than its predictive-label expectation as the model fits its labels, so that the scale of matching the curvature grows with the confidence of the model, which explains our choice of (Proposition 3). For CoeF-J, we show that the loss upper bound of Proposition 2 remains valid with the adaptive coefficient, even though it can be smaller than , and holds in expectation over the random projections, with a violation whose expected amount decreases as (Proposition 4); in practice, violations are rare with (Appendix F.2). Moreover, since Eq. (14) holds for any , the discrepancy between the compensated gradient and the true server-side gradient remains bounded by the curvature mismatch $\Vert M_i\widehat P_i-H_{0,i}\Vert $ for every realization of the projections. Finally, we have added a convergence analysis of CoeF-SFL in Appendix E, which requires only a finite bound on this mismatch and therefore applies equally to the implemented CoeF-J with and the adaptive coefficient, which converges at the rate . For details, please refer to the propositions and appendices above, or to our responses to Reviewer V8MQ (Q2 on the guarantees of CoeF-J and Q3 on the sensitivity to and ).
[W.3] Communication efficiency: Comparison with auxiliary-network-based methods
We appreciate the reviewer's distinction between fewer exchanges and overall communication efficiency. We agree that CoeF-SFL transmits more data and uses more server computation than auxiliary-network-based methods. However, these savings of auxiliary-network-based methods come at the cost of the end-to-end objective and a large loss in accuracy, whereas the additional cost of CoeF-SFL is small in wall-clock time. We therefore believe that the two should be compared in terms of the trade-off between accuracy and communication, in which CoeF-SFL offers a substantially better trade-off.
Trade-off between accuracy and cost. Auxiliary-network-based methods save communication and server computation by training the client with a local objective instead of the end-to-end objective, which limits their accuracy; e.g., in Table 2, AccSFL achieves on Tiny-ImageNet, whereas CoeF-SFL achieves , close to vanilla SFL. Enlarging the auxiliary network improves their accuracy, but it shifts the cost to the memory and computation of the clients, the scarcest resources in SFL, and the accuracy still remains below that of CoeF-SFL (Table 15). In contrast, the additional cost of CoeF-SFL lies only in the downlink and on the server: CoeF-SFL uploads the same activations as AccSFL and FSL-SAGE (Table 9 of the original manuscript, Table 14 in the revised manuscript), and the additional download takes only a small fraction of the round time, since the downlink is typically much faster than the uplink in cellular networks. Compared with vanilla SFL, CoeF-SFL exchanges data once per round instead of once per mini-batch, which shortens the round time, e.g., from s to s on a commercial 5G mmWave network with two local epochs (Table R2 (Reviewer V8MQ, Q1)). The round-time model and the comparison with vanilla SFL are detailed in our response to Reviewer V8MQ (Q1) and Appendix G.3.
Response to Reviewer 3tbx (3/6)
[Q.1] Longer training horizons: Displacement and staleness across training stages
We thank the reviewer for this thoughtful suggestion. To answer this question, we extended training to rounds on ViT-Tiny/CIFAR-100 and examined the accuracy, activation displacement, and feedback alignment over the training horizon. Both CoeF-D and CoeF-J continue to compensate the staleness effectively and maintain their advantage over gradient reuse without compensation throughout the 200 rounds. We have added these results as Appendix F.3 of the revised manuscript.
Accuracy over training. As shown in Table R1, under the IID split, CoeF-D and CoeF-J reach and at round 200, respectively, whereas gradient reuse without compensation trains only with a ten times smaller learning rate, converges more slowly, and remains below CoeF-D throughout. Under , gradient reuse diverges and stays at accuracy, whereas CoeF-D and CoeF-J reach and , even exceeding the of the oracle with perfect compensation.
Displacement and feedback alignment. Table R2 reports the displacement $\Vert \delta\Vert $ of the activation and the cosine similarity of the received feedback and the applied feedback with , the gradient of the round-initial server at the current activation. With CoeF-SFL, the displacement does not grow as training progresses, e.g., it stays around under the IID split for both variants, and the received feedback remains well aligned with at every stage, so the staleness does not accumulate over longer horizons. Under , the displacement of gradient reuse grows explosively, whereas it remains bounded with CoeF-SFL, and CoeF-J keeps it the smallest () with . The low cosine similarity of the applied feedback of CoeF-J is expected: CoeF-J does not approximate but minimizes a majorizer of the end-to-end loss under the conditions of Proposition 5, and its correction penalizes the displacement in the directions to which the server output is sensitive, which keeps the received feedback accurate. CoeF-D instead prevents the feedback from deviating far from rather than raising the alignment at every step, as will be discussed in our response to Q2. Note that is evaluated along the trajectory of each method, so the cosine values indicate the staleness within each method and should not be compared across methods.
[Table R1. Test accuracy (%) on ViT-Tiny / CIFAR-100 (, ). Learning rate , except IID reuse ().]
| IID | ||||||||
| Round | 50 | 100 | 150 | 200 | 50 | 100 | 150 | 200 |
| Gradient reuse (with perfect comp.) | 84.19 | 84.63 | 84.62 | 84.50 | 61.79 | 65.96 | 67.85 | 66.54 |
| Gradient reuse (w/o comp.) | 69.82 | 78.66 | 80.67 | 82.07 | 1.00 | 1.00 | 1.00 | 1.00 |
| CoeF-D | 84.04 | 84.68 | 85.09 | 84.79 | 61.17 | 69.73 | 69.60 | 69.20 |
| CoeF-J | 83.54 | 84.16 | 84.20 | 83.90 | 65.49 | 71.49 | 71.30 | 72.38 |
Response to Reviewer 3tbx (1/6)
Please also see our general response (Summary of Revisions).
We thank the reviewer for pointing out these related studies, which helped us sharpen the contribution of this work. We clarify below that (i) we distinguish the client-side staleness from model drift as the main cause of the degradation, (ii) existing methods only mitigate it heuristically, and (iii) the novelty of our compensation lies mainly in CoeF-J.
The problem and its characterization. As the reviewer pointed out, CELU-VFL (Fu et al., 2022) addresses a similar problem in vertical federated learning: it also reuses cached derivatives over multiple local updates, but treats the resulting staleness as a generic approximation error. In contrast, we distinguish the client-side staleness caused by gradient reuse from model drift, and characterize it both theoretically and empirically (see our response to Reviewer 4GpS (Q2) and Eq. (6) in Section 3). In particular, to first order, the staleness of the gradient equals the curvature of the server-side loss at the transmitted smashed data applied to the displacement of the activation caused by the client's own updates, and hence it accumulates over the local updates. DC-ASGD (Zheng et al., 2017) instead addresses model drift, where the gradient is exact for an outdated model that actually existed, and FedLite (Wang et al., 2022) addresses a one-shot quantization error that does not accumulate. Under gradient reuse, however, the client is trained against a fictitious server-side model that never existed, which deviates further as the smashed data move. Accordingly, in Table 1, gradient reuse without compensation drops to 37.1 on MNLI-m, whereas gradient reuse with perfect compensation, subject only to model drift, performs close to vanilla SFL.
Differences in addressing the staleness. CELU-VFL only down-weights stale instances by cosine similarity and does not correct the stale gradient. Correction is difficult in SFL because the required curvature belongs to the server-side model, which the client does not hold, and no further exchange is allowed within the round. FedLite (Wang et al., 2022) uses a scaled identity, which carries no curvature information of the server-side loss. Hence, only the DC-ASGD-type surrogate, built solely from the received gradient, can be applied directly.
CoeF-D and CoeF-J. In terms of the compensation itself, CoeF-D adopts the surrogate of DC-ASGD as it is. However, by analyzing it theoretically in our setting, we obtain interpretations that DC-ASGD does not provide: the diagonal outer product of the received gradient agrees in expectation with the GGN diagonal with respect to the activation, it reduces the feedback error of gradient reuse if and only if λ<2λi⋆ (Proposition 1), and the scale of λ required for the observed labels grows as the model fits them, which explains our choice of λ=3000 (Proposition 3). In contrast, CoeF-J combines techniques that none of the above methods uses: the Jacobian of the server-side model, its random projection to reduce the communication, and the closed-form curvature bound of the loss with an adaptive coefficient, from which it builds an upper-bound surrogate of the loss instead of approximating the curvature (Proposition 2). We believe that this design has sufficient methodological novelty, and our theory supports it: under the conditions of Proposition 2, the client performs majorization–minimization of the end-to-end loss (Proposition 5), and with the adaptive coefficient and random projections, the bound holds in expectation with a violation decreasing as 1/R (Proposition 4).
We have added a paragraph titled “Gradient compensation and reuse” to Section 2, where we introduce DC-ASGD, CELU-VFL, and FedLite and explain how the client-side staleness and our compensation differ from theirs.
References
Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. Towards communication-efficient vertical federated learning training via cache-enabled local updates. Proceedings of the VLDB Endowment, 15(10):2111–2120, 2022.
Jianyu Wang, Hang Qi, Ankit Singh Rawat, Sashank Reddi, Sagar Waghmare, Felix X. Yu, and Gauri Joshi. FedLite: A scalable approach for federated learning on resource-constrained clients. arXiv preprint arXiv:2201.11865, 2022.
Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In International Conference on Machine Learning (ICML), pp. 4120–4129, 2017.