acceptodds
Under review as a conference paper at ICLR 2027

CommuteProp: Decoupled Training for Communication-Bound Split LLM Fine-Tuning

Abstract

Split learning has emerged as a promising paradigm for privacy-preserving LLM fine-tuning, yet its practical deployment is severely hindered by the sequential communication-computation bottleneck. In conventional synchronous pipelines, clients remain idle while waiting for server-side gradients, resulting in substantial training inefficiency. We propose CommuteProp, an asynchronous split-learning algorithm that decouples the training process into two concurrent phases: a cross-block forward-backward pass and an in-block weight update. By overlapping computation with communication, CommuteProp reduces the marginal cycle time from a sum of all stage latencies to the dominant computational bottleneck. We provide a rigorous asynchronous error and convergence analysis. Moreover, we derive an NS preconditioner method based on our analysis to further mitigate staleness-induced noise. Comprehensive experiments indicate that our algorithm achieves substantial throughput gains while maintaining accuracy comparable to synchronous methods; furthermore, it functions as a plug-and-play module that not only accommodates but actively enhances existing privacy enhancement methods.

Primary area:
optimization
Keywords:
Split Learning, Decoupled Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_G68bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CommuteProp for communication-bound split LLM fine-tuning. It separates cross-block computation from local parameter updates and overlaps them across batches without storing historical weights. A blockwise preconditioner is introduced to address staleness. The evaluation covers NLP tasks, client–server deployment, and integration with DualGuard.

Critical strengths
  1. The paper addresses a practical bottleneck in split learning. Improving communication–computation overlap without retaining multiple weight versions is a useful goal.

  2. The evaluation includes real two-GPU measurements and a PipeDream-2BW comparison under a fixed time budget, providing evidence beyond simulation.

  3. The DualGuard experiments demonstrate compatibility with an existing defense while largely preserving task performance.

Critical weaknesses
  1. The proof of Lemma 4.3 does not account for the cached backward signal’s dependence on the local block’s old output. Using current local weights therefore does not justify eliminating the local staleness term.

  2. The novelty over Zero Bubble Pipeline Parallelism’s computation splitting and Shampoo-style preconditioning needs to be stated more clearly.

  3. The Nesterov Async settings differ from the cited work, and it is unclear whether its stage-dependent adjustments were retained. This weakens confidence in the baseline comparison.

  4. The main equal-time comparison excludes optimizer costs, including NS preconditioning. Actual staleness and peak client memory are also needed to assess the efficiency trade-off.

  5. Table 11 reports element-wise Hessian statistics, whereas the theory requires operator-norm bounds. These measurements do not directly validate the assumption.

  6. The experiments do not isolate whether NS specifically mitigates staleness or improves optimization more generally. A synchronous NS baseline would help distinguish these effects.

Questions for authors
  1. Could you justify the mixed-state gradient equivalence in Lemma 4.3, given that the cached backward signal depends on the local block’s old output?

  2. What distinguishes CommuteProp’s decomposition and preconditioner from Zero Bubble Pipeline Parallelism and Shampoo, respectively?

  3. Which momentum and learning-rate adjustments were used for Nesterov Async? Were the stage-dependent settings of the original no-weight-stashing variant retained?

  4. Do the deployment timings include optimizer and NS preconditioning costs? Please also clarify the queue depths and actual parameter staleness in these experiments.

  5. How do the element-wise Hessian statistics in Table 11 support the operator-norm assumption used in the analysis?

  6. Do the existing results distinguish NS’s general optimization benefits from its effect on staleness-induced error?

5 comments
AuthorsOpenReview ↗

Response to Reviewer G68b

Response to W1 and Q1:

Thanks for your question. As shown in Eq. (5), our method computes the surrogate gradient through a local forward-backward pass. We have further clarified this process in the revised manuscript. Specifically, in Phase 2, each block independently performs a local forward-backward pass using the cached boundary activation hl(t−τ)\boldsymbol{h}_{l}^{(t-\tau)}, backward signal gl+1(t−τ)\boldsymbol{g}_{l+1}^{(t-\tau)}, and current local parameters θl(t)\boldsymbol{\theta}_{l}^{(t)}, followed by the parameter update. This can be interpreted as a mixed-version parameter state θ~=(θ<l(t−τ),θl(t),θ>l(t−τ))\widetilde{\boldsymbol{\theta}} = (\boldsymbol{\theta}_{\lt l}^{(t-\tau)}, \boldsymbol{\theta}_{l}^{(t)}, \boldsymbol{\theta}_{\gt l}^{(t-\tau)}).

Response to W2 and Q2:

Thanks for your question, Zero Bubble Pipeline Parallelism is a synchronous algorithm. Thus, the gradient of zero bubble parallelism’s splitting method is the same as the classical synchronous algorithm without any asynchronous errors. However, commuteprop considers the asynchronous scenario, and decouple the forward backward pass into two phases. The two methods may share some similar design philosophy, but they are two different kinds of algorithms.

As for Shampoo-style preconditioning, the NS preconditioning can be regarded as a shampoo-style preconditioning, and we cite the reference paper in line 376. Our contribution is not the introduction of inverse-root preconditioning itself, but its application to mitigating asynchronous cross-block errors. Our focus is to investigate whether NS preconditioning can improve optimization under asynchronous execution, rather than to study its contribution to synchronous algorithms. This is the motivation for introducing NS into CommuteProp.

Finally, we emphasize that our work focuses on split-learning scenarios. Zero Bubble Pipeline Parallelism and Shampoo-style preconditioning are developed under synchronous training settings. Sharing similar design principles does not, by itself, imply a lack of novelty. For example, Nesterov-based asynchronous training introduces a different learning schedule, but this does not preclude other algorithms from making novel contributions to asynchronous optimization.

Response to W3:

Thanks for your question, The utility comparison between Nesterov Async and Zero-shot indicates that Nesterov Async works on most datasets. As for the unexpected performance of Nesterov Async on SciERC dataset, we believe that the hyperparemeter contributes to the outlier. Please check the Figure 4c) and 4d), where NesterovAsync shares similar results as CommuteProp(Muon).

Moreover, avoiding weight stashing is an important motivation for memory-constrained split learning. Therefore, we choose synchronous training as a strong reference for training quality and NesterovAsync without weight stashing as our primary asynchronous baseline. Generally, synchronous training is expected to achieve better training quality due to the absence of stale updates.

Response to W4:

Thanks for your question, we have already provided the memory comparison in the revised version, specifically, we have:

MethodClient Peak Memory (GB)Server Peak Memory (GB)
Sync SL and NesterovAsync5.011.8
PipeDream-2BW (τ=4)6.017.0
CommuteProp (τ=4)5.018.2
CommuteProp (NS) (τ=4)5.420.1
CommuteProp (τ=8)5.024.7
CommuteProp (NS) (τ=8)5.426.5

The results show that CommuteProp achieves the same client-side peak memory consumption as synchronous SL and NesterovAsync, while PipeDream-2BW incurs higher memory overhead. Moreover, NS preconditioning introduces only a small additional memory cost.

The result is also updated in the revised version.

AuthorsOpenReview ↗

Response to Reviewer G68b (Cont.)

Response to W5 and Q5:

Answer: The results show that cross-block interactions are substantially weaker than the dominant within-block terms, supporting the assumption used in our analysis. This observation is also consistent with prior studies on the approximately block-diagonal Hessian structure of large neural networks [1–3].

Following the empirical protocols in [3,4], we use element-wise Hessian statistics to support the structural assumption rather than directly estimating the block operator norm, whose computation is prohibitive at the billion-parameter scale. Moreover, we would like to emphasize that our convergence and performance results are obtained from full-training runs and are independent of this estimate.

[1] Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. “Why Transformers Need Adam: A Hessian Perspective.” Advances in Neural Information Processing Systems, 37:131786–131823, 2024.

[2]Ivan Ilin. “Hessian of Perplexity for Large Language Models by PyTorch Autograd (Open Source).” arXiv preprint arXiv:2504.04520, 2025.

[3] Zhaorui Dong, Yushun Zhang, Jianfeng Yao, and Ruoyu Sun. “Towards Quantifying the Hessian Structure of Neural Networks.” In OPT 2025: Optimization for Machine Learning, 2025.

[4] Tianyue H. Zhang et al. “Understanding Adam Requires Better Rotation-Dependent Assumptions.” Advances in Neural Information Processing Systems, 38:115111–115152, 2025.

Response to W6andQ6W6 and Q6: Thanks for your question, we provide the synchronous adamw with synchronous, the result indicates that NS precondition does not improve model’s utility.

MethodEmotion (Acc)AgNews (Acc)Samsum (ROUGE)Samsum (METEOR)SciERC (F1)Spider (EM)
Sync (NS)92.0394.5736.2139.5173.2579.20
Sync91.1594.6436.6139.7272.8981.40

Response to Q3:

We use the default momentum factor 0.9 and Nesterov correction as mentioned in Appendix line 731, and the stage-dependent setting of the original no-weight-stashing variant was therefore retained.

Response to Q4:

Yes, we consider the NS preconditioning cost in our experiments (Please refer Table3 in line 758). The queue depth depends on the staleness tau. Larger tau indicates larger queue depths.

AuthorsOpenReview ↗

[Author Follow-up] Request for Feedback on Rebuttal Responses

Dear Reviewer

Thank you again for your time and thoughtful feedback. As the discussion period approaches its close, we would greatly appreciate any thoughts on our responses and the additional evidence provided.

In particular, we would be grateful to know whether any of your concerns remain unresolved, and we would be happy to provide further clarification. If our responses have addressed your main concerns, we would also appreciate your reconsideration of the overall assessment.

Thank you for your time and consideration.

Best regards,
Authors

Reviewer_ruWtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They propose CommuteProp, an asynchronous framework that splits LLM finetuning into overlapping communication and computation. The work uses cross-block forward and backward propogation, with a Newton-Schulz preconditioner that mitigates gradient staleness. They then provide a convergence gaurantee of such split scheme and provide experimental results on how this works against other synchronous and asynchrnous training methods under pre-defined training budget.

Critical strengths

The paper addresses a practical bottleneck in split LLM fine-tuning: idle computation while waiting for communication. Its separation of cross-block propagation from local parameter updates provides a promising way to overlap those operations, with experiments demonstrating throughput improvements under several network settings.

Critical weaknesses
  1. The main evaluation emphasizes outperforming NesterovAsync and preserving synchronous training quality, but these establish different claims. Matching synchronous quality demonstrates tolerance to staleness; superiority over competitive asynchronous pipelines requires a separate comparison. PipeDream-2BW appears only in Appendix Table 5, where the advantage is mixed. Bringing this comparison into the main evaluation would better establish when and why CommuteProp should be preferred.

  2. The authors should clarify baseline implementation and tuning. NesterovAsync’s severe degradation for example, 7.73 versus synchronous AdamW’s 72.89 on SciERC—warrants additional discussion of implementation validation and hyperparameter tuning. Appendix B.2 also describes synchronous Muon using β1,β2,ϵ\beta_1,\beta_2,\epsilon, while CommuteProp(Muon) uses momentum 0.95 and Nesterov acceleration. It is unclear whether this is a reporting error or an actual configuration difference. The authors should clarify the implementations and establish that competing methods receive comparable tuning effort.

  3. Avoiding weight stashing is an important motivation for memory-constrained split learning, but CommuteProp also stores queued activations and gradients, and its NS variant introduces additional optimizer state. Reporting peak client and server memory alongside throughput and model quality, across queue depths and against PipeDream-2BW, would substantiate the practical advantage. Without these measurements, the benefit of avoiding weight stashing remains insufficiently quantified.

Questions for authors

Please address the three weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ruWt

Response to W1:

Thanks for your question, as quested in your weakness 3, avoiding weight stashing is an important motivation for memory-constrained split learning. Therefore, we choose synchronous training as a strong reference for training quality and NesterovAsync without weight stashing as our primary asynchronous baseline. Generally, synchronous training is expected to achieve better training quality due to the absence of stale updates.

Moreover, existing asynchronous pipelines employ different weight-stashing strategies, resulting in different memory requirements and sensitivity to communication latency. Determining which pipeline best suits memory-constrained, high-latency split learning is a separate research question. Thus, we choose the PipeDream-2BW, as an example of asynchronous algorithm, in the Appendix, and choose NesterovAsync without weight stashing as baseline in the experiment part.

Response to W2:

Thanks for your suggestion, we provide the wrong hyper-parameter in Appendix B.2, and we already fixed the typo in our revised version.

For Nesterov async baseline, it provides comparable improvements in most tasks. Regarding the SciERC outlier for NesterovAsync, we believe it is likely due to hyperparameter sensitivity. As shown in Figures 4(c) and 4(d), NesterovAsync achieves results comparable to CommuteProp (Muon).

In our experiments, we use synchronous training configuration as the reference and intentionally avoid additional hyperparameter tuning for asynchronous methods. This reflects our practical expectation that hyperparameters tuned for synchronous training should transfer to asynchronous training without requiring separate tuning.

Response to W3:

Thanks for your question, we have already provided the memory comparison in the revised version, specifically, we have:

MethodClient Peak Memory (GB)Server Peak Memory (GB)
Sync SL and NesterovAsync5.011.8
PipeDream-2BW (τ=4)6.017.0
CommuteProp (τ=4)5.018.2
CommuteProp (NS) (τ=4)5.420.1
CommuteProp (τ=8)5.024.7
CommuteProp (NS) (τ=8)5.426.5

The results show that CommuteProp achieves the same client-side peak memory consumption as synchronous SL and NesterovAsync, while PipeDream-2BW incurs higher memory overhead. Moreover, NS preconditioning introduces only a small additional memory cost.

The result is also updated in the revised version.

AuthorsOpenReview ↗

[Author Follow-up] Request for Feedback on Rebuttal Responses

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period approaches its close, we would greatly appreciate any thoughts on our responses and the additional evidence provided.

In particular, we would be grateful to know whether any of your concerns remain unresolved, and we would be happy to provide further clarification. If our responses have addressed your main concerns, we would also appreciate your reconsideration of the overall assessment.

Thank you for your time and consideration.

Best regards,
Authors

Thank you for clarifying the typo and providing peak-memory measurements. These measurements substantially address my request. The explanation regarding asynchronous pipelines also makes sense. I have no more concerns and will therefore raise my score to a 3.

Reviewer_CHuxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CommuteProp, an asynchronous split-learning framework for communication-constrained LLM fine-tuning. It decouples training into two phases: cross-block forward/backward propagation and in-block parameter updates. By executing these phases asynchronously, the method aims to overlap communication and computation without storing historical model parameters. The paper introduces a queue-based dispatcher, derives an asynchronous gradient-error bound and a convergence result, and proposes a NS preconditioner to mitigate staleness. Experiments using Qwen2.5-1.5B-Instruct and Llama-3.2-3B-Instruct examine training utility, throughput, scheduling configurations, and integration with the DualGuard privacy defense.

Critical strengths
  1. Practical motivation and system-oriented problem formulation.
  2. An attempt to connect asynchronous system design with optimization analysis.
  3. Empirical evaluation across multiple settings.
Critical weaknesses
  1. The novelty over existing asynchronous pipeline training approaches is insufficiently established. PipeMare (Yang et al., 2021), Nesterov-based asynchronous pipeline training (Ajanthan et al., 2025), and PipeDream have already explored asynchronous execution, delayed updates, and pipeline scheduling. Although CommuteProp scales their method to larger models, the paper does not fundamentally advance over existing approaches. (Q1)

  2. The asynchronous convergence guarantees rely on insufficiently justified assumptions about stale gradients and block updates. Lemma 4.3 assumes that the surrogate gradient computed from delayed activations and backward signals can be represented as the true gradient evaluated at a mixed parameter state containing fresh local weights and stale external weights. However, in an asynchronous pipeline, these signals may originate from different parameter versions and execution times (Q2-3)

  3. Section 4.3 argues that applying a preconditioner approximating the inverse square root of the within-block Hessian can reduce cross-block error propagation. However, Eq. (13) only characterizes the error in terms of preconditioned update directions. It does not establish that the proposed preconditioner necessarily decreases the bound. (Q4-5)

  4. Fairness of efficiency measurements. Although Appendix B.3 claims that optimizer update, gradient zeroing, and learning-rate scheduling result in nearly equivalent overhead across methods, CommuteProp introduces additional forward computation, asynchronous task management, and potentially expensive NS preconditioning. A detailed breakdown of computation, communication, queue waiting, optimizer overhead, and memory consumption is needed. (Q6)

  5. Fairness and statistical reliability of experimental comparisons. The Nesterov Async baseline performs substantially worse than the other methods on several tasks. The equal-time comparisons also show different synchronous baseline accuracies across columns corresponding to different asynchronous staleness settings (Table 5), without explaining the source of this variation. (Q7)

  6. Validity of the privacy-enhancement claim. The paper claims that CommuteProp also improves privacy protection. However, the evidence is limited to integration with DualGuard and a small set of reconstruction attacks. The proposed mechanism also provides no formal privacy guarantee, and the causal relationship between staleness and attack resistance is not isolated through controlled experiments. (Q8)

  7. A bunch of mathematical notations and definitions are not explained or ambiguous. For example, the intermediate activations and their boundary conditions (Eq (2)), the parameter index ranges (Eq (2)-(3)), the dimensions and Jacobian conventions underlying the surrogate gradients (Eqs (4)-(5)), and the micro-batch and node-count variables in the pipeline efficiency analysis (Eq (6)). More seriously, the manuscript inconsistently uses the same loss notation for the original task objective, the auxiliary penalized objective, and block-restricted objectives in the convergence analysis (Eqs (8)-(12)). These ambiguities complicate verification of the derivations and undermine the clarity and reproducibility of the proposed method.

  8. The manuscript needs substantial proofreading and editorial revision before publication. Several grammatical and formatting errors include but are not limited to:

  • on page 1, "bulk part of parameters include in" -> the bulk of ... included in
  • on page 1, "sever communication overhead" -> severe
  • on page 2, "we provide the ablation study, further shows the" -> we provide an ..., which further shows
  • on page 4, "Data Batchs" -> Batches
  • on page 5, "Figure 3a demonstrate the" -> demonstrates
  • on page 8, "AdamW and Muon in Table 1 reports" -> report
  • on page 8, "Result on other base model are" -> Results
  • on page 9, "More experiment result are deferred to" -> results
  • on page 14, "Our experiment use one single" -> experiments use
  • on page 18, "The results, reported in Table 10, — consistently demonstrates" -> demonstrate
Questions for authors
  1. What are the essential differences between CommuteProp and existing asynchronous pipeline training methods? Beyond combining delayed updates with computation-communication overlap, what new optimization or scheduling principle does CommuteProp introduce? Could the authors provide controlled ablations that independently isolate the contributions of two-phase decoupling, critical-path scheduling, and NS preconditioning under matched memory and communication constraints?

  2. How can the stale activations and backward signals used in Eq. (5) be rigorously represented as a gradient evaluated at one consistent mixed parameter state? How does the queue-length limit ensure the assumed maximum parameter-update delay under asynchronous execution?

  3. How does the proof of Theorem 4.4 account for the fact that different blocks are updated at different steps in the virtual serialized sequence? Could the authors clarify the block-update schedule and extend the convergence analysis to the actual AdamW, Muon, and preconditioned optimization rules used in the experiments?

  4. Under what assumptions does the proposed NS preconditioner reduce the asynchronous error bound in Eq. (13)? Could the authors provide direct measurements of asynchronous gradient error before and after preconditioning, together with controlled comparisons against AdamW, Muon, and representative existing preconditioning methods such as SOAP?

  5. Could the authors clarify how the element-wise Hessian measurements in Table 11 support the structural assumption?

  6. Could the authors report end-to-end training throughput, GPU utilization, peak memory, optimizer and NS-preconditioner costs, queue waiting time, and actual transmitted bytes under matched hardware and training workloads? Could the authors also evaluate sustained performance across different bandwidths, client/server computation ratios, and queue-depth limits?

  7. Were all baselines independently tuned under comparable optimization and staleness budgets? Why does Nesterov Async exhibit such substantial accuracy degradation on several tasks? Could the authors report multiple independent runs with confidence intervals, provide matched hyperparameter searches, and explain why the synchronous baseline results in Table 5 vary across columns corresponding to different values of the asynchronous staleness parameter?

  8. Under what conditions does asynchronous staleness improve resistance to reconstruction attacks, rather than merely perturbing the intermediate representations? How do the authors explain cases in Table 8 where CommuteProp (NS) performs worse than synchronous DualGuard? Could the authors conduct controlled experiments varying staleness while holding the defense mechanism, optimization budget, and model utility fixed, and evaluate additional attacks and multiple random seeds?

6 comments
AuthorsOpenReview ↗

Response to Reviewer CHux

Thanks for your question, we would like to answer your question one by one

Response to W1

Thanks for your question, asynchronous execution, delayed updates, and pipeline scheduling are established ideas, but sharing these board concepts does not necessarily mean that the methods make the same technical contribution, especially these concepts are broadly used in asynchronous optimization algorithms[1,2,3]. Moreover, CommuteProp employs a fundamentally different mechanism from PipeMare, Nesterov-based asynchronous training, and PipeDream. As detailed in Section 3.1, CommuteProp adopts a two-phase asynchronous algorithm. In Phase 1, cross-block forward and backward computations are performed to obtain and cache the boundary activations and backward signals. In Phase 2, each block independently recomputes its local forward and backward passes using the cached boundary activation hl(t−τ)\boldsymbol{h}l^{(t-\tau)}, backward signal gl+1(t−τ)\boldsymbol{g}{l+1}^{(t-\tau)}, and current local parameters θl(t)\boldsymbol{\theta}_l^{(t)}, followed by the parameter update in Eq. (5).

[1] Bertsekas D, Tsitsiklis J. Parallel and distributed computation: numerical methods[M]. Athena Scientific, 2015.

[2] Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C]//Proceedings of the 27th ACM symposium on operating systems principles. 2019: 1-15.

[3] Tsitsiklis J, Bertsekas D, Athans M. Distributed asynchronous deterministic and stochastic gradient optimization algorithms[J]. IEEE transactions on automatic control, 1986, 31(9): 803-812.

Response to W2:

Thanks for your question. Our method uses a mixed-version parameter state for surrogate gradient computation in Phase 2. As shown in Eq. (5), the surrogate gradient is computed through a local forward-backward pass. We provide further clarification in the revised version. Specifically, in Phase 2, each block independently performs a local forward-backward pass using the cached boundary activation hl(t−τ)\boldsymbol{h}_l^{(t-\tau)}, backward signal gl+1(t−τ)\boldsymbol{g}_{l+1}^{(t-\tau)}, and current local parameters θl(t)\boldsymbol{\theta}_l^{(t)}, followed by the parameter update in Eq. (5). This can be interpreted as a mixed-version parameter state θ~=(θ<l(t−τ),θl(t),θ>l(t−τ))\widetilde{\boldsymbol{\theta}} = (\boldsymbol{\theta}_{\lt l}^{(t-\tau)}, \boldsymbol{\theta}_l^{(t)}, \boldsymbol{\theta}_{\gt l}^{(t-\tau)}).

Response to W3:

Our paper does not claim a theoretical guarantee for decreasing the bound in Equation(13). Please refer to line 369 and 375 in our paper. The preconditioner is designed for AdamW. In standard AdamW, the original precondition serves as a purely diagonal approximation of the local inverse covariance. As shown in Eq. (13), the asynchronous error is governed by the interaction between the cross-block Hessian and the local update direction. Our NS preconditioner aims to reduce this error by whitening the update direction, assuming the estimated local curvature captures the directions responsible for cross-block error amplification. We do not claim that preconditioning universally reduces this bound for arbitrary curvature structures. Moreover, as stated in Lines 377 and 379, NS preconditioning is an empirical method motivated by the theoretical insights from Lemma 4.3 and Theorem 4.4. We do not claim a theoretical bound that rigorously establishes its reduction of asynchronous error.

Response to W4:

In Appendix B.3, we provide the computation and communication time analysis for our algorithm under the worst case scenario. We also provide the peak memory consumption in the updated version. However, we cannot provide a detailed version of queue waiting time, as it depends on many practical factors such as bandwidth. Thus, we also evaluate all methods under the same 15-minute wall-clock budget at Appendix B.4. For detailed information, please check response to Q6.

Response to W5:

For Nesterov async baseline, it provides comparable improvements in most tasks. For the outlier claimed in Table1 SciERC, please refer to Figure4. Figure 4 shows that Nesterov async has comparable performance under different configurations (Figure 1c, 1d). Moreover, our paper also provides the synchronous baseline, the result indicates the CommuteProp provides comparable performance with synchronous algorithms.

For equal time comparison, Table 5 shows the performance under a fixed time budget. Under different latencies (τ=4\tau=4 and τ=16\tau=16), the baseline should differ, as larger latency indicates smaller training iterations for synchronous algorithms.

AuthorsOpenReview ↗

Response to Reviewer CHux (Cont.)

Response to W6 and Q8

CommuteProp is not designed to provide an independent privacy guarantee. Since CommuteProp stores intermediate activations on the server side, our goal is to use it as a plug-and-play optimization module alongside existing privacy-preserving methods without compromising their effectiveness. Accordingly, our experiments focus on evaluating its compatibility with these methods rather than its robustness against newly designed adaptive attacks. We have clarified this part in the revised version.

Response to W7:

Thanks for your suggestion. The parameter index ranges (Eq (2)-(3)) is denoted in line 122, the intermediate activations definition is based on equation (2). The surrogate gradient is denoted in line139, where we do not intend to involve Jacobian. The node count is denoted in line 254-255. The loss notation difference is because we would like to focus the updated parameter, which is explained on the first difference happens(line 306~lin307)

Response to W8:

Thanks for your suggestion, we have fixed all typos in the revised version

Response to Q1:

Two-phase decoupling and critical-path scheduling is the necessity component of our algorithm and the key contribution of our algorithm. CommuteProp decoupled the forward-backward of traditional asynchronous pipeline training methods into two phases by employing a two-phase Gaussian-Seidel optimization scheme. The phase1 calculates the cross intermediate blocks gradient with detached parameter gradient, while the phase2 updates the in-block parameters based on per-block activations and gradients. This, the two-phase decoupling and critical-path scheduling is the necessity component of our algorithm. As for the ablation study on NS preconditioning, we provide it in Table 5.2, and the memory consumption of NS precondition is provided in the revised version. The result indicates that NS precondition involves 8% peak memory overhead.

Thanks for your question. As shown in Eq. (5), our method computes the surrogate gradient through a local forward-backward pass. We have further clarified this process in the revised manuscript. Specifically, in Phase 2, each block independently performs a local forward-backward pass using the cached boundary activation hl(t−τ)\boldsymbol{h}_{l}^{(t-\tau)}, backward signal gl+1(t−τ)\boldsymbol{g}_{l+1}^{(t-\tau)}, and current local parameters θl(t)\boldsymbol{\theta}_{l}^{(t)}, followed by the parameter update. This can be interpreted as a mixed-version parameter state θ~=(θ<l(t−τ),θl(t),θ>l(t−τ))\widetilde{\boldsymbol{\theta}} = (\boldsymbol{\theta}_{\lt l}^{(t-\tau)}, \boldsymbol{\theta}_{l}^{(t)}, \boldsymbol{\theta}_{\gt l}^{(t-\tau)}).

Regarding the delay constraint, Algorithm 2 prioritizes Phase-2 updates when the queue reaches τmax⁡\tau_{\max}.

Response to Q2:

Thanks for your question. As shown in Eq. (5), our method computes the surrogate gradient through a local forward-backward pass. We have further clarified this process in the revised manuscript. Specifically, in Phase 2, each block independently performs a local forward-backward pass using the cached boundary activation hl(t−τ)\boldsymbol{h}_{l}^{(t-\tau)}, backward signal gl+1(t−τ)\boldsymbol{g}_{l+1}^{(t-\tau)}, and current local parameters θl(t)\boldsymbol{\theta}_{l}^{(t)}, followed by the parameter update. This can be interpreted as a mixed-version parameter state θ~=(θ<l(t−τ),θl(t),θ>l(t−τ))\widetilde{\boldsymbol{\theta}} = (\boldsymbol{\theta}_{\lt l}^{(t-\tau)}, \boldsymbol{\theta}_{l}^{(t)}, \boldsymbol{\theta}_{\gt l}^{(t-\tau)}).

Regarding the delay constraint, Algorithm 2 prioritizes Phase-2 updates when the queue reaches τmax⁡\tau_{\max}.

Response to Q3:

Answer: As explicitly described in Section 4.2, our convergence analysis follows the standard block coordinate descent framework. We construct a virtual serialized sequence in which blocks are updated one at a time, while all other blocks remain fixed. Therefore, the block-wise descent inequality in Eq. (9) applies to each individual update, and summing these inequalities yields the telescoping relation in Eq. (8).

Regarding the optimizers, Theorem 4.4 analyzes the convergence of CommuteProp under the first-order update in Eq. (5), with the primary goal of characterizing the impact of asynchronous errors. Extending this guarantee to AdamW, Muon, and NS would require analyzing their distinct optimizer-specific dynamics, including momentum, adaptive scaling, and preconditioning. Such analysis concerns the convergence properties of individual optimizers rather than the asynchronous mechanism proposed in this paper.

AuthorsOpenReview ↗

Response to Reviewer CHux (Cont.)

Response to Q4:

Answer: As we explained in line 357 to 373, the preconditioner is designed for AdamW. In standard AdamW, the original precondition serves as a purely diagonal approximation of the local inverse covariance. As shown in Eq. (13), the asynchronous error is governed by the interaction between the cross-block Hessian and the local update direction. Our NS preconditioner aims to reduce this error by whitening the update direction, assuming the estimated local curvature captures the directions responsible for cross-block error amplification.

As for the direct measurements of asynchronous gradient error before and after preconditioning, the Figure 4 in our paper already isolates NS preconditioning’s contribution by comparing CommuteProp with and without NS under identical configurations. Directly comparing asynchronous gradient errors is less informative, as NS changes parameter updates, leading to different parameters and subsequent gradients. Therefore, the training trajectories in Figure 4 provide a more meaningful empirical comparison.

Response to Q5:

Answer: The results show that cross-block interactions are substantially weaker than the dominant within-block terms, supporting the assumption used in our analysis. This observation is also consistent with prior studies on the approximately block-diagonal Hessian structure of large neural networks [1–3].

Following the empirical protocols in [3,4], we use element-wise Hessian statistics to support the structural assumption rather than directly estimating the block operator norm, whose computation is prohibitive at the billion-parameter scale. Moreover, we would like to emphasize that our convergence and performance results are obtained from full-training runs and are independent of this estimate.

[1] Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. “Why Transformers Need Adam: A Hessian Perspective.” Advances in Neural Information Processing Systems, 37:131786–131823, 2024.

[2]Ivan Ilin. “Hessian of Perplexity for Large Language Models by PyTorch Autograd (Open Source).” arXiv preprint arXiv:2504.04520, 2025.

[3] Zhaorui Dong, Yushun Zhang, Jianfeng Yao, and Ruoyu Sun. “Towards Quantifying the Hessian Structure of Neural Networks.” In OPT 2025: Optimization for Machine Learning, 2025.

[4] Tianyue H. Zhang et al. “Understanding Adam Requires Better Rotation-Dependent Assumptions.” Advances in Neural Information Processing Systems, 38:115111–115152, 2025.

Response to Q6:

In Appendix B.3, we provide the computation and communication time analysis for our algorithm under the worst case scenario. Specifically, Table 3 reports the per-stage computation costs, while Table 4 provides real two-GPU client-server measurements under different network latencies, including optimizer and NS-preconditioner overhead. Furthermore, Eq. (6) characterizes the effects of communication latency, client/server computation ratios, and queue-depth limits on pipeline efficiency. We also provide the peak memory consumption in the updated version. However, we cannot provide a detailed version of queue waiting time, as it depends on many practical factors such as bandwidth. Thus, we also evaluate all methods under the same 15-minute wall-clock budget at Appendix B.4.

Response to Q7:

  1. Not independently. All baselines—synchronous and asynchronous—were run with identical hyperparameters, including learning rate, momentum, batch size, and staleness budget where applicable. We deliberately did not give asynchronous methods an extra hyperparameter search: the comparison is meant to isolate synchronization/staleness effects, and our primary comparison is each async variant vs. its sync counterpart under the same setting.

  2. Regarding Table 5, latency differences also affect the synchronous algorithm, since larger communication overhead results in fewer training iterations under a fixed time budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their valuable suggestions. We would like to clarify the target setting of CommuteProp, its key algorithmic differences from classical asynchronous algorithms, and our rationale for selecting the baselines.

1. Target setting and algorithmic design.

CommuteProp is designed for split learning scenarios, where client-side memory is constrained, as discussed in Lines 36–40 and 170–175. We have further clarified this target setting in the revised manuscript (Lines 193–196).

Inspired by the Gauss–Seidel update scheme, we propose CommuteProp, which decouples the conventional forward–backward computation into two phases. In Phase 1, the cross-block forward–backward computation is performed to obtain and cache the boundary activation and backward signal. In Phase 2, each block independently performs a local forward-backward pass using the cached boundary activation hl(t−τ)\boldsymbol{h}_{l}^{(t-\tau)}, backward signal gl+1(t−τ)\boldsymbol{g}_{l+1}^{(t-\tau)}, and current local parameters θl(t)\boldsymbol{\theta}_{l}^{(t)}, followed by the parameter update. This can be interpreted as a mixed-version parameter state θ~=(θ<l(t−τ),θl(t),θ>l(t−τ))\widetilde{\boldsymbol{\theta}} = (\boldsymbol{\theta}_{\lt l}^{(t-\tau)}, \boldsymbol{\theta}_{l}^{(t)}, \boldsymbol{\theta}_{\gt l}^{(t-\tau)}).

Therefore, CommuteProp adopts a fundamentally different update mechanism from classical asynchronous pipeline algorithms such as PipeDream: it uses cached boundary activations and backward signals instead of retaining multiple historical versions of model weights.

2. Baseline selection under split learning constraints.

This algorithmic distinction also motivates our baseline selection. Classical asynchronous pipeline algorithms such as PipeDream rely on weight stashing to maintain consistency between forward and backward computations. However, maintaining multiple historical weight versions introduces additional memory overhead, making such methods less suitable for our target memory-constrained split learning setting. On the other hand, CommuteProp adopts a two-phase update scheme that avoids weight stashing and allows intermediate activations to be cached on the server side.

Therefore, we select synchronous training as a reference for training quality and NesterovAsync without weight stashing as our primary asynchronous baseline. This allows us to evaluate CommuteProp under a consistent no-weight-stashing constraint.

Moreover, existing asynchronous pipeline methods adopt different weight-stashing strategies, leading to different memory requirements and sensitivity to communication latency. A comprehensive comparison of these strategies under memory-constrained, high-latency split learning is beyond the scope of this work. We therefore include PipeDream-2BW as an additional asynchronous pipeline example in the Appendix, while retaining NesterovAsync without weight stashing as our primary asynchronous baseline in the main experiments.

3. Compatibility with privacy-preserving methods.

CommuteProp stores intermediate activations on the server side, which may introduce potential privacy risks. However, CommuteProp is a plug-and-play optimization method that can be integrated with existing privacy-preserving mechanisms. Our DualGuard experiments demonstrate that CommuteProp can work alongside an existing activation protection method. We emphasize that CommuteProp itself does not provide an independent privacy guarantee, and our experiments focus on its compatibility with existing protection mechanisms rather than robustness against newly designed adaptive attacks. We have clarified this distinction in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.