- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes CommuteProp for communication-bound split LLM fine-tuning. It separates cross-block computation from local parameter updates and overlaps them across batches without storing historical weights. A blockwise preconditioner is introduced to address staleness. The evaluation covers NLP tasks, client–server deployment, and integration with DualGuard.
Critical strengths
-
The paper addresses a practical bottleneck in split learning. Improving communication–computation overlap without retaining multiple weight versions is a useful goal.
-
The evaluation includes real two-GPU measurements and a PipeDream-2BW comparison under a fixed time budget, providing evidence beyond simulation.
-
The DualGuard experiments demonstrate compatibility with an existing defense while largely preserving task performance.
Critical weaknesses
-
The proof of Lemma 4.3 does not account for the cached backward signal’s dependence on the local block’s old output. Using current local weights therefore does not justify eliminating the local staleness term.
-
The novelty over Zero Bubble Pipeline Parallelism’s computation splitting and Shampoo-style preconditioning needs to be stated more clearly.
-
The Nesterov Async settings differ from the cited work, and it is unclear whether its stage-dependent adjustments were retained. This weakens confidence in the baseline comparison.
-
The main equal-time comparison excludes optimizer costs, including NS preconditioning. Actual staleness and peak client memory are also needed to assess the efficiency trade-off.
-
Table 11 reports element-wise Hessian statistics, whereas the theory requires operator-norm bounds. These measurements do not directly validate the assumption.
-
The experiments do not isolate whether NS specifically mitigates staleness or improves optimization more generally. A synchronous NS baseline would help distinguish these effects.
Questions for authors
-
Could you justify the mixed-state gradient equivalence in Lemma 4.3, given that the cached backward signal depends on the local block’s old output?
-
What distinguishes CommuteProp’s decomposition and preconditioner from Zero Bubble Pipeline Parallelism and Shampoo, respectively?
-
Which momentum and learning-rate adjustments were used for Nesterov Async? Were the stage-dependent settings of the original no-weight-stashing variant retained?
-
Do the deployment timings include optimizer and NS preconditioning costs? Please also clarify the queue depths and actual parameter staleness in these experiments.
-
How do the element-wise Hessian statistics in Table 11 support the operator-norm assumption used in the analysis?
-
Do the existing results distinguish NS’s general optimization benefits from its effect on staleness-induced error?
5 comments
Response to Reviewer G68b (Cont.)
Response to W5 and Q5:
Answer: The results show that cross-block interactions are substantially weaker than the dominant within-block terms, supporting the assumption used in our analysis. This observation is also consistent with prior studies on the approximately block-diagonal Hessian structure of large neural networks [1–3].
Following the empirical protocols in [3,4], we use element-wise Hessian statistics to support the structural assumption rather than directly estimating the block operator norm, whose computation is prohibitive at the billion-parameter scale. Moreover, we would like to emphasize that our convergence and performance results are obtained from full-training runs and are independent of this estimate.
[1] Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. “Why Transformers Need Adam: A Hessian Perspective.” Advances in Neural Information Processing Systems, 37:131786–131823, 2024.
[2]Ivan Ilin. “Hessian of Perplexity for Large Language Models by PyTorch Autograd (Open Source).” arXiv preprint arXiv:2504.04520, 2025.
[3] Zhaorui Dong, Yushun Zhang, Jianfeng Yao, and Ruoyu Sun. “Towards Quantifying the Hessian Structure of Neural Networks.” In OPT 2025: Optimization for Machine Learning, 2025.
[4] Tianyue H. Zhang et al. “Understanding Adam Requires Better Rotation-Dependent Assumptions.” Advances in Neural Information Processing Systems, 38:115111–115152, 2025.
Response to : Thanks for your question, we provide the synchronous adamw with synchronous, the result indicates that NS precondition does not improve model’s utility.
| Method | Emotion (Acc) | AgNews (Acc) | Samsum (ROUGE) | Samsum (METEOR) | SciERC (F1) | Spider (EM) |
|---|---|---|---|---|---|---|
| Sync (NS) | 92.03 | 94.57 | 36.21 | 39.51 | 73.25 | 79.20 |
| Sync | 91.15 | 94.64 | 36.61 | 39.72 | 72.89 | 81.40 |
Response to Q3:
We use the default momentum factor 0.9 and Nesterov correction as mentioned in Appendix line 731, and the stage-dependent setting of the original no-weight-stashing variant was therefore retained.
Response to Q4:
Yes, we consider the NS preconditioning cost in our experiments (Please refer Table3 in line 758). The queue depth depends on the staleness tau. Larger tau indicates larger queue depths.
[Author Follow-up] Request for Feedback on Rebuttal Responses
Dear Reviewer
Thank you again for your time and thoughtful feedback. As the discussion period approaches its close, we would greatly appreciate any thoughts on our responses and the additional evidence provided.
In particular, we would be grateful to know whether any of your concerns remain unresolved, and we would be happy to provide further clarification. If our responses have addressed your main concerns, we would also appreciate your reconsideration of the overall assessment.
Thank you for your time and consideration.
Best regards,
Authors
Response to Reviewer G68b
Response to W1 and Q1:
Thanks for your question. As shown in Eq. (5), our method computes the surrogate gradient through a local forward-backward pass. We have further clarified this process in the revised manuscript. Specifically, in Phase 2, each block independently performs a local forward-backward pass using the cached boundary activation hl(t−τ), backward signal gl+1(t−τ), and current local parameters θl(t), followed by the parameter update. This can be interpreted as a mixed-version parameter state θ=(θ<l(t−τ),θl(t),θ>l(t−τ)).
Response to W2 and Q2:
Thanks for your question, Zero Bubble Pipeline Parallelism is a synchronous algorithm. Thus, the gradient of zero bubble parallelism’s splitting method is the same as the classical synchronous algorithm without any asynchronous errors. However, commuteprop considers the asynchronous scenario, and decouple the forward backward pass into two phases. The two methods may share some similar design philosophy, but they are two different kinds of algorithms.
As for Shampoo-style preconditioning, the NS preconditioning can be regarded as a shampoo-style preconditioning, and we cite the reference paper in line 376. Our contribution is not the introduction of inverse-root preconditioning itself, but its application to mitigating asynchronous cross-block errors. Our focus is to investigate whether NS preconditioning can improve optimization under asynchronous execution, rather than to study its contribution to synchronous algorithms. This is the motivation for introducing NS into CommuteProp.
Finally, we emphasize that our work focuses on split-learning scenarios. Zero Bubble Pipeline Parallelism and Shampoo-style preconditioning are developed under synchronous training settings. Sharing similar design principles does not, by itself, imply a lack of novelty. For example, Nesterov-based asynchronous training introduces a different learning schedule, but this does not preclude other algorithms from making novel contributions to asynchronous optimization.
Response to W3:
Thanks for your question, The utility comparison between Nesterov Async and Zero-shot indicates that Nesterov Async works on most datasets. As for the unexpected performance of Nesterov Async on SciERC dataset, we believe that the hyperparemeter contributes to the outlier. Please check the Figure 4c) and 4d), where NesterovAsync shares similar results as CommuteProp(Muon).
Moreover, avoiding weight stashing is an important motivation for memory-constrained split learning. Therefore, we choose synchronous training as a strong reference for training quality and NesterovAsync without weight stashing as our primary asynchronous baseline. Generally, synchronous training is expected to achieve better training quality due to the absence of stale updates.
Response to W4:
Thanks for your question, we have already provided the memory comparison in the revised version, specifically, we have:
The results show that CommuteProp achieves the same client-side peak memory consumption as synchronous SL and NesterovAsync, while PipeDream-2BW incurs higher memory overhead. Moreover, NS preconditioning introduces only a small additional memory cost.
The result is also updated in the revised version.