- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ParaMoLE, a framework for efficient fine-tuning of Mixture of LoRA Experts (MoLE) on tensor-parallel (TP) backbones. The framework combines partially replicated LoRA factor placement, low-rank synchronization for joint expert and router updates, and grouped GEMM for variable-sized expert batches. The authors provide an exact-arithmetic equivalence analysis and evaluate the framework on 3B and 14B language models. The reported results demonstrate improvements in training time, peak GPU memory, and communication overhead while maintaining comparable downstream performance.
Critical strengths
-
The authors provide clear motivation on the practical situation. Efficient MoLE fine-tuning under limited GPU memory is a meaningful problem. The paper identifies relevant challenges involving expert-state storage, communication, and irregular GPU computation.
-
The framework has technically sound design. The proposed low-rank synchronization scheme is supported by detailed gradient derivations and an equivalence analysis. Reusing synchronized intermediate tensors for both expert and router updates is a useful design choice.
Critical weaknesses
-
I acknowledge that ParaMoLE addresses a different setting, especially the joint training of experts and routers. However, the paper could more clearly distinguish its novel synchronization mechanism from established techniques. Specifically, some previous work, such as S-LoRA, has already explored efficient tensor-parallel LoRA execution through communication on low-dimensional intermediate tensors. Similarly, efficient processing of variable-sized expert batches has been extensively studied, and Grouped GEMM is an existing GPU optimization technique. The authors should explain whether the three components in ParaMoLE involve nontrivial co-design beyond their integration.
-
For experimental results, my concern is about potentially unfair baseline configurations and insufficient attribution of the reported speedups. Specifically, the main baselines (FR, FS, and EP) use a per-expert loop backend, whereas ParaMoLE employs grouped GEMM, as stated in Appendix F.1. Since grouped GEMM is an established optimization, it is unclear how much of the large end-to-end improvement over existing parallelization strategies is attributable to ParaMoLE's novel contributions.
-
The proposed synchronization scheme substantially reduces the reported communication payload. However, the relative advantage may depend on the synchronization frequency and gradient accumulation strategy. It would be helpful to clarify whether the measured GradSync baseline employs delayed parameter-gradient synchronization and to evaluate how the communication and runtime advantages vary with gradient accumulation steps and batch sizes.
Questions for authors
-
How much of ParaMoLE's improvement remains when FR/FS baselines use an optimized grouped GEMM backend?
-
What is the specific technical novelty of low-rank synchronization relative to existing TP-LoRA methods, beyond adapting the gradient computations to joint expert/router training?
-
Are the three components in ParaMoLE’s independently applicable, or do they provide measurable synergistic benefits?
-
Does ParaMoLE-GradSync accumulate parameter gradients locally across micro-batches before synchronization? How does this choice affect the reported communication savings?
5 comments
Rebuttal Part 2 by Authors
Response to W2/Q1: Comparison with optimized FR, FS, and EP baselines
We agree that matching the optimized backend is important for attributing the gains. We therefore evaluated FR, FS, and EP with the same grouped GEMM backend in forward and backward. ParaMoLE retains its advantage over FR and FS under this comparison. Table R2 includes both Loop and Grouped versions to show the batching gains within each method and the differences that remain with the backend matched.
We use Qwen2.5-3B/XSum with four H100 GPUs, , top-, , sequence length 1024, local micro-batch size 4, and global batch size 64. FR, FS, and ParaMoLE use TP=4 and accumulation 16. Native EP uses TP=1, four data streams, and accumulation 4. All methods process the same global examples per update with BF16 computation and seed 42.
Table R2. FR, FS, and EP comparisons with Loop and Grouped GEMM backends. Time is per complete optimizer step, and memory is peak allocated memory per GPU.
| Method | Backbone TP | Loop time (s) | Grouped time (s) | Loop memory (GiB) | Grouped memory (GiB) |
|---|---|---|---|---|---|
| FR | 4 | 51.319 | 11.9192 | 62.489 | 55.168 |
| FS | 4 | 39.226 | 9.9821 | 23.043 | 21.176 |
| Native EP | 1 | 5.989 | 4.746 | 78.291 | 78.311 |
| ParaMoLE | 4 | 21.988 | 4.637 | 16.231 | 16.233 |
With grouped GEMM, ParaMoLE achieves 2.57× and 2.15× measured speedups over FR and FS, with 70.6% and 23.3% lower peak memory, respectively. With Loop, the corresponding speedups are 2.33× and 1.78×. The advantage over FR and FS therefore persists with either backend.
Switching from Loop to Grouped gives a complete-step speedup of 4.74× for ParaMoLE, compared with 4.31× for FR and 3.93× for FS. These comparisons measure the complete executions with the same local backend. Table R1 separately isolates the synchronization choice within the ParaMoLE layout.
Against grouped Native EP, measured step times are close, at 4.637 versus 4.746 s, while peak memory is 16.233 versus 78.311 GiB/GPU. This comparison matches the backend but includes different backbone layouts, since Native EP retains a full backbone on each GPU. We report the comparison with EP at the same backbone TP=4 separately in our response to Reviewer a1ac, W4.
Rebuttal Part 3 by Authors
Response to W3/Q4: Synchronization frequency and gradient accumulation
We clarify that the original Table 3 uses the Micro schedule defined in W1/Q2/Q3, with factor-gradient synchronization after each microbatch. To address your question about delayed synchronization, we compare ParaMoLE with both Micro and Step in terms of communication and runtime.
We use the Qwen2.5-3B/XSum setup in W2/Q1, keeping factor placement, grouped GEMM, and output aggregation fixed. First, we fix micro-batch size at 4 and increase accumulation (Table R3).
Table R3. Gradient accumulation with fixed micro-batch size 4. Commu. is in GiB/GPU/step.
| Accumulation | Global batch | Strategy | Step time (s) | Peak memory (GiB/GPU) | Commu. (MoLE) | Commu. (+ Base) |
|---|---|---|---|---|---|---|
| 1 | 4 | ParaMoLE-GradSync-Micro | 0.396 | 16.229 | 0.528 | 2.546 |
| 1 | 4 | ParaMoLE-GradSync-Step | 0.367 | 16.229 | 0.528 | 2.546 |
| 1 | 4 | ParaMoLE | 0.301 | 16.229 | 0.035 | 2.053 |
| 4 | 16 | ParaMoLE-GradSync-Micro | 1.370 | 16.222 | 2.314 | 10.384 |
| 4 | 16 | ParaMoLE-GradSync-Step | 1.276 | 16.222 | 1.012 | 9.082 |
| 4 | 16 | ParaMoLE | 1.146 | 16.222 | 0.141 | 8.211 |
| 16 | 64 | ParaMoLE-GradSync-Micro | 5.229 | 16.226 | 9.844 | 42.124 |
| 16 | 64 | ParaMoLE-GradSync-Step | 4.963 | 16.226 | 1.146 | 33.426 |
| 16 | 64 | ParaMoLE | 4.637 | 16.226 | 0.563 | 32.843 |
| 64 | 256 | ParaMoLE-GradSync-Micro | 20.411 | 16.227 | 33.721 | 162.842 |
| 64 | 256 | ParaMoLE-GradSync-Step | 19.514 | 16.227 | 2.300 | 131.421 |
| 64 | 256 | ParaMoLE | 18.233 | 16.227 | 2.250 | 131.371 |
At accumulation 16, allowing delayed synchronization changes the MoLE communication saving from 94.3% to 50.9% and the measured step-time reduction from 11.3% to 6.6%. ParaMoLE therefore still roughly halves MoLE communication against the delayed baseline. As accumulation and global batch increase together, communication savings over Step narrow to 2.2% at accumulation 64, where total communication is nearly equal. Measured step-time reductions range from 6.6% to 18.0% across this sweep.
We also hold the workload constant by fixing global batch at 64 and varying micro-batch size (Table R4).
Table R4. Micro-batch size with fixed global batch size 64. Commu. is in GiB/GPU/step.
| Micro-batch | Accumulation | Strategy | Step time (s) | Peak memory (GiB/GPU) | Commu. (MoLE) | Commu. (+ Base) |
|---|---|---|---|---|---|---|
| 1 | 64 | ParaMoLE-GradSync-Micro | 17.984 | 10.101 | 29.887 | 62.167 |
| 1 | 64 | ParaMoLE-GradSync-Step | 16.083 | 10.101 | 1.146 | 33.426 |
| 1 | 64 | ParaMoLE | 15.705 | 10.101 | 0.563 | 32.843 |
| 2 | 32 | ParaMoLE-GradSync-Micro | 9.127 | 12.161 | 16.052 | 48.332 |
| 2 | 32 | ParaMoLE-GradSync-Step | 8.811 | 12.161 | 1.146 | 33.426 |
| 2 | 32 | ParaMoLE | 7.988 | 12.161 | 0.563 | 32.843 |
| 4 | 16 | ParaMoLE-GradSync-Micro | 5.229 | 16.226 | 9.844 | 42.124 |
| 4 | 16 | ParaMoLE-GradSync-Step | 4.963 | 16.226 | 1.146 | 33.426 |
| 4 | 16 | ParaMoLE | 4.637 | 16.226 | 0.563 | 32.843 |
| 8 | 8 | ParaMoLE-GradSync-Micro | 3.955 | 24.228 | 4.449 | 36.729 |
| 8 | 8 | ParaMoLE-GradSync-Step | 3.843 | 24.228 | 1.146 | 33.426 |
| 8 | 8 | ParaMoLE | 3.526 | 24.228 | 0.563 | 32.843 |
With global batch fixed, ParaMoLE retains 50.9% lower MoLE communication than Step at every micro-batch size, with 2.4%–9.3% shorter measured steps. Its communication remains 0.563 GiB/GPU/step throughout. Splitting the same workload into more microbatches produces more, smaller low-rank exchanges with the same total bytes. We will add both comparisons and clarify the two GradSync schedules.
Rebuttal Part 1 by Authors
We thank you for the thoughtful questions about the technical distinction and the contributions of individual components.
Response to W1/Q2/Q3: Technical distinction and component contributions
We clarify that our contribution is the recovery and reuse of low-rank information for joint expert and router updates. S-LoRA partitions both LoRA factors and exchanges intermediate activations between its forward products (Section 6.1). ParaMoLE uses partial replication to keep both expert products local. At DOWN, partial expert outputs join the backbone's output reduction, while partial low-rank activations are cached for backward.
The key distinction is that a reduced factor-gradient matrix alone does not generally provide each token's mixing-weight derivative. In DOWN, forming factor gradients first sums over tokens. We instead recover the low-rank activations before forming the replicated-factor gradient (Eq. (1)). The recovered tensor retains one vector per selected token–expert pair and, together with local tensors, supports both factor gradients and mixing-weight derivatives. In UP, activations are already complete but their gradients are partial, so we recover low-rank activation gradients. Each projection thus reuses one backward low-rank reduction for both updates, avoiding a separate replicated-factor gradient reduction.
Factor placement determines which low-rank tensor requires recovery. We reuse the expert grouping in forward and backward and align corresponding token–expert rows across GPUs before reduction. Grouped GEMM accelerates the local products, while shared recovery provides the information needed to complete both updates. Changing the matrix-multiplication backend alone does not provide this recovery scheme.
Regarding Q3, grouped computation and low-rank recovery can be varied separately, although the recovery formulas depend on factor placement. ParaMoLE-Loop retains low-rank recovery without grouped GEMM. The GradSync variants retain grouped GEMM but replace shared recovery with parameter-gradient and mixing-weight-derivative reductions. ParaMoLE-GradSync-Micro, used in the original Table 3, synchronizes replicated-factor gradients after each microbatch and then accumulates them for the optimizer update. ParaMoLE-GradSync-Step accumulates these gradients locally and synchronizes once per optimizer step (Appendix E.1, Eq. (60)). Both retain the router communication required within each microbatch.
Table R1. Component ablations on Qwen2.5-3B/XSum at global batch 64 and accumulation 16. All variants retain the same partially replicated factor layout and forward output aggregation. Commu. is in GiB/GPU/step.
With low-rank recovery retained, grouped execution gives a 4.74× measured step-time speedup without increasing MoLE communication. With grouped execution retained, shared recovery reduces measured step time by 11.3% against GradSync-Micro and 6.6% against GradSync-Step. Against the delayed Step variant, MoLE communication also falls by 50.9%. Each execution choice therefore provides a measurable benefit when the other is retained. We further examine synchronization frequency in W3/Q4.