acceptodds
Under review as a conference paper at ICLR 2027

ParaMoLE: Efficient Tensor-Parallel Fine-Tuning of Mixture of LoRA Experts

Abstract

Mixtures of LoRA experts (MoLE) expand the adaptation capacity of large language models through sparse routing, but fine-tuning retains training states for the entire expert pool. Distributing these states can reduce memory use, yet memory feasibility alone does not ensure efficient training. Joint expert and router updates require cross-GPU coordination, while uneven expert batches can limit the efficiency of sparse computation. To address these challenges, we propose **ParaMoLE**, a framework for memory-efficient MoLE fine-tuning on tensor-parallel (TP) backbones under a fixed GPU budget. ParaMoLE adopts partial factor replication to reduce expert-state storage relative to full replication while jointly evaluating each selected expert within the backbone's TP group. During backward, ParaMoLE reuses reduced low-rank activations or activation gradients for both expert and router updates, avoiding separate synchronization of replicated-factor parameter gradients. Grouped matrix multiplication processes variable-sized expert batches in both forward and backward without padding them to a common length. We establish equivalence to the original MoLE outputs and gradients in exact arithmetic. Experiments across five NLP datasets show lower memory, training-step time, and communication on 3B and 14B LLMs, with comparable downstream quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Tensor Parallelism, Mixture of Experts, LoRA, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_4zveOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ParaMoLE, a framework for efficient fine-tuning of Mixture of LoRA Experts (MoLE) on tensor-parallel (TP) backbones. The framework combines partially replicated LoRA factor placement, low-rank synchronization for joint expert and router updates, and grouped GEMM for variable-sized expert batches. The authors provide an exact-arithmetic equivalence analysis and evaluate the framework on 3B and 14B language models. The reported results demonstrate improvements in training time, peak GPU memory, and communication overhead while maintaining comparable downstream performance.

Critical strengths
  • The authors provide clear motivation on the practical situation. Efficient MoLE fine-tuning under limited GPU memory is a meaningful problem. The paper identifies relevant challenges involving expert-state storage, communication, and irregular GPU computation.

  • The framework has technically sound design. The proposed low-rank synchronization scheme is supported by detailed gradient derivations and an equivalence analysis. Reusing synchronized intermediate tensors for both expert and router updates is a useful design choice.

Critical weaknesses
  • I acknowledge that ParaMoLE addresses a different setting, especially the joint training of experts and routers. However, the paper could more clearly distinguish its novel synchronization mechanism from established techniques. Specifically, some previous work, such as S-LoRA, has already explored efficient tensor-parallel LoRA execution through communication on low-dimensional intermediate tensors. Similarly, efficient processing of variable-sized expert batches has been extensively studied, and Grouped GEMM is an existing GPU optimization technique. The authors should explain whether the three components in ParaMoLE involve nontrivial co-design beyond their integration.

  • For experimental results, my concern is about potentially unfair baseline configurations and insufficient attribution of the reported speedups. Specifically, the main baselines (FR, FS, and EP) use a per-expert loop backend, whereas ParaMoLE employs grouped GEMM, as stated in Appendix F.1. Since grouped GEMM is an established optimization, it is unclear how much of the large end-to-end improvement over existing parallelization strategies is attributable to ParaMoLE's novel contributions.

  • The proposed synchronization scheme substantially reduces the reported communication payload. However, the relative advantage may depend on the synchronization frequency and gradient accumulation strategy. It would be helpful to clarify whether the measured GradSync baseline employs delayed parameter-gradient synchronization and to evaluate how the communication and runtime advantages vary with gradient accumulation steps and batch sizes.

Questions for authors
  • How much of ParaMoLE's improvement remains when FR/FS baselines use an optimized grouped GEMM backend?

  • What is the specific technical novelty of low-rank synchronization relative to existing TP-LoRA methods, beyond adapting the gradient computations to joint expert/router training?

  • Are the three components in ParaMoLE’s independently applicable, or do they provide measurable synergistic benefits?

  • Does ParaMoLE-GradSync accumulate parameter gradients locally across micro-batches before synchronization? How does this choice affect the reported communication savings?

5 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

We thank you for the thoughtful questions about the technical distinction and the contributions of individual components.

Response to W1/Q2/Q3: Technical distinction and component contributions

We clarify that our contribution is the recovery and reuse of low-rank information for joint expert and router updates. S-LoRA partitions both LoRA factors and exchanges intermediate activations between its forward products (Section 6.1). ParaMoLE uses partial replication to keep both expert products local. At DOWN, partial expert outputs join the backbone's output reduction, while partial low-rank activations are cached for backward.

The key distinction is that a reduced factor-gradient matrix alone does not generally provide each token's mixing-weight derivative. In DOWN, forming factor gradients first sums over tokens. We instead recover the low-rank activations before forming the replicated-factor gradient (Eq. (1)). The recovered tensor retains one vector per selected token–expert pair and, together with local tensors, supports both factor gradients and mixing-weight derivatives. In UP, activations are already complete but their gradients are partial, so we recover low-rank activation gradients. Each projection thus reuses one backward low-rank reduction for both updates, avoiding a separate replicated-factor gradient reduction.

Factor placement determines which low-rank tensor requires recovery. We reuse the expert grouping in forward and backward and align corresponding token–expert rows across GPUs before reduction. Grouped GEMM accelerates the local products, while shared recovery provides the information needed to complete both updates. Changing the matrix-multiplication backend alone does not provide this recovery scheme.

Regarding Q3, grouped computation and low-rank recovery can be varied separately, although the recovery formulas depend on factor placement. ParaMoLE-Loop retains low-rank recovery without grouped GEMM. The GradSync variants retain grouped GEMM but replace shared recovery with parameter-gradient and mixing-weight-derivative reductions. ParaMoLE-GradSync-Micro, used in the original Table 3, synchronizes replicated-factor gradients after each microbatch and then accumulates them for the optimizer update. ParaMoLE-GradSync-Step accumulates these gradients locally and synchronizes once per optimizer step (Appendix E.1, Eq. (60)). Both retain the router communication required within each microbatch.

Table R1. Component ablations on Qwen2.5-3B/XSum at global batch 64 and accumulation 16. All variants retain the same partially replicated factor layout and forward output aggregation. Commu. is in GiB/GPU/step.

MethodGrouped GEMMShared low-rank recoveryStep time (s)Commu. (MoLE)
ParaMoLE-LoopNoYes21.9880.563
ParaMoLE-GradSync-MicroYesNo5.2299.844
ParaMoLE-GradSync-StepYesNo4.9631.146
ParaMoLEYesYes4.6370.563

With low-rank recovery retained, grouped execution gives a 4.74× measured step-time speedup without increasing MoLE communication. With grouped execution retained, shared recovery reduces measured step time by 11.3% against GradSync-Micro and 6.6% against GradSync-Step. Against the delayed Step variant, MoLE communication also falls by 50.9%. Each execution choice therefore provides a measurable benefit when the other is retained. We further examine synchronization frequency in W3/Q4.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to W2/Q1: Comparison with optimized FR, FS, and EP baselines

We agree that matching the optimized backend is important for attributing the gains. We therefore evaluated FR, FS, and EP with the same grouped GEMM backend in forward and backward. ParaMoLE retains its advantage over FR and FS under this comparison. Table R2 includes both Loop and Grouped versions to show the batching gains within each method and the differences that remain with the backend matched.

We use Qwen2.5-3B/XSum with four H100 GPUs, E=64E=64, top-k=2k=2, r=α=16r=\alpha=16, sequence length 1024, local micro-batch size 4, and global batch size 64. FR, FS, and ParaMoLE use TP=4 and accumulation 16. Native EP uses TP=1, four data streams, and accumulation 4. All methods process the same global examples per update with BF16 computation and seed 42.

Table R2. FR, FS, and EP comparisons with Loop and Grouped GEMM backends. Time is per complete optimizer step, and memory is peak allocated memory per GPU.

MethodBackbone TPLoop time (s)Grouped time (s)Loop memory (GiB)Grouped memory (GiB)
FR451.31911.919262.48955.168
FS439.2269.982123.04321.176
Native EP15.9894.74678.29178.311
ParaMoLE421.9884.63716.23116.233

With grouped GEMM, ParaMoLE achieves 2.57× and 2.15× measured speedups over FR and FS, with 70.6% and 23.3% lower peak memory, respectively. With Loop, the corresponding speedups are 2.33× and 1.78×. The advantage over FR and FS therefore persists with either backend.

Switching from Loop to Grouped gives a complete-step speedup of 4.74× for ParaMoLE, compared with 4.31× for FR and 3.93× for FS. These comparisons measure the complete executions with the same local backend. Table R1 separately isolates the synchronization choice within the ParaMoLE layout.

Against grouped Native EP, measured step times are close, at 4.637 versus 4.746 s, while peak memory is 16.233 versus 78.311 GiB/GPU. This comparison matches the backend but includes different backbone layouts, since Native EP retains a full backbone on each GPU. We report the comparison with EP at the same backbone TP=4 separately in our response to Reviewer a1ac, W4.

AuthorsOpenReview ↗

Rebuttal Part 3 by Authors

Response to W3/Q4: Synchronization frequency and gradient accumulation

We clarify that the original Table 3 uses the Micro schedule defined in W1/Q2/Q3, with factor-gradient synchronization after each microbatch. To address your question about delayed synchronization, we compare ParaMoLE with both Micro and Step in terms of communication and runtime.

We use the Qwen2.5-3B/XSum setup in W2/Q1, keeping factor placement, grouped GEMM, and output aggregation fixed. First, we fix micro-batch size at 4 and increase accumulation (Table R3).

Table R3. Gradient accumulation with fixed micro-batch size 4. Commu. is in GiB/GPU/step.

AccumulationGlobal batchStrategyStep time (s)Peak memory (GiB/GPU)Commu. (MoLE)Commu. (+ Base)
14ParaMoLE-GradSync-Micro0.39616.2290.5282.546
14ParaMoLE-GradSync-Step0.36716.2290.5282.546
14ParaMoLE0.30116.2290.0352.053
416ParaMoLE-GradSync-Micro1.37016.2222.31410.384
416ParaMoLE-GradSync-Step1.27616.2221.0129.082
416ParaMoLE1.14616.2220.1418.211
1664ParaMoLE-GradSync-Micro5.22916.2269.84442.124
1664ParaMoLE-GradSync-Step4.96316.2261.14633.426
1664ParaMoLE4.63716.2260.56332.843
64256ParaMoLE-GradSync-Micro20.41116.22733.721162.842
64256ParaMoLE-GradSync-Step19.51416.2272.300131.421
64256ParaMoLE18.23316.2272.250131.371

At accumulation 16, allowing delayed synchronization changes the MoLE communication saving from 94.3% to 50.9% and the measured step-time reduction from 11.3% to 6.6%. ParaMoLE therefore still roughly halves MoLE communication against the delayed baseline. As accumulation and global batch increase together, communication savings over Step narrow to 2.2% at accumulation 64, where total communication is nearly equal. Measured step-time reductions range from 6.6% to 18.0% across this sweep.

We also hold the workload constant by fixing global batch at 64 and varying micro-batch size (Table R4).

Table R4. Micro-batch size with fixed global batch size 64. Commu. is in GiB/GPU/step.

Micro-batchAccumulationStrategyStep time (s)Peak memory (GiB/GPU)Commu. (MoLE)Commu. (+ Base)
164ParaMoLE-GradSync-Micro17.98410.10129.88762.167
164ParaMoLE-GradSync-Step16.08310.1011.14633.426
164ParaMoLE15.70510.1010.56332.843
232ParaMoLE-GradSync-Micro9.12712.16116.05248.332
232ParaMoLE-GradSync-Step8.81112.1611.14633.426
232ParaMoLE7.98812.1610.56332.843
416ParaMoLE-GradSync-Micro5.22916.2269.84442.124
416ParaMoLE-GradSync-Step4.96316.2261.14633.426
416ParaMoLE4.63716.2260.56332.843
88ParaMoLE-GradSync-Micro3.95524.2284.44936.729
88ParaMoLE-GradSync-Step3.84324.2281.14633.426
88ParaMoLE3.52624.2280.56332.843

With global batch fixed, ParaMoLE retains 50.9% lower MoLE communication than Step at every micro-batch size, with 2.4%–9.3% shorter measured steps. Its communication remains 0.563 GiB/GPU/step throughout. Splitting the same workload into more microbatches produces more, smaller low-rank exchanges with the same total bytes. We will add both comparisons and clarify the two GradSync schedules.

Reviewer_22N7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ParaMoLE, an efficient tensor-parallel fine-tuning framework for mixtures of LoRA experts. The framework combines partially replicated LoRA factor placement, low-rank activation/gradient synchronization for joint expert and router updates, and grouped GEMM for processing variable-sized expert batches. The authors provide an exact-arithmetic equivalence analysis and evaluate training efficiency, memory consumption, communication payload, and downstream quality on 3B and 14B language models.

Critical strengths
  1. The paper addresses an important systems problem: reducing expert-state replication does not automatically result in efficient end-to-end MoLE training. The joint consideration of memory efficiency, communication, and irregular computation is well motivated.
  2. The derivations cover expert-factor updates, router gradients, and input-gradient propagation under tensor parallelism. The distinction between reducing low-rank activation gradients for column-parallel layers and reducing low-rank activations for row-parallel layers is clearly explained.
  3. ParaMoLE achieves substantially lower memory consumption and training-step latency than the evaluated baselines in several configurations. The ablations help distinguish the contributions of grouped computation and low-rank synchronization.
Critical weaknesses
  1. The baselines use different execution backends and parallelism configurations. In particular, FR/FS use per-expert loops, while ParaMoLE uses grouped GEMM. Table 3 shows that grouped computation contributes substantially more to the speedup than low-rank synchronization. A comparison with an optimized grouped-GEMM baseline would better establish the incremental benefit of ParaMoLE.
  2. Partial factor replication and grouped GEMM are established techniques, making low-rank synchronization the main novel contribution. Although it significantly reduces communication volume, its end-to-end speedup over GradSync is relatively modest (11.3%). Further experiments varying sequence length and gradient accumulation would help clarify when this approach is most beneficial.
  3. The equivalence analysis assumes exact arithmetic, while experiments use BF16. Although the authors discuss numerical effects in Appendix B.5, direct comparisons of routing decisions and gradient errors against a reference implementation would strengthen the correctness claims.
Questions for authors
  1. Can the authors provide an FS baseline with grouped GEMM, or a controlled comparison that better isolates the effects of grouped computation and factor placement?
  2. How does the advantage of low-rank synchronization over delayed parameter-gradient synchronization change with sequence length and gradient accumulation steps?
  3. Have the authors measured numerical differences in router selections or gradients between ParaMoLE and a non-distributed reference implementation under BF16?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

Thank you for the detailed feedback and suggestions for evaluating synchronization benefits and numerical agreement.

Response to W1/Q1: Optimized FR and FS baselines

Following your suggestion, we evaluated FR and FS with grouped GEMM in both forward and backward, using the same TP=4 backbone as ParaMoLE. ParaMoLE retains its advantage over both baselines under these matched conditions. Table R1 reports both Loop and Grouped versions, showing the batching benefit within each method and the differences between methods under matched backends and backbone partitioning.

We use Qwen2.5-3B on XSum with four H100 GPUs, E=64E=64, top-k=2k=2, 4r=\alpha=16$, sequence length 1024, micro-batch size 4, global batch size 64, and accumulation 16. All methods process the same input examples with BF16 computation and seed 42.

Table R1. Complete optimizer-step time and peak allocated memory per GPU.

MethodBackbone TPLoop time (s)Grouped time (s)Loop memory (GiB)Grouped memory (GiB)
FR451.31911.919262.48955.168
FS439.2269.982123.04321.176
ParaMoLE421.9884.63716.23116.233

With grouped GEMM, ParaMoLE is 2.57× faster than FR and 2.15× faster than FS, with 70.6% and 23.3% lower peak memory, respectively. The corresponding Loop speedups are 2.33× and 1.78×. The advantage therefore holds with either backend.

Switching from Loop to Grouped gives a complete-step speedup of 4.74× for ParaMoLE, compared with 4.31× for FR and 3.93× for FS. These comparisons measure the complete training executions and do not attribute the remaining speedups entirely to low-rank synchronization. Our response to W2/Q2 examines synchronization separately using ParaMoLE-GradSync-Micro and ParaMoLE-GradSync-Step, with factor placement and the grouped backend fixed. We will add the matched-backend results to the main text and distinguish the batching gains from the remaining differences between execution strategies.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to W2/Q2: Effects of sequence length and gradient accumulation

We agree that sequence length and accumulation are important to this comparison. Our added experiments show that the benefit persists with delayed synchronization, while communication savings narrow as accumulation or sequence length increases. ParaMoLE-GradSync is a controlled variant within our framework, with the same factor placement, grouped GEMM backend, and output aggregation. ParaMoLE reuses reduced low-rank tensors together with local tensors for both replicated-factor and routing derivatives. The reported 11.3% step-time reduction measures this additional backward-execution benefit, rather than the full-system improvement over FR/FS.

The original Table 3 variant synchronizes factor gradients after each microbatch. We will name it ParaMoLE-GradSync-Micro. We also evaluate ParaMoLE-GradSync-Step, which accumulates factor gradients locally and synchronizes once per optimizer step (Appendix E.1, Eq. (60)). Both variants retain the router communication required in each backward pass.

We use Qwen2.5-3B/XSum with four H100 GPUs, TP=4, E=64E=64, top-k=2k=2, r=16r=16, BF16 computation, and micro-batch size 4. All strategies process the same examples at each setting, with one run, 5 warmup steps, and 20 measured optimizer steps. Table R2 fixes sequence length at 1024 and increases microbatches per step from 1 to 64, with global batch growing accordingly.

Table R2. Effect of gradient accumulation. Commu. is in GiB/GPU/step.

AccumulationGlobal batchStrategyStep time (s)Peak memory (GiB/GPU)Commu. (MoLE)Commu. (+ Base)
14ParaMoLE-GradSync-Micro0.39616.2290.5282.546
14ParaMoLE-GradSync-Step0.36716.2290.5282.546
14ParaMoLE0.30116.2290.0352.053
416ParaMoLE-GradSync-Micro1.37016.2222.31410.384
416ParaMoLE-GradSync-Step1.27616.2221.0129.082
416ParaMoLE1.14616.2220.1418.211
1664ParaMoLE-GradSync-Micro5.22916.2269.84442.124
1664ParaMoLE-GradSync-Step4.96316.2261.14633.426
1664ParaMoLE4.63716.2260.56332.843
64256ParaMoLE-GradSync-Micro20.41116.22733.721162.842
64256ParaMoLE-GradSync-Step19.51416.2272.300131.421
64256ParaMoLE18.23316.2272.250131.371

At accumulation 16, ParaMoLE reduces MoLE communication from 1.146 to 0.563 GiB/GPU/step (50.9%) and measured step time from 4.963 to 4.637 s (6.6%) relative to ParaMoLE-GradSync-Step. Thus, MoLE communication is roughly halved even when the baseline accumulates all 16 microbatches before synchronizing factor gradients.

As accumulation increases, ParaMoLE exchanges low-rank information for more tokens, while the Step variant accumulates factor gradients before sending them. MoLE communication savings decrease from 93.4% at accumulation 1 to 2.2% at 64, where total communication is nearly equal. Measured steps remain 6.6%–18.0% shorter across the four settings.

Table R3 varies sequence length from 256 to 2048, with global batch fixed at 64 and accumulation at 16.

Table R3. Effect of sequence length. Commu. is in GiB/GPU/step.

Sequence lengthStrategyStep time (s)Peak memory (GiB/GPU)Commu. (MoLE)Commu. (+ Base)
256ParaMoLE-GradSync-Step4.00219.8330.7068.777
256ParaMoLE-GradSync-Micro4.35639.8337.97216.043
256ParaMoLE3.97159.8330.1408.211
512ParaMoLE-GradSync-Step4.148211.8460.98717.127
512ParaMoLE-GradSync-Micro4.392511.8468.22224.363
512ParaMoLE3.998811.8460.28116.421
1024ParaMoLE-GradSync-Step4.96316.2261.14633.426
1024ParaMoLE-GradSync-Micro5.22916.2269.84442.124
1024ParaMoLE4.63716.2260.56332.843
2048ParaMoLE-GradSync-Step8.726026.6401.96366.523
2048ParaMoLE-GradSync-Micro9.09226.64010.06674.626
2048ParaMoLE7.60626.6401.12565.685

Relative to ParaMoLE-GradSync-Step, MoLE communication decreases by 80.2% at length 256 and 42.7% at 2048, with 0.8%–12.8% shorter measured steps across the four lengths. Longer sequences increase the low-rank information exchanged, while each accumulated factor-gradient matrix retains the same shape. Communication savings therefore narrow. Runtime gains need not follow the same trend because step time also includes backbone computation and optimizer work. We will add both sweeps and clarify the two GradSync schedules.

AuthorsOpenReview ↗

Rebuttal Part 3 by Authors

Response to W3/Q3: Numerical routing consistency and gradient accuracy

To address your question about numerical agreement under BF16, we directly compare routing decisions and gradients with independent, non-distributed references. Routing agrees in the tested FP32 score-reduction configuration, while the original BF16 DOWN reduction produces mismatches. As stated in Appendix B.5 (original paper in line 812), our BF16 execution computes local router scores in FP32. Lines 812–813 specify that DOWN scores are communicated in the activation dtype. Here, we additionally retain FP32 through the DOWN score reduction and compare this configuration with the original BF16 reduction. Expert computation and low-rank backward communication remain BF16 in both settings.

We use Qwen2.5-3B/XSum with TP=4, E=64E=64, k=2k=2, r=α=16r=\alpha=16, sequence length 1024, micro-batch size 4, and global batch size 64. We capture the first microbatch at updates 1, 5, and 25 from all 36 layers, giving 108 UP and 108 DOWN states. Comparisons use identical captured inputs, parameters, and upstream gradients. The independent, unpartitioned references use ordinary matrix multiplication and autograd, without our custom backward.

To check routing consistency, we let each implementation select experts independently and compare the complete top-kk sets and their ordering (Table R4). Local scoring, selection, and softmax use FP32 throughout. Only DOWN score-reduction precision changes; UP requires no score reduction.

Table R4. Natural-routing agreement with the non-distributed references. Each projection contains 186,372 non-padding token observations. Expert computation and low-rank communication remain BF16 throughout.

ProjectionPrecision configurationSet agreement (%)Ordered agreement (%)Set mismatches
UPFP32 scoring, no score reduction100.000000100.0000000
DOWNFP32 score reduction100.000000100.0000000
DOWNBF16 score reduction (ablation)98.53089597.7029812,738

With FP32 score reduction, all observed expert selections and their ordering match the references. The original BF16 reduction produces 2,738 DOWN set mismatches on the same states. This comparison isolates score-reduction precision while preserving BF16 expert computation and low-rank synchronization.

To assess gradient accuracy, we compare the gradients of LoRA factors AA and BB and router parameters RR against the independent references (Table R5). Natural routing uses each implementation’s own selections and weights. Fixed routing uses common reference selections and weights while retaining router differentiation. For each parameter type, we aggregate errors over the 108 states ss as RelErr⁡(G)  (%)=100∑s∥GsParaMoLE−Gsref∥22∑s∥Gsref∥22.\operatorname{RelErr}(G)\;(\%)=100\sqrt{\frac{\sum_s\|G_s^{\mathrm{ParaMoLE}}-G_s^{\mathrm{ref}}\|_2^2}{\sum_s\|G_s^{\mathrm{ref}}\|_2^2}}.

Table R5. Relative L2L_2 parameter-gradient errors (%). ParaMoLE uses FP32 router-score computation/reduction and BF16 expert computation/low-rank communication.

ReferenceProjectionRoutingGradient of AAGradient of BBGradient of RR
BF16UPNatural0.276392050.000000000.38592597
BF16UPFixed0.276392050.000000000.38592597
BF16DOWNNatural0.000025650.365934530.34969475
BF16DOWNFixed0.000000000.365918530.34969412
FP32UPNatural0.266352500.310527740.29142714
FP32UPFixed0.266352500.310527740.29142714
FP32DOWNNatural0.225150290.370776950.19582198
FP32DOWNFixed0.225150290.370778360.19582294

BF16/FP32 identifies the reference’s input, parameter, and returned-gradient tensor precision. Both references use FP32 router scoring and softmax and select identical experts. The FP32 reference upcasts the same captured states, rather than using a separately trained model.

All reported aggregate gradient errors are below 0.4%, including under natural routing. Natural and fixed routing give closely matching results, so the small errors do not depend on forcing common routes. Tables R4–R5 support numerical agreement for the tested configuration with FP32 router-score reduction and BF16 expert computation and low-rank communication.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your valuable comments and constructive suggestions. We have carefully addressed your concerns in our rebuttal and provided additional clarifications where necessary. We understand that this is a busy time. As the discussion period draws to a close, we would be very grateful if you could find a moment to review our responses and let us know whether any concerns remain. Your feedback would help us further improve the paper, and we would be happy to provide any additional clarification. We sincerely appreciate your time and effort.

Reviewer_a1acOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the memory and efficiency bottlenecks of fine-tuning Mixtures of LoRA Experts (MoLE) on tensor-parallel (TP) backbones. The authors observed that expert and router updates share low-rank intermediate computations in the backward pass, while uneven routing creates variable-sized expert batches. Motivated by this, they propose ParaMoLE, a framework combining partial factor replication, low-rank backward synchronization for joint expert and router updates, and grouped GEMM for unpadded expert batches. Evaluations on 3B and 14B models across five NLP datasets show reduced memory and step time over several baselines while preserving downstream quality.

Critical strengths
  1. The low-rank backward synchronization scheme is well-designed. By reusing reduced low-rank activations or activation gradients to compute both replicated LoRA factor gradients and router derivatives, ParaMoLE avoids redundant collective communication and largely cuts MoLE communication volume.

  2. The theoretical derivations and systems analysis are thorough. The authors clearly establish exact-arithmetic equivalence to standard MoLE, provide clear analytical models for memory, FLOPs, and communication payloads.

Critical weaknesses
  1. The paper overclaims grouped GEMM over variable-sized expert batches as a core contribution alongside low-rank synchronization (line087-090). However, processing variable-sized expert batches via grouped or block-sparse matrix multiplication is already standard in prior MoE and multi-LoRA systems such as MegaBlocks and ALTO.

  2. The comparison is unfair. Appendix F.1 mentioned that FR, FS, and EP using per-expert loop implementation while ParaMoLE uses grouped GEMM. These comparisons mix batching improvements with the proposed distributed scheme (as shown in table3, where grouped GEMM yield 4.74x speedup). Benchmarking EP and FS with grouped GEMM is needed to isolate the benefit of the parallel layout.

  3. Appendix E.1 shows that GradSync only needs to synchronize parameter gradients once per optimizer step, while ParaMoLE must synchronize low-rank vectors at every micro-batch. As the number of micro-batches grows, ParaMoLE's communication increases linearly. The paper lacks an analysis of this trade-off under high gradient accumulation.

  4. The memory comparison with EP in Table 1 is unfair. Appendix F.1 shows that ParaMoLE shards the frozen backbone across 4 GPUs (TP=4), while EP replicates the full backbone on each GPU (TP=1). Both methods should use the partitioning strategy to isolate the memory of the expert layouts.

Questions for authors
  1. How does ParaMoLE compare in step time and peak memory against EP and FS when all methods use the same grouped GEMM backend?

  2. Why does FS consume more peak memory than ParaMoLE in Table 1 despite storing strictly fewer expert parameters per GPU?

  3. Fow does the synchronization benefit change with different gradient accumulation setting?

5 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

We appreciate your careful assessment and suggestions for clarifying the contribution and strengthening the comparisons.

Response to W1: Clarifying the second contribution

We agree that grouped GEMM for unequal expert batches is established. Our intended second contribution is the shared low-rank backward synchronization mechanism and its integration with grouped execution. Lines 087–090 describe both the derivation of our synchronization scheme and its integration with grouped forward and backward computation. ParaMoLE recovers low-rank activation gradients at UP and cached low-rank activations at DOWN. Together with local tensors, each recovered result supports both replicated-factor gradients and token-level mixing-weight derivatives, avoiding a separate TP reduction of those factor gradients. Grouped GEMM efficiently evaluates the narrow local products with unequal expert batch sizes, while the grouping preserves the token–expert correspondence required by the reductions.

MegaBlocks provides block-sparse operations for unequal expert batches, including backward computation. Applying these operations to the local LoRA products would still require combining contributions across GPUs. Our mechanism specifies which low-rank tensor to reduce and how to use the result for both factor and routing derivatives. ALTO’s rank-local adapter parallelism keeps each independent adapter’s computation and gradients on its assigned GPU. ParaMoLE evaluates each selected expert across a TP group, so its updates depend on contributions from multiple GPUs. We will revise the second contribution to distinguish the recovery mechanism we derive from the established grouped computation we use. The comparisons in W2/Q1 evaluate all methods with the same grouped backend.

Response to W2/Q1: Comparisons with the same grouped GEMM backend

To address your concern about the comparison, we evaluated FR, FS, and EP with the same grouped GEMM backend in both forward and backward. ParaMoLE’s advantage over FR and FS remains with the backend matched. Table R1 compares Loop and Grouped versions within each method, as well as the different execution strategies under the same backend.

The original FR/FS/EP implementations used loops, as stated in Appendix F.1. We use Qwen2.5-3B/XSum with four H100 GPUs, E=64E=64, natural top-k=2k=2 routing, r=α=16r=\alpha=16, sequence length 1024, local micro-batch size 4, global batch size 64, BF16 computation, and seed 42. FR, FS, and ParaMoLE use TP=4 and accumulation 16. Native EP uses TP=1, four data streams, and accumulation 4. All methods process the same global examples per update.

Table R1. Complete optimizer-update time and peak allocated memory per GPU.

MethodBackbone TPLoop time (s)Grouped time (s)Loop memory (GiB)Grouped memory (GiB)
FR451.31911.919262.48955.168
FS439.2269.982123.04321.176
Native EP15.9894.74678.29178.311
ParaMoLE421.9884.63716.23116.233

With Loop, ParaMoLE is 2.33× and 1.78× faster than FR and FS. With Grouped GEMM, the corresponding speedups are 2.57× and 2.15×. Switching from Loop to Grouped gives a measured complete-step speedup of 4.74× for ParaMoLE, compared with 4.31× for FR and 3.93× for FS. ParaMoLE thus benefits from grouped computation while retaining its advantage when the baselines receive the same optimization. These comparisons control the local backend but do not isolate factor placement from the other execution differences.

With Grouped GEMM, ParaMoLE uses 70.6% and 23.3% less peak memory than FR and FS. Against grouped Native EP, measured step times are close at 4.637 versus 4.746 s, while peak memory is 16.233 versus 78.311 GiB/GPU. This small timing difference does not establish a reliable speed advantage over EP. Because Native EP replicates the backbone, we address the comparison with matched backbone partitioning separately in W4. We will add Table R1 to the main text and distinguish the batching gains within each method from the remaining differences between execution strategies.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to Q2: Why does FS use more peak memory despite fewer expert parameters?

We agree that FS stores fewer expert parameters. Its larger activation and cache footprint explains the higher peak training memory in the evaluated implementation. To examine this difference, we profiled the original FS in Table 1 and ParaMoLE on Qwen2.5-3B/XSum with the same TP=4 backbone, sequence length 1024, micro-batch size 4, and global batch size 64. Table R4 records the components present at each method’s diagnostic peak, measured in a separate update after warmup.

Table R4. Memory breakdown at the observed peak, in GiB/GPU.

Memory componentFully ShardedParaMoLE
Frozen backbone1.4371.437
Adapter/router parameters, gradients, master weights, and optimizer states4.1606.118
Persistent caches1.6340.378
Saved activations14.4697.189
Other live allocations and allocator rounding1.3421.103
Total diagnostic peak23.04216.225

FS uses 1.958 GiB less for adapter/router training states, consistent with its smaller parameter storage. However, its saved activations occupy 7.280 GiB more than ParaMoLE’s, more than offsetting that saving. FS also retains larger persistent caches, giving a diagnostic peak of 23.042 GiB versus 16.225 GiB for ParaMoLE. ParaMoLE’s lower peak therefore reflects the smaller activation and cache footprint of the evaluated execution.

Response to W3/Q3: Synchronization frequency and gradient accumulation

We clarify that the original ParaMoLE-GradSync ablation synchronizes replicated-factor gradients after each microbatch. We will name this variant ParaMoLE-GradSync-Micro to make its schedule explicit. We have also evaluated ParaMoLE-GradSync-Step, which accumulates factor gradients locally and synchronizes them once per optimizer step, as discussed in Appendix E.1, Eq. (60). This schedule delays factor-gradient synchronization, while the routing derivatives are still needed in each backward pass. ParaMoLE reuses each low-rank reduction for both factor and routing derivatives.

Table R2 compares the three strategies with the same factor placement, grouped GEMM backend, and output aggregation. We use Qwen2.5-3B/XSum with four H100 GPUs, TP=4, E=64E=64, top-k=2k=2, r=16r=16, sequence length 1024, and BF16 computation. We fix micro-batch size at 4 and vary accumulation over 1, 4, 16, and 64, giving global batch sizes of 4, 16, 64, and 256. All strategies process the same examples at each setting.

Table R2. Effect of gradient accumulation. Commu. is in GiB/GPU/step.

AccumulationGlobal batchStrategyStep time (s)Peak memory (GiB/GPU)Commu. (MoLE)Commu. (+ Base)
14ParaMoLE-GradSync-Micro0.39616.2290.5282.546
14ParaMoLE-GradSync-Step0.36716.2290.5282.546
14ParaMoLE0.30116.2290.0352.053
416ParaMoLE-GradSync-Micro1.37016.2222.31410.384
416ParaMoLE-GradSync-Step1.27616.2221.0129.082
416ParaMoLE1.14616.2220.1418.211
1664ParaMoLE-GradSync-Micro5.22916.2269.84442.124
1664ParaMoLE-GradSync-Step4.962716.2261.14633.426
1664ParaMoLE4.63716.2260.56332.843
64256ParaMoLE-GradSync-Micro20.41116.22733.721162.842
64256ParaMoLE-GradSync-Step19.51416.2272.300131.421
64256ParaMoLE18.23316.2272.250131.371

The benefit remains when factor-gradient synchronization is delayed until the optimizer step. At accumulation 16, as in Table 3, ParaMoLE reduces MoLE communication from 1.146 to 0.563 GiB/GPU/step (50.9%) and measured step time from 4.9627 to 4.637 s (6.6%) relative to ParaMoLE-GradSync-Step. It therefore roughly halves MoLE communication even when the baseline accumulates all 16 microbatches before synchronizing factor gradients.

As your comment suggests, increasing accumulation narrows this communication saving. With micro-batch size fixed, ParaMoLE exchanges low-rank information for more tokens, whereas the Step variant combines factor gradients before transmission. MoLE communication savings decrease from 93.4% at accumulation 1 to 2.2% at accumulation 64, where total communication is nearly equal. ParaMoLE also records 6.6%–18.0% shorter measured steps across the four settings, including 18.233 versus 19.514 s at accumulation 64. We will add this comparison to the main text and explain how the communication saving changes with accumulation.

AuthorsOpenReview ↗

Rebuttal Part 3 by Authors

Response to W4: Memory comparison with matched backbone partitioning

To address your concern about backbone partitioning, we evaluated EP with backbone TP=4 on the same four GPUs, retaining the same whole-expert assignment. ParaMoLE still uses less peak memory when backbone partitioning and the grouped GEMM backend are matched. In the original EP baseline, each GPU holds a full frozen backbone and 16 complete LoRA experts per adapted projection.

Table R3 covers all three models and five datasets in Table 1. EP with TP=1 retains the original Loop baseline. EP with TP=4 uses the same grouped GEMM backend as ParaMoLE.

Table R3. EP with replicated and TP-sharded backbones, compared with ParaMoLE. Each cell reports Step (s) / Mem. (GiB/GPU) / Commu. (GiB/GPU/step).

ModelMethodSST-2SQuADGSM8KXSumCNN/DM
Qwen2.5-3BEP, TP=10.989 / 42.77 / 16.4323.078 / 76.35 / 61.7443.076 / 75.14 / 59.5725.989 / 78.29 / 120.66813.015 / 73.65 / 239.298
EP, TP=41.5398 / 14.982 / 27.7753.4750 / 24.093 / 106.5593.4848 / 23.792 / 104.4046.8886 / 24.648 / 210.59115.0076 / 24.080 / 418.660
ParaMoLE, TP=41.001 / 11.61 / 5.9302.215 / 15.61 / 19.6412.205 / 15.61 / 19.6414.637 / 16.23 / 32.8439.993 / 15.57 / 74.486
Llama-3.2-3BEP, TP=10.699 / 29.33 / 10.6702.257 / 51.82 / 40.9822.228 / 51.39 / 40.7494.500 / 54.18 / 79.82010.058 / 49.66 / 159.788
EP, TP=41.1653 / 10.554 / 20.2562.5297 / 16.901 / 78.9142.5506 / 16.792 / 78.6735.0764 / 17.612 / 155.55411.1705 / 16.467 / 311.293
ParaMoLE, TP=40.756 / 9.48 / 6.4331.675 / 12.28 / 22.3471.683 / 12.28 / 22.3473.517 / 12.95 / 43.5668.065 / 11.71 / 86.004
Qwen2.5-14BEP, TP=1OOMOOMOOMOOMOOM
EP, TP=42.1020 / 33.345 / 57.7885.8437 / 51.839 / 224.2755.8303 / 51.691 / 224.01011.9254 / 53.858 / 445.05627.6018 / 50.159 / 892.992
ParaMoLE, TP=41.390 / 30.84 / 18.3334.435 / 39.73 / 63.6144.361 / 39.73 / 63.6149.258 / 41.64 / 123.99021.705 / 37.87 / 244.740

The two EP configurations exhibit a speed–memory trade-off. The evaluated TP=1 configuration avoids backbone TP synchronization and has shorter measured steps and less communication on the two 3B models, but requires substantially more memory. The TP=4 configuration reduces memory and enables all five 14B tasks, while incurring more communication and longer steps. With backbone TP=4 and grouped GEMM matched, ParaMoLE uses less peak memory than EP across all 15 settings. We will add these results to Table 1.

Reviewer_FKdjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ParaMoLE, an efficient tensor-parallel fine-tuning framework for Mixture of LoRA Experts (MoLE). ParaMoLE reduces communication overhead by low-rank synchronization, utilizing low-rank gradients instead of full expert parameter gradients, while employing partial factor replication to align with tensor-parallel execution. It also utilizes grouped GEMM to reduce computational overhead. Experimental results show substantial improvements in training speed and memory efficiency, along with significantly reduced communication payload, while maintaining comparable downstream task performance.

Critical strengths
  • This paper proposes two method for efficient TP finetuning of mixture of LoRA experts: eusing synchronized low-rank activations or their gradients, and utilizing grouped GEMM for faster matrix calculation. These techniques effectively reduce communication overhead and improve GPU utilization. Moreover, the proposed approach preserves mathematical equivalence to the original MoLE formulation under the stated assumptions.
  • The experiment results shows substantial improvement of training efficiency without loosing the performance, which shows high speedup and almost 90 percent reduction of communication payload. Especially, ParaMoLE achieves substantial training speedups and approximately 94% reduction in MoLE communication payload compared to the gradient-synchronization variant. The ablation studies further demonstrate the individual contributions of low-rank synchronization and grouped GEMM to the overall efficiency gains.
Critical weaknesses
  • In Table 3, grouped computation contributes for a reduction in step time than low-rank synchronization, which means computational efficiency play a major role in the overall speedup. Still, the grouped GEMM is an existing technique, and it is unclear whether comparable optimizations have been applied to the FR, FS, and EP baselines. For a fair comparison, it would be helpful if the authors could clarify whether these baselines also utilize grouped GEMM or report their performance when similar optimizations are applied.
  • Proposed low-rank synchronization constructs the router's weight gradient using hi,e=⟨zi,e,δzi,e⟩/gi,eh_{i,e} = \langle z_{i,e}, \delta z_{i,e} \rangle / g_{i,e}, but the division with the router's weight may introduce numerical instability. I wonder a quantitative analysis of how much the numerical error introduced by this recovery step, and how its effect on training stability. It would be helpful if the authors could report the distribution of selected routing weights during training and measure the gradient recovery error against an FP32 reference, particularly for low-weight experts.
Questions for authors

Could the authors quantify the numerical error introduced by router-gradient recovery, particularly for small routing weights, and assess its impact on training stability?

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1 by Authors

We thank you for the constructive comments on baseline comparisons and numerical stability.

Response to W1: Grouped computation and optimized baselines

Following your suggestion, we have evaluated FR, FS, and EP with the same grouped GEMM backend as ParaMoLE in both forward and backward. ParaMoLE retains its advantage over FR and FS under this comparison. We also compare each method's Loop and Grouped versions to quantify its benefit from grouped computation.

We clarify that ParaMoLE's partially replicated layout keeps both LoRA forward products local to each GPU. We group these products using each expert's actual token count and reuse the grouping in backward. When complete low-rank information is needed, we align the corresponding token–expert contributions across GPUs before reduction. The recovered DOWN activations or UP activation gradients, together with local tensors, support both replicated-factor gradients and mixing-weight derivatives. Grouped GEMM accelerates the local calculations, while low-rank recovery supplies the information needed for both updates.

The original FR/FS/EP implementations used loops, as stated in Appendix F.1. Table R1 compares both backends for every method on Qwen2.5-3B/XSum with four H100 GPUs, E=64E=64, natural top-k=2k=2 routing, r=α=16r=\alpha=16, sequence length 1024, local micro-batch 4, and global batch 64. Runs use BF16 and seed 42. FR, FS, and ParaMoLE use TP=4 and accumulation 16. Native EP uses TP=1, four data streams, and accumulation 4. All methods process the same global examples per update.

Table R1. Complete optimizer-update time and peak allocated memory per GPU.

MethodBackbone TPLoop time (s)Grouped time (s)Loop memory (GiB)Grouped memory (GiB)
FR451.31911.919262.48955.168
FS439.2269.982123.04321.176
Native EP15.9894.74678.29178.311
ParaMoLE421.9884.63716.23116.233

With Loop, ParaMoLE is 2.33× and 1.78× faster than FR and FS, respectively. With grouped GEMM applied to every method, the corresponding speedups are 2.57× and 2.15×. Switching from Loop to Grouped yields a 4.74× complete-step speedup for ParaMoLE, compared with 4.31× for FR and 3.93× for FS. The advantage over FR and FS therefore persists with either backend. These complete-step measurements do not isolate kernel efficiency or the contribution of factor placement alone.

With grouped GEMM, ParaMoLE also uses 70.6% and 23.3% less peak memory than FR and FS. Against grouped Native EP, measured step times are close, at 4.637 versus 4.746 s, while peak memory is 16.233 versus 78.311 GiB/GPU. We will add the matched-backend results to the main text and distinguish local batching gains from differences in the complete distributed execution.

AuthorsOpenReview ↗

Rebuttal Part 2 by Authors

Response to W2/Q1: Router-gradient accuracy and training stability

To address your concern about division by small routing weights, we examined the weight distribution, gradient errors against FP32, and training losses. The sampled small-weight UP gradients show similar errors with and without division, and all three 100-update runs completed without nonfinite losses or skipped updates.

Table R2 summarizes six sampled updates (1, 5, 10, 25, 50, and 100) from a 100-update Qwen2.5-3B run on XSum with TP=4, E=64E=64, top-k=2k=2, r=16r=16, and BF16 computation. We observed 41,917 UP pairs with 10−4≤g<10−210^{-4}\le g<10^{-2} and none below 10−410^{-4}. The explicit division is used in UP, whereas DOWN uses direct contraction.

Table R2. Selected-weight distribution pooled across the six sampled updates. Each projection type has 13,125,816 observations.

Weight intervalUP count (%)DOWN count (%)
10−2≤g≤110^{-2}\le g\le113,083,899 (99.68065223525913%)13,119,979 (99.95553038378719%)
10−4≤g<10−210^{-4}\le g<10^{-2}41,917 (0.319347764740874%)4,598 (0.03503020307461266%)
10−6≤g<10−410^{-6}\le g<10^{-4}0 (0%)389 (0.002963625270992676%)
10−8≤g<10−610^{-8}\le g<10^{-6}0 (0%)309 (0.002354139354079015%)
0<g<10−80<g<10^{-8}0 (0%)541 (0.004121648513128631%)

To assess gradient accuracy (Table R3), we compare Current, the original recovery with division by the routing weight in UP, with Division-free (Appendix B.5), which computes the UP router-logit gradient without this division. DOWN computation is unchanged. Both strategies are evaluated against direct FP32 derivatives using the same cached states. For each weight interval, we concatenate the sampled router-logit gradients into a vector Γ\Gamma and compute the relative L2L_2 error as Relativeerror⁡(\operatorname{Relative error}(\\%)=100\times\frac{\lVert\Gamma-\Gamma_{\mathrm{FP32}}\rVert_2}{\lVert\Gamma_{\mathrm{FP32}}\rVert_2}, where ΓFP32\Gamma_{\mathrm{FP32}} contains the corresponding FP32 reference gradients. For the observed small-weight UP pairs (10−4≤g<10−210^{-4}\le g<10^{-2}), both strategies have an error of approximately 0.1734%. Their gradient accuracy is therefore similar on the observed small-weight inputs.

Table R3. Pooled relative L2L_2 error of router-logit gradients against FP32, in percent.

ProjectionWeight intervalPairsCurrentDivision-free
UP10−2≤g≤110^{-2}\le g\le1812,2810.222045934465762230.2220456597143065
UP10−4≤g<10−210^{-4}\le g<10^{-2}41,9170.173375445833314960.17337532351552967
UP0<g<10−40<g<10^{-4}0——
DOWN10−2≤g≤110^{-2}\le g\le1814,7350.0451706936126379040.045170693612637904
DOWN10−4≤g<10−210^{-4}\le g<10^{-2}4,5980.040377096580094740.04037709658009474
DOWN10−6≤g<10−410^{-6}\le g<10^{-4}3890.087292161448258870.08729216144825887
DOWN10−8≤g<10−610^{-8}\le g<10^{-6}3090.041636561118538890.04163656111853889
DOWN0<g<10−80<g<10^{-8}5410.0362356071748618640.036235607174861864

The division-free variant changes UP only; DOWN retains the same direct-contraction computation, so its two columns are identical. Low-weight bins include all 16 microbatches, whereas the ordinary-weight control uses the first microbatch at each sampled update. A dash denotes an unobserved interval.

To check training stability, we compare three separate 100-update runs with the same initialization and data order (Table R4). Only router-derivative computation changes between Current, Division-free, and direct FP32 computation; the rest of the mixed-precision execution is unchanged. All three runs have similar average losses, with no nonfinite losses or skipped updates. These results support stable training over the tested 100 updates. We will add these measurements to the revised manuscript.

Table R4. Training losses and update failures.

Update / summaryCurrentDivision-freeFP32 router derivative
11.7018661.7018661.701866
51.4722851.4717381.473355
101.4750221.4745531.475325
251.5696101.5690471.568365
501.4897221.4906151.487779
1001.5775631.5793991.584918
Mean, all 100 updates1.5444981.5438991.544798
Mean, final 20 updates1.5064791.5063551.506864
Nonfinite loss updates000
Skipped updates000

Thank you for the rebuttal. Since the authors have addressed my concerns, I will keep my positive score.

AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your positive feedback and for recognizing our efforts to address your concerns. We sincerely appreciate your continued support.

Comments on the paper

1 comment

As only a few days remain in the author-reviewer discussion period, I encourage reviewers to review the authors' rebuttal and actively participate in the discussion. In particular, please clarify which concerns have been addressed and identify any remaining concerns or questions that may require further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.