acceptodds
Under review as a conference paper at ICLR 2027

Multiplicative Mixture of Experts for Rank-Efficient LLM Finetuning

Abstract

Large language models (LLMs) have achieved impressive results in many general-purpose domains, but their performance on specific tasks can still be improved through finetuning. Parameter-efficient finetuning (PEFT) tailors an LLM to one or more tasks through a small amount of trainable parameters, requiring reduced computational resources. On one hand, techniques like low-rank adaptation (LoRA) provide the required parameter efficiency with additive adapters of low, and fixed, rank, which limits their flexibility. On the other hand, mixture of experts (MoEs) enhance the flexibility of a model at the cost of parameter count and memory budget. The combination of the two approaches, parameter-efficient MoEfication, has shown promise in addressing the issues of both. In this work, we show that {replacing additive with multiplicative interactions improves the efficiency of PEFT adapters, increasing the flexibility and reducing the number of parameters involved in MoEfication. We find that our quantum-inspired multiplicative method, OperA, is optimal given the same parameter budget for 7/8 models considered, using fewer or equal parameters than the baseline. In a multi-task setting, OperA fully surpasses all baselines on all models. We also propose a class of DMRG-based algorithms to accelerate OperA in large-scale deployments by alleviating its computational complexity, resulting in up to a 2.5x throughput improvement and up to a 8.9x memory reduction over the naive implementation. Finally, we provide evidence that OperA surpasses the rank of competing solutions by more than two orders of magnitude.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
rank-efficient, moe, llm, finetuning, mpo, lora

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_a6D7OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OperA, a PEFT MoE method that uses TT-SVD to decompose the weights of a model into Matric Product Operator (MPO) tensors. It then freezes the shared core of the weight and trains exxpert-specific auxiliary tensors. The authors evaluate their proposed approach on a number or LLMS and tasks, and show that OperA can get competitive accuracy with fewer trainable parameters than the evaluated LoRA-based baselines.

Critical strengths

1- The paper provides some evidence of comparable performance with the baselines with fewer trainable parameters. In the scenarios where storage space for LoRA parameters is a bottleneck, this direction is practical.

2- There are some useful ablations in the paper.

Critical weaknesses

1- There is not sufficient methodological novelty in this work. Both MPO finetuning with a frozen center and a MPO experts with shared center has been explored in previous work. MPOE (which the paper cites) includes a discussion of frozen central tensor in section 3.2: "In particular, when pb is equal to 1, it means that the parameters of the central tensor are frozen for each input of the data. The computational cost of central tensor update can be also reduced with this trick." This fact makes MPOE a direct baseline of this work. OperA should compare with MPOE and explain what might be the source of potential improved performance over MPOE. The authors treat this work a not an analysis paper. The paper is positioned as a new method so novelty is a concern. And there is not even enough analytical insights provided in the paper.

2- There doesn't seem to be significant practical advantage in this work. The average accuracy improvement over the baseline are very modest. More importantly, Table 9 shows that the peak VRAM usage of OperA is more than MixLoRA. The main goal of using PEFT in any work (including Mixture of PEFT experts) is memory savings. The advantage of this paper is therefore only in case where storage space is a bottleneck (as I mentioned in the strengths), which are not common. Besides the paper is motivated around memory usage, but the evidence does not support the claim. The memory improvements are over the naive implementation of OperA rather than other baselines.

3- Since the accuracy values are very close, the paper should present more evidence (more models, baselines, and benchmarks) of its performance advantage. From table 10, LoRAMOE seems to be a stronger baseline, so that must be included in the multi-seed comparisons.

Questions for authors

In addition to the questions and required clarifications in the weakness section:

1- Can you add a comparison with a single expert MPO baseline, so readers can see how much improvement is the result of MPO and how much from multiple experts?

2- Since the main advantage seems to be in lora storage requirements, can you provide a study where you increase the number of experts and compare against the strongest baseline to see whether performance improvements persist at large number of experts?

3- Can you provide results on more tasks? The number of tasks considered in this work seems limited compared to other PEFT papers.

6 comments

We thank the Reviewer for recognizing that OperA reaches comparable performance with fewer trainable parameters, and for appreciating our ablations.

Following the Reviewer's suggestions, we now clarify both the methodological and practical contributions of this work, especially with respect to other MPO baselines. We also provide a new optimized implementation of OperA that closes the performance gap with additive baselines, and include new benchmarks and ablations.

We address each point in detail below.


W1.1 Methodological novelty

We thank the Reviewer for raising this point, which allows us to distinguish OperA from other MPO methods more precisely. While OperA shares with MPOE the decomposition into central and auxiliary tensors, this previous work does not address the question at the core of OperA: parameter-efficient finetuning of LLMs.

MPOE scales the capacity of MoEs with fewer total parameters, on small (<1B parameters) models, as required by its high computational expense. Indeed, MPOE "fine-tunes all parameters of the model" (Sec. 4.1 of [2]), and trains the shared center. The frozen center (pb=1p_b = 1) is mentioned in a single sentence as a potential computational saving, and is never evaluated. Moreover, the MoE layer of MPOE, y=∑iG(x)iEi(x)y = \sum_i G(x)_i E_i(x) (Eq. 1), has no residual path through the pretrained weights.

Specifically, OperA distinguishes itself in four ways (stated here briefly, see the General response for details):

  1. Objective: OperA does PEFT MoEfication of large frozen LLMs, up to 32B parameters;
  2. Architecture: OperA keeps the pretrained weights on the residual path and initializes every expert such that the model is preserved exactly at initialization, while incorporating all the best-performing practices from the PEFT literature;
  3. Rank analysis: we provide the stable rank analysis of PEFT models, now further enhanced by a theoretical argument about the stable rank of the updates (added in Section 3);
  4. Efficient implementation: we provide a new efficient implementation that exploits the frozen center by design, with a >3x throughput improvement over naive contraction and using less VRAM than additive adapters (added in Section 3.1);

Beyond the conceptual distance, following the Reviewer's suggestion we now benchmark MPOE directly both in its original trainable core version and a frozen core version based on the suggestions in the original paper. We add the full results in Appendix D.11 and report a summary below.

ModelAdapterCenterTrainable params ↓Peak memory (GiB) ↓Time per sample (ms) ↓Avg. acc. ↑
Qwen2.5-3BMPOETrained (pb=0.5p_b = 0.5)2.51B96.347.179.81
MPOEFrozen (pb=1p_b = 1)74M69.045.880.45
OperAFrozen74M64.423.282.79
Llama3.2-3BMPOETrained (pb=0.5p_b = 0.5)2.17B84.042.475.36
MPOEFrozen (pb=1p_b = 1)59M60.439.075.84
OperAFrozen59M56.523.077.46

As can be seen, OperA is the most accurate variant on both models, with 34x to 37x fewer trainable parameters than MPOE, and half its time per sample. Our MPOE baseline finetunes the FFNs only (while the rest is kept PEFT as with OperA). This is because full finetuning is too expensive: training the FFNs alone already requires 2.17B to 2.51B trainable parameters and 84.0 to 96.3 GiB of peak memory at batch 128 on the 3B models, while 7B and larger models do not fit in memory.

W1.2 Analytical insight

We thank the Reviewer for allowing us to clarify and improve upon our analysis of the effects of multiplicative experts. In the manuscript we argue that OperA, leveraging its multiplicative nature, provides a notably higher-dimensional subspace, as measured by the stable rank, over which adaptation can occur during finetuning over additive adapters. We show that this is the case in Figure 3, where OperA achieves >100x stable rank in its expert updates, corroborating our hypothesis. We note that this analysis is a central part of our contribution, and that it is absent from previous MPO-based works to the best of our knowledge. Following the Reviewer's suggestions, we now highlight it more clearly in the Introduction and strengthen it in three ways.

  1. Insights over previous MPO work: no previous MPO work analyzes the rank of the update, nor compares with low-rank adapters. On the contrary, MPOP [1] considers MPO "a special low-rank approximation method", and argues that, as "most of the information in an original matrix will be concentrated in the central tensor", "the overall variations are small" when only the auxiliary tensors are tuned. MPOE [2] states that "only tuning the auxiliary tensors (keeping the central tensor fixed) can lead to the same effect as tuning the whole matrix", but provides no derivation. Our analysis shows the opposite of MPOP's view: the update obtained by tuning only the auxiliary tensors is not low-rank, but full-rank, with a stable rank orders of magnitude higher than that of additive adapters. Among non-MPO adapters, HiRA reaches a related conclusion for the Hadamard product with the pretrained weights. In contrast, OperA obtains its high rank through the MPO structure of each expert, and, to the best of our knowledge, is the first to establish and measure this property for MoEs.

  2. A new theoretical argument: while the measurements of Figure 3 already provided an empirical evaluation of our insight, we now formalize it and add it as Proposition 1 in Section 3. Writing the MPO as a sum over the bond index of an auxiliary core, W=∑αL(α)⊗K(α)W = \sum_\alpha L^{(\alpha)} \otimes K^{(\alpha)}, where K(α)K^{(\alpha)} is the contraction of the remaining frozen cores, an update of one slice gives ΔW=ΔL(α)⊗K(α)\Delta W = \Delta L^{(\alpha)} \otimes K^{(\alpha)}, and therefore rank(ΔW)=rank(ΔL(α))⋅rank(K(α))\mathrm{rank}(\Delta W) = \mathrm{rank}(\Delta L^{(\alpha)}) \cdot \mathrm{rank}(K^{(\alpha)}), with the same product rule for the stable rank. In other words, the rank of OperA's update is inherited from the pretrained weights through the frozen cores, while a LoRA update with pp parameters has rank at most p/(M+N)p/(M+N). This also provides the mechanism behind the statement of MPOE. We verify it on the real MLP weights of Llama3-8B and Qwen2.5-7B (three layers, all MLP projections): the product rule holds exactly on all 18 matrices, and an update of the auxiliary core next to the center produces a full-rank update with a stable rank of 715 to 1,710 (Llama3-8B, 50,176 parameters) and 96 to 1,714 (Qwen2.5-7B, 65,536 parameters), whereas a LoRA with the same number of parameters is limited to rank 2.

  3. The stable rank of every trained update: we also extend Figure 3, which tracked three layers of Llama3-8B, to every MLP projection of every layer of Llama3.1-8B and Qwen2.5-7B, and now include the parameter-matched non-MoE LoRA. We report the results in Appendix D.10 and provide a summary here for ease of reading:

ModelAdapterTrainable params per update ↓Rank of the update ↑Stable rank of the update ↑
Llama3.1-8BLoRA (rank 40)737,280402.3
MixLoRA expert (rank 12)221,184121.7
OperA expert115,8404096 (full)1,147
Qwen2.5-7BLoRA (rank 44)991,232442.0
MixLoRA expert (rank 12)270,336121.5
OperA expert131,1523584 (full)1,146

As can be seen, the updates of OperA's experts have an about 500x to 760x higher stable rank than those of additive adapters, with 2x to 8x fewer trainable parameters per update. Moreover, the additive updates are concentrated in very few directions (stable rank of 1.5 to 2.3, even with rank 40 or 44), confirming that additive adapters use only a small fraction of their nominal rank.

W2. Practical advantage

As the Reviewer correctly mentions, the main focus of OperA is in the reduction of trainable parameters. At the same time, following the Reviewer's suggestion, we now provide a new optimized implementation of OperA. OperA now directly benefits from its smaller optimizer states, thanks to its fewer trainable parameters, leading to lower VRAM consumption than additive adapters.

We compare OperA with MixLoRA on one Nvidia H200, with MoE-PEFT's full training step (forward, recomputation, backward, and optimizer step). We provide a summary here for ease of reading:

ModelParams ↓ (OperA / MixLoRA)Peak memory (GiB) ↓ (OperA / MixLoRA)Time per sample (ms) ↓ (OperA / MixLoRA)
Llama3.2-3B92M / 131M56.5 / 56.723.0 / 27.2
Qwen2.5-3B74M / 188M64.4 / 64.823.2 / 32.5
Llama3-8B173M / 241M68.0 / 68.547.0 / 48.9
Qwen2.5-7B192M / 253M74.9 / 75.443.4 / 46.9
Qwen2.5-14B200M / 376M95.5 / 96.586.7 / 82.8
Phi4-14B182M / 336M73.9 / 74.883.5 / 77.0
Qwen2.5-32B343M / 841M134.0 / 135.6189.5 / 171.3

OperA uses less peak memory than MixLoRA on all 7 models, and is faster on the 3B to 8B models (by 4% to 40%), and 5% to 11% slower on the 14B to 32B models. Beyond peak memory, the practical benefits of OperA are: (i) 24% to 61% fewer trainable parameters than additive adapters, which directly reduces optimizer-state memory and the storage of the adapters, especially when many task-specific adapters are kept for the same base model; and (ii) the highest accuracy on 7 of the 8 models at the same or lower parameter budget (Table 2), and on every model in the multi-task setting (Table 3).

W3. Further evidence

In the original manuscript we provide a large amount of experiments and benchmarks to robustly determine the performance of OperA, including 8 single-task benchmarks, 1 multi-task benchmark, 8 models from 3 families with sizes from 3B to 32B parameters, 5 baselines (now 9 with DoRA, HiRA, and MPOE), and ablations of each component.

Following the Reviewer's suggestion, we now include LoRAMoE in the multi-seed comparison of Table 10, with the same parameter budget as MixLoRA. We report the extended Table 10 below (accuracy, mean ± sd over seeds).

ModelAdapterParams ↓ARC-cARC-ePIQAOBQABoolQHSWGAvg. ↑
Llama3.1-8BMixLoRA106M77.84 ± 0.4487.03 ± 0.8686.20 ± 2.0085.60 ± 1.4875.25 ± 0.5194.85 ± 0.4084.13 ± 0.7884.41 ± 0.14
LoRAMoE111M78.33 ± 1.1387.26 ± 0.3887.87 ± 0.9984.87 ± 0.3174.51 ± 0.6395.36 ± 0.3984.82 ± 0.0984.66 ± 0.40
OperA95M77.97 ± 0.1986.81 ± 0.8288.03 ± 0.6285.65 ± 0.3875.24 ± 0.4295.66 ± 0.3283.41 ± 2.3484.68 ± 0.34
Qwen2.5-7BMixLoRA111M87.14 ± 0.7792.09 ± 0.4488.96 ± 0.4391.75 ± 1.9174.12 ± 1.0594.93 ± 0.4585.52 ± 0.2887.79 ± 0.29
LoRAMoE116M87.54 ± 0.6592.40 ± 0.1889.83 ± 0.1192.07 ± 0.5074.30 ± 0.6995.37 ± 0.2285.08 ± 0.4488.08 ± 0.09
OperA93M87.56 ± 0.5792.08 ± 0.4590.62 ± 0.2293.00 ± 0.7874.09 ± 1.0495.55 ± 0.2584.10 ± 0.8088.14 ± 0.24

The Reviewer is indeed correct that LoRAMoE is a strong baseline: it is stronger than MixLoRA and closest to OperA, while using 17% to 25% more trainable parameters. However, LoRAMoE uses a larger computation budget: it routes every token densely to all 8 experts (top-8), while OperA and MixLoRA evaluate only the top-2 experts per token, so that LoRAMoE performs 4x more expert evaluations per token (added to Section 4). Therefore, OperA remains Pareto-optimal in this comparison. The same holds across all 8 models: against MixLoRA and LoRAMoE with the same or a larger parameter budget, no adapter is either smaller or more accurate than OperA on 7 of the 8 models.

Reviewer_PvVMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the concept of Matrix-Product Operators (MPO)-style decomposition to represent original big parameter matrices and turn each into a tensor contract form with a frozen central matrix shared by different experts as well as several trainable left/right matrix (freely trained by different experts). According to the theoretical analysis provided by the authors, such form can represent update matrix with higher stable ranks with fewer or same number of trainable parameters in the LLM layers. Experimental results show that the proposed method can achieve competitive performance compared with existing methods while saving trainable parameters, yet with higher computational complexities against additive counterparts.

Critical strengths
  1. This paper brings a new idea of LoRA-style reparameterization for parameter-efficient MoE finetuning, which is significantly motivated by tensor contraction or decomposition. The proposed structure not only provides convenience for shared knowledge between such experts, but also ensure higher expressiveness (such as stable ranks) with strictly controllable number of trainable parameters.

  2. In addition to theoretical insights or technical innovations, the authors provide some improvements for tensor contraction, which can practically reduce memory usage compared with the naive implementation.

Critical weaknesses
  1. While the proposed method can save some adapter parameter storage, it still suffers from higher end-to-end space complexities (requiring more than 10%10\% space compared with MixLoRA). If little justification is provided, readers would be confused with the practical benefits of this technique, despite its theoretical beauty.

  2. Sometimes, the proposed technique runs much slower than the counterparts, e.g., on Llama3-8B (OperA vs MixLoRA).

Questions for authors

See the Weakness.

3 comments

We thank the Reviewer for recognizing the novelty of our tensor-contraction approach to parameter-efficient MoE finetuning, and for appreciating the theoretical beauty of our work.

Following the Reviewer's suggestions, we have developed a new optimized implementation of OperA, compare its runtime and memory usage directly with additive adapters on all models, and clarify the practical benefits of OperA.

We address each point in detail below.


W1. End-to-end computational complexity

Following the Reviewer's suggestion, we provide a new optimized implementation of OperA in TileLang (see Section 3.1 for full details). It reconstructs the weights of each expert once per step, processes all the tokens routed to the experts with a single grouped matrix multiplication, and folds the frozen pretrained weights into the reconstruction, so that they are not stored twice. OperA now directly benefits from its smaller optimizer states, thanks to its fewer trainable parameters, leading to lower VRAM consumption than additive adapters.

We compare OperA with MixLoRA on one Nvidia H200, with MoE-PEFT's full training step (forward, recomputation, backward, and optimizer step). We provide a summary here for ease of reading:

ModelParams ↓ (OperA / MixLoRA)Peak memory (GiB) ↓ (OperA / MixLoRA)Time per sample (ms) ↓ (OperA / MixLoRA)
Llama3.2-3B92M / 131M56.5 / 56.723.0 / 27.2
Qwen2.5-3B74M / 188M64.4 / 64.823.2 / 32.5
Llama3-8B173M / 241M68.0 / 68.547.0 / 48.9
Qwen2.5-7B192M / 253M74.9 / 75.443.4 / 46.9
Qwen2.5-14B200M / 376M95.5 / 96.586.7 / 82.8
Phi4-14B182M / 336M73.9 / 74.883.5 / 77.0
Qwen2.5-32B343M / 841M134.0 / 135.6189.5 / 171.3

OperA uses less peak memory than MixLoRA on all 7 models, and is faster on the 3B to 8B models (by 4% to 40%), and 5% to 11% slower on the 14B to 32B models. Beyond peak memory, the practical benefits of OperA are: (i) 24% to 61% fewer trainable parameters than additive adapters, which directly reduces optimizer-state memory and the storage of the adapters, especially when many task-specific adapters are kept for the same base model; and (ii) the highest accuracy on 7 of the 8 models at the same or lower parameter budget (Table 2), and on every model in the multi-task setting (Table 3)

W2. Runtime

The Reviewer is correct in that the naive implementation of OperA is slower than the additive counterparts, as each expert must be reconstructed at every step. Our new TileLang implementation addresses exactly this: it is now >3x faster than the naive implementation. As a result, OperA is now faster than MixLoRA on the 3B to 8B models, and in particular 4% faster on Llama3-8B mentioned by the Reviewer, while training 24% to 61% fewer parameters (see the Table in W1).

We have added the new implementation in Section 3.1 and Appendix C.

We greatly appreciate the time and effort dedicated by the Reviewer to improve our work. As the discussion period is coming to an end, we kindly ask the Reviewer to acknowledge whether we have successfully addressed their outstanding concerns. We remain available to further discuss any remaining question or weakness in our work.

Thanks for the authors' effort for the rebuttal. I hope the supplementary material can be added into the paper. And, I will maintain my original positive rating.

Reviewer_iFKAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OperA, a parameter-efficient mixture-of-experts method based on matrix product operator (MPO) decompositions. Each expert shares a frozen central tensor derived from pretrained weights and learns expert-specific auxiliary tensors. This permits high-rank weight updates with relatively few trainable parameters.

Critical strengths
  1. OperA achieves competitive accuracy with fewer adapter parameters. The experiments cover several model families, sizes from 3B to 32B, and both single-task and multitask training. In Table 2, Qwen2.5-32B reaches 90.22 with 343M trainable parameters, compared with LoRAMoE’s 90.09 with 439M.

  2. The authors measure stable rank on the actual updates, W−EiW-E_i, which makes the comparison meaningful. They also test parameter budgets, additive-adapter ranks, initialization, and contraction optimizations.

Critical weaknesses
  1. The paper does not clearly establish its contribution over earlier MPO methods. Liu et al. (2021) already fine-tune auxiliary MPO tensors while fixing the central tensor. MPOE also describes a pb=1p_b=1 setting that freezes the shared central tensor.

  2. The comparisons do not establish OperA’s advantage over strong non-MoE PEFT methods. The paper compares accuracy mainly against LoRA-based MoE methods and includes only vanilla LoRA as a non-MoE baseline. Methods such as DoRA, HiRA, PiSSA, MiLora, Lora-one already support joint training on Commonsense170K and evaluation across eight tasks.

[1] Liu et al. (2021), Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

[2] Gao et al. (2022) Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models

Questions for authors
  1. How does OperA differ from MPOE’s frozen-center setting with a frozen backbone?

  2. Equation (7) assigns equal weights to the pretrained and expert branches. However, any combination (1λ)W+λ∑iTi(x)Ei(1 \lambda)W+\lambda\sum_iT_i(x)E_i preserves the pretrained model at initialization. Why choose λ=0.5\lambda=0.5? How sensitive are the results to this choice, and would a learnable weight improve performance?

3 comments

We thank the Reviewer for highlighting the thoroughness of our benchmarks and the usefulness of our method, and for appreciating our approach to the analysis of the stable rank.

Following the Reviewer's suggestions, we now clarify our contributions with respect to MPOP and MPOE, and compare directly with MPOE in both its default and its untested frozen-center settings. Moreover, we also broaden our comparisons to include LoRA, DoRA, and HiRA on the Commonsense170K benchmark.

We address each point in detail below.


W1. Contribution over other MPO methods

We thank the Reviewer for raising this point, which allows us to distinguish OperA from other MPO methods more precisely. While OperA shares with MPOP [1] and MPOE [2] the decomposition into central and auxiliary tensors, neither work addresses the question at the core of OperA: parameter-efficient finetuning of LLMs.

MPOP is mainly a compression method, which reduces the size of small (<1B parameters) BERT models by truncating the bond dimensions of the MPO and then recovers accuracy by training the auxiliary cores. In contrast, we keep the MPOs at their full bond size to ensure lossless initialization (as in LoRA with B=0), and train the auxiliary cores for parameter-efficient fine-tuning. Moreover, MPOP uses a single MPO per matrix, without experts, and it is not compared with any PEFT adapter. Finally, it describes the MPO as "a special low-rank approximation method", whereas we show that the update of a frozen-center MPO is in fact high-rank (see the General response and W1.2 of Reviewer a6D7).

MPOE scales the capacity of MoEs with fewer total parameters, again on small (<1B parameters) models, as required by its high computational expense. Indeed, MPOE "fine-tunes all parameters of the model" (Sec. 4.1 of [2]), and trains the shared center. The frozen center (pb=1p_b = 1) is mentioned in a single sentence as a potential computational saving, and is never evaluated. Moreover, the MoE layer of MPOE, y=∑iG(x)iEi(x)y = \sum_i G(x)_i E_i(x) (Eq. 1), has no residual path through the pretrained weights.

Specifically, OperA distinguishes itself in four ways (stated here briefly, see the General response for details):

  1. Objective: OperA does PEFT MoEfication of large frozen LLMs, up to 32B parameters;
  2. Architecture: OperA keeps the pretrained weights on the residual path and initializes every expert such that the model is preserved exactly at initialization, while incorporating all the best-performing practices from the PEFT literature;
  3. Rank analysis: we provide the stable rank analysis of PEFT models, now further enhanced by a theoretical argument about the stable rank of the updates (added in Section 3);
  4. Efficient implementation: we provide a new efficient implementation that exploits the frozen center by design, with a >3x throughput improvement over naive contraction and using less VRAM than additive adapters (added in Section 3.1);

Beyond the methodological distance, following the Reviewer's suggestion we now benchmark MPOE directly both in its original trainable core version and a frozen core version based on the suggestions in the original paper. We add the full results in Appendix D.11 and report a summary below.

ModelAdapterCenterTrainable params ↓Peak memory (GiB) ↓Time per sample (ms) ↓Avg. acc. ↑
Qwen2.5-3BMPOETrained (pb=0.5p_b = 0.5)2.51B96.347.179.81
MPOEFrozen (pb=1p_b = 1)74M69.045.880.45
OperAFrozen74M64.423.282.79
Llama3.2-3BMPOETrained (pb=0.5p_b = 0.5)2.17B84.042.475.36
MPOEFrozen (pb=1p_b = 1)59M60.439.075.84
OperAFrozen59M56.523.077.46

As can be seen, OperA is the most accurate variant on both models, with 34x to 37x fewer trainable parameters than MPOE, and half its time per sample. Our MPOE baseline finetunes the FFNs only (while the rest is kept PEFT as with OperA). This is because full finetuning is too expensive: training the FFNs alone already requires 2.17B to 2.51B trainable parameters and 84.0 to 96.3 GiB of peak memory at batch 128 on the 3B models, while 7B and larger models do not fit in memory.

W2. Comparison with non-MoE PEFT

The Reviewer points to a relevant discussion in the literature, whether MoEfication provides clear advantages over plain (i.e., single expert) PEFT. This question has been considered in many recent papers [3] [4], and MoEs have shown to consistently outperform alternatives. In the context of our baselines, MixLoRA compares against single LoRA and DoRA adapters with the same parameter budget, and outperforms them by 5.8 and 2.3 points on LLaMA-2 7B. Therefore, OperA builds on the established literature and focuses on the MoEfication case, where MPO's parameter efficiency is most prevalent. At the same time, we now also perform a single-expert ablation (see Reviewer a6D7, Q1) that isolates the effects of the multiplicative interactions from the effect of the MoE: a single OperA expert, which is itself a non-MoE adapter, already matches the 8-expert additive expert MoE with about 1/11 to 1/13 of their trainable parameters.

Finally, following the Reviewer's suggestion, we now train OperA and the non-MoE adapters LoRA, DoRA, and HiRA jointly on Commonsense170K with Llama3-8B, and evaluate them on its eight commonsense reasoning tasks, following the training and evaluation recipe of DoRA. We provide a summary of the results here (see Appendix D.9 for the full details):

AdapterParams ↓BoolQPIQASIQAHellaS.WinoG.ARC-eARC-cOBQAAvg. ↑
LoRA105M67.0082.7578.1586.5280.3579.0065.7878.2077.22
DoRA106M72.6388.5780.9195.2086.1990.1980.1286.6085.05
HiRA105M73.1288.3080.0995.6085.6491.2979.6987.2085.12
MixLoRA181M73.2487.3881.3794.7485.8788.9778.9284.0084.31
LoRAMoE186M74.4388.9680.0995.2485.4091.3779.6185.4085.06
OperA95M74.1089.1781.3295.3785.7989.9880.3886.2085.29

As can be seen, OperA is more accurate than LoRA, DoRA, HiRA, MixLoRA, and LoRAMoE with fewer trainable parameters than all of them. Therefore, OperA remains on the Pareto frontier of accuracy and trainable parameters also against strong non-MoE adapters.

Q1. MPOE's frozen-center setting

As mentioned in W1., MPOE only refers to this setting in passing, and we now instantiate it to form a new baseline. We follow MPOE's setting wherever it is specified (8 experts, five cores, experts from the decomposition of the pretrained FFN). With a frozen center and a frozen backbone, the MPOE layer is y=∑iT(x)iEixy = \sum_i T(x)_i E_i x (Eq. 1 of MPOE), without a residual path. OperA instead keeps the pretrained weights as a residual path, y=λWx+(1−λ)∑iT(x)iEixy = \lambda W x + (1-\lambda) \sum_i T(x)_i E_i x with λ=1/2\lambda = 1/2. Therefore, MPOE with a frozen center is a special case of OperA with λ=0\lambda = 0, everything else being equal. Importantly, MPOE provides no optimized implementation for this setting. On Llama3.2-3B, the naive contraction is 1.7x slower than our optimized implementation at batch 128 (39.0 against 23.0 ms per sample), and needs more peak memory (60.4 against 56.5 GiB). The results we obtain here are extremely relevant also for Q2, so we refer to the results Table in Q2 for a summary. In particular, it is clear that λ=0\lambda = 0 is a suboptimal choice for this setting, with OperA outperforming frozen-core MPOE in all cases.

Q2. Choice of λ\lambda

We thank the Reviewer for pointing out this learnable λ\lambda optimization. Specifically, as every expert is initialized to WW, any λ\lambda preserves the pretrained model at initialization. We chose λ=1/2\lambda = 1/2 as the analogue of setting B=0B = 0 in LoRA, with the pretrained path and the experts on the same scale. Following the Reviewer's suggestion, we now evaluate λ∈{0,0.25,0.5,0.75}\lambda \in \{0, 0.25, 0.5, 0.75\} and a learnable λ\lambda (one per MoE layer, initialized at 0.5). We report the summary Table below:

λ\lambda0 (MPOE, pb=1p_b = 1)0.250.5 (OperA)0.75Learnable
Llama3.1-8B82.4882.6682.8682.1382.56
Qwen2.5-7B78.9287.0887.7687.5387.16
Qwen2.5-3B80.4582.4382.7982.7782.54
Llama3.2-3B75.8476.3677.4675.9476.07

As can be seen, the results are not particularly sensitive to λ\lambda in the range 0.25 to 0.75. Only λ=0\lambda = 0, i.e. removing the pretrained path as in MPOE, is notably worse. We therefore keep λ=0.5\lambda = 0.5, and report the learnable variant in Appendix D.8.

[1] Liu, Peiyu, et al. "Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators." ACL (2021).

[2] Gao, Ze-Feng, et al. "Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models." COLING (2022).

[3] Zadouri, Ted, et al. "Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning." ICLR (2024).

[4] Cai, Weilin, et al. "A Survey on Mixture of Experts in Large Language Models." IEEE Transactions on Knowledge and Data Engineering (2024).

Reviewer_iFKAOpenReview ↗

Thank you for the detailed response and additional experiments

Thank you for the additional experiments. The frozen-center MPOE comparison clarifies OperA’s added value, and the Commonsense170K results address my concern about strong non-MoE baselines. The mixing-weight ablation also answers my question about the equal-weight choice. These additions substantially address my main concerns, and I will raise my rating to 3.

Comments on the paper

2 comments

We are grateful to all Reviewers for their dedicated service and for providing thoughtful and interesting feedback. We have taken all the reviews to heart and have updated the manuscript to reflect the changes made following the Reviewers' suggestions, with changes highlighted in blue.

Most importantly, our new optimized implementation turns the parameter efficiency of OperA into a concrete practical advantage: OperA now trains 24% to 61% fewer parameters than MixLoRA while using less peak memory on all tested models, it is faster on the 3B to 8B models, and it is Pareto-optimal in accuracy and trainable parameters on 7 of the 8 models against parameter-matched additive MoE adapters.

  1. Novelty over MPO-based methods. As suggested by Reviewers iFKA and a6D7, we clarify the relation of OperA with MPOP and MPOE. The three share only the MPO decomposition into central and auxiliary tensors, which MPOP uses to compress BERT models and MPOE to scale the experts of small MoEs by training a shared center. OperA is the first to use a frozen MPO for PEFT MoEfication of LLMs, and the first to show that its updates are full-rank, contrary to the low-rank view of MPOP:
MPOPMPOEOperA
Central tensorFrozen, truncated (lossy)Shared, trainedShared, frozen, lossless
Rank of the update"low-rank approximation"NoneFull (Prop. 1), >500x stable rank
FinetuningLossy compression recoveryFullPEFT
Trainable params, 3B (ours)n/a2.17B to 2.51B59M to 74M
Largest modelBERT (<1B)T5 (<1B)Llama, Phi, Qwen, up to 32B
PEFT baselinesNoneHyperformerLoRA, DoRA, HiRA, 4 MoE adapters
Optimized implementationNoNoYes (>3x faster)

We now also benchmark MPOE directly. With a trained center, MPOE needs 34x to 37x more trainable parameters, 1.5x more peak memory, and twice the time per sample, and is still less accurate than OperA. With a frozen center, which MPOE mentions but never evaluates, MPOE is a special case of OperA with λ=0\lambda = 0, and is less accurate than OperA on all four tested models.

  1. Full-rank updates with a fraction of the parameters. As suggested by Reviewer a6D7, we strengthen the analysis at the core of our work: multiplicative experts adapt the model over a far higher-dimensional subspace than additive ones. We now measure the stable rank of every trained MLP update of Llama3.1-8B and Qwen2.5-7B:
UpdateParams per update ↓Rank ↑Stable rank ↑
LoRA (parameter-matched)737K to 991K40 to 442.0 to 2.3
MixLoRA expert221K to 270K121.5 to 1.7
OperA expert116K to 131KFull (3,584 to 4,096)1,146 to 1,147

OperA's updates have a 500x to 760x higher stable rank than those of additive adapters, with 1.9x to 7.6x fewer trainable parameters per update. Additive updates concentrate on about two directions, even at rank 40 or 44, while OperA's span the full space. We add Proposition 1 (Section 3) to explain why: the rank of an OperA update is inherited from the pretrained weights through the frozen cores, whereas a LoRA update is explicitly limited by its rank.

  1. Optimized implementation. As suggested by Reviewers PvVM and a6D7, we provide a new optimized implementation of OperA in TileLang. This implementation is more than 3x faster than the naive contraction and uses considerably less memory. OperA now uses less memory than additive adapters on all models, is faster than MixLoRA on the 3B to 8B models (by 4% to 40%), and only 5% to 11% slower on the 14B to 32B models, due to the increase in FLOPs, which could be mitigated with quantized deployment.

  2. More benchmarks. As suggested by Reviewers iFKA and a6D7, we now evaluate 10 single tasks, 8 tasks in a multi-task setting, and Commonsense170K on 8 models from 3 families, from 3B to 32B parameters: this is a notably broader setup than comparable works (Reviewer a6D7, W3).

  • LoRAMoE baseline. We now include LoRAMoE in the multi-seed comparison as requested. OperA remains Pareto-optimal on 7 of the 8 models in the parameter-matched setting: no adapter is either smaller or more accurate, with OperA using 10% to 37% fewer trainable parameters (Reviewer a6D7, W3).
  • Two more tasks. We add SIQA and CommonsenseQA on all 8 models, bringing the total to 10 tasks (Reviewer a6D7, Q3). The new tasks do not change the picture: on the 10-task average OperA is the most accurate on 7 of the 8 models with the same or fewer parameters (Table 2).
  • Commonsense170K. We train OperA, LoRA, DoRA, HiRA, MixLoRA, and LoRAMoE jointly on Commonsense170K with Llama3-8B (Reviewer iFKA, W2): OperA is more accurate than all baselines with fewer trainable parameters.
  • We also ablate λ\lambda (Reviewer iFKA, Q2) and the number of experts, from 1 to 32 (Reviewer a6D7, Q1/2).

We are convinced that our work is now better and clearer thanks to the Reviewers' feedback, and look forward to engaging in interesting discussions with all the Reviewers.

To all Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

we are deeply grateful to all Reviewers for the interesting and thoughtful discussions over the past two weeks. We would like to individually thank each Reviewer, specifically:

  • Reviewer iFKA for highlighting the added value of OperA, together with the importance of our stable rank analysis and the thoroughness of our benchmarks;
  • Reviewer PvVM for recognizing the significance of our MPO to the field and both our theoretical insights and technical innovation;
  • Reviewer a6D7 for finding that OperA is a practical solution, backed by thorough experimental analysis;

We are especially delighted to read that our efforts during the rebuttal phase to clarify the role of OperA, its theoretical novelty, and its experimental thoroughness have been highly appreciated by the Reviewers, who now all recommend our work for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.