- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OperA, a PEFT MoE method that uses TT-SVD to decompose the weights of a model into Matric Product Operator (MPO) tensors. It then freezes the shared core of the weight and trains exxpert-specific auxiliary tensors. The authors evaluate their proposed approach on a number or LLMS and tasks, and show that OperA can get competitive accuracy with fewer trainable parameters than the evaluated LoRA-based baselines.
Critical strengths
1- The paper provides some evidence of comparable performance with the baselines with fewer trainable parameters. In the scenarios where storage space for LoRA parameters is a bottleneck, this direction is practical.
2- There are some useful ablations in the paper.
Critical weaknesses
1- There is not sufficient methodological novelty in this work. Both MPO finetuning with a frozen center and a MPO experts with shared center has been explored in previous work. MPOE (which the paper cites) includes a discussion of frozen central tensor in section 3.2: "In particular, when pb is equal to 1, it means that the parameters of the central tensor are frozen for each input of the data. The computational cost of central tensor update can be also reduced with this trick." This fact makes MPOE a direct baseline of this work. OperA should compare with MPOE and explain what might be the source of potential improved performance over MPOE. The authors treat this work a not an analysis paper. The paper is positioned as a new method so novelty is a concern. And there is not even enough analytical insights provided in the paper.
2- There doesn't seem to be significant practical advantage in this work. The average accuracy improvement over the baseline are very modest. More importantly, Table 9 shows that the peak VRAM usage of OperA is more than MixLoRA. The main goal of using PEFT in any work (including Mixture of PEFT experts) is memory savings. The advantage of this paper is therefore only in case where storage space is a bottleneck (as I mentioned in the strengths), which are not common. Besides the paper is motivated around memory usage, but the evidence does not support the claim. The memory improvements are over the naive implementation of OperA rather than other baselines.
3- Since the accuracy values are very close, the paper should present more evidence (more models, baselines, and benchmarks) of its performance advantage. From table 10, LoRAMOE seems to be a stronger baseline, so that must be included in the multi-seed comparisons.
Questions for authors
In addition to the questions and required clarifications in the weakness section:
1- Can you add a comparison with a single expert MPO baseline, so readers can see how much improvement is the result of MPO and how much from multiple experts?
2- Since the main advantage seems to be in lora storage requirements, can you provide a study where you increase the number of experts and compare against the strongest baseline to see whether performance improvements persist at large number of experts?
3- Can you provide results on more tasks? The number of tasks considered in this work seems limited compared to other PEFT papers.
6 comments
W1.2 Analytical insight
We thank the Reviewer for allowing us to clarify and improve upon our analysis of the effects of multiplicative experts. In the manuscript we argue that OperA, leveraging its multiplicative nature, provides a notably higher-dimensional subspace, as measured by the stable rank, over which adaptation can occur during finetuning over additive adapters. We show that this is the case in Figure 3, where OperA achieves >100x stable rank in its expert updates, corroborating our hypothesis. We note that this analysis is a central part of our contribution, and that it is absent from previous MPO-based works to the best of our knowledge. Following the Reviewer's suggestions, we now highlight it more clearly in the Introduction and strengthen it in three ways.
-
Insights over previous MPO work: no previous MPO work analyzes the rank of the update, nor compares with low-rank adapters. On the contrary, MPOP [1] considers MPO "a special low-rank approximation method", and argues that, as "most of the information in an original matrix will be concentrated in the central tensor", "the overall variations are small" when only the auxiliary tensors are tuned. MPOE [2] states that "only tuning the auxiliary tensors (keeping the central tensor fixed) can lead to the same effect as tuning the whole matrix", but provides no derivation. Our analysis shows the opposite of MPOP's view: the update obtained by tuning only the auxiliary tensors is not low-rank, but full-rank, with a stable rank orders of magnitude higher than that of additive adapters. Among non-MPO adapters, HiRA reaches a related conclusion for the Hadamard product with the pretrained weights. In contrast, OperA obtains its high rank through the MPO structure of each expert, and, to the best of our knowledge, is the first to establish and measure this property for MoEs.
-
A new theoretical argument: while the measurements of Figure 3 already provided an empirical evaluation of our insight, we now formalize it and add it as Proposition 1 in Section 3. Writing the MPO as a sum over the bond index of an auxiliary core, , where is the contraction of the remaining frozen cores, an update of one slice gives , and therefore , with the same product rule for the stable rank. In other words, the rank of OperA's update is inherited from the pretrained weights through the frozen cores, while a LoRA update with parameters has rank at most . This also provides the mechanism behind the statement of MPOE. We verify it on the real MLP weights of Llama3-8B and Qwen2.5-7B (three layers, all MLP projections): the product rule holds exactly on all 18 matrices, and an update of the auxiliary core next to the center produces a full-rank update with a stable rank of 715 to 1,710 (Llama3-8B, 50,176 parameters) and 96 to 1,714 (Qwen2.5-7B, 65,536 parameters), whereas a LoRA with the same number of parameters is limited to rank 2.
-
The stable rank of every trained update: we also extend Figure 3, which tracked three layers of Llama3-8B, to every MLP projection of every layer of Llama3.1-8B and Qwen2.5-7B, and now include the parameter-matched non-MoE LoRA. We report the results in Appendix D.10 and provide a summary here for ease of reading:
| Model | Adapter | Trainable params per update ↓ | Rank of the update ↑ | Stable rank of the update ↑ |
|---|---|---|---|---|
| Llama3.1-8B | LoRA (rank 40) | 737,280 | 40 | 2.3 |
| MixLoRA expert (rank 12) | 221,184 | 12 | 1.7 | |
| OperA expert | 115,840 | 4096 (full) | 1,147 | |
| Qwen2.5-7B | LoRA (rank 44) | 991,232 | 44 | 2.0 |
| MixLoRA expert (rank 12) | 270,336 | 12 | 1.5 | |
| OperA expert | 131,152 | 3584 (full) | 1,146 |
As can be seen, the updates of OperA's experts have an about 500x to 760x higher stable rank than those of additive adapters, with 2x to 8x fewer trainable parameters per update. Moreover, the additive updates are concentrated in very few directions (stable rank of 1.5 to 2.3, even with rank 40 or 44), confirming that additive adapters use only a small fraction of their nominal rank.
W2. Practical advantage
As the Reviewer correctly mentions, the main focus of OperA is in the reduction of trainable parameters. At the same time, following the Reviewer's suggestion, we now provide a new optimized implementation of OperA. OperA now directly benefits from its smaller optimizer states, thanks to its fewer trainable parameters, leading to lower VRAM consumption than additive adapters.
We compare OperA with MixLoRA on one Nvidia H200, with MoE-PEFT's full training step (forward, recomputation, backward, and optimizer step). We provide a summary here for ease of reading:
| Model | Params ↓ (OperA / MixLoRA) | Peak memory (GiB) ↓ (OperA / MixLoRA) | Time per sample (ms) ↓ (OperA / MixLoRA) |
|---|---|---|---|
| Llama3.2-3B | 92M / 131M | 56.5 / 56.7 | 23.0 / 27.2 |
| Qwen2.5-3B | 74M / 188M | 64.4 / 64.8 | 23.2 / 32.5 |
| Llama3-8B | 173M / 241M | 68.0 / 68.5 | 47.0 / 48.9 |
| Qwen2.5-7B | 192M / 253M | 74.9 / 75.4 | 43.4 / 46.9 |
| Qwen2.5-14B | 200M / 376M | 95.5 / 96.5 | 86.7 / 82.8 |
| Phi4-14B | 182M / 336M | 73.9 / 74.8 | 83.5 / 77.0 |
| Qwen2.5-32B | 343M / 841M | 134.0 / 135.6 | 189.5 / 171.3 |
OperA uses less peak memory than MixLoRA on all 7 models, and is faster on the 3B to 8B models (by 4% to 40%), and 5% to 11% slower on the 14B to 32B models. Beyond peak memory, the practical benefits of OperA are: (i) 24% to 61% fewer trainable parameters than additive adapters, which directly reduces optimizer-state memory and the storage of the adapters, especially when many task-specific adapters are kept for the same base model; and (ii) the highest accuracy on 7 of the 8 models at the same or lower parameter budget (Table 2), and on every model in the multi-task setting (Table 3).
W3. Further evidence
In the original manuscript we provide a large amount of experiments and benchmarks to robustly determine the performance of OperA, including 8 single-task benchmarks, 1 multi-task benchmark, 8 models from 3 families with sizes from 3B to 32B parameters, 5 baselines (now 9 with DoRA, HiRA, and MPOE), and ablations of each component.
Following the Reviewer's suggestion, we now include LoRAMoE in the multi-seed comparison of Table 10, with the same parameter budget as MixLoRA. We report the extended Table 10 below (accuracy, mean ± sd over seeds).
| Model | Adapter | Params ↓ | ARC-c | ARC-e | PIQA | OBQA | BoolQ | HS | WG | Avg. ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Llama3.1-8B | MixLoRA | 106M | 77.84 ± 0.44 | 87.03 ± 0.86 | 86.20 ± 2.00 | 85.60 ± 1.48 | 75.25 ± 0.51 | 94.85 ± 0.40 | 84.13 ± 0.78 | 84.41 ± 0.14 |
| LoRAMoE | 111M | 78.33 ± 1.13 | 87.26 ± 0.38 | 87.87 ± 0.99 | 84.87 ± 0.31 | 74.51 ± 0.63 | 95.36 ± 0.39 | 84.82 ± 0.09 | 84.66 ± 0.40 | |
| OperA | 95M | 77.97 ± 0.19 | 86.81 ± 0.82 | 88.03 ± 0.62 | 85.65 ± 0.38 | 75.24 ± 0.42 | 95.66 ± 0.32 | 83.41 ± 2.34 | 84.68 ± 0.34 | |
| Qwen2.5-7B | MixLoRA | 111M | 87.14 ± 0.77 | 92.09 ± 0.44 | 88.96 ± 0.43 | 91.75 ± 1.91 | 74.12 ± 1.05 | 94.93 ± 0.45 | 85.52 ± 0.28 | 87.79 ± 0.29 |
| LoRAMoE | 116M | 87.54 ± 0.65 | 92.40 ± 0.18 | 89.83 ± 0.11 | 92.07 ± 0.50 | 74.30 ± 0.69 | 95.37 ± 0.22 | 85.08 ± 0.44 | 88.08 ± 0.09 | |
| OperA | 93M | 87.56 ± 0.57 | 92.08 ± 0.45 | 90.62 ± 0.22 | 93.00 ± 0.78 | 74.09 ± 1.04 | 95.55 ± 0.25 | 84.10 ± 0.80 | 88.14 ± 0.24 |
The Reviewer is indeed correct that LoRAMoE is a strong baseline: it is stronger than MixLoRA and closest to OperA, while using 17% to 25% more trainable parameters. However, LoRAMoE uses a larger computation budget: it routes every token densely to all 8 experts (top-8), while OperA and MixLoRA evaluate only the top-2 experts per token, so that LoRAMoE performs 4x more expert evaluations per token (added to Section 4). Therefore, OperA remains Pareto-optimal in this comparison. The same holds across all 8 models: against MixLoRA and LoRAMoE with the same or a larger parameter budget, no adapter is either smaller or more accurate than OperA on 7 of the 8 models.
We thank the Reviewer for recognizing that OperA reaches comparable performance with fewer trainable parameters, and for appreciating our ablations.
Following the Reviewer's suggestions, we now clarify both the methodological and practical contributions of this work, especially with respect to other MPO baselines. We also provide a new optimized implementation of OperA that closes the performance gap with additive baselines, and include new benchmarks and ablations.
We address each point in detail below.
W1.1 Methodological novelty
We thank the Reviewer for raising this point, which allows us to distinguish OperA from other MPO methods more precisely. While OperA shares with MPOE the decomposition into central and auxiliary tensors, this previous work does not address the question at the core of OperA: parameter-efficient finetuning of LLMs.
MPOE scales the capacity of MoEs with fewer total parameters, on small (<1B parameters) models, as required by its high computational expense. Indeed, MPOE "fine-tunes all parameters of the model" (Sec. 4.1 of [2]), and trains the shared center. The frozen center (pb=1) is mentioned in a single sentence as a potential computational saving, and is never evaluated. Moreover, the MoE layer of MPOE, y=∑iG(x)iEi(x) (Eq. 1), has no residual path through the pretrained weights.
Specifically, OperA distinguishes itself in four ways (stated here briefly, see the General response for details):
Beyond the conceptual distance, following the Reviewer's suggestion we now benchmark MPOE directly both in its original trainable core version and a frozen core version based on the suggestions in the original paper. We add the full results in Appendix D.11 and report a summary below.
As can be seen, OperA is the most accurate variant on both models, with 34x to 37x fewer trainable parameters than MPOE, and half its time per sample. Our MPOE baseline finetunes the FFNs only (while the rest is kept PEFT as with OperA). This is because full finetuning is too expensive: training the FFNs alone already requires 2.17B to 2.51B trainable parameters and 84.0 to 96.3 GiB of peak memory at batch 128 on the 3B models, while 7B and larger models do not fit in memory.