- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the setting of LoRA continual merging, i.e. where LoRA adapters arrive over time and must be merged under a fixed parameter budget (without having access to the data). MEPC incrementally builds a shared orthonormal basis using the adapter's factors and a set of task specific coefficients. When a new update arrives, to reduce the storage of redundant directions, its factors are projected onto the existing subspace and only its residual directions are appended. When the budget is exceeded, the method uses the eigendecomposition of the Gram matrix of the coefficients and frees up space by removing the low-eigenvalue directions. The experimental setting covers both vision and language modalities across several models.
Critical strengths
- The paper has a solid experimental setting, with three LLM families and vision tasks, multiple storage budgets, mixed task order etc.
- The method is well motivated, sound, and coherent with the task.
- MEPC outperforms Share, its main competitor, while reducing update time and memory usage thanks to its incremental approach.
Critical weaknesses
- The approach (and its intuition) is quite simple, however, I found Section 3 hard to follow, and the notation is unnecessarily heavy in my opinion. The role of the Gram matrix could be explained much more clearly, and an overview of the update cycle might help.
- An empirical analysis of how the shared rank evolves with the task stream and how the selected budget K impacts it is not provided (Fig. 7 shows only the final allocation, its evolution would be informative).
- I would have considered other decompositions instead of QR. The SVD for example, would have sorted the new directions by importance for free and allowed discarding negligible ones before appending. The choice of the QR decomposition is not discussed or ablated enough.
Questions for authors
See weaknesses.
- How many directions are retained per module? How does this compare with the rank r of the incoming LoRA adapters?
- When a new task arrives but the budget is exhausted, how do the authors ensure that the loss does not sacrifice the just-arrived contribution?
- Why not consider the whole update as a single matrix? This would overcome the non uniqueness problem of the LoRA decomposition.
- Have the authors considered weighting the deletion loss per task? This could help retain very specific tasks' updates.
7 comments
W3: Choice of QR for Residual Factorization
I would have considered other decompositions instead of QR. The SVD for example, would have sorted the new directions by importance for free and allowed discarding negligible ones before appending. The choice of the QR decomposition is not discussed or ablated enough.
We ran additional experiments with three SVD variants. Replacing QR directly with SVD yields nearly identical performance, while using SVD to truncate residual directions early either reduces performance or offers no gain. We therefore retain QR. In MEPC, QR serves solely to identify the residual subspace of the incoming LoRA that is not covered by the current shared subspace and to construct an orthogonal basis for it. MEPC assesses the importance of each direction later using the joint deletion loss. In the expanded shared basis, this loss measures how deleting a direction affects the full weight updates, taking into account the coefficients of all tasks and those on the opposite factor side. The singular values reflect only the energy of the corresponding directions in one factor of the incoming LoRA and cannot substitute for this global importance measure. Early truncation with SVD therefore acts as a local, irreversible screening step before information from all tasks is considered. The subsequent global deletion step serves a separate purpose.
To test this, we compared three variants. SVD-Full replaces QR with SVD, retains all nonzero residual directions, and leaves the subsequent joint deletion procedure unchanged. SVD-75 and SVD-90 retain only the residual directions needed to reach and of the total squared singular-value energy, respectively, before applying the original global compression procedure. All variants use the same 50-task stream, input rank , scaling factor , and storage budgets. The results are shown below.
| Model | QR (MEPC) | SVD-Full | SVD-75 | SVD-90 | |
|---|---|---|---|---|---|
| Llama-3.2-1B-Instruct | 3 | 0.9446 | 0.9441 | 0.9418 | 0.9392 |
| Llama-3.2-1B-Instruct | 5 | 0.9750 | 0.9763 | 0.9719 | 0.9749 |
| Qwen2.5-1.5B-Instruct | 3 | 0.9703 | 0.9686 | 0.9673 | 0.9667 |
| Qwen2.5-1.5B-Instruct | 5 | 0.9869 | 0.9878 | 0.9867 | 0.9879 |
The score difference between SVD-Full and QR ranges from to percentage points, indicating that replacing QR with SVD alone does not consistently improve performance. This is consistent with the method: SVD-Full and QR span the same expanded subspace, with their bases differing only by an orthogonal rotation within the block of newly added directions. The subsequent eigendecomposition of the coefficient Gram matrix rotates the entire basis again, so both methods retain the same subspace in exact arithmetic. The small differences observed in the experiments are likely due to numerical error.
SVD-75 scores below QR in all settings, with drops of to percentage points. SVD-90 shows larger drops of to percentage points at . These results indicate that early truncation of residual directions does not consistently improve performance and may discard important directions under tighter budgets. We therefore use QR to preserve the full residual subspace contributed by the incoming task and leave direction selection to the joint deletion loss. This avoids introducing an additional energy-threshold hyperparameter and an irreversible local screening step.
We have explained our choice of QR in Appendix C.11 of the revised manuscript and reported the ablation results in Table 16.
Q1: Retained Directions per Module
How many directions are retained per module? How does this compare with the rank r of the incoming LoRA adapters?
We measured the shared ranks retained at the end of the task stream for each attention projection and factor side. The number of retained directions varies substantially across projections and factor sides. On the A side, the shared ranks are approximately to times the input rank . On the B side, they are approximately to times for the query and output projections and to times for the key and value projections.
We use the same settings as in the main experiments, with an input LoRA rank of , a storage budget of , and a stream of 50 tasks ordered using random seed 42. For each projection and factor side, we record the final shared rank, given by the number of columns in the shared basis. We average these ranks across all Transformer layers and report the corresponding multiples of the input rank .
| Model | Projection | A-side rank | A-side / | B-side rank | B-side / |
|---|---|---|---|---|---|
| Llama-3.2-1B | Query | 19.1 | 1.19 | 65.5 | 4.09 |
| Llama-3.2-1B | Key | 16.0 | 1.00 | 14.8 | 0.93 |
| Llama-3.2-1B | Value | 16.6 | 1.04 | 20.7 | 1.29 |
| Llama-3.2-1B | Output | 23.0 | 1.44 | 67.9 | 4.24 |
| Qwen2.5-1.5B | Query | 18.6 | 1.16 | 59.6 | 3.73 |
| Qwen2.5-1.5B | Key | 14.6 | 0.91 | 7.3 | 0.46 |
| Qwen2.5-1.5B | Value | 13.0 | 0.81 | 8.4 | 0.52 |
| Qwen2.5-1.5B | Output | 22.9 | 1.43 | 63.9 | 3.99 |
| Mistral-7B | Query | 19.6 | 1.22 | 87.7 | 5.48 |
| Mistral-7B | Key | 15.9 | 0.99 | 17.8 | 1.11 |
| Mistral-7B | Value | 16.0 | 1.00 | 22.7 | 1.42 |
| Mistral-7B | Output | 21.7 | 1.36 | 85.1 | 5.32 |
We have added these results to Appendix C.5 of the revised manuscript.
We appreciate the reviewer's careful reading of our manuscript and the suggestions for clarifying the method and extending the analysis. We respond to each point below.
W1: Notation, the Gram Matrix, and the Update Cycle
We have reorganized Section 3 and simplified the mathematical notation in the revised manuscript.
We added an overview of the full update cycle at the beginning of Section 3.2. When the first LoRA arrives, MEPC initializes the merged state. For each subsequent LoRA, MEPC first expands and transforms the shared basis while updating the task-specific coefficients. If the updated representation exceeds the budget, MEPC applies importance-aware compression. The resulting merged state, which satisfies the budget, is then used for inference and the next update.
In Section 3.2.2 of the revised manuscript, we clarify why MEPC uses the Gram matrix. After basis expansion, the direction that minimizes the one-sided deletion loss may be a linear combination of several basis vectors. MEPC constructs the Gram matrix from the task-specific coefficients of all observed tasks and expresses the one-sided deletion loss as a quadratic form in the coordinates of the direction. The eigenvector associated with the smallest eigenvalue gives the coordinates, in the current basis, of the direction that minimizes this loss over the entire expanded subspace. MEPC uses this result to transform the shared basis and the task-specific coefficients together, preserving each task's factor reconstruction and making this direction the local candidate for compression.
W2: Shared-Rank Evolution under Different Storage Budgets
We recorded MEPC's actual shared ranks after each of the 50 updates and present the results in Figure 8 of Appendix C.5 in the revised manuscript. Figure 8 shows three phases of shared-rank evolution: growth, redistribution, and stabilization. The storage budget K determines when each phase occurs and how much the shared ranks change. We ran the experiment on Mistral-7B using a 50-task stream, with input LoRA rank r=16, scaling factor α=32, and storage budgets K∈1,3,5,7. Figure 8 shows how the shared ranks of the query, key, value, and output projections evolve with task step t. The shaded regions indicate the 25th to 75th percentiles of shared rank across layers. During the growth phase, the storage budget has yet to be reached, and the shared ranks increase approximately linearly as tasks arrive. Increasing K delays the onset of compression and raises the peak A-side shared ranks. During redistribution, the shared ranks of different projections gradually diverge. The A-side shared ranks drop rapidly across all budgets. The B-side shared ranks of the key and value projections also rise and then decline markedly, while those of the query and output projections remain relatively high after peaking.
As tasks accumulate, the storage cost qt of the task-specific coefficients associated with each shared direction increases (Equation 8), reducing the number of shared directions that can be retained under the fixed total budget. MEPC frees capacity primarily from the A side and from the key and value projections, retaining more directions on the B side of the query and output projections, which are more important to the full weight updates. Figure 8 shows how this non-uniform allocation gradually develops over the task stream. During stabilization, the curves generally flatten after step 30 (t>30), while the B-side shared ranks of the query and output projections continue to decline slowly as more tasks arrive. We have added the results and analysis to Appendix C.5 of the revised manuscript. While preparing this analysis, we discovered that the runs used to collect the data for Figure 7 in the original submission had imposed an additional rank cap on the shared bases. We removed the cap, reran the experiment under otherwise identical conditions, and collected the data for Figure 7 again. This error was confined to the experiment for Figure 7 in the original submission.