acceptodds
Withdrawn from ICLR 2027

Make Every Parameter Count: On-Device Shared-Subspace Streaming LoRA Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Edge-deployed pretrained models typically use LoRA to continually extend their task capabilities. However, retaining a separate adapter for each task can quickly exhaust the limited storage budget on edge devices. Prior work has explored adapter merging to reduce storage costs, but often struggles to avoid interference between task-specific information after merging under a limited budget. Shared-subspace methods can preserve finer-grained task differences through shared bases and task-specific coefficients. However, these methods struggle both to maintain efficient updates and to fully utilize the limited storage budget. We therefore propose MEPC (Make Every Parameter Count), a shared-subspace method for streaming LoRA merging. MEPC uses incremental subspace updates to reduce update overhead and performs importance-aware compression based on joint deletion loss, allocating the limited parameters preferentially to preserving more important cross-task information. Experiments across multiple storage budgets and task-stream settings show that MEPC substantially outperforms baselines without shared subspaces. Compared with the strongest shared-subspace baseline, MEPC improves performance by up to percentage points, achieves a per-step update speedup of up to , and reduces peak memory usage by up to .

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA Merging, Shared Subspaces, Continual Adaptation, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_A5UkOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the setting of LoRA continual merging, i.e. where LoRA adapters arrive over time and must be merged under a fixed parameter budget (without having access to the data). MEPC incrementally builds a shared orthonormal basis using the adapter's factors and a set of task specific coefficients. When a new update arrives, to reduce the storage of redundant directions, its factors are projected onto the existing subspace and only its residual directions are appended. When the budget is exceeded, the method uses the eigendecomposition of the Gram matrix of the coefficients and frees up space by removing the low-eigenvalue directions. The experimental setting covers both vision and language modalities across several models.

Critical strengths
  • The paper has a solid experimental setting, with three LLM families and vision tasks, multiple storage budgets, mixed task order etc.
  • The method is well motivated, sound, and coherent with the task.
  • MEPC outperforms Share, its main competitor, while reducing update time and memory usage thanks to its incremental approach.
Critical weaknesses
  • The approach (and its intuition) is quite simple, however, I found Section 3 hard to follow, and the notation is unnecessarily heavy in my opinion. The role of the Gram matrix could be explained much more clearly, and an overview of the update cycle might help.
  • An empirical analysis of how the shared rank evolves with the task stream and how the selected budget K impacts it is not provided (Fig. 7 shows only the final allocation, its evolution would be informative).
  • I would have considered other decompositions instead of QR. The SVD for example, would have sorted the new directions by importance for free and allowed discarding negligible ones before appending. The choice of the QR decomposition is not discussed or ablated enough.
Questions for authors

See weaknesses.

  • How many directions are retained per module? How does this compare with the rank r of the incoming LoRA adapters?
  • When a new task arrives but the budget is exhausted, how do the authors ensure that the loss does not sacrifice the just-arrived contribution?
  • Why not consider the whole update ΔW\Delta W as a single matrix? This would overcome the non uniqueness problem of the LoRA decomposition.
  • Have the authors considered weighting the deletion loss per task? This could help retain very specific tasks' updates.
7 comments

We appreciate the reviewer's careful reading of our manuscript and the suggestions for clarifying the method and extending the analysis. We respond to each point below.

W1: Notation, the Gram Matrix, and the Update Cycle

The approach (and its intuition) is quite simple, however, I found Section 3 hard to follow, and the notation is unnecessarily heavy in my opinion. The role of the Gram matrix could be explained much more clearly, and an overview of the update cycle might help.

We have reorganized Section 3 and simplified the mathematical notation in the revised manuscript.

We added an overview of the full update cycle at the beginning of Section 3.2. When the first LoRA arrives, MEPC initializes the merged state. For each subsequent LoRA, MEPC first expands and transforms the shared basis while updating the task-specific coefficients. If the updated representation exceeds the budget, MEPC applies importance-aware compression. The resulting merged state, which satisfies the budget, is then used for inference and the next update.

In Section 3.2.2 of the revised manuscript, we clarify why MEPC uses the Gram matrix. After basis expansion, the direction that minimizes the one-sided deletion loss may be a linear combination of several basis vectors. MEPC constructs the Gram matrix from the task-specific coefficients of all observed tasks and expresses the one-sided deletion loss as a quadratic form in the coordinates of the direction. The eigenvector associated with the smallest eigenvalue gives the coordinates, in the current basis, of the direction that minimizes this loss over the entire expanded subspace. MEPC uses this result to transform the shared basis and the task-specific coefficients together, preserving each task's factor reconstruction and making this direction the local candidate for compression.

W2: Shared-Rank Evolution under Different Storage Budgets

An empirical analysis of how the shared rank evolves with the task stream and how the selected budget K impacts it is not provided (Fig. 7 shows only the final allocation, its evolution would be informative).

We recorded MEPC's actual shared ranks after each of the 50 updates and present the results in Figure 8 of Appendix C.5 in the revised manuscript. Figure 8 shows three phases of shared-rank evolution: growth, redistribution, and stabilization. The storage budget KK determines when each phase occurs and how much the shared ranks change. We ran the experiment on Mistral-7B using a 50-task stream, with input LoRA rank r=16r=16, scaling factor α=32\alpha=32, and storage budgets K∈1,3,5,7K\in\\{1,3,5,7\\}. Figure 8 shows how the shared ranks of the query, key, value, and output projections evolve with task step tt. The shaded regions indicate the 25th to 75th percentiles of shared rank across layers. During the growth phase, the storage budget has yet to be reached, and the shared ranks increase approximately linearly as tasks arrive. Increasing KK delays the onset of compression and raises the peak A-side shared ranks. During redistribution, the shared ranks of different projections gradually diverge. The A-side shared ranks drop rapidly across all budgets. The B-side shared ranks of the key and value projections also rise and then decline markedly, while those of the query and output projections remain relatively high after peaking.

As tasks accumulate, the storage cost qtq_t of the task-specific coefficients associated with each shared direction increases (Equation 8), reducing the number of shared directions that can be retained under the fixed total budget. MEPC frees capacity primarily from the A side and from the key and value projections, retaining more directions on the B side of the query and output projections, which are more important to the full weight updates. Figure 8 shows how this non-uniform allocation gradually develops over the task stream. During stabilization, the curves generally flatten after step 30 (t>30t>30), while the B-side shared ranks of the query and output projections continue to decline slowly as more tasks arrive. We have added the results and analysis to Appendix C.5 of the revised manuscript. While preparing this analysis, we discovered that the runs used to collect the data for Figure 7 in the original submission had imposed an additional rank cap on the shared bases. We removed the cap, reran the experiment under otherwise identical conditions, and collected the data for Figure 7 again. This error was confined to the experiment for Figure 7 in the original submission.

W3: Choice of QR for Residual Factorization

I would have considered other decompositions instead of QR. The SVD for example, would have sorted the new directions by importance for free and allowed discarding negligible ones before appending. The choice of the QR decomposition is not discussed or ablated enough.

We ran additional experiments with three SVD variants. Replacing QR directly with SVD yields nearly identical performance, while using SVD to truncate residual directions early either reduces performance or offers no gain. We therefore retain QR. In MEPC, QR serves solely to identify the residual subspace of the incoming LoRA that is not covered by the current shared subspace and to construct an orthogonal basis for it. MEPC assesses the importance of each direction later using the joint deletion loss. In the expanded shared basis, this loss measures how deleting a direction affects the full weight updates, taking into account the coefficients of all tasks and those on the opposite factor side. The singular values reflect only the energy of the corresponding directions in one factor of the incoming LoRA and cannot substitute for this global importance measure. Early truncation with SVD therefore acts as a local, irreversible screening step before information from all tasks is considered. The subsequent global deletion step serves a separate purpose.

To test this, we compared three variants. SVD-Full replaces QR with SVD, retains all nonzero residual directions, and leaves the subsequent joint deletion procedure unchanged. SVD-75 and SVD-90 retain only the residual directions needed to reach 75%75\% and 90%90\% of the total squared singular-value energy, respectively, before applying the original global compression procedure. All variants use the same 50-task stream, input rank r=16r=16, scaling factor α=32\alpha=32, and storage budgets. The results are shown below.

ModelKKQR (MEPC)SVD-FullSVD-75SVD-90
Llama-3.2-1B-Instruct30.94460.94410.94180.9392
Llama-3.2-1B-Instruct50.97500.97630.97190.9749
Qwen2.5-1.5B-Instruct30.97030.96860.96730.9667
Qwen2.5-1.5B-Instruct50.98690.98780.98670.9879

The score difference between SVD-Full and QR ranges from −0.17-0.17 to +0.13+0.13 percentage points, indicating that replacing QR with SVD alone does not consistently improve performance. This is consistent with the method: SVD-Full and QR span the same expanded subspace, with their bases differing only by an orthogonal rotation within the block of newly added directions. The subsequent eigendecomposition of the coefficient Gram matrix rotates the entire basis again, so both methods retain the same subspace in exact arithmetic. The small differences observed in the experiments are likely due to numerical error.

SVD-75 scores below QR in all settings, with drops of 0.020.02 to 0.310.31 percentage points. SVD-90 shows larger drops of 0.360.36 to 0.540.54 percentage points at K=3K=3. These results indicate that early truncation of residual directions does not consistently improve performance and may discard important directions under tighter budgets. We therefore use QR to preserve the full residual subspace contributed by the incoming task and leave direction selection to the joint deletion loss. This avoids introducing an additional energy-threshold hyperparameter and an irreversible local screening step.

We have explained our choice of QR in Appendix C.11 of the revised manuscript and reported the ablation results in Table 16.

Q1: Retained Directions per Module

How many directions are retained per module? How does this compare with the rank r of the incoming LoRA adapters?

We measured the shared ranks retained at the end of the task stream for each attention projection and factor side. The number of retained directions varies substantially across projections and factor sides. On the A side, the shared ranks are approximately 0.80.8 to 1.41.4 times the input rank rr. On the B side, they are approximately 3.73.7 to 5.55.5 times rr for the query and output projections and 0.50.5 to 1.41.4 times rr for the key and value projections.

We use the same settings as in the main experiments, with an input LoRA rank of r=16r=16, a storage budget of K=3K=3, and a stream of 50 tasks ordered using random seed 42. For each projection and factor side, we record the final shared rank, given by the number of columns km,fk_{m,f} in the shared basis. We average these ranks across all Transformer layers and report the corresponding multiples of the input rank r=16r=16.

ModelProjectionA-side rankA-side / rrB-side rankB-side / rr
Llama-3.2-1BQuery19.11.1965.54.09
Llama-3.2-1BKey16.01.0014.80.93
Llama-3.2-1BValue16.61.0420.71.29
Llama-3.2-1BOutput23.01.4467.94.24
Qwen2.5-1.5BQuery18.61.1659.63.73
Qwen2.5-1.5BKey14.60.917.30.46
Qwen2.5-1.5BValue13.00.818.40.52
Qwen2.5-1.5BOutput22.91.4363.93.99
Mistral-7BQuery19.61.2287.75.48
Mistral-7BKey15.90.9917.81.11
Mistral-7BValue16.01.0022.71.42
Mistral-7BOutput21.71.3685.15.32

We have added these results to Appendix C.5 of the revised manuscript.

Reviewer_UJwUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MEPC, a shared-subspace approach for streaming LoRA merging under a fixed storage budget. MEPC incrementally incorporates only the residual directions of incoming adapters, avoiding reconstruction of historical LoRAs, and uses importance-aware compression with joint deletion loss to selectively remove less important subspace directions. Experiments on language and vision tasks show that MEPC consistently improves task retention over existing baselines while substantially reducing update time and peak memory usage.

Critical strengths
  • There is a theoretical complexity analysis.
  • The experiments are comprehensive.
Critical weaknesses
  • The main concern is the practicality of this method. First, it assumes the task ID during inference, which is not practical. Second, it is not a standard LoRA merging setting, where all adapters are merged into a unified one. Third, there are no experiments comparing the storage of different methods. Finally, Fig.2 shows a trade-off between speed and memory usage. As this paper focuses on an on-device setting and targets the storage cost, I think memory usage (and the lacked storage usage) is of higher priority.
  • Another concern is the writing of this paper. I find Sec.3.2.1 and 3.2.2 hard to follow. It lacks too many details or explanations. Also, I am unclear about how to make an inference with MEPC after reading Sec.3, which I think is not mentioned in this paper.

I may misunderstand this paper, and will raise my score if my concerns are addressed.

Questions for authors
  • Line 112: params(⋅)params(\cdot) is not defined
  • In Eq.4, where does Um,B(t)U^{(t)}_{m,B} come from? Sec.3.2.1 only mentions storing CC, but not UU.
  • What does the merged state SS include? Explicitly defining it may help better understand the whole method.
  • Removing the least important direction makes sense, but ignores the distribution of those important directions in tasks. What if most unimportant directions come from a single task? Deleting those directions would destroy the performance on that task.
3 comments

We thank the reviewer for the thoughtful feedback on the practical setting and the presentation of MEPC. We address the comments and questions below.

W1-1: Task IDs at Inference Time

First, it assumes the task ID during inference, which is not practical.

We assume access to task IDs at inference time to evaluate consolidation quality under controlled conditions. In deployment, existing routing methods can handle task identification separately from MEPC, so users can submit requests without explicitly providing a task ID.

Our focus is streaming LoRA consolidation under a fixed storage budget. The goal is to continually incorporate incoming LoRAs and retain information about each task within the available capacity, without keeping historical task data or the original LoRAs. Identifying the task at inference time is a separate routing problem. Evaluating both components together would make it difficult to attribute performance differences to routing or consolidation. We therefore provide the ground-truth task ID at inference time to isolate how well each consolidation method retains task information. MEPC uses the task ID solely to retrieve the corresponding task-specific coefficients and reconstruct the LoRA for that task. Any module that outputs a task index can therefore be used with MEPC.

In practice, task identity can be obtained in two ways. In on-device applications with dedicated service endpoints, application modules, or request metadata, the calling context can identify the task directly. For mixed requests whose tasks are unknown, an input-aware LoRA router can be placed upstream of MEPC. Existing work offers several ways to select LoRAs without explicit task IDs. LoraRetriever uses input representations to retrieve and compose relevant LoRAs [1]. Arrow routes inputs to relevant LoRAs in a zero-shot manner by scoring their alignment with adapter directions, without training a separate router [2]. Adapters Selector uses a lightweight selector to infer the task and domain from the input and select the corresponding LoRA [3]. These routing approaches can be combined with MEPC.

References

[1] Zhao et al., 2024. LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild. Findings of ACL.

[2] Ostapenko et al., 2024. Towards Modular LLMs by Building and Reusing a Library of LoRAs. ICML.

[3] Tian et al., 2025. Adapters Selector: Cross-domains and Multi-tasks LoRA Modules Integration Usage Method. COLING.

W1-2: LoRA Merging Setting

Second, it is not a standard LoRA merging setting, where all adapters are merged into a unified one.

The reviewer notes that standard LoRA merging combines all adapters into a single unified adapter and that our setting differs from this formulation. We view single-adapter merging as one form of LoRA merging. Existing work can be broadly grouped into three categories based on how the merged result is represented. MEPC falls into the shared-representation category and thus remains within the scope of LoRA merging.

Single-adapter merging. These methods merge the parameters or full weight updates of multiple LoRAs and compress the result into a single low-rank adapter. LoRA-LEGO decomposes, matches, and recombines low-rank components from multiple LoRAs. CtM reconstructs the full weight-update matrices and compresses them jointly to produce a single adapter that satisfies a target rank constraint [4, 5].

Multi-slot merging. These methods retain several slots, each consolidating a subset of the LoRAs. D²C clusters LoRAs by similarity and produces one merged adapter per cluster. K-Merge assigns each incoming LoRA to a slot and merges it with the adapter already stored there, retaining a fixed number of merged adapters [6, 7].

Shared-representation merging. These methods extract reusable low-rank bases from multiple LoRAs and represent each LoRA with lightweight adapter-specific coefficients. Compress then Serve compresses multiple LoRAs into shared bases and retains scaling parameters for each adapter. Share consolidates the low-rank factors of different LoRAs into shared subspaces and reconstructs each task's weight updates using task-specific coefficients [8, 9].

All three categories consolidate information from multiple LoRAs within limited storage and address interference among them. They differ in how they organize the merged result. Single-adapter merging can perform inference without task IDs because all tasks use the same adapter, which removes task-specific distinctions in the merged representation. Shared-representation merging preserves those distinctions through task-specific coefficients. At inference time, it requires either a known task ID or a routing mechanism, which can introduce routing errors.

References

[4] Zhao et al., 2025. LoRA-LEGO: A Flexible Framework for Efficiently Unified Low-Rank Adaptation. ICLR.

[5] He et al., 2026. Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter.

[6] Bohdal et al., 2026. Data-driven Clustering and Merging of Adapters for On-device LLMs. ICASSP.

[7] Shenaj et al., 2026. K-Merge: Online Continual Merging of Adapters for On-device LLMs. ACL.

[8] Gabrielsson et al., 2025. Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead. ICML.

[9] Kaushik et al., 2026. Shared LoRA Subspaces for Almost Strict Continual Learning. ECCV.

W1-3: Storage Usage and Budget Utilization

Third, there are no experiments comparing the storage of different methods. Finally, Fig.2 shows a trade-off between speed and memory usage. As this paper focuses on an on-device setting and targets the storage cost, I think memory usage (and the lacked storage usage) is of higher priority.

We compare all methods under the same fixed storage budget, which controls for storage cost by design. The reported performance differences are therefore measured under a common storage constraint. To show how much of the available budget each method actually uses, we measured budget utilization. After budget saturation, MEPC achieves an average utilization of at least 99.99%, compared with 98.62% to 99.30% for Share.

Following the protocol in Section 4.1, we set the same persistent parameter budget bb for all methods, equal to the parameter count of KK complete LoRAs. We define budget utilization at step tt as u(t)=params⁡(S(t))b×100u^{(t)}=\frac{\operatorname{params}(\mathcal{S}^{(t)})}{b}\times 100\\%, where params⁡(S(t))\operatorname{params}(\mathcal{S}^{(t)}) is the number of parameters persistently stored in the merged state at step tt, and bb is the parameter budget. Single-adapter and fixed-slot methods store parameters in units of complete merged adapters and complete LoRA slots, respectively. Because the experimental budgets are defined in units of complete LoRAs, both types of method reach 100% utilization by construction once the budget is saturated. The table therefore reports results for Share and MEPC, whose storage usage varies across updates. We use a stream of 50 tasks with input LoRA rank r=16r=16 and the task arrival order determined by seed 4242. We report average utilization over the update steps after budget saturation.

ModelKKShareMEPC
Llama-3.2-1B-Instruct398.63%99.99%
Llama-3.2-1B-Instruct599.30%99.99%
Qwen2.5-1.5B-Instruct398.62%99.99%
Qwen2.5-1.5B-Instruct599.21%100.00%

MEPC uses nearly the entire budget, with utilization approximately 0.7 to 1.4 percentage points higher than Share. Single-adapter and fixed-slot methods achieve 100% utilization because the experimental budgets are integer multiples of the parameter count of a complete LoRA. In deployment, available storage may leave a remainder too small to hold another complete adapter or slot. MEPC adjusts storage usage one shared direction at a time, allowing it to fit a given storage limit at a finer granularity. We have added the budget utilization results to Appendix C.10 of the revised manuscript (Table 15).

W2: Method Description and Inference

Another concern is the writing of this paper. I find Sec.3.2.1 and 3.2.2 hard to follow. It lacks too many details or explanations. Also, I am unclear about how to make an inference with MEPC after reading Sec.3, which I think is not mentioned in this paper.

We have rewritten Sections 3.2.1–3.2.3 and added an overview at the beginning of Section 3.2 to explain how initialization, incremental subspace updates, and importance-aware compression fit together. Section 3.2.1 explains how MEPC represents LoRAs using shared bases and task-specific coefficients and initializes the merged state from the first LoRA. Section 3.2.2 explains how MEPC projects the incoming LoRA factors onto the existing shared subspaces, expands the bases using the orthogonal residuals, and transforms the shared bases and task-specific coefficients. Section 3.2.3 describes the two-stage compression procedure used when the budget is exceeded.

We have also added Section 3.2.4 to explain inference. Given a task ID, MEPC retrieves the shared bases on both factor sides of each module and the corresponding task-specific coefficients from the current merged state. It then reconstructs the LoRA factors using Equation (2) and applies the reconstructed LoRA to the frozen pretrained model. This procedure implements the reconstruction map ψ\psi defined in Section 3.1.

Q1: Definition of $ \operatorname{params}⋅\cdot $

params⁡(⋅)\operatorname{params}(\cdot) is not defined.

We have added its definition to Section 3.1, where params⁡(⋅)\operatorname{params}(\cdot) denotes the total number of parameters in a persistently stored object.

Q2: Definition of the Shared Basis Um,B(t)\mathbf U_{m,\mathrm B}^{(t)}

In Eq. 4, where does Um,B(t)\mathbf U_{m,\mathrm B}^{(t)} come from? Section 3.2.1 only mentions storing C\mathbf C, but not U\mathbf U.

We have added the definition of Um,f(t)\mathbf U_{m,f}^{(t)} to Section 3.2.1. It denotes the shared basis for LoRA factor side f∈{A,B}f \in \{\mathrm A,\mathrm B\} of module mm to which LoRA is applied in the pretrained model.

Q3: Definition of the Merged State S(t)\mathcal S^{(t)}

What does the merged state S\mathcal S include? Explicitly defining it may help better understand the whole method.

We have added an explicit definition of MEPC’s merged state at time step tt to Section 3.2.1:

S(t)={(Um,f(t),{Ci,m,f(t)}i=1t)}m∈M,f∈F.\mathcal{S}^{(t)}=\left\lbrace\left(\mathbf{U}_{m,f}^{(t)},\lbrace\mathbf{C}_{i,m,f}^{(t)}\rbrace_{i=1}^{t}\right)\right\rbrace_{m\in\mathcal{M},f\in\mathcal{F}}.

Here, M\mathcal M is the set of modules to which LoRA is applied, F={A,B}\mathcal F=\{\mathrm A,\mathrm B\} is the set of LoRA factor sides, and Ci,m,f(t)\mathbf C_{i,m,f}^{(t)} contains the task-specific coefficients for task τi\tau_i on LoRA factor side ff of module mm. Thus, the merged state contains the shared bases on both sides of every module and the task-specific coefficients of all observed tasks in each shared basis.

Q4: Shared Directions and Task-Level Performance

Removing the least important direction makes sense, but ignores the distribution of those important directions in tasks. What if most unimportant directions come from a single task? Deleting those directions would destroy the performance on that task.

We acknowledge that deleting shared directions can reduce performance on individual tasks. MEPC's deletion loss measures the overall effect of removing a candidate direction on the full LoRA weight updates of all observed tasks, so it cannot provide a performance guarantee for each task individually. MEPC selects deletion candidates from the shared subspace, allowing it to search for low-loss directions in a space shared across tasks. This approach can mitigate the reviewer's concern to some extent. We also computed the worst-task performance retention for each method. Under the same storage budget, MEPC achieves substantially higher worst-task retention than all other methods on both Llama and Qwen.

Before compression, MEPC transforms the shared basis using the task-specific coefficients of all observed tasks. This change of basis preserves each task's LoRA reconstruction and makes the direction that minimizes the one-sided deletion loss within each shared subspace its local candidate. This direction is a linear combination of basis vectors in the expanded basis, so the candidates come from the entire shared subspace rather than from task-specific directions introduced by individual tasks. MEPC then compares the local candidates using the joint deletion loss and selects the candidate whose removal has the smallest overall effect on the full LoRA weight updates of all observed tasks. This procedure generally avoids restricting the candidates to the original directions introduced by any single task.

As shown in Figure 6, under the same storage budget, MEPC has more tasks than Share with final performance retention of at least 90% on all three models. On each model, at most one task has final retention below 75% under MEPC. To examine the lowest-retention tasks more closely, we use the per-task data underlying Figure 6 to compute the minimum final performance retention across all 50 tasks for each method on Llama and Qwen at K=3K=3.

MethodLlama-3.2-1B-InstructQwen2.5-1.5B-Instruct
Linear4.53%15.65%
TIES8.30%27.73%
DARE15.09%\underline{45.61\\%}
DARE-TIES1.82%31.78%
K-Merge6.19%18.88%
K-Merge++6.53%18.88%
OPCM11.13%4.20%
CtM14.11%26.92%
LoRA-LEGO5.57%5.04%
Share\underline{59.20\\%}45.45%
MEPC64.11%69.93%

MEPC's worst-task retention exceeds that of the second-best method by 4.91 and 24.32 percentage points on Llama and Qwen, respectively. Together with the task-level distributions in Figure 6, these results show that, on the task streams evaluated, MEPC achieves higher aggregate scores while retaining task performance substantially better than the other methods.

Reviewer_6GTCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies data-free streaming compression of task-specific LoRA adapters under a fixed budget for persisted adapter parameters. The proposed method, MEPC, represents adapters using shared orthonormal bases and task-specific coefficients. Incoming adapters are incorporated through projection and residual QR decomposition, avoiding explicit reconstruction of historical factors. When the representation exceeds the budget, a two-stage pruning procedure first identifies a low-energy candidate within each factor subspace and then compares candidates using the reconstruction error induced in the full LoRA weight update. This produces nonuniform rank allocations across modules and factor sides. Experiments cover 50 language tasks with three language models, together with 19 VTAB-1k tasks, and examine different storage budgets, source ranks, mixed-rank streams, and task orders. The reported results show improved task-performance retention and reduced update overhead relative to Share.

Critical strengths
  1. A practically relevant and clearly specified problem. Maintaining multiple task-specific adaptations under a fixed storage budget is useful for resource-constrained deployment. The paper explicitly specifies the availability of task identity, the absence of historical task data and raw adapters, and the inclusion of both shared bases and task-specific coefficients in the parameter budget.

  2. A coherent combination of incremental updates and product-aware compression. The incremental construction preserves existing factor reconstructions before compression and avoids repeatedly reconstructing historical factors. The joint deletion score measures the effect of removing a direction on the full weight update, rather than considering either LoRA factor in isolation. Under the stated orthonormality assumptions, this score can be computed directly from the coefficient matrices.

  3. Consistent improvements over the closest shared-subspace baseline across several settings. MEPC improves on Share on all three language models in Table 1, across the reported budget sweep and task orders, and in the mixed-rank and vision experiments.

Critical weaknesses
  1. The efficiency contribution would benefit from a more closely controlled comparison with Share. The reported speedup is measured against a baseline that explicitly reconstructs historical factors before decomposition. Given an orthonormal shared basis and its coefficient matrix, the corresponding decomposition can instead be performed through a compact matrix after incorporating the incoming residual, without materializing the historical factors. Comparing against this reconstruction-free implementation while retaining the same truncation rule would help separate the benefit of the computational implementation from that of the proposed allocation strategy.

  2. The global deletion criterion does not explicitly account for unequal parameter savings. Equation (9) compares absolute joint deletion losses, but Equation (11) and Algorithm 1 show that deleting an A-side direction frees d_m + q_{t+1} parameters, whereas deleting a B-side direction frees n_m + q_{t+1} parameters. These costs can differ across modules and factor sides.

  3. Some baseline comparisons conflate compression quality with differences in representation and inference functionality. Appendix B.3 implements Linear, TIES, and DARE variants by merging the A and B factors independently. Such operations are generally not equivalent to merging the actual weight updates BA and can introduce cross terms or dependence on the chosen factorization. A budget-matched control operating on the weight updates would help assess the effect of these adaptations.

Questions for authors

See Above.

3 comments

We thank the reviewer for the detailed and constructive feedback. We address each concern below and indicate where the corresponding experiments and analyses appear in the revised manuscript.

W1: Controlled Efficiency Comparison with Share

The efficiency contribution would benefit from a more closely controlled comparison with Share. The reported speedup is measured against a baseline that explicitly reconstructs historical factors before decomposition. Given an orthonormal shared basis and its coefficient matrix, the corresponding decomposition can instead be performed through a compact matrix after incorporating the incoming residual, without materializing the historical factors. Comparing against this reconstruction-free implementation while retaining the same truncation rule would help separate the benefit of the computational implementation from that of the proposed allocation strategy.

We agree with the reviewer's observation and implemented Share-opt, a variant of Share that updates the SVD incrementally by factorizing a compact matrix. We evaluated Share-opt alongside Share and MEPC. We use the same 50-task stream as in the main experiments, with input LoRA rank r=16r=16 and scaling factor α=32\alpha=32. We evaluate Llama-3.2-1B-Instruct and Qwen2.5-1.5B-Instruct under storage budgets K∈3,5K\in\\{3,5\\}. The table reports total update time over the full task stream, peak resident set size (Peak RSS), and the final aggregate score s(50)s^{(50)}. The efficiency measurements for Share and MEPC come from the original experiment logs, and those for Share-opt come from the supplementary experiments.

ModelKKMethodTotal update time (s)Peak RSS (GB)s(50)s^{(50)}
Llama-3.2-1B-Instruct3Share1297.64302.17500.9186
Llama-3.2-1B-Instruct3Share-opt340.47700.73800.9189
Llama-3.2-1B-Instruct3MEPC87.89200.71600.9446
Llama-3.2-1B-Instruct5Share1425.06102.20100.9623
Llama-3.2-1B-Instruct5Share-opt575.77100.78000.9598
Llama-3.2-1B-Instruct5MEPC140.99900.78900.9750
Qwen2.5-1.5B-Instruct3Share1780.80102.30300.9564
Qwen2.5-1.5B-Instruct3Share-opt477.45000.78100.9557
Qwen2.5-1.5B-Instruct3MEPC122.10400.77800.9703
Qwen2.5-1.5B-Instruct5Share1904.88302.31800.9842
Qwen2.5-1.5B-Instruct5Share-opt844.78700.89300.9846
Qwen2.5-1.5B-Instruct5MEPC179.84000.86900.9869

Share-opt takes 26.2%26.2\% to 44.3%44.3\% of Share's total update time and reduces peak RSS by 61.5%61.5\% to 66.1%66.1\%. Changes in the final aggregate score range from −0.25-0.25 to +0.04+0.04 percentage points. These results support the reviewer's observation that a compact-matrix implementation substantially reduces Share's update overhead while largely preserving its final score. Across all four settings, Share-opt's total update time is 3.873.87 to 4.704.70 times that of MEPC. The two methods have similar peak RSS. MEPC achieves a higher final aggregate score in all four settings, with a gain of up to 2.572.57 percentage points. We have added the full results to Appendix C.7 of the revised manuscript (Table 12).

W2: Parameter Cost in Global Deletion

The global deletion criterion does not explicitly account for unequal parameter savings. Equation (9) compares absolute joint deletion losses, but Equation (11) and Algorithm 1 show that deleting an A-side direction frees dm+qt+1d_m + q_{t+1} parameters, whereas deleting a B-side direction frees nm+qt+1n_m + q_{t+1} parameters. These costs can differ across modules and factor sides.

As the reviewer points out, the number of parameters freed by deleting a direction can vary across modules and factor sides. We consider accounting for parameter cost in the global comparison a reasonable alternative objective and evaluated a cost-normalized variant of MEPC. The two criteria yield similar final aggregate scores, and cost normalization does not consistently outperform the original criterion.

In each compression iteration, MEPC compares the joint deletion losses ℓm,f(t+1)\ell_{m,f}^{(t+1)} of the local candidates and removes the direction whose deletion has the smallest effect on the full LoRA weight update. The suggested criterion minimizes the joint deletion loss per parameter freed. To evaluate this criterion, we implemented Cost-normalized MEPC, which uses the same local candidate selection procedure and defines the global deletion score as ℓm,fcost,(t+1)=ℓm,f(t+1)Δpm,f(t+1),Δpm,f(t+1)=dm,f+qt+1\ell_{m,f}^{\mathrm{cost},(t+1)}=\frac{\ell_{m,f}^{(t+1)}}{\Delta p_{m,f}^{(t+1)}}, \Delta p_{m,f}^{(t+1)}=d_{m,f}+q_{t+1}. Here, Δpm,f(t+1)\Delta p_{m,f}^{(t+1)} is the number of parameters freed by deleting the candidate direction. The basis vector contains dm,fd_{m,f} parameters, and the corresponding rows of the task-specific coefficient matrices contain qt+1=∑i=1t+1riq_{t+1}=\sum_{i=1}^{t+1}r_i parameters in total.

We evaluate both methods on Llama-3.2-1B-Instruct and Qwen2.5-1.5B-Instruct using the same 50-task stream and task order (seed 4242) as in the main experiments. The input LoRA rank is r=16r=16, the scaling factor is α=32\alpha=32, and the storage budgets are K∈1,3,5,7K\in\\{1,3,5,7\\}. The table reports the final aggregate score s(50)s^{(50)}, computed on the full test sets after all 50 tasks have been processed.

ModelKKMEPCCost-normalized MEPC
Llama-3.2-1B10.74260.7463
Llama-3.2-1B30.94460.9429
Llama-3.2-1B50.97500.9734
Llama-3.2-1B70.98480.9848
Qwen2.5-1.5B10.85930.8602
Qwen2.5-1.5B30.97030.9711
Qwen2.5-1.5B50.98690.9864
Qwen2.5-1.5B70.99450.9937

The final aggregate score of Cost-normalized MEPC differs from that of MEPC by −0.17-0.17 to +0.37+0.37 percentage points. At the tightest budget, K=1K=1, cost normalization slightly improves the score on both models. As the budget increases, this advantage does not persist across both models. Normalizing by the number of parameters freed measures the joint deletion loss per parameter freed. This criterion may therefore prioritize deleting directions that free more parameters but have a greater impact on the full LoRA weight update, which can lead to lower performance. MEPC directly compares the joint deletion losses of candidate directions and always removes the candidate with the lowest loss, maintaining comparable or higher overall performance than Cost-normalized MEPC across different budgets.

Based on these results, we retain the original criterion as the default. We have added this comparison to Appendix C.8 of the revised manuscript (Table 13).

W3: Budget-Matched Baselines in Weight-Update Space

Some baseline comparisons conflate compression quality with differences in representation and inference functionality. Appendix B.3 implements Linear, TIES, and DARE variants by merging the A and B factors independently. Such operations are generally not equivalent to merging the actual weight updates BA and can introduce cross terms or dependence on the chosen factorization. A budget-matched control operating on the weight updates would help assess the effect of these adaptations.

We agree that merging the two LoRA factors separately is not equivalent to merging the full weight updates directly. We therefore evaluated two matrix-space configurations. Even against the strongest matrix-space baseline, MEPC leads by 9.89.8 to 18.618.6 percentage points across all four settings, indicating that its advantage cannot be attributed to weaker factor-space implementations of the baselines.

We implemented versions of Linear, TIES, DARE, and DARE-TIES that merge directly in the space of full weight-update matrices ΔW\Delta\mathbf{W}. We evaluated two configurations:

  • Matrix, single-slot. For each incoming LoRA, we reconstruct ΔW\Delta\mathbf{W} from its two low-rank factors and merge it with the weight update already stored in the single slot using the corresponding merging method. We then apply truncated SVD to the merged matrix and store the result as a rank-KrKr LoRA in that slot. This configuration uses the same single-LoRA inference scheme and storage budget as the original factor-space baselines, isolating the effect of the merging space.
  • Matrix, multi-slot. We maintain KK low-rank slots and use routing to select a slot for each incoming LoRA. We reconstruct the full weight-update matrices of the incoming LoRA and the LoRA in the selected slot, merge them in weight-update space, and apply truncated SVD before storing the result in the selected slot. Each slot has the capacity of one input LoRA, giving this configuration the same total storage budget as the single-slot configuration while retaining multi-slot routing.

We evaluate Llama-3.2-1B-Instruct and Qwen2.5-1.5B-Instruct on the same 50-task stream and task order (seed 4242) as in the main experiments. Each task provides a LoRA with rank r=16r=16 and scaling factor α=32\alpha=32. We use storage budgets K∈3,5K\in\\{3,5\\}. In the single-slot configuration, these budgets correspond to a single slot of rank 4848 or 8080, respectively. In the multi-slot configuration, they correspond to 33 or 55 slots of rank 1616, respectively. The table reports the final aggregate score s(50)s^{(50)}.

MethodImplementationLlama, K=3K=3Llama, K=5K=5Qwen, K=3K=3Qwen, K=5K=5
LinearFactor-space0.67770.76140.82750.8425
LinearMatrix, single-slot0.66920.66850.80340.8011
LinearMatrix, multi-slot0.67940.76720.83360.8613
TIESFactor-space0.72260.78930.82840.8748
TIESMatrix, single-slot0.67660.67760.81650.8163
TIESMatrix, multi-slot0.7586‾\underline{0.7586}0.8180‾\underline{0.8180}0.8694‾\underline{0.8694}0.8887‾\underline{0.8887}
DAREFactor-space0.70090.74920.83690.8484
DAREMatrix, single-slot0.66780.66910.80030.8007
DAREMatrix, multi-slot0.67930.76680.83330.8612
DARE-TIESFactor-space0.57500.63330.79490.7990
DARE-TIESMatrix, single-slot0.61670.61590.79020.7920
DARE-TIESMatrix, multi-slot0.63140.66760.80160.8256
MEPCShared subspace0.94460.97500.97030.9869

The single-slot matrix-space variants score lower than the original factor-space baselines in most settings, and their scores change little as the budget increases. This suggests that consolidating all tasks into a single slot limits how much these variants benefit from additional storage. The multi-slot matrix-space variants outperform their factor-space counterparts in most settings. Even against multi-slot Matrix-TIES, the strongest matrix-space baseline, MEPC leads by 18.618.6, 15.715.7, 10.110.1, and 9.89.8 percentage points, respectively. These results show that MEPC's advantage persists when baselines merge full weight updates directly and retain multi-slot routing under the same storage budget.

We have added these matrix-space baselines and the corresponding analysis to Appendix C.9 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.