Make Every Parameter Count: On-Device Shared-Subspace Streaming LoRA Merging

Edge-deployed pretrained models typically use LoRA to continually extend their task capabilities. However, retaining a separate adapter for each task can quickly exhaust the limited storage budget on edge devices. Prior work has explored adapter merging to reduce storage costs, but often struggles to avoid interference between task-specific information after merging under a limited budget. Shared-subspace methods can preserve finer-grained task differences through shared bases and task-specific coefficients. However, these methods struggle both to maintain efficient updates and to fully utilize the limited storage budget. We therefore propose MEPC (Make Every Parameter Count), a shared-subspace method for streaming LoRA merging. MEPC uses incremental subspace updates to reduce update overhead and performs importance-aware compression based on joint deletion loss, allocating the limited parameters preferentially to preserving more important cross-task information. Experiments across multiple storage budgets and task-stream settings show that MEPC substantially outperforms baselines without shared subspaces. Compared with the strongest shared-subspace baseline, MEPC improves performance by up to $3.13$ percentage points, achieves a per-step update speedup of up to $28.34\times$, and reduces peak memory usage by up to $66.2\%$.