MOSAIC: Continual LoRA Merging via Hypernetwork Structure

Hypernetworks offer an efficient alternative to LoRA fine-tuning by directly generating adapters from conditioning inputs, such as document chunks or task descriptions. However, in streaming settings, retaining every generated LoRAs increases storage and leaves information distributed across separate LoRAs. Existing weight-space merging approaches consolidate these adapters without exploiting their shared hypernetwork, whose decoder relates generating coordinates to adapter behavior. To address this, we propose MOSAIC, a continual merging method that uses the hypernetwork structure to merge generated LoRAs in hypernetwork coordinates. MOSAIC measures adapter differences through their layer-output contributions on frozen target-model activation probes and pulls the resulting activation-induced metric back to hypernetwork coordinates through the frozen decoder's Jacobian, yielding a matrix-weighted barycenter formulation with additive sufficient statistics. This formulation enables one-pass updates and order-invariant accumulation with memory independent of stream length, producing a single deployable LoRA after each arrival. MOSAIC further applies covariance shrinkage to stabilize the accumulated functional metric. Experiments across multiple hypernetwork-based LoRA generation families and target LLMs demonstrate the effectiveness of MOSAIC in continually merging generated LoRAs.