H-OM: Hadamard-Orthogonal Model Merging with Recoverable Task Components

Model merging combines task-specific updates into a single model without joint retraining. However, existing independent and sequential methods for model merging cannot adapt to real-world deployments due to dependence on original adapters or preconditioned tasks. In this paper, to address the problem, we propose Hadamard Orthogonal Merging (H-OM) that constructs mutually orthogonal output subspaces for Low-Rank Adaptation (LoRA). H-OM learns to adapt to each task learns within its assigned subspace to simultaneously allow parallel adaptation and maintain orthogonal updates. Furthermore, we develop Coordinate-Exposure Uniformity (CEU) to guide subspace construction for effective adaptation, and induce subspaces with uniform coordinate exposure from disjoint sets of randomized Hadamard basis vectors. H-OM is flexible to enable component removal and reweighting without retaining the original adapters, since it can exactly recover the weight updates for tasks from their projection in the assigned subspaces under linear merging. Experiments on VTAB-1K and NLP-15 demonstrate that H-OM yields state-of-the-art accuracy in model merging with evidently fewer trainable parameters than standard LoRA. Remarkably, H-OM recovers all 19 VTAB-1K task updates from the merged model using the pretrained base model and reproducible basis metadata, without compromising single-task accuracy.