Basis-to-LoRA: Learning to Generate LoRA Adapters for Task Adaptation

Parameter-efficient fine-tuning methods such as Low-Rank Adapterss (LoRAs) have become a standard approach for adapting large language models to downstream tasks. Nevertheless, training a separate LoRA adapter for every new task remains computationally expensive. Weight-space learning offers an alternative by training a hypernetwork to generate adapter weights directly. However, predicting full weight updates is challenging due to their high dimensionality, while predicting LoRA factors introduces non-identifiability from invariance to a group of linear transformations. In this work, we show that LoRA updates trained on related datasets approximately share a low-dimensional subspace, and leverage this structure to improve weight-space learning. Our method learns a shared basis from a collection of trained LoRA checkpoints and constrains the hypernetwork to predict only the coordinates of a new adapter within this basis. This formulation provides an identifiable representation while substantially reducing the hypernetwork’s output dimension. We evaluate our approach on commonsense reasoning and mathematical reasoning benchmarks, where it consistently outperforms weight-space learning baselines. On Qwen2.5-1.5B, our method achieves 33.7% accuracy on MATH and 69.5% on GSM8K, improving over Drag-and-Drop by 9.8 and 3.2 percentage points, respectively.