Instance-Customized LoRA Generation via Discrete Priors for Efficient LLM Adaptation

As a mainstream Parameter-Efficient Fine-Tuning (PEFT) method, Low-Rank Adaptation (LoRA) significantly reduces the optimization and deployment costs of large language models by introducing trainable low-rank matrices. Despite the progress, existing LoRA and its variants typically learn a universal adapter at the entire task or dataset level, failing to achieve fine-grained personalized adaptation for individual instances, and thus limiting the performance in complex scenarios. In this paper, we propose InLoRA, a novel dynamic, instance-customized low-rank adapter considering individual idiosyncrasies. In particular, our InLoRA first maintains a discrete codebook with vector quantization to store the prior patterns of adapters. More importantly, we retrieve from the codebook for individual instances and then utilize lightweight projections to generate instance-customized yet well-structured adapters, which would be combined with task-specific adapters for efficient fine-tuning. Extensive experiments demonstrate that our InLoRA not only outperforms state-of-the-art PEFT methods, but also matches or even surpasses the performance of full fine-tuning on multiple benchmarks. The code and pre-trained models will be made publicly available.