GA-LoRA: A Gradient-Free Path to Robust Fine-Tuning

Fine-tuning a pre-trained model on a target distribution can improve in-distribution accuracy at the cost of the out-of-distribution robustness acquired during pre-training. Parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) mitigate this tension by restricting updates to a compact subspace, yet the choice of optimizer still governs the geometry of the resulting solution. We introduce GA-LoRA, a gradient-free fine-tuning framework that evolves LoRA parameters with a Genetic Algorithm (GA), directly maximizing task accuracy as its fitness objective without backpropagation. We show analytically that the mutation–selection dynamics of GA-LoRA preferentially retain solutions in broader basins characterized by lower average-case sharpness. Empirically, across multiple model architectures and distribution-shift benchmarks, GA-LoRA preserves competitive in-distribution accuracy while consistently improving out-of-distribution generalization relative to gradient-based optimizers. These results establish evolutionary optimization as a principled and effective pathway toward robust fine-tuning of large-scale pre-trained models.