acceptodds
Under review as a conference paper at ICLR 2027

GA-LoRA: A Gradient-Free Path to Robust Fine-Tuning

Abstract

Fine-tuning a pre-trained model on a target distribution can improve in-distribution accuracy at the cost of the out-of-distribution robustness acquired during pre-training. Parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) mitigate this tension by restricting updates to a compact subspace, yet the choice of optimizer still governs the geometry of the resulting solution. We introduce GA-LoRA, a gradient-free fine-tuning framework that evolves LoRA parameters with a Genetic Algorithm (GA), directly maximizing task accuracy as its fitness objective without backpropagation. We show analytically that the mutation–selection dynamics of GA-LoRA preferentially retain solutions in broader basins characterized by lower average-case sharpness. Empirically, across multiple model architectures and distribution-shift benchmarks, GA-LoRA preserves competitive in-distribution accuracy while consistently improving out-of-distribution generalization relative to gradient-based optimizers. These results establish evolutionary optimization as a principled and effective pathway toward robust fine-tuning of large-scale pre-trained models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Out-of-Distribution Generalization, Genetic Algorithm

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MqfdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the use of generic algorithm for finetuning a pretrained model via the LoRA scheme.

They claim that the proposed method may generalise to different architectures and domain shift datasets.

The evaluations show some improved performance for some OOD datasets.

Critical strengths
  • Overall this paper is complete with necessary components presented.

  • The evaluations provide some insights of incorporating generic algorithm in model finetuning for OOD tasks.

  • Some figures provide some interesting comparisons, such as fig 5.

Critical weaknesses

Major:

  • Simply incorporating the concepts of generic algorithm into model finetuning is not a solid work grounded for novelty. Generic algorithms have been widely applied into many different domains and areas, and this sounds like an application to finetuning problem. The novelty of this work appears to be very limited.

  • There is no motivating figure showing why and how the proposed model has insights or contributes to the current literature.

  • Fig 2 as the main pipeline overview, caption only contains one sentence and does not provide any descriptions to show why, how and what are the key innovations for the current literature. The figure is also organised in a quite hard to understand way.

  • The method section is written in an unnecessary complex way. In most places, those equations could be explained via plain texts, such as L2-norm, mean over indicators, weighted fusion of weights etc. The method section is also not well organised, and does not clearly show the novelty of this work.

  • This work also introduces many hyperparameters, making the practical application side much weaker, and hard to tune those parameters, although section 3.3 provides some mechanisms for mutation-selection mechanism, but those wordings are unnessary complex and misleading.

  • The implementation details eg in page 6 bottom, is not detailed, and the authors refer to appendix, which also does not provide enough info to deeply assess the setups. The appendix reads like a list and are very concise and compact, more like GPT generated with long and dense maths equations but many bullet points in short, although I would not use this for evaluation.

  • The evaluations appear to be very limited and biased. The authors should perform evaluations on at least 3 large scale/new/challenging datasets rather than old-fashioned datasets. Also with the introduce of LLM/VLM, the evaluations should start shifting to language-driven evaluation tasks rather than classification / determinstic tasks.

  • The discussions and analysis regarding the presented experiments appear to be very limited.

Minor:

  • Figure font sizes are too small to read.

  • The paper reads like a condensed paper (more like GPT generated). With only skeletons and no any solid meats on top.

  • The paper reads quite dry and boring, with unnessary maths equations and complexity, making the core innovations unclear to reviewer.

Questions for authors

See my comments above.

Reviewer_s2UFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GA-LoRA, a gradient-free fine-tuning method that evolves all LoRA matrices as one chromosome using elitism, arithmetic crossover, and annealed masked Gaussian mutation, with ID top-1 accuracy as fitness. The motivation is that mutation plus selection favors low-loss perturbed offspring, yielding flatter minima and better OOD robustness than gradient-based optimizers; a local second-order analysis and mean-field update support this, and LoRA’s low dimensionality makes a GA tractable. Experiments apply low-rank LoRA to vision encoders, evaluating few-shot ID classification and multiple OOD benchmarks against standard and robustness-oriented optimizers, with additional results on other backbones and a language task. GA-LoRA reports competitive ID accuracy, higher average OOD accuracy, and effective robustness close to zero-shot; sharpness analysis shows lower average-case but not worst-case sharpness, with average-case sharpness correlating more strongly with effective robustness. GA training is slower.

Critical strengths
  1. The paper investigates a relevant problem in parameter-efficient fine-tuning. Rather than introducing another parameterization method, the authors examine whether the choice of optimizer can improve out-of-distribution generalization within the restricted parameter space of LoRA. The proposed gradient-free framework provides a straightforward way to investigate this question.
  2. The paper attempts to connect empirical performance with optimization geometry. In addition to accuracy comparisons, the authors examine worst-case and average-case sharpness under different perturbation radii and investigate the robustness–efficiency trade-off. These analyses provide useful complementary perspectives on the behavior of the proposed method.
Critical weaknesses
  1. The theoretical analysis does not fully characterize the behavior of the implemented algorithm. The analysis in Appendix C considers a simplified local mutation–selection process with smooth loss functions, isotropic Gaussian perturbations, and offspring selection based on local viability. In contrast, the actual method optimizes training accuracy and incorporates masked mutation, arithmetic crossover, and an annealing schedule. Consequently, the analysis provides an intuitive explanation for why flatter regions might be favored, but does not rigorously establish that the implemented algorithm systematically converges to such regions or that this mechanism accounts for the observed OOD improvements.
  2. The computational cost and scalability of the approach remain concerns. GA-LoRA uses a population of 100 individuals over 500 generations, requiring a substantial number of forward evaluations even without backpropagation. Although candidate evaluations are parallelizable, the paper acknowledges higher single-GPU training time than standard gradient-based optimizers. More systematic comparisons under matched computational budgets would help determine whether the robustness gains justify the additional cost. Moreover, the evaluation of language models is limited to two GPT-2 sizes and a single sentiment-analysis protocol, leaving the scalability of the approach to larger language models and more diverse tasks insufficiently established.
Questions for authors
  1. The paper argues that LoRA’s low dimensionality is what makes the GA work, yet all main results use r=2 (about 10⁵ parameters). In Table 2, GA’s ID accuracy at r=8 (70.1) is on par with SGD and SAM. Was the evolution budget (P, G, N_eff) scaled with rank? Does the OOD advantage persist at larger ranks or on larger models? If not, the abstract’s claim about “large-scale pre-trained models” should be narrowed.
  2. Could the authors compare GA-LoRA with random-weight-perturbation training, other evolution strategies (e.g., OpenAI-ES and CMA-ES), and zeroth-order optimization methods such as MeZO under comparable computational budgets? Given that the proposed mechanism attributes improved generalization to the preference for flatter regions under random perturbations, these comparisons would help clarify whether the observed gains are specific to the genetic algorithm or can also be achieved by other gradient-free optimization methods or perturbation-based approaches.
Reviewer_CJVjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a genetic algorithm (GA-LoRA) to finetune LoRa adapters for neural networks. The algorithm uses a combination of selection, crossover, and mutation to adapt the parameters, optimizing for training accuracy using only forward passes through the network. The authors run experiments on CLIP, BLIP-2 and GPT-2, and report improved OOD accuracy while maintaining competitive ID performance. The paper also includes an analysis of the sharpness induced by this algorithm compared to other optimization algorithms. They show it results in lower average sharpness and higher worst case sharpness, and that lower average sharpness is correlated with higher robustness in their experiments.

Critical strengths
  • The paper was well written and easy to read. That was much appreciated.
  • The method itself is simple and well motivated.
  • The motivation of the idea that evolutionary algorithms lead to flatter minima is intuitive and makes sense.
  • The method shows consistent improvements on OOD accuracy across several different settings and architectures.
Critical weaknesses
  • There are several related works on evolutionary optimization of lora adapters [1,2,3] that were not cited/discussed. The novelty is less so about using an evolutionary algorithm, but more about the robustness/geometry effects.
  • There is no spread reported on any of the numbers.
  • Overall, there is a lot about the dynamics of this evolutionary process that is missing (see questions). A detailed account of the process would significantly strengthen the paper.
  • The sharpness vs robustness analysis is a good starting point, but I would have liked that section to be a bit deeper as well. For example, why does the genetic algorithm result in essentially the highest worst case sharpness but one of the lower average case sharpness. Also, while there is a better line fit through average case sharpness vs robustness, looking at the plot, I am not sure it's not being affected by the low outliers from the Adam and SAM runs. That also means I am not sure that the cause of the robustness gains are because of the sharpness mitigations.

[1] Feihu Jin, Yin Liu, and Ying Tan. Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models. arXiv:2403.01754, 2024. Introduces C-LoRA and F-LoRA.
[2] Daria Korotyshova et al. ESSA: Evolutionary Strategies for Scalable Alignment. arXiv:2507.04453, July 2025.
[3] Iheb Gafsi. NoRA: Neuro-Evolution of Low-Rank Adaptation of Language Models. EasyChair Preprint 15133, September 2024.

Questions for authors
  • It seems that the genetic algorithm falls behind on in domain accuracy. Would being able to follow a gradient signal at the end of genetic training potentially help with that?
  • What is the spread on the numbers reported? Are the results statistically significant?
  • What are the dynamics of this process? How fast do the networks improve? How much do they change? How often are the networks selected in the next generation crossed with another network vs dynamically mutated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.