The Path Matters: Retention-Budget Continuation for Low-Rank LLM Post-Training

Low-rank adaptation (LoRA) enables efficient specialization of large language models, but low rank alone does not ensure capability retention. How the optimization path shapes specialization under a fixed behavioral allowance and finite computation remains unclear. We introduce Retention-Budget Continuation for LoRA (RBC-LoRA), a homotopy method that constrains forward sequence Kullback-Leibler divergence from the original model on reference prompts. A temporary quadratic factor anchor supplies a unique starting solution; continuation then strengthens the task objective, removes the anchor, and expands the retention allowance. First-order primal-dual correction and empirical acceptance checks guide progression toward the prescribed constrained endpoint while preserving adapter rank. Under local regularity and correction assumptions, we characterize solution-path sensitivity and tracking error; a divergence bound also controls changes in bounded expected scores on the reference distribution. Experiments show that RBC-LoRA improves target-task scores over independently tuned direct constrained training on math and code tasks. These gains accompany less measured degradation of pretrained behaviour, reflected in lower audit divergence and smaller retained-benchmark losses. These findings establish the optimization path as a practical design choice for improving the specialization–retention trade-off without increasing adapter capacity.