PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Models

Efficiently updating Large Language Models (LLMs) with new or evolving factual knowledge can erode previously acquired reasoning abilities. Spectral parameter-efficient fine-tuning methods control this interference through a fixed-rank boundary in the singular value decomposition of pre-trained weights, treating singular-value rank as a proxy for skill relevance. We show, both theoretically and empirically, that this premise is incomplete: information essential for reasoning is distributed across the singular spectrum of multilayer perceptron weights rather than confined to either end of it. We therefore introduce PALoRA, a two-stage framework for relevance-based knowledge injection. PALoRA first trains a Singular Value Fine-Tuning (SVF) expert on a target reasoning skill and uses its learned singular scaling vector as a frozen probe of how strongly the skill relies on each singular component. It then injects factual knowledge with Low-Rank Adaptation (LoRA) while penalizing the projection of the update onto every singular direction in proportion to this reliance. We prove that this graded penalty bounds the first-order skill degradation of the update, with fixed-rank protection as its binary case. Across Llama 3.1 8B and Mistral 7B, and across mathematical, coding, and scientific reasoning benchmarks, PALoRA preserves on average 96% of the SVF expert's reasoning performance while maintaining competitive factual recall. It matches the strongest of five baselines without a fixed rank boundary, with the only positive backward transfer, retains more of the skill under sequential injection, and adds less than 0.006% parameter overhead.