Stable-LoRA+: Regularizing Low-Rank Adaptation for Robust Scalin

The efficiency of low-rank adaptation does not ensure stable training as language models increase in size. Settings that are effective at one scale may degrade accuracy or lead to divergence at another. We study robust scaling of adaptation for mathematical reasoning, evaluating training stability and predictive accuracy across model sizes under a fixed training configuration, separately from performance after tuning. We introduce Stable-LoRA⁺, a training method that regularizes the factorized adaptation process. Its premise is that scaling the adapter output alone does not fully constrain factor evolution. The method therefore controls factor growth during training while penalizing update components aligned with the pretrained weight's leading singular subspaces. The same regularization is explicitly implemented for 4-bit backbones. Across more than 130 training-and-evaluation runs on five Qwen backbones (0.5B–14B), the strict fixed-configuration recipe matches vanilla LoRA's macro-average GSM8K accuracy (69.55%). With protocol and seed selection, the broader best-protocol comparison reaches 70.14% and mean rank 1.50. At lr = 10⁻³, vanilla LoRA diverges on 3B/7B, losing at least 76 percentage points, while our shared 10⁻⁴ configuration remains stable across the evaluated backbones. A quantization audit also identifies a silently inactive anchor, whose correction recovers 1.36 points at 7B. These results motivate regularization of factorized updates as an approach to robust adaptation across the evaluated model scales.