Differentially Private Fine-Tuning in Learned Subspaces of LoRA

Parameter-efficient fine-tuning (PEFT) has emerged as an effective paradigm for adapting large language models to downstream tasks involving privacy-sensitive data. Differential privacy (DP) mitigates privacy leakage during fine-tuning but often introduces substantial utility degradation. Focusing on low-rank adaptation (LoRA), a representative PEFT method, we observe that although LoRA constrains the rank of weight updates, its parameterization remains highly redundant, which can result in large noise and impaired fine-tuning utility. To address this limitation, we propose a two-stage framework for differentially private fine-tuning within a learned low-dimensional subspace of LoRA parameters. Our method first identifies task-relevant update subspaces by extracting dominant directions from LoRA training trajectories, and then reparameterizes LoRA factors to perform private fine-tuning directly in this compact subspace. By restricting noise-perturbed updates to a low-dimensional, task-aligned space, the proposed method reduces the effective impact of DP noise while preserving the same privacy guarantees. We theoretically characterize how subspace-based updates reduce the effective magnitude of injected DP noise compared to high-dimensional parameter updates. Extensive experiments on multiple benchmark datasets demonstrate that our method consistently outperforms representative differentially private fine-tuning baselines across a broad range of privacy settings. Our code is available at https://anonymous.4open.science/r/8BA2.