BASE: Bilateral Subspace Projection for Differentially Private LLM Fine-Tuning

Large language models (LLMs) are commonly adapted to downstream tasks through fine-tuning, where task-specific data often contain sensitive information. Differential privacy (DP) provides rigorous privacy guarantees, but noise injected into high-dimensional update spaces can severely degrade model utility. Existing approaches typically mitigate this issue by restricting optimization to low-dimensional spaces. However, these spaces are typically predefined and may not align well with task-relevant update directions, limiting their effectiveness in private fine-tuning. To address this challenge, we propose BASE, a differentially private bilateral subspace fine-tuning framework that constructs task-aligned low-dimensional update spaces for private fine-tuning. BASE first learns a structured bilateral subspace from public auxiliary data by capturing the principal variation patterns of model updates. This construction is motivated by a gradient energy preservation objective under a Kronecker-structured approximation, leading to a principled construction of input-side and output-side subspaces from update statistics. Private fine-tuning is then constrained within the resulting subspace, preserving informative update directions under DP noise while maintaining computational efficiency. Extensive experiments across diverse models, privacy budgets, and benchmark tasks demonstrate that BASE consistently outperforms strong DP fine-tuning baselines, achieving superior privacy–utility trade-offs while substantially reducing training time and memory overhead. Our code is available at https://anonymous.4open.science/r/ksf-anonymous-2ee2/.