Privacy Needs the Right Space: Rethinking Federated LoRA Beyond Rank

Low-Rank Adaptation (LoRA) can still expose many coordinates to private optimization, while reducing the number of coordinates can discard useful adaptation directions. We study how both private dimension and retained update directions shape a compact trainable space for federated LoRA under clipping and DP noise. We propose FedTIE, a differentially private federated LoRA framework built on Tied Isometric Embedding (TIE), which uses privatized warm-up profiles to determine which factor coordinates across layers and factors share latent variables, with statistical load penalties regularizing the resulting assignment. Its sparse embedding preserves the Euclidean energy of latent perturbations when lifted to the concatenated factor space. The embedding is then frozen, and main-stage optimization and communication operate in the latent space. We establish an end-to-end sample-level differential privacy guarantee covering space selection and main training, together with a finite-horizon convergence bound for the induced subspace objective. On GLUE, FedTIE improves average accuracy over the strongest evaluated baseline by 3.40% and 4.44% at $\epsilon =6$ and $3$, respectively, with $\delta =10^{-5}$, while also improving non-private average accuracy and achieving competitive generation performance. On RoBERTa-base, FedTIE reduces per-round communication by 51.9% relative to FFA-LoRA and 85.2% relative to FedASK. Code is available at https://anonymous.4open.science/r/fedhsip/.