- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
DP-LoRA applies DP-SGD to optimize the Low Rank Adapters (LoRA) , , which first clips the gradient and then adds Gaussian noise proportional to the clipping threshold. This introduces noise scaling with to the gradient dimension. Hence, reducing the gradient dimension decreases the noise required to satisfy the privacy constraint. Additionally, in the federated setting, reducing the gradient dimension also lowers communication costs. Therefore, this paper proposes a method to further reduce the dimension of the LoRAs in the federated setting, thereby improving the privacy-accuracy tradeoff and reducing the communication cost.
To achieve this, they concatenate the LoRAs for all layers into . Instead of updating , they update , where is an orthonormal projection matrix with . Hence, this method reduces the gradient dimension from to , which decreases the privacy-induced noise and the communication costs, at the expense of a slight subspace approximation error. To mitigate this approximation error, they choose by first doing a private warmup training on all clients to get a update for each client and then selecting a sparse and orthonormal that minimizes the gradient conflict among the projected updates 's.
Empirical results show that their method outperforms several baselines on four datasets across different .
Critical strengths
The method is intuitive and provably reduces communication cost; empirically, it also improves the privacy–accuracy tradeoff over multiple baselines.
Critical weaknesses
The paper chooses the projection matrix P in the form specified in equation (5), however, it does not provide sufficient justification for this particular choice. Equation (3) provides a decomposition that holds for every orthonormal P. Hence, the optimality of the proposed private selection mechanism is not clear.
Questions for authors
-
Under a limited privacy budget, is TIE's sparse signed grouping structure easier to estimate accurately than other private subspace estimation methods (e.g., DP-PCA), and hence leading to a better privacy–accuracy tradeoff?
-
How would the results change if a small public dataset is available, allowing one to estimate the best subspace () from the public data?
-
Is this method a private version of Uni-LoRA?
5 comments
Response to Question 1
We believe that the potential advantage of TIE over DP-PCA primarily stems from its structural prior, which is specifically designed to address the parameter-sharing problem in federated LoRA. DP-PCA focuses on estimating a general dominant subspace, whereas TIE formulates subspace estimation as a sparse, signed coordinate-grouping problem. It only requires per-parameter update profiles across different clients, together with statistics such as sensitivity, update energy, and heterogeneity, without the need to estimate a full high-dimensional covariance matrix. Therefore, when the effective update directions in a task indeed exhibit coordinate-sharing structures across layers and factors, this constrained estimation approach is expected to reduce the statistical estimation difficulty under a limited privacy budget. Furthermore, TIE obtains its warm-up statistics through a differentially private mechanism, while the grouping procedure constitutes post-processing, and the privacy cost incurred during the warm-up phase is already accounted for in the end-to-end privacy budget. Experimentally, TIE significantly outperforms random isometric projections under the same projection dimension, providing further empirical evidence, to some extent, of the effectiveness of using data-driven structured grouping to select the subspace.
Response to Question 2
We believe this is a highly valuable research direction. If a small-scale but representative public dataset is available, it can be used to estimate the update profiles of individual LoRA parameters and construct the subspace through TIE's sparse signed grouping mechanism. This approach could potentially offer two benefits:
- Improving the quality of subspace estimation: Since public data is not affected by differential privacy noise, it can provide more accurate parameter-update statistics, thereby enabling the construction of more appropriate parameter-sharing structures and reducing the loss of informative model update directions caused by incorrect grouping.
- Reducing privacy budget consumption: Since public data does not involve user privacy, constructing based on such data does not require any differential privacy budget. This allows more of the privacy budget to be allocated to subsequent model training, potentially improving the trade-off between privacy protection and model accuracy. Of course, the effectiveness of this approach also depends on the consistency between the public data distribution and the actual client data distributions. If there is a substantial discrepancy between them, the subspace estimated from public data may fail to adequately capture the actual client heterogeneity, potentially affecting model performance. We will include additional experiments in the revised manuscript to compare the performance of subspaces constructed using public data with those constructed using private warm-up data, thereby further validating the effectiveness of this approach.
Response to Weakness
Equation (3) provides an error decomposition that applies to any orthogonal projection matrix; however, this does not imply that different projection matrices yield the same model performance. In fact, both the discarded-signal error and the projection-induced clipping bias depend on the specific choice of P. Therefore, selecting an appropriate low-dimensional subspace remains an important problem. In this work, we adopt the sparse signed grouping structure in Equation (5). We do not claim that this structure is globally optimal among all possible orthogonal projection matrices. Rather, it is designed to simultaneously ensure exact orthogonality, enable cross-layer parameter sharing, and maintain low storage and computational overhead. Building on this structure, we leverage privacy-preserving warm-up statistics to perform data-driven subspace selection based on factors such as client heterogeneity patterns and parameter sensitivity. Experiments comparing against random projections further validate the effectiveness of this selection strategy on practical tasks.