- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes a two-stage framework for differentially private fine-tuning in learned subspaces of LoRA parameters. By restricting private optimization to low-dimensional subspaces, their method reduces the effective impact of differential privacy noise while preserving formal privacy guarantees. They provide a theoretical interpretation showing that subspace optimization is equivalent to gradient projection–based optimization and leads to reduced noise magnitude. Extensive experiments on multiple benchmarks demonstrate consistent improvements over existing DP baselines, especially under strict privacy budgets. The authors further show that learned subspaces can transfer across datasets from related task, enabling practical private fine-tuning when target trajectories are unavailable.
Critical strengths
The paper combines efficient training with a focus on differential privacy by proposing a two-stage framework. Private optimization is restricted to low-dimensional subspaces, which is effective in reducing noise introduced by differential privacy. They have proposed a theoretical interpretation of their choice. Empirical results are strong.
Critical weaknesses
The paper is generally hard to read. There is a strong technical contribution aspect that is loosely connected with the motivation. Therefore, the paper is not written for a general audience but for an expert reader in the field.
Questions for authors
- "Parameter-efficient fine-tuning (PEFT) has emerged as an effective paradigm for adapting large language models to downstream tasks involving privacy-sensitive data." Since when is PEFT proposed for privacy-sensitive data? The abstract does not read well from the beginning. Many problems are introduced in parallel, such as privacy leakage, utility degradation, noise and impaired utility, etc., making us wonder what specific problems the paper is focusing on.
2 comments
Thanks for the clarification. The first line of the abstract still starts with a confusing connection between PEFT and privacy-sensitive data, which is wrong. I would appreciate the authors reflecting the additional clarifications in the paper.
We respectfully disagree the questions and would like to clarify the following points.
1. Clarification of PEFT and Privacy
PEFT was not originally proposed specifically for privacy-sensitive data, nor does PEFT itself provide formal privacy guarantees. Our statement is intended to emphasize that PEFT has become a widely adopted approach for adapting large language models to downstream tasks, including scenarios involving sensitive data. In such scenarios, additional privacy-preserving mechanisms, such as DP-SGD, are required to provide formal privacy guarantees. Our work investigates how to improve the effectiveness of PEFT under these differential privacy constraints, rather than treating PEFT itself as a privacy-preserving mechanism.
2. Clarification of the Central Research Problem
The central problem addressed in our paper is the utility degradation caused by differentially private optimization, particularly under strict privacy budgets. Privacy leakage motivates the need for differential privacy, while gradient clipping and noise injection are mechanisms used in DP-SGD that can impair optimization. These are therefore not separate research objectives, but interconnected aspects of the same privacy–utility trade-off.
Our two-stage framework is designed to address this problem directly. In the first stage, we leverage public optimization trajectories of LoRA parameters to identify task-relevant low-dimensional subspaces without consuming any privacy budget. In the second stage, private optimization is restricted to these learned subspaces, reducing the effective dimensionality of the injected noise while maintaining formal differential privacy guarantees. This design aims to improve model utility under a given privacy budget, as supported by our theoretical analysis and experimental results.