- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces BASE, a differentially private fine-tuning framework that learns a low-dimensional subspace along the input side and another along the output side. These subspaces are built from public auxiliary data: the model is briefly fine-tuned on the public data, the weight displacements along the way are collected, and the leading eigenvectors of their input-side and output-side covariance matrices are used as fixed projection bases. Private fine-tuning then reloads the original pretrained model and trains only a small core matrix per module with DP-SGD, so every update stays inside the learned subspace. Experiments on language understanding, dialogue summarization and instruction following show better privacy-utility trade-offs than several DP fine-tuning baselines.
Critical strengths
- The method is simple and the privacy argument is sound. The bases and the core initialization depend only on public data, and Stage 2 is plain DP-SGD with global clipping, so the guarantee is easy to verify.
- The method uses the matrix structure of linear layers to build separate input-side and output-side subspaces, making it fairly lightweight and applicable to models where the global-projection approach does not fit in memory.
- The results on a billion-scale decoder model also show that the approach can scale to settings where the global-subspace baseline cannot be run.
Critical weaknesses
- The core idea (freeze fixed projection bases on both sides of a weight matrix and train only a small core matrix in between) closely resembles existing parameter-efficient methods with fixed low-rank bases (e.g., LoRA-XS-type methods), as well as public-data-derived gradient subspaces for DP. The paper does not discuss this relation, so the new part is mainly the use of warm-up displacement statistics to choose the bases.
- The theoretical motivation does not connect to what is actually computed. The Kronecker-factored objective relies on an approximation of the gradient second moment that the authors say need not hold, and the bases are then taken from the eigenvectors of weight-displacement statistics from a short warm-up run, which is only a "surrogate". There is no argument and/or evidence that these directions are close to the leading eigenspaces of the activation and error statistics, or that they capture the private-task gradient. The "equivalence to bilateral gradient projection" is essentially the chain rule, and it does not hold exactly with Adam, which is the optimizer used. The theory therefore gives little support for the claimed benefit.
- The empirical evaluation relies on a strong and favorable assumption that the public data is drawn from the same distribution as the private data (a random split of the same training set, with a sizeable public fraction). It is not clear what happens when the public data is small, out-of-distribution, or from a different task, which is the realistic setting for public-data-assisted DP.
- The bilateral ranks, learning rates, clipping thresholds and epochs appear to be selected per task, and the paper does not say how (on private data or not). If the parameters are selected on private data, then the tuning is not accounted for in the privacy budget.
Questions for authors
How are the bilateral ranks, learning rates and clipping thresholds chosen, and is private data used for this selection? If so, how is the privacy cost of the selection handled?
5 comments
Response to Weakness 2:
We clarify that the Kronecker-factored approximation provides the structural motivation for constructing separate input-side and output-side subspaces, rather than asserting an exact equivalence between gradient covariance and weight-displacement statistics. To directly examine whether trajectory-derived bases capture informative gradient directions, we conducted additional experiments comparing BASE with Gradient-Covariance PCA (G-PCA), which constructs bilateral bases directly from public gradient second-moment statistics.
Both methods use identical bilateral ranks, and gradient retention is evaluated on independent held-out public batches. For a gradient matrix , we define the gradient-norm retention ratio as
=\frac{\|P_{\mathrm{out}}^\top G_B P_{\mathrm{in}}\|_F}{\|G_B\|_F}.$$ The relative retention of BASE against G-PCA is calculated as $$R_{\mathrm{BASE/G\text{-}PCA}} =\frac{\rho_{\mathrm{BASE}}}{\rho_{\mathrm{G\text{-}PCA}}}\times100\%.$$ For each method, the retention rates are averaged over the evaluated held-out batches before computing the relative ratio. The results for all six evaluated modules are reported below. | Dataset | Module | BASE / G-PCA | |---|---|---:| | SST-2 | Layer 1 / V | 99.83% | | SST-2 | Layer 12 / O | 96.85% | | QNLI | Layer 1 / V | 98.42% | | QNLI | Layer 1 / K | 94.45% | | QNLI | Layer 1 / Q | 107.57% | | QNLI | Layer 12 / O | 88.47% | Across all six evaluated modules, BASE achieves an average relative gradient-norm retention of **97.60%** compared with G-PCA. These results provide direct empirical evidence that bilateral subspaces derived from public optimization trajectories can retain gradient information comparable to subspaces constructed explicitly from gradient second-moment statistics. While this does not establish equivalence between the underlying covariance matrices, it supports the effectiveness of trajectory-based construction as a computationally efficient surrogate for identifying task-relevant update directions. ### **Response to Weakness 3:** We thank the reviewer for highlighting the importance of public–private distribution alignment. To address this concern, we conducted additional cross-dataset transfer experiments using SST-2 and IMDB. Although both datasets involve sentiment classification, they originate from different data sources and exhibit distinct data characteristics. This setting better reflects practical scenarios where auxiliary public data is related to the private task but is not drawn from the same distribution. Specifically, we evaluate two transfer directions: IMDB → SST-2 and SST-2 → IMDB. In each setting, BASE constructs bilateral subspaces using 5,000 public samples from the source dataset, followed by private fine-tuning on the target dataset. For a fair comparison, DP-LoRA is evaluated under the same cross-dataset public warm-up protocol. All results are averaged over three random seeds. | Public → Private | Method | $\varepsilon = 0.5$ | $\varepsilon = 4.0$ | |:---|:---|:---:|:---:| | IMDB → SST-2 | DP-LoRA | 91.23 ± 0.34 | 91.66 ± 0.43 | | IMDB → SST-2 | **BASE** | **92.67 ± 0.15** | **92.73 ± 0.37** | | SST-2 → IMDB | DP-LoRA | 91.37 ± 0.07 | 91.73 ± 0.04 | | SST-2 → IMDB | **BASE** | **92.00 ± 0.22** | **92.15 ± 0.05** | The results show that BASE consistently outperforms DP-LoRA across both transfer directions and privacy budgets. Specifically, BASE achieves improvements of 1.44/1.07 percentage points on IMDB → SST-2 and 0.63/0.42 percentage points on SST-2 → IMDB under $\epsilon = 0.5/4.0$, respectively. These findings demonstrate that BASE can effectively exploit task-relevant information from external public data to construct informative bilateral subspaces, even when the public and private datasets differ in distribution. Furthermore, Figure 2(b) of the main paper already examines public-data ratios ranging from 1% to 20%, demonstrating the effectiveness of BASE with limited auxiliary data. Together, these results provide evidence that BASE's effectiveness is not confined to the idealized in-distribution public/private split.Response to Weakness 4 and Question 1:
We clarify that no private data was used for hyperparameter selection, and thus this process incurred no additional privacy cost.
Specifically, all hyperparameter selection was conducted using public auxiliary data. The primary learning-rate search grids were {1e-5, 3e-5, 5e-5, 7e-5} for public warm-up and {1e-4, 3e-4, 5e-4, 7e-4, 1e-3, 3e-3, 5e-3, 7e-3} for private-stage optimization. Additional learning-rate candidates were explored when necessary, exclusively using public data. Bilateral ranks were also determined through public-data experiments based on optimization performance and training stability, without accessing private training or validation data.
Clipping thresholds and training epochs were predetermined before private optimization. All private training runs followed fixed epoch schedules and used the final checkpoint, without private-data-based early stopping or checkpoint selection.
Once the hyperparameters were fixed, private fine-tuning was performed using DP-SGD. The noise multiplier was calibrated according to the target privacy budget, sampling rate, and total number of private optimization steps, with the privacy loss accounted for using RDP. Therefore, the reported privacy guarantees cover the entire private optimization procedure, without any unaccounted privacy cost from hyperparameter selection.
We thank the reviewer for the comments and address the concerns as follows.
Response to Weakness 1:
We would like to clarify the fundamental distinctions between BASE and the existing approaches mentioned by the reviewer.
Regarding LoRA-XS, the similarity is primarily limited to the fixed-basis, low-dimensional core parameterization, rather than the underlying subspace construction mechanism. LoRA-XS derives its bases from the singular vectors of pretrained weights, without incorporating downstream task information. However, dominant singular directions of pretrained weights are not necessarily aligned with task-relevant optimization directions, which is particularly consequential under differential privacy, where optimization is restricted to a small subspace and subject to gradient clipping and noise perturbation. In contrast, BASE explicitly identifies task-relevant bilateral directions from public optimization trajectories. To substantiate this distinction, we replaced BASE's trajectory-derived bases with pretrained-weight SVD bases while keeping the same private optimization procedure. Under ϵ=0.5, the pretrained-SVD variant achieves only 50.92% and 54.90% accuracy on SST-2 and QNLI, respectively, compared with 93.69% and 91.68% for BASE. Furthermore, BASE's bases retain 17.9$\times$ and 14.9$\times$ more gradient energy on held-out batches, respectively. These results demonstrate that the choice of subspace is not a minor implementation detail but a critical factor in effective differentially private optimization.
Regarding existing public-data-assisted DP subspace methods, the reviewer's characterization overlooks substantial differences in subspace construction, optimization mechanisms, and computational scalability, which we have already discussed in the Related Work section of the main paper. Methods such as GEP and PE-SGD incorporate gradient-subspace estimation, projection, or evolution into private training, introducing additional computational overhead beyond standard DP optimization. Such operations become increasingly expensive as model dimensionality grows, posing substantial challenges for billion-parameter models. Other approaches, such as DP-SFT, avoid repeated subspace estimation but rely on a global projection matrix whose storage and computational costs scale with the full parameter dimension, making this implementation prohibitively expensive at billion-parameter scale. In contrast, BASE constructs task-adaptive, module-wise bilateral bases only once from public optimization trajectories and subsequently performs private optimization directly on compact, fixed core matrices. This design eliminates repeated basis estimation during private training and avoids materializing high-dimensional global projection matrices. As demonstrated in Table 4, BASE reduces peak memory from 91.68 GiB to 5.72 GiB and end-to-end training time from 488.13 to 65.17 minutes compared with DP-SFT, corresponding to a 16.0$\times$ memory reduction and a 7.5$\times$ speedup. Therefore, BASE is not merely an alternative basis-selection strategy, but a structurally distinct and computationally scalable framework for task-adaptive differentially private fine-tuning.