acceptodds
Under review as a conference paper at ICLR 2027

BASE: Bilateral Subspace Projection for Differentially Private LLM Fine-Tuning

Abstract

Large language models (LLMs) are commonly adapted to downstream tasks through fine-tuning, where task-specific data often contain sensitive information. Differential privacy (DP) provides rigorous privacy guarantees, but noise injected into high-dimensional update spaces can severely degrade model utility. Existing approaches typically mitigate this issue by restricting optimization to low-dimensional spaces. However, these spaces are typically predefined and may not align well with task-relevant update directions, limiting their effectiveness in private fine-tuning. To address this challenge, we propose BASE, a differentially private bilateral subspace fine-tuning framework that constructs task-aligned low-dimensional update spaces for private fine-tuning. BASE first learns a structured bilateral subspace from public auxiliary data by capturing the principal variation patterns of model updates. This construction is motivated by a gradient energy preservation objective under a Kronecker-structured approximation, leading to a principled construction of input-side and output-side subspaces from update statistics. Private fine-tuning is then constrained within the resulting subspace, preserving informative update directions under DP noise while maintaining computational efficiency. Extensive experiments across diverse models, privacy budgets, and benchmark tasks demonstrate that BASE consistently outperforms strong DP fine-tuning baselines, achieving superior privacy–utility trade-offs while substantially reducing training time and memory overhead. Our code is available at https://anonymous.4open.science/r/ksf-anonymous-2ee2/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differetial privacy, Large language models, Private fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_m6DaOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BASE, a differentially private fine-tuning framework that learns a low-dimensional subspace along the input side and another along the output side. These subspaces are built from public auxiliary data: the model is briefly fine-tuned on the public data, the weight displacements along the way are collected, and the leading eigenvectors of their input-side and output-side covariance matrices are used as fixed projection bases. Private fine-tuning then reloads the original pretrained model and trains only a small core matrix per module with DP-SGD, so every update stays inside the learned subspace. Experiments on language understanding, dialogue summarization and instruction following show better privacy-utility trade-offs than several DP fine-tuning baselines.

Critical strengths
  • The method is simple and the privacy argument is sound. The bases and the core initialization depend only on public data, and Stage 2 is plain DP-SGD with global clipping, so the guarantee is easy to verify.
  • The method uses the matrix structure of linear layers to build separate input-side and output-side subspaces, making it fairly lightweight and applicable to models where the global-projection approach does not fit in memory.
  • The results on a billion-scale decoder model also show that the approach can scale to settings where the global-subspace baseline cannot be run.
Critical weaknesses
  • The core idea (freeze fixed projection bases on both sides of a weight matrix and train only a small core matrix in between) closely resembles existing parameter-efficient methods with fixed low-rank bases (e.g., LoRA-XS-type methods), as well as public-data-derived gradient subspaces for DP. The paper does not discuss this relation, so the new part is mainly the use of warm-up displacement statistics to choose the bases.
  • The theoretical motivation does not connect to what is actually computed. The Kronecker-factored objective relies on an approximation of the gradient second moment that the authors say need not hold, and the bases are then taken from the eigenvectors of weight-displacement statistics from a short warm-up run, which is only a "surrogate". There is no argument and/or evidence that these directions are close to the leading eigenspaces of the activation and error statistics, or that they capture the private-task gradient. The "equivalence to bilateral gradient projection" is essentially the chain rule, and it does not hold exactly with Adam, which is the optimizer used. The theory therefore gives little support for the claimed benefit.
  • The empirical evaluation relies on a strong and favorable assumption that the public data is drawn from the same distribution as the private data (a random split of the same training set, with a sizeable public fraction). It is not clear what happens when the public data is small, out-of-distribution, or from a different task, which is the realistic setting for public-data-assisted DP.
  • The bilateral ranks, learning rates, clipping thresholds and epochs appear to be selected per task, and the paper does not say how (on private data or not). If the parameters are selected on private data, then the tuning is not accounted for in the privacy budget.
Questions for authors

How are the bilateral ranks, learning rates and clipping thresholds chosen, and is private data used for this selection? If so, how is the privacy cost of the selection handled?

5 comments

We thank the reviewer for the comments and address the concerns as follows.

Response to Weakness 1:

We would like to clarify the fundamental distinctions between BASE and the existing approaches mentioned by the reviewer.

Regarding LoRA-XS, the similarity is primarily limited to the fixed-basis, low-dimensional core parameterization, rather than the underlying subspace construction mechanism. LoRA-XS derives its bases from the singular vectors of pretrained weights, without incorporating downstream task information. However, dominant singular directions of pretrained weights are not necessarily aligned with task-relevant optimization directions, which is particularly consequential under differential privacy, where optimization is restricted to a small subspace and subject to gradient clipping and noise perturbation. In contrast, BASE explicitly identifies task-relevant bilateral directions from public optimization trajectories. To substantiate this distinction, we replaced BASE's trajectory-derived bases with pretrained-weight SVD bases while keeping the same private optimization procedure. Under ϵ=0.5\epsilon = 0.5, the pretrained-SVD variant achieves only 50.92% and 54.90% accuracy on SST-2 and QNLI, respectively, compared with 93.69% and 91.68% for BASE. Furthermore, BASE's bases retain 17.9$\times$ and 14.9$\times$ more gradient energy on held-out batches, respectively. These results demonstrate that the choice of subspace is not a minor implementation detail but a critical factor in effective differentially private optimization.

Regarding existing public-data-assisted DP subspace methods, the reviewer's characterization overlooks substantial differences in subspace construction, optimization mechanisms, and computational scalability, which we have already discussed in the Related Work section of the main paper. Methods such as GEP and PE-SGD incorporate gradient-subspace estimation, projection, or evolution into private training, introducing additional computational overhead beyond standard DP optimization. Such operations become increasingly expensive as model dimensionality grows, posing substantial challenges for billion-parameter models. Other approaches, such as DP-SFT, avoid repeated subspace estimation but rely on a global projection matrix whose storage and computational costs scale with the full parameter dimension, making this implementation prohibitively expensive at billion-parameter scale. In contrast, BASE constructs task-adaptive, module-wise bilateral bases only once from public optimization trajectories and subsequently performs private optimization directly on compact, fixed core matrices. This design eliminates repeated basis estimation during private training and avoids materializing high-dimensional global projection matrices. As demonstrated in Table 4, BASE reduces peak memory from 91.68 GiB to 5.72 GiB and end-to-end training time from 488.13 to 65.17 minutes compared with DP-SFT, corresponding to a 16.0$\times$ memory reduction and a 7.5$\times$ speedup. Therefore, BASE is not merely an alternative basis-selection strategy, but a structurally distinct and computationally scalable framework for task-adaptive differentially private fine-tuning.

Response to Weakness 2:

We clarify that the Kronecker-factored approximation provides the structural motivation for constructing separate input-side and output-side subspaces, rather than asserting an exact equivalence between gradient covariance and weight-displacement statistics. To directly examine whether trajectory-derived bases capture informative gradient directions, we conducted additional experiments comparing BASE with Gradient-Covariance PCA (G-PCA), which constructs bilateral bases directly from public gradient second-moment statistics.

Both methods use identical bilateral ranks, and gradient retention is evaluated on independent held-out public batches. For a gradient matrix GBG_B, we define the gradient-norm retention ratio as

=\frac{\|P_{\mathrm{out}}^\top G_B P_{\mathrm{in}}\|_F}{\|G_B\|_F}.$$ The relative retention of BASE against G-PCA is calculated as $$R_{\mathrm{BASE/G\text{-}PCA}} =\frac{\rho_{\mathrm{BASE}}}{\rho_{\mathrm{G\text{-}PCA}}}\times100\%.$$ For each method, the retention rates are averaged over the evaluated held-out batches before computing the relative ratio. The results for all six evaluated modules are reported below. | Dataset | Module | BASE / G-PCA | |---|---|---:| | SST-2 | Layer 1 / V | 99.83% | | SST-2 | Layer 12 / O | 96.85% | | QNLI | Layer 1 / V | 98.42% | | QNLI | Layer 1 / K | 94.45% | | QNLI | Layer 1 / Q | 107.57% | | QNLI | Layer 12 / O | 88.47% | Across all six evaluated modules, BASE achieves an average relative gradient-norm retention of **97.60%** compared with G-PCA. These results provide direct empirical evidence that bilateral subspaces derived from public optimization trajectories can retain gradient information comparable to subspaces constructed explicitly from gradient second-moment statistics. While this does not establish equivalence between the underlying covariance matrices, it supports the effectiveness of trajectory-based construction as a computationally efficient surrogate for identifying task-relevant update directions. ### **Response to Weakness 3:** We thank the reviewer for highlighting the importance of public–private distribution alignment. To address this concern, we conducted additional cross-dataset transfer experiments using SST-2 and IMDB. Although both datasets involve sentiment classification, they originate from different data sources and exhibit distinct data characteristics. This setting better reflects practical scenarios where auxiliary public data is related to the private task but is not drawn from the same distribution. Specifically, we evaluate two transfer directions: IMDB → SST-2 and SST-2 → IMDB. In each setting, BASE constructs bilateral subspaces using 5,000 public samples from the source dataset, followed by private fine-tuning on the target dataset. For a fair comparison, DP-LoRA is evaluated under the same cross-dataset public warm-up protocol. All results are averaged over three random seeds. | Public → Private | Method | $\varepsilon = 0.5$ | $\varepsilon = 4.0$ | |:---|:---|:---:|:---:| | IMDB → SST-2 | DP-LoRA | 91.23 ± 0.34 | 91.66 ± 0.43 | | IMDB → SST-2 | **BASE** | **92.67 ± 0.15** | **92.73 ± 0.37** | | SST-2 → IMDB | DP-LoRA | 91.37 ± 0.07 | 91.73 ± 0.04 | | SST-2 → IMDB | **BASE** | **92.00 ± 0.22** | **92.15 ± 0.05** | The results show that BASE consistently outperforms DP-LoRA across both transfer directions and privacy budgets. Specifically, BASE achieves improvements of 1.44/1.07 percentage points on IMDB → SST-2 and 0.63/0.42 percentage points on SST-2 → IMDB under $\epsilon = 0.5/4.0$, respectively. These findings demonstrate that BASE can effectively exploit task-relevant information from external public data to construct informative bilateral subspaces, even when the public and private datasets differ in distribution. Furthermore, Figure 2(b) of the main paper already examines public-data ratios ranging from 1% to 20%, demonstrating the effectiveness of BASE with limited auxiliary data. Together, these results provide evidence that BASE's effectiveness is not confined to the idealized in-distribution public/private split.

Response to Weakness 4 and Question 1:

We clarify that no private data was used for hyperparameter selection, and thus this process incurred no additional privacy cost.

Specifically, all hyperparameter selection was conducted using public auxiliary data. The primary learning-rate search grids were {1e-5, 3e-5, 5e-5, 7e-5} for public warm-up and {1e-4, 3e-4, 5e-4, 7e-4, 1e-3, 3e-3, 5e-3, 7e-3} for private-stage optimization. Additional learning-rate candidates were explored when necessary, exclusively using public data. Bilateral ranks were also determined through public-data experiments based on optimization performance and training stability, without accessing private training or validation data.

Clipping thresholds and training epochs were predetermined before private optimization. All private training runs followed fixed epoch schedules and used the final checkpoint, without private-data-based early stopping or checkpoint selection.

Once the hyperparameters were fixed, private fine-tuning was performed using DP-SGD. The noise multiplier was calibrated according to the target privacy budget, sampling rate, and total number of private optimization steps, with the privacy loss accounted for using RDP. Therefore, the reported privacy guarantees cover the entire private optimization procedure, without any unaccounted privacy cost from hyperparameter selection.

Reviewer_1tqzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BASE, a method that leverages public data to facilitate the differentially private (DP) fine-tuning of language models. The approach utilizes public "warm-up trajectories" to construct fixed input and output bases for each target linear module. During private training, only a small core matrix—situated between these bases—requires optimization. The training process incorporates per-sample gradient clipping and the injection of Gaussian noise. We build this framework upon a "Kronecker-factored gradient-energy" objective function, derive its projection and norm-preserving properties, and provide a privacy guarantee based on standard DP-SGD (Differentially Private Stochastic Gradient Descent) for this fixed, publicly-parameterized formulation. Experimental results across classification, summarization, and instruction-following tasks demonstrate that BASE not only outperforms existing differentially private baselines but also incurs significantly lower computational costs than global subspace fine-tuning methods.

Critical strengths
  1. This paper identifies a notable gap between "architecture-defined parameter-efficient fine-tuning (DP-PEFT) parameterizations" and "high-dimensional global learning subspaces." Leveraging the matrix structure of the target modules offers a natural approach to addressing this scalability challenge.

  2. The update form W=W0+PoutSPinTW = W_0 + P_{out} S P_{in}^T confines privacy-related optimization to a compact "core," while keeping the task-adapted bases fixed. The paper’s derivations regarding gradient projection relationships and norm-preservation clarify how this parameterization interacts with global gradient clipping mechanisms.

  3. The process consists of two stages: the first utilizes only a designated public subset, while the second applies Differentially Private Stochastic Gradient Descent (DP-SGD) to the joint core-gradient vector. Privacy guarantees are strictly bounded within the scope of the private subset. Appendix B also carefully qualifies the optimization findings: while Adam updates remain confined to the subspace, they are not equivalent to performing Adam updates followed by a projection back from the ambient space.

Critical weaknesses
  1. The default protocol randomly partitions the original training data into 10% public data and 90% private data. While this is a reasonable, controlled setup—and one that does not invalidate the paper’s guarantees regarding the “private subset”—it fails to verify whether the selected bases retain their transferability when the public data source is independent or exhibits a distribution mismatch. Varying the amount of public data through ablation studies does not address the issue of distribution shift. Given that “task alignment” is central to this method, its performance under conditions of weak alignment represents a significant limitation that could constrain its general-purpose deployment.

  2. Section 4.2 and Appendix B.1 analyze the second-order moment of the gradient—approximated via Kronecker factorization—and subsequently replace the constituent factors with statistics derived from the “cumulative weight displacement during the warm-up phase.” The appendix explicitly clarifies that these statistics are not estimates of the activation/error factors, and that the “separable trajectory objective” is not equivalent to the “joint bidirectional trajectory energy.” While this transparency is commendable, the paper does not analyze the extent to which the selected bases preserve gradient information specific to the private tasks. Directly measuring the gradient energy or comparing this approach against alternative trajectory construction methods would provide stronger empirical support for the paper’s mechanistic claims.

  3. DP-SFT employs a global subspace rank of 32, whereas the BASE model in Table 4 contains 24,576 trainable core parameters. For DP-LoRA, the rank is fixed at 16, while the bidirectional rank and optimization configuration for the BASE model are selected on a task-specific basis. Although these differences in scalability are inherently significant, the current comparison fails to clearly disentangle “structural advantages” from “model capacity” and “tuning-related variations.” Constructing “privacy-utility” curves—while keeping dimensions or resource budgets constant—would allow for a more precise attribution of the observed performance gains. Furthermore, DP-SFT is absent at the 1.5B parameter scale, which makes it impossible to effectively compare this method against the strongest "learned-subspace" baseline models at that scale.

Questions for authors
  1. Can effective results be achieved when using independent sources or mismatched public data? To what extent must public and private tasks be aligned for "learned bases" to outperform more flexible DP-PEFT parameterization schemes?

  2. What data were used during rank selection, hyperparameter tuning, checkpoint saving, and final evaluation? How should the potential impact be considered if private records are involved in the selection process? Furthermore, how is consistency in the "tuning budget" maintained across different baseline methods?

  3. Are there comparisons against DP-SFT and DP-LoRA that account for alignment in terms of "core dimension" or "memory footprint"? Once these budget constraints are controlled for, how much of the utility gain remains?

  4. Does this trajectory construction method perform better than using bases derived solely from the final public update? Have the authors measured metrics such as the "ratio of retained public gradient energy to private gradient energy" or "subspace alignment"?

  5. Does the total runtime reported in Table 4 include the time required for trajectory storage, basis construction/eigenvalue decomposition, and checkpoint reloading? Could the authors explicitly quantify the memory and time advantages of this method relative to each baseline, rather than merely implying that it has the lowest memory footprint among all approaches?

4 comments

We thank the reviewer for the constructive comments and address the concerns as follows.

Response to Weakness 1 and Question 1:

We thank the reviewer for raising this important question. To address this concern, we conducted additional cross-dataset transfer experiments using SST-2 and IMDB, which share the same sentiment classification objective but differ in data sources and distributions.

Specifically, we evaluate two transfer directions, IMDB → SST-2 and SST-2 → IMDB, using 5,000 public samples from the source dataset for subspace construction, followed by differentially private fine-tuning on the target dataset. For a fair comparison, DP-LoRA is also evaluated under the same cross-dataset public warm-up protocol. All results are averaged over three random seeds.

Public → PrivateMethodε=0.5\varepsilon = 0.5ε=4.0\varepsilon = 4.0
IMDB → SST-2DP-LoRA91.23 ± 0.3491.66 ± 0.43
IMDB → SST-2BASE92.67 ± 0.1592.73 ± 0.37
SST-2 → IMDBDP-LoRA91.37 ± 0.0791.73 ± 0.04
SST-2 → IMDBBASE92.00 ± 0.2292.15 ± 0.05

As shown above, BASE consistently outperforms DP-LoRA across both transfer directions and privacy budgets, achieving improvements of 0.42–1.44 percentage points. These results demonstrate that exact dataset alignment is not required for the learned bilateral bases to remain effective. Public data from a related task, even when originating from a different dataset and distribution, can provide useful update directions for private fine-tuning, allowing BASE to maintain its advantage over DP-LoRA.

Response to Question 2:

Regarding the reviewer's concerns, we provide the following clarifications.

Data usage and privacy accounting. All hyperparameters, including learning rates and ranks, were selected exclusively using the public auxiliary dataset DauxD_{\mathrm{aux}} and fixed before private training. All methods followed predefined training schedules, with the final checkpoint taken from the last epoch rather than selected based on private data. Final evaluation was performed on held-out evaluation sets. Thus, hyperparameter tuning and checkpoint selection incur no additional privacy cost with respect to DprivD_{\mathrm{priv}}.

Tuning fairness. All methods underwent systematic hyperparameter searches. The primary learning-rate grids were {1,3,5,7}×10−5\{1,3,5,7\}\times10^{-5} for public warm-up and {1,3,5,7,10,30,50,70}×10−4\{1,3,5,7,10,30,50,70\}\times10^{-4} for private optimization, with additional searches when necessary. We explored ranks {8,16,32}\{8,16,32\} for DP-LoRA and DP-SFT, reduction factors {16,32,48}\{16,32,48\} for DP-Adapter, and symmetric/asymmetric bilateral ranks for BASE (Appendix C). Given their different parameterizations, we used method-appropriate search spaces rather than enforcing identical structural hyperparameters. We will clarify these tuning protocols in the revised manuscript.

Response to Question 3:

Regarding dimension- or memory-matched comparisons, we clarify that such constraints are not directly applicable to fundamentally different parameterizations. DP-SFT optimizes a global subspace coefficient vector, DP-LoRA updates low-rank factor matrices, while BASE optimizes module-wise bilateral cores. For example, DP-SFT uses a 32-dimensional global subspace, whereas BASE distributes 24,576 core parameters across 96 target modules on RoBERTa-large. Enforcing identical optimization dimensions would either severely restrict BASE's module-wise structure or require a prohibitively large global projection matrix for DP-SFT. Similarly, matching peak memory usage does not imply equivalent optimization capacity, as memory overhead also depends on method-specific computational structures.

Therefore, our comparisons focus on the privacy–utility–efficiency trade-off under each method's native parameterization, rather than artificially enforcing identical parameter counts. As shown in Appendix Table 6, reducing BASE's private optimization dimension by 75%, from 24,576 to 6,144, still achieves 90.74% on QNLI and 87.76% on MNLI-matched, outperforming DP-SFT at 89.33% and 86.96%, and DP-LoRA at 86.15% and 84.64%, respectively. These results indicate that BASE's utility advantage is not solely dependent on its largest evaluated core dimension.

Response to Weakness 2 and Question 4:

We thank the reviewer for this valuable suggestion. To address the concerns regarding trajectory construction and gradient information preservation, we conducted three targeted experiments within the limited rebuttal period.

First, we compare Final Update, which constructs bilateral bases from the final public displacement ΔWT=WT−W0\Delta W_T = W_T - W_0, with BASE, which uses the accumulated trajectory statistics in Eq. (10). All other configurations remain unchanged. The results on RoBERTa-large under ε=0.5\varepsilon = 0.5, averaged over three random seeds, are presented below.

MethodQNLIMNLI-mMNLI-mm
Final Update91.08 ± 1.4287.83 ± 0.3287.94 ± 0.26
BASE91.68 ± 0.5888.50 ± 0.1688.16 ± 0.10

BASE achieves higher mean accuracy across all three metrics, with improvements of 0.60, 0.67, and 0.22 percentage points, respectively. These results support the benefit of aggregating intermediate public updates rather than relying solely on the final displacement.

Second, we examine how effectively the learned bilateral bases preserve task-relevant gradient information. We measure the retained gradient energy using

R=∥Pout⊤GPin∥F2∥G∥F2.R = \frac{\|P_{\mathrm{out}}^\top G P_{\mathrm{in}}\|_F^2}{\|G\|_F^2}.

Using held-out task gradients evaluated at the pretrained model state, we report the ratios of mean retained energy between BASE and alternative basis constructions.

DatasetBASE / RandomBASE / Weight SVDBASE / Final Update
SST-213.56×17.88×1.09×
QNLI7.33×14.95×1.03×

BASE retains substantially more gradient energy than random bases and bases derived from pretrained-weight SVD, while also showing consistent improvements over Final Update. These findings support the effectiveness of public trajectory-based construction in capturing task-relevant gradient directions.

Third, we compare BASE with bilateral bases constructed from Gradient-Covariance PCA to further assess the quality of the learned subspaces. We measure the projected gradient-norm retention ratio

ρ=∥Pout⊤GPin∥F∥G∥F\rho = \frac{\|P_{\mathrm{out}}^\top G P_{\mathrm{in}}\|_F}{\|G\|_F}

and report the relative retention ratio ρBASE/ρPCA\rho_{\mathrm{BASE}}/\rho_{\mathrm{PCA}} across six evaluated attention projections.

DatasetTransformer LayerAttention ProjectionBASE / Gradient PCA
SST-2Layer 1Value99.83%
SST-2Layer 12Output96.85%
QNLILayer 1Value98.42%
QNLILayer 1Key94.45%
QNLILayer 1Query107.57%
QNLILayer 12Output88.47%
Average97.60%

Across these modules, BASE achieves an average relative gradient-norm retention ratio of 97.60% compared with Gradient-Covariance PCA, indicating that trajectory-derived bases can preserve gradient information comparably to gradient-statistics-based bases in the evaluated settings, without explicitly estimating gradient covariance during subspace construction.

As discussed in Appendix B.1, our trajectory statistics serve as empirical surrogates rather than direct estimates of private gradient second moments. The theoretical analysis motivates the bilateral structure, while these additional experiments provide complementary evidence for the effectiveness of trajectory-based subspace construction and its ability to preserve task-relevant gradient information. We will incorporate these findings into the revised manuscript.

Response to Weakness 3 and Question 5:

Regarding computational overhead, as already stated in the efficiency analysis in Section 5, Table 4 reports end-to-end training time. We further clarify that the measurements cover the entire execution process, including public warm-up, trajectory collection and storage, covariance construction, eigendecomposition, checkpoint reloading, and private DP-SGD optimization. As explicitly specified in Table 4, all methods use the same 10%/90% public/private data split, 3 public warm-up epochs, 3 private training epochs, and logical/physical batch sizes of 32/8.

As reported in Table 4, BASE achieves the shortest end-to-end runtime of 65.17 minutes, corresponding to speedups of 1.94×, 1.63×, 1.17×, 1.14×, and 7.49× over DP-Full, DP-LoRA, DP-Adapter, DP-BiTFiT, and DP-SFT, respectively. Its peak memory of 5.72 GiB is approximately 3.29× and 16.03× lower than DP-Full and DP-SFT, respectively, while DP-LoRA, DP-Adapter, and DP-BiTFiT have lower memory requirements. Thus, BASE achieves the lowest end-to-end training time, rather than the lowest peak memory usage among all methods.

Regarding the absence of DP-SFT at the 1.5B scale, this limitation is already discussed alongside Table 2 and quantitatively analyzed in Appendix D.1. For Qwen2.5-1.5B, DP-SFT's global projection matrix alone requires approximately 197.1 GB in FP32 at rank 32, excluding model parameters, optimizer states, and intermediate computations. This makes its original global-subspace implementation impractical. In contrast, BASE avoids the global projection matrix through module-wise bilateral subspaces, enabling efficient private fine-tuning at this scale.

Together with the hyperparameter tuning protocol clarified in Q2 and the dimensionality analysis in Q3, these results support BASE's practical advantages in privacy–utility–efficiency trade-offs.

Dear Reviewer 1tqz:

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We sincerely appreciate your constructive feedback and would be grateful to know whether our responses have adequately addressed your concerns. Thank you again for your time and consideration.

Reviewer_azjJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to leverage public data to improve privacy-utility tradeoff for language tasks. In particular, they first estimate a "task direction" using the public data: they conduct a few steps of warmup training on the public data and define the task direction as the resulting weight update matrix. Then, they compute the left- and right-side low-rank projection bases of the task direction. Finally, they perform DP-SGD in the low-rank space.

The bilateral projection structure is motivated by the Kronecker product identity, and it requires less computation than using one single projection matrix for the same target rank. For private learning, dimension reduction also implies less noise needed for achieving the same privacy guarantee, and potentially improving the utility. Experiments on three model class and multiple datasets demonstrate this improvement in utility compared with several other baselines (random projection, doing one fixed projection from public data, etc.)

Critical strengths
  • The paper combines bilateral subspace projection using public data with DP training. The combination is new, and the proposed improves computational efficiency while reducing the total amount of DP noise required.
  • The paper is well-written and easy to read.
Critical weaknesses
  1. The novelty of the proposed method is limited. Bilateral projection is an established method [1][2]. This paper claims that they propose a surrogate, the weight matrix's update direction during a public warmup training, of the task space using public data. Similar ideas are used in previous works like [3][4][5].
  2. The comparison between DP-SFT and BASE seems unfair. Table 4 shows that DP-SFT and BASE use different DP dimension. How do you select the dimension for each method? Additionally, DP-SFT seems to project to the subspace that captures the most approximate gradient energy arg$\max_P ||vec(G) - P vec(G)||^2$. BASE estimates of the same term, under a structural constraint of the projection matrices (i.e. arg$\max_{P_{in}, P_{out}}) \Vert vec(G) - (P_{in} \otimes P_{out}) vec(G)\Vert^2 $).

It seems that the DP-SFT always preserve more gradient energy than BASE, when restricting PP and Pin⊗PoutP_{in}\otimes P_{out} to the same rank.

[1] Bałazy et al. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. [2] Ye, J. Generalized Low Rank Approximations of Matrices. Mach Learn 61, 167–191 (2005). [3] Yu et al. Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning. ICLR 2021. [4] Yu et al. Large Scale Private Learning via Low-rank Reparametrization. ICML 2021. [5] Zheng et al. Differentially Private Subspace Fine-Tuning for Large Language Models. ArXiv: 2601.11113

Questions for authors

See the second point in weaknesses: can you elaborate the conditions under which bilateral projection leads to better privacy utility tradeoff than a single projection?

4 comments

We thank the reviewer for the constructive comments and address the concerns as follows.

Response to Weakness 1:

We thank the reviewer for the comments. We respectfully disagree that BASE lacks substantive methodological novelty. Although the cited works share individual components with BASE, they differ fundamentally in their objectives, subspace construction, and private optimization mechanisms. Their key limitations, and how BASE addresses them, are summarized below.

MethodPublic Task TrajectoriesBilateral Matrix StructureNo Pretrain-Alignment AssumptionNo Basis Refresh During DPNo Global Projection MatrixFixed Bilateral-Core DP
LoRA-XS [1]✗✓✗—✓—
GLRAM [2]✗✓————
GEP [3]✗✗✓✗✗✗
RGP [4]✗✓✓✗✓✗
DP-SFT [5]✓✗✓✓✗✗
BASE (Ours)✓✓✓✓✓✓

✓: satisfied; ✗: not satisfied; —: not applicable. Public Task Trajectories refers specifically to public parameter-update trajectories used for basis construction. LoRA-XS and GLRAM are not originally DP methods.

Distinctions from LoRA-XS and GLRAM. LoRA-XS constructs fixed bilateral bases from pretrained-weight SVD, relying on the assumption that dominant pretrained directions are informative for downstream adaptation, which is not generally guaranteed. GLRAM focuses on iterative low-rank reconstruction of matrix collections, without considering task-specific optimization directions or differential privacy. In contrast, BASE employs bilateral decomposition to construct task-aligned update subspaces from public optimization trajectories, rather than merely approximating pretrained weights or input matrices.

Distinctions from existing DP subspace methods. As already discussed in the Related Work section, GEP and RGP require repeated estimation of gradient subspaces or carriers during private training, introducing additional computation. RGP also reconstructs updates in the original parameter space. DP-SFT learns task-relevant subspaces from public trajectories but requires costly global projection matrices. BASE instead constructs module-wise bilateral bases once using public data and performs DP optimization directly on fixed, compact cores, avoiding repeated basis estimation and global projection overhead.

Additional experimental evidence. To examine whether the pretrained-weight SVD basis construction strategy adopted by LoRA-XS is effective in our DP setting, we replaced BASE's trajectory-derived bases with pretrained-SVD bases while retaining the same optimization procedure. Under ε=0.5\varepsilon=0.5, this variant achieves only 50.92% and 54.90% accuracy on SST-2 and QNLI (three-seed averages), respectively, compared with 93.69% and 91.68% for BASE. Furthermore, at the pretrained model state, evaluation on three held-out batches shows that BASE preserves 17.9× and 14.9× more gradient energy than pretrained-SVD bases on SST-2 and QNLI, respectively. These results demonstrate that pretrained-weight singular directions are not necessarily aligned with downstream optimization, highlighting the importance of task-adaptive basis construction.

Therefore, BASE's contribution lies in learning task-relevant bilateral subspaces from public optimization trajectories and integrating them with efficient, fixed-core differentially private optimization, rather than simply reusing existing bilateral projection techniques. We will further clarify these distinctions, particularly the relationship to LoRA-XS and GLRAM, in the revised manuscript.

[1] Balazy et al. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. [2] Ye, J. Generalized Low Rank Approximations of Matrices. Mach Learn 61, 167–191 (2005). [3] Yu et al. Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning. ICLR 2021. [4] Yu et al. Large Scale Private Learning via Low-rank Reparametrization. ICML 2021. [5] Zheng et al. Differentially Private Subspace Fine-Tuning for Large Language Models. ArXiv: 2601.11113

Response to Weakness 2 and Question 1:

We thank the reviewer for raising this question. We clarify the dimension selection protocol and explain why the theoretical comparison under identical subspace dimensions does not adequately reflect the practical advantages of BASE.

1. Selection of DP dimensions.

The subspace dimensions were selected empirically using only public auxiliary data. Specifically, we evaluated candidate ranks based on whether they supported stable training-loss reduction during one epoch of public warm-up, while considering computational feasibility. No private data were used for rank selection.

Importantly, the rank definitions differ between the two methods. DP-SFT uses a global subspace dimension of k=32k=32, whereas BASE uses module-wise bilateral ranks (rout,rin)(r_{\mathrm{out}},r_{\mathrm{in}}). For example, on RoBERTa-large/QNLI, BASE adopts ranks (16, 16) across 96 attention projection modules, yielding 24,576 private optimization dimensions. These configurations reflect the different parameterizations and their practical computational constraints.

2. Theoretical comparison versus practical feasibility.

The reviewer's argument assumes identical effective subspace dimensions for unconstrained and bilaterally structured projections. While an optimal unconstrained projection has at least as much gradient-energy preservation capacity under this assumption, matching these dimensions is computationally impractical for the explicit global projection representation used by DP-SFT.

For instance, in Table 4, BASE operates in a 24,576-dimensional module-wise optimization space, whereas DP-SFT uses only 32 global dimensions. Matching BASE's dimension would require DP-SFT to store a global projection matrix occupying approximately 34.9 TB in FP32 on RoBERTa-large. This computational infeasibility is precisely one of the motivations for BASE, rather than a limitation of its bilateral design.

More generally, DP-SFT requires O(dk)O(dk) storage for its explicit global projection matrix, whereas BASE represents its bilateral bases using only O(∑m(dout(m)rout(m)+din(m)rin(m)))O\left(\sum_m (d_{\mathrm{out}}^{(m)}r_{\mathrm{out}}^{(m)} + d_{\mathrm{in}}^{(m)}r_{\mathrm{in}}^{(m)})\right) storage.

This distinction becomes even more pronounced at larger scales. As already quantified in Appendix D.1, even a 32-dimensional global projection for Qwen2.5-1.5B requires approximately 197.1 GB of FP32 storage.

Furthermore, the theoretical optimality of unconstrained projection does not guarantee that a subspace estimated from finite public optimization trajectories will preserve more unseen task gradients or achieve better DP utility.

3. Practical privacy–utility trade-off.

Bilateral projection is particularly advantageous when task-relevant update directions exhibit approximately separable input- and output-side structure, allowing compact module-wise representations to capture useful gradient information without constructing prohibitively large global projection matrices.

Our experimental results demonstrate this practical advantage. As reported in Table 4, BASE reduces peak memory from 91.68 to 5.72 GiB (16.0×) and end-to-end training time from 488.13 to 65.17 minutes (7.5×) compared with DP-SFT, while achieving higher QNLI accuracy (91.73% vs. 90.11%, under ε=4.0\varepsilon=4.0).

Therefore, BASE does not aim to surpass the theoretically optimal unconstrained projection under identical subspace dimensions. Instead, its contribution is to exploit the intrinsic matrix structure to efficiently represent substantially larger task-aligned optimization spaces under realistic computational constraints, achieving a more favorable practical trade-off between privacy, utility, and efficiency.

Thank you for the detailed response and the additional experimental results. The response has cleared up my questions about the comparison with existing methods, and I have raised my score accordingly.

We sincerely thank you for your thoughtful feedback and for reconsidering your assessment of our work. We greatly appreciate your constructive comments and the time you have devoted to reviewing our manuscript. Your suggestions will help us further improve the clarity and quality of our paper.

Reviewer_PSNcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a differentially private fine-tuning method that learns module-wise bilateral subspaces from public data. The method first performs public warm-up and collects weight-update trajectories to identify important input-side and output-side directions. It then freezes these directions and privately optimizes a small core matrix for each target module. Experiments on RoBERTa and Qwen2.5-1.5B report improvements over existing DP fine-tuning methods on several tasks.

Critical strengths

The core idea of using public data to identify useful update directions before private optimization makes sense to me.

The reported efficiency improvements over DP-SFT are indeed substantial and impressive. However, it seems that BASE does not use less memory than DP-LoRA, DP-Adapter, or DP-BiTFiT.

Critical weaknesses

W1. The IFEval results do not appear consistent with the stated evaluation protocol. Table 2 reports mean accuracy over three seeds, and Appendix D specifies strict prompt-level accuracy. The standard IFEval set contains 541 prompts. With one response per prompt per run, the mean percentage must equal 100K(3 x 541) for an integer K. Under this setup, 10 of the 11 reported prompt-accuracy means cannot be obtained after rounding to two decimals. For example, Full FT reports 31.69%, but it is not possible; the adjacent attainable means are 31.67% and 31.73%. This could reflect an undocumented evaluation setup, which needs to be explained.

W2. The privacy proof does not explain whether its accountant covers the batch normalization used in the algorithm. Algorithm 3 samples each example independently, so the batch size varies between steps. It then divides the noisy gradient sum by the actual number of sampled examples, rather than a fixed expected batch size. If I understand it correctly, with an expected batch size of 32, a step that samples 20 examples divides both the gradient sum and noise by 20, not 32. These are different randomized mechanisms. Appendix B.3 simply calls the written mechanism standard DP-SGD and applies its accountant without addressing this difference. Not a big deal for sure, but it would be great if the authors could clarify what the code actually does. If it uses the expected batch size, the algorithm and proof should be corrected. If it uses the actual sampled size, the authors need to justify the privacy accounting for that mechanism.

W3. The privacy implementation needs clarification. Algorithm 3 and Eq. 54 use Poisson sampling but divide the noisy gradient sum by the realized batch size |B_t|. This differs from the expected-batch-size normalization used by implementations like Opacus. Appendix B.3 then invokes standard DP-SGD accounting without explaining whether the accountant covers this normalization. This may simply be a pseudocode issue, but it would be great if you could clarify which denominator the code uses and provide the actual noise multipliers and accountant configuration.

Questions for authors

See W1-W3.

2 comments

We thank the reviewer for the constructive comments and address the concerns as follows.

Response to Weakness 1 and Question 1:

We thank the reviewer for pointing out the inconsistency in our IFEval results. Upon verification, we found that the original evaluation used an incomplete local copy of IFEval containing 489 prompts, which was not explicitly stated in the manuscript. Additionally, the reported accuracies were truncated to two decimal places instead of rounded, and three values were incorrectly reported.

To address these issues, we re-evaluated the final checkpoints of all compared methods on the complete 541-prompt IFEval benchmark. We report the number of correctly evaluated prompts for each of the three random seeds, together with the recalculated mean accuracy and standard deviation.

MethodεSeed 42Seed 43Seed 44Mean ± Std (%)
Full FT–17217417131.85 ± 0.28
DP-Full0.514514114326.43 ± 0.37
DP-LoRA0.514014114126.00 ± 0.11
DP-Adapter0.513913613825.45 ± 0.28
DP-BiTFiT0.514814514627.05 ± 0.28
BASE0.515715615728.96 ± 0.11
DP-Full4.014714514827.11 ± 0.28
DP-LoRA4.014414614326.68 ± 0.28
DP-Adapter4.014814414426.86 ± 0.43
DP-BiTFiT4.015114915327.91 ± 0.37
BASE4.016016216129.76 ± 0.18

All mean accuracies are calculated directly from the integer counts over 541 prompts and three seeds. Standard deviations are computed across the three seed-level accuracies, and all results are rounded to two decimal places.

The re-evaluation confirms that BASE consistently outperforms all evaluated DP baselines under both privacy budgets. Specifically, BASE achieves 28.96% and 29.76% prompt-level accuracy at ε = 0.5 and ε = 4.0, respectively, exceeding the strongest DP baseline by 1.91 and 1.85 percentage points. Furthermore, the changes in mean accuracy across all 11 entries are within 0.25 percentage points, and the main comparative conclusions remain unchanged.

Response to Weaknesses 2–3 and Questions 2–3:

We thank the reviewer for raising concerns about the consistency between our mathematical description and privacy accounting. We provide the following clarifications based on our actual implementation.

Implementation details. Our implementation is publicly available in the anonymous repository linked at the end of the abstract. As shown in Lines 304–313 of train.py, we implement private optimization using Opacus, which employs Poisson sampling, global per-example ℓ2\ell_2 gradient clipping, and Gaussian noise addition. Under the default loss_reduction="mean" setting, Opacus normalizes the noisy gradient sum by the fixed expected batch size, rather than the realized batch size ∣Bt∣|B_t|. Furthermore, Lines 323–337 of train.py show that physical micro-batches are handled by BatchMemoryManager, preserving the underlying logical DP updates.

We acknowledge that the normalization by ∣Bt∣|B_t| in Eq. (3), Algorithm 3, and Eq. (54) does not accurately reflect our implementation. The correct denominator is the fixed expected batch size. Accordingly, the privacy argument in Appendix B.3 remains applicable to the implemented mechanism. Specifically, the globally clipped joint subspace gradients are processed using the standard Poisson-subsampled Gaussian mechanism, while the subsequent normalization by the fixed expected batch size constitutes deterministic post-processing and incurs no additional privacy loss. We will correct these expressions in the revised manuscript.

Privacy accounting configuration. We use the RDP accountant provided by Opacus and calibrate the noise multiplier through make_private_with_epsilon, as implemented in Lines 304–313 of train.py. The calibration accounts for the target privacy budget, sampling rate, training epochs, and δ\delta. We independently recalculated the noise multipliers using the original experimental configurations. The results are summarized below:

DatasetPrivate Epochsδ\deltaσ\sigma (ϵ=0.5\epsilon=0.5)σ\sigma (ϵ=4.0\epsilon=4.0)
SAMSum51e-51.3281250.599365
Dolly-15k51e-51.3476560.605164
SST-231e-51.0937500.509949
QNLI31e-51.0546880.493774
MNLI31e-61.0742190.481873
QQP31e-61.0937500.484009

The discrepancy is therefore limited to the mathematical description and does not affect the implemented DP-SGD mechanism or its privacy accounting. We will clarify these details and correct the corresponding expressions in the revised manuscript.

Dear Reviewer PSNc,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We sincerely appreciate your constructive feedback and would be grateful to know whether our responses have adequately addressed your concerns. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.