acceptodds
Under review as a conference paper at ICLR 2027

Privacy Needs the Right Space: Rethinking Federated LoRA Beyond Rank

Abstract

Low-Rank Adaptation (LoRA) can still expose many coordinates to private optimization, while reducing the number of coordinates can discard useful adaptation directions. We study how both private dimension and retained update directions shape a compact trainable space for federated LoRA under clipping and DP noise. We propose FedTIE, a differentially private federated LoRA framework built on Tied Isometric Embedding (TIE), which uses privatized warm-up profiles to determine which factor coordinates across layers and factors share latent variables, with statistical load penalties regularizing the resulting assignment. Its sparse embedding preserves the Euclidean energy of latent perturbations when lifted to the concatenated factor space. The embedding is then frozen, and main-stage optimization and communication operate in the latent space. We establish an end-to-end sample-level differential privacy guarantee covering space selection and main training, together with a finite-horizon convergence bound for the induced subspace objective. On GLUE, FedTIE improves average accuracy over the strongest evaluated baseline by 3.40% and 4.44% at and , respectively, with , while also improving non-private average accuracy and achieving competitive generation performance. On RoBERTa-base, FedTIE reduces per-round communication by 51.9% relative to FFA-LoRA and 85.2% relative to FedASK. Code is available at https://anonymous.4open.science/r/fedhsip/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Low-rank adaptation, Federated learning, Differential privacy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_X6nHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

DP-LoRA applies DP-SGD to optimize the Low Rank Adapters (LoRA) AA, BB, which first clips the gradient and then adds Gaussian noise proportional to the clipping threshold. This introduces noise scaling with to the gradient dimension. Hence, reducing the gradient dimension decreases the noise required to satisfy the privacy constraint. Additionally, in the federated setting, reducing the gradient dimension also lowers communication costs. Therefore, this paper proposes a method to further reduce the dimension of the LoRAs in the federated setting, thereby improving the privacy-accuracy tradeoff and reducing the communication cost.

To achieve this, they concatenate the LoRAs for all layers into θ=[A1,B1,…,Aℓ,Bℓ]∈RD\theta = [A_1, B_1, …, A_{\ell}, B_{\ell}] \in \mathbb{R}^D. Instead of updating θ\theta, they update PθP \theta, where P∈Rd×DP\in \mathbb{R}^{d\times D} is an orthonormal projection matrix with d≪Dd\ll D. Hence, this method reduces the gradient dimension from DD to dd, which decreases the privacy-induced noise and the communication costs, at the expense of a slight subspace approximation error. To mitigate this approximation error, they choose PP by first doing a private warmup training on all clients to get a update Δθi\Delta \theta_i for each client ii and then selecting a sparse and orthonormal PP that minimizes the gradient conflict among the projected updates {PΔθi}\{P\Delta\theta_i\}'s.

Empirical results show that their method outperforms several baselines on four datasets across different ϵ′s\epsilon's.

Critical strengths

The method is intuitive and provably reduces communication cost; empirically, it also improves the privacy–accuracy tradeoff over multiple baselines.

Critical weaknesses

The paper chooses the projection matrix P in the form specified in equation (5), however, it does not provide sufficient justification for this particular choice. Equation (3) provides a decomposition that holds for every orthonormal P. Hence, the optimality of the proposed private selection mechanism is not clear.

Questions for authors
  1. Under a limited privacy budget, is TIE's sparse signed grouping structure easier to estimate accurately than other private subspace estimation methods (e.g., DP-PCA), and hence leading to a better privacy–accuracy tradeoff?

  2. How would the results change if a small public dataset is available, allowing one to estimate the best subspace (PP) from the public data?

  3. Is this method a private version of Uni-LoRA?

5 comments
AuthorsOpenReview ↗

Response to Weakness

Equation (3) provides an error decomposition that applies to any orthogonal projection matrix; however, this does not imply that different projection matrices yield the same model performance. In fact, both the discarded-signal error and the projection-induced clipping bias depend on the specific choice of PP. Therefore, selecting an appropriate low-dimensional subspace remains an important problem. In this work, we adopt the sparse signed grouping structure in Equation (5). We do not claim that this structure is globally optimal among all possible orthogonal projection matrices. Rather, it is designed to simultaneously ensure exact orthogonality, enable cross-layer parameter sharing, and maintain low storage and computational overhead. Building on this structure, we leverage privacy-preserving warm-up statistics to perform data-driven subspace selection based on factors such as client heterogeneity patterns and parameter sensitivity. Experiments comparing against random projections further validate the effectiveness of this selection strategy on practical tasks.

AuthorsOpenReview ↗

Response to Question 1

We believe that the potential advantage of TIE over DP-PCA primarily stems from its structural prior, which is specifically designed to address the parameter-sharing problem in federated LoRA. DP-PCA focuses on estimating a general dominant subspace, whereas TIE formulates subspace estimation as a sparse, signed coordinate-grouping problem. It only requires per-parameter update profiles across different clients, together with statistics such as sensitivity, update energy, and heterogeneity, without the need to estimate a full high-dimensional covariance matrix. Therefore, when the effective update directions in a task indeed exhibit coordinate-sharing structures across layers and factors, this constrained estimation approach is expected to reduce the statistical estimation difficulty under a limited privacy budget. Furthermore, TIE obtains its warm-up statistics through a differentially private mechanism, while the grouping procedure constitutes post-processing, and the privacy cost incurred during the warm-up phase is already accounted for in the end-to-end privacy budget. Experimentally, TIE significantly outperforms random isometric projections under the same projection dimension, providing further empirical evidence, to some extent, of the effectiveness of using data-driven structured grouping to select the subspace.

AuthorsOpenReview ↗

Response to Question 2

We believe this is a highly valuable research direction. If a small-scale but representative public dataset is available, it can be used to estimate the update profiles of individual LoRA parameters and construct the subspace PP through TIE's sparse signed grouping mechanism. This approach could potentially offer two benefits:

  1. Improving the quality of subspace estimation: Since public data is not affected by differential privacy noise, it can provide more accurate parameter-update statistics, thereby enabling the construction of more appropriate parameter-sharing structures and reducing the loss of informative model update directions caused by incorrect grouping.
  2. Reducing privacy budget consumption: Since public data does not involve user privacy, constructing PP based on such data does not require any differential privacy budget. This allows more of the privacy budget to be allocated to subsequent model training, potentially improving the trade-off between privacy protection and model accuracy. Of course, the effectiveness of this approach also depends on the consistency between the public data distribution and the actual client data distributions. If there is a substantial discrepancy between them, the subspace estimated from public data may fail to adequately capture the actual client heterogeneity, potentially affecting model performance. We will include additional experiments in the revised manuscript to compare the performance of subspaces constructed using public data with those constructed using private warm-up data, thereby further validating the effectiveness of this approach.
Reviewer_MNW9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FedTIE studies federated LoRA with sample-level differential privacy. It uses a private warm-up to group LoRA factor coordinates into shared latent variables, then freezes this sparse isometric embedding and restarts training in the smaller space. The paper separates discarded signal, clipping bias, and dimension-dependent noise, and provides privacy accounting and a conditional convergence analysis. Evaluation covers GLUE with RoBERTa-base and generation with GPT-2, reporting improved utility and reduced communication.

Critical strengths
  1. Useful formulation. The decomposition in Eq. 3 clearly explains why reducing dimension alone does not strictly lead to improved utility: the retained directions and clipping bias also matter.
  2. Explicit accounting for space selection. The formal construction charges the private warm-up and subsequent training to one end-to-end sample-level guarantee, addressing the potential leakage from learning the embedding itself.
Critical weaknesses
  1. The empirical reporting is not currently trustworthy. Table 1's caption says "Standard deviations marked with dagger are randomly simulated placeholders for formatting purposes". There are also numerical inconsistencies independent of that sentence. If the Average column summarizes the same seeded runs as the five task columns, its SD cannot exceed the mean of their SDs. For FedTIE at ε = 6, this bound is (2.76 + 2.53 + 1.99 + 2.23 + 1.95)/5 = 2.292, yet the reported SD is 3.20. FFA-LoRA at ε = 3 similarly reports 6.47 against a bound of 2.224. Experimental descriptions also conflict: Section 4.2 uses 15 warm-up rounds while Appendix E.2 uses 20; generation is called non-private in Section 4.2 but private at ε = 6 in Appendix E.2. These issues affect the central evidence, privacy interpretation, and communication claims, and honestly the whole paper.
  2. The evaluation does not sufficiently isolate the assignment's contribution. The random-map comparison is limited to one task, dimension, and privacy budget, without reported variability, and changes both the sharing directions and load balance. There is no dimension sweep or dimension-matched standard LoRA baseline. Consequently, the evidence does not establish how much improvement comes from the learned assignment rather than reduced dimension or balancing. The convergence analysis does not supply that missing comparison; it analyzes optimization within the selected subspace.
Questions for authors
  1. Reporting clarification: Which uncertainty entries are measured, and which are placeholders? Please provide the three per-seed task scores and the precise calculation of Average ± SD, explaining the incompatible values above.
  2. Configuration clarification: Which warm-up schedule, local-step counts, and noise multipliers produced the tables? Please reconcile Sections 4.2, B.5, and E.2, identify Table 3's actual privacy setting, and recompute communication including the transmitted warm-up statistic blocks.
  3. Evidence needed to reconsider: Can you provide repeated-seed results against a balanced random assignment and a dimension-matched standard baseline, at the same end-to-end privacy budget?
5 comments
AuthorsOpenReview ↗

Response to Question 1

We thank the reviewer for pointing out the issues in the reporting of our experimental results. We have re-examined the experimental results and the underlying run records, and would like to provide the following clarifications:

(1) Regarding the placeholder value for the standard deviation. We confirm that, in Table 1, only the standard deviation in the Average column was a placeholder value temporarily inserted during the manuscript formatting process. Due to an oversight during final typesetting, it was not replaced with the actual computed result. This was a formatting oversight on our part, rather than an actual experimental statistical result. We would like to emphasize that the means and standard deviations reported for all five task columns were calculated from the actual experimental results, and the mean in the Average column was also computed from the actual experimental results. We sincerely apologize for this oversight and thank the reviewer for helping us identify this issue.

(2) Regarding the experimental results from three random seeds and the statistical methods. We have additionally provided the exact experimental scores obtained by each method on all five tasks using three random seeds (4242, 4343, and 4444), with all scores reported to two decimal places. The detailed results are presented in the table below. The statistical procedures are as follows:

  • Means and standard deviations for each task: The arithmetic mean and sample standard deviation are calculated separately based on the experimental scores obtained from the three random seeds.
  • Mean and standard deviation for the Average column: We first calculate the average score across the five tasks for each random seed, yielding three values. We then calculate the arithmetic mean and sample standard deviation of these three values. All statistical calculations are performed using the original experimental data retained to 16 decimal places, with rounding to two decimal places applied only when presenting the final results.

(3) Regarding the underlying experimental run records. To address the reviewer's concerns about reproducibility and data reliability, we have provided the complete raw experimental records for all three random seeds, retaining their original 16-decimal-place precision, in the Underlying Run Records subsection of the Details of Ethics Concerns section of the review. This allows the reviewer to verify the specific scores for each task and the procedure used to calculate the statistical results in the Average column. We once again thank the reviewer for bringing this issue to our attention. In the revised manuscript, we will correct the standard deviation in the Average column and revise the table notes to avoid similar ambiguities.

AuthorsOpenReview ↗

Supplementary Experimental Results for Question 1: Per-Seed Scores

non_i.i.d.method-seedmnli_matchedmnli_mismatchedqnliqqpsst2
Non Privatefedavg-seed4282.0084.8347.1885.4649.29
Non Privatefedavg-seed4386.5885.8552.3287.6451.77
Non Privatefedavg-seed4486.4685.8352.1987.5851.71
Non Privateffa-seed4286.8484.3086.0285.2195.10
Non Privateffa-seed4382.0783.2084.3384.3892.48
Non Privateffa-seed4480.1682.7683.6584.0491.43
Non Privatefedsvd-seed4283.9584.6789.0787.0694.55
Non Privatefedsvd-seed4383.7284.2388.6186.1993.55
Non Privatefedsvd-seed4482.8382.4786.7582.7189.56
Non Privatefedask-seed4288.0386.2552.6189.5853.18
Non Privatefedask-seed4386.3485.4149.9087.7452.28
Non Privatefedask-seed4481.8985.2949.7184.0747.31
Non Privatela-lora-seed4278.2979.1185.7980.7590.83
Non Privatela-lora-seed4379.3579.8486.4681.9791.10
Non Privatela-lora-seed4482.5482.0488.4885.6391.91
Non Privateas-lora-seed4284.3584.1189.5585.8891.37
Non Privateas-lora-seed4382.5583.3489.9685.7790.66
Non Privateas-lora-seed4477.1278.5585.0081.6789.41
Non Privatefedtie-seed4285.3384.6992.3485.9694.35
Non Privatefedtie-seed4386.7185.3093.5986.2794.66
Non Privatefedtie-seed4479.7982.2587.3584.7293.12
ϵ = 6fedavg-seed4268.8169.8049.9271.7977.98
ϵ = 6fedavg-seed4376.8977.2454.4781.2383.82
ϵ = 6fedavg-seed4479.7779.8956.0984.6185.91
ϵ = 6ffa-seed4271.0171.4884.2883.6895.71
ϵ = 6ffa-seed4368.8169.7582.3682.0493.44
ϵ = 6ffa-seed4460.0262.8174.6675.4684.37
ϵ = 6fedsvd-seed4272.4073.1978.6876.8486.78
ϵ = 6fedsvd-seed4376.9378.5381.5781.4989.60
ϵ = 6fedsvd-seed4475.5276.2981.2879.8288.88
ϵ = 6fedask-seed4281.3380.0264.3381.7983.77
ϵ = 6fedask-seed4373.9575.1260.3278.3078.81
ϵ = 6fedask-seed4472.6174.2359.5977.6677.91
ϵ = 6la-lora-seed4260.7363.4570.2373.9387.21
ϵ = 6la-lora-seed4357.1058.4478.2977.8389.82
ϵ = 6la-lora-seed4461.9765.1677.4574.9387.88
ϵ = 6as-lora-seed4279.8179.6777.6782.0387.32
ϵ = 6as-lora-seed4375.1776.3574.4279.6186.16
ϵ = 6as-lora-seed4466.4668.8767.1075.0683.54
ϵ = 6fedtie-seed4281.1481.9586.3482.2193.80
ϵ = 6fedtie-seed4379.8280.7485.4081.1492.84
ϵ = 6fedtie-seed4475.8477.0882.5977.9289.97
ϵ = 3fedavg-seed4273.5972.7255.7477.8277.36
ϵ = 3fedavg-seed4380.3670.7152.5082.8375.85
ϵ = 3fedavg-seed4466.8879.6963.9372.8883.82
ϵ = 3ffa-seed4256.6457.7580.6682.7894.31
ϵ = 3ffa-seed4352.3655.5676.5878.1089.11
ϵ = 3ffa-seed4452.8955.8477.0978.6889.76
ϵ = 3fedsvd-seed4279.6577.7184.8681.9994.20
ϵ = 3fedsvd-seed4370.6573.5075.8177.1987.00
ϵ = 3fedsvd-seed4468.0772.3372.7475.6985.10
ϵ = 3fedask-seed4275.1276.8466.3580.7986.32
ϵ = 3fedask-seed4373.9175.6764.2579.5884.28
ϵ = 3fedask-seed4470.2472.1757.9275.9478.13
ϵ = 3la-lora-seed4251.8154.7875.6676.5187.51
ϵ = 3la-lora-seed4349.6551.4472.2674.0586.42
ϵ = 3la-lora-seed4447.7948.3969.1671.8885.47
ϵ = 3as-lora-seed4273.2876.6077.6580.0170.25
ϵ = 3as-lora-seed4372.2775.1676.3679.2769.76
ϵ = 3as-lora-seed4465.9966.1468.2974.6466.76
ϵ = 3fedtie-seed4279.7080.4585.0381.5292.81
ϵ = 3fedtie-seed4380.8281.2285.5582.8693.41
ϵ = 3fedtie-seed4474.5976.9682.6875.4390.05
AuthorsOpenReview ↗

Response to Question 2

1. Regarding the Inconsistency in the Number of Warm-up Rounds and Local Training Steps. The different warm-up schedules are adopted for different experimental purposes and do not represent a change in the underlying algorithm. For the main GLUE performance evaluation, we use Rw=20R_w=20 warm-up rounds and Rm=130R_m=130 main-training rounds. Each warm-up round involves 3 local optimization steps, while each main-training round uses τ=10\tau=10 local steps. In contrast, the communication cost analysis considers a separate 150-round configuration with Rw=15R_w=15 warm-up rounds and Rm=135R_m=135 main-training rounds. We will clarify these experimental settings and their respective purposes in the revised manuscript.

2. Regarding Noise Multipliers and Privacy Budget Allocation. Importantly, the noise multipliers νw\nu_w and νm\nu_m are automatically calibrated rather than manually specified, as stated in the experimental details. Specifically, they are determined using the analytical sampled-Gaussian RDP accountant according to the target end-to-end privacy budget (ϵ,δ)(\epsilon,\delta), sampling rates, clipping thresholds, and the release schedules of both stages. All privacy-relevant releases during warm-up and main training are included in the same privacy accounting procedure. Consequently, different warm-up schedules lead to corresponding recalibration of the noise multipliers, while maintaining the prescribed end-to-end privacy guarantee. We will further clarify the local-step counts, privacy budget allocation, and calibrated noise multipliers for each experimental configuration in the revised manuscript.

3. Regarding Whether Table 3 Uses Differential Privacy. The description of the experiments in Table 3 as non-private experiments in Section 4.2 is indeed a typographical error. The actual experimental configuration should be based on Appendix E.2: the experiments do employ a differential privacy mechanism, with a privacy budget of ϵ=6\epsilon=6 and δ=10−5\delta=10^{-5}. Specifically, the experiments use 10 communication rounds, comprising 3 warm-up rounds and 7 main-training rounds, with 3 local epochs per round and one gradient-accumulation step. The noise multiplier is automatically calibrated to satisfy the specified privacy budget. We will correct this description in the revised manuscript.

4. Regarding the Communication Cost of Warm-up. We acknowledge that the previous communication cost calculation did not explicitly include the additional warm-up statistic blocks. According to our method, each warm-up release is a jointly clipped and Gaussian-perturbed message containing both the provisional optimization block and the statistic blocks required for TIE construction. These blocks are transmitted as part of the same joint message and are covered by the privacy accountant. We will therefore revise the communication cost calculation to include the complete warm-up message, rather than counting only the provisional LoRA parameter exchange. The main-stage communication cost remains unchanged: after the embedding is constructed and cached, each communication round requires transmitting only the d=46,080d=46,080-dimensional latent parameter vector in each direction. We will update the overall communication costs and corresponding reduction percentages accordingly, while retaining the main-stage communication savings.

Reviewer_usZkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FedTIE, which has its core design on private dimensionality and update directions for private federated low-rank adaptation. FedTIE works in two stages: 1) a warm-up stage, which constructs the sketching matrix PP with each client's per-example statistics, and 2) a training stage, which restarts training using latent-space DP-SGD. An end-to-end RDP guarantee covers both stages. Empirically, FedTIE improves average accuracy and reduces communication cost than the baselines.

Critical strengths
  1. The paper treats private federated LoRA as two coupled choices: how many coordinates are optimized under DP, and which update directions those coordinates span. Eq. 3 cleanly separates the resulting error into discarded signal, projected clipping bias, and Gaussian noise energy. The design of the Tied Isometric Embedding follows directly from this decomposition as the learned assignment targets which directions are retained.

  2. The privacy analysis charges every data-dependent component to the budget. This includes the costs specific to this design: the warm-up statistics and the embedding constructed from them. All statistics are concatenated and jointly clipped into a single sampled-Gaussian release per step, which avoids unaccounted composition within a step.

Critical weaknesses
  1. The experiments confound private dimension with direction choice. Table 1 holds the privacy budget fixed, but FedTIE differs from the baselines in two ways at once: it trains far fewer private coordinates and it uses learned directions. By Eq. 3, the Gaussian noise term scales with dd, so part of the gain should come from compactness alone. Also, a single dd is used throughout all experiments. The evaluation is also limited to three clients with full participation. This leaves open how the method behaves at realistic federation sizes and under partial participation.

  2. The convergence analysis gives limited insight into the proposed method. Theorem 3 is a standard local-SGD analysis with a biased oracle. Its bound contains a 10BP210B_P^2 clipping-bias term that does not vanish with TT. More importantly, Theorem 3 only bounds the expected average squared norm of the induced subspace gradient, which is within the selected dd-dimensional subspace.

Questions for authors
  1. Please explain the following inconsistencies in this paper. First, Section 4.2 and Table 4 compute communication with 15 warm-up and 135 main rounds, while E.2 states that experiments use Rw=20R_w = 20 and Rm=130R_m = 130. Second, Section 4.2 describes Table 3 as non-private, while E.2 states DP was enabled with ε=6ε = 6. Third, if Stage 2 restarts from θd=0θ_d = 0 around the public reference, accuracy should drop to near chance at the transition step. However, the drop in Figure 3 does not support the analysis.

  2. This paper gives the target privacy budget but none of the parameters that produce it, for FedTIE or for any baseline. Can the authors provide the essential factors, including the noise multipliers and clipping thresholds? Without these, readers cannot verify that the comparison at equal ε\varepsilon is calibrated consistently across methods.

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

1. Regarding the confounding effects of private dimensionality and update direction selection. The primary goal of our main experiments is to evaluate the overall effectiveness of the FedTIE framework, rather than to isolate the gains solely attributable to dimensionality reduction. In fact, we specifically designed a fixed-dimensional ablation experiment: we kept the training procedure unchanged, replaced TIE with a randomly sampled isotropic embedding, and maintained the same private dimensionality of d=46,080d=46,080. Both embeddings satisfy P⊤P=IdP^\top P=I_d. Therefore, under the same per-coordinate noise variance, they incur the same Gaussian noise energy of dv2dv^2. The experimental results show that, on MNLI with ϵ=6\epsilon=6, the random embedding achieves only 55.05% accuracy, whereas FedTIE achieves 78.93%, representing an improvement of 23.88 percentage points. This demonstrates that, even when controlling for dimensionality and the isometric property, learning update directions from client statistics provides substantial performance gains. Therefore, the advantages of FedTIE cannot be attributed solely to reducing the number of private optimization coordinates. Furthermore, similar design differences also exist among the baseline methods: compared with the classical FedAvg, FFA-LoRA already substantially reduces the number of coordinates optimized under differential privacy; compared with FFA-LoRA, FedSVD further introduces learned update directions. Therefore, the primary objective of our comparisons is to evaluate the overall effectiveness of different methods under their respective design mechanisms, i.e., to assess their overall performance under the same privacy budget, rather than to isolate the contribution of each individual design component. The aforementioned fixed-dimensional ablation experiment further validates the independent contribution of the learned update direction mechanism in FedTIE.

2. Regarding the use of a fixed private dimensionality across all experiments. We adopt d=46,080d=46,080 as a unified experimental configuration primarily to evaluate the effectiveness of TIE in learning update directions under the same latent parameter dimensionality, while avoiding additional dimension tuning for different tasks. We do not claim that this dimensionality is optimal for all tasks. In fact, the dimensionality dd and the embedding matrix PP play distinct roles in controlling the privacy noise magnitude and determining the retained update directions, respectively. Therefore, further investigating the performance variations and trade-offs under different private dimensionalities is a worthwhile direction for future research.

3. Regarding the use of only three clients with full participation. We would like to clarify that using three clients with full participation is merely the experimental configuration adopted in this work, rather than an inherent limitation of the FedTIE algorithm. In fact, the algorithm allows K′K' clients to be sampled from a total of KK clients in each communication round. The convergence analysis in the appendix also accounts for partial client participation and explicitly characterizes the variance introduced by client sampling. Therefore, FedTIE supports partial client participation at both the algorithmic and theoretical levels. Furthermore, we evaluate its performance under different levels of data heterogeneity by varying the Dirichlet distribution parameter α\alpha. Nevertheless, we acknowledge that empirical evaluations involving a larger number of clients and partial participation settings are currently lacking, and we will further investigate these scenarios in future work.

AuthorsOpenReview ↗

Response to Weakness 2

1. Regarding the Issue That the Clipping Bias Term Does Not Vanish as the Number of Training Rounds TT Increases.

We acknowledge that the 10BP210B_P^2 term in Theorem 3, where BPB_P denotes an upper bound on the clipping bias in the projected space, does not automatically vanish as TT increases. However, this does not necessarily imply that the method must incur substantial performance degradation in practical training. Rather, it reflects the inherent bias introduced by gradient clipping in differentially private SGD. Since gradient clipping changes the expectation of the gradient estimator, this bias cannot be guaranteed to vanish with increasing training rounds without additional assumptions. It is worth noting that when clipping does not alter the gradients, i.e., BP=0B_P = 0, Theorem 3 yields the standard O(T−1/2)O(T^{-1/2}) convergence guarantee in terms of first-order stationarity. Therefore, this term primarily characterizes the impact of persistent clipping bias on the attainable convergence accuracy, rather than a convergence deficiency specific to FedTIE.

2. Regarding the Issue That Theorem 3 Only Guarantees Convergence Within the Selected dd-Dimensional Subspace.

This is actually consistent with the design objective of FedTIE. FedTIE does not directly optimize the parameters in the original DD-dimensional parameter space. Instead, it constructs a low-dimensional embedding matrix PP through TIE and restricts the optimization to the dd-dimensional subspace spanned by its columns:

θD=θD,ref+Pθd.\theta_D = \theta_{D,\mathrm{ref}} + P\theta_d.

Therefore, the actual objective function that needs to be analyzed is the constrained objective:

F(θd)=F(θD,ref+Pθd),F(\theta_d) = F(\theta_{D,\mathrm{ref}} + P\theta_d),

whose gradient satisfies:

∇F(θd)=P⊤∇F(θD).\nabla F(\theta_d) = P^\top \nabla F(\theta_D).

Theorem 3 establishes a bound on the expected squared norm of this gradient, thereby establishing a first-order stationarity guarantee for FedTIE within its feasible optimization subspace.

Furthermore, the appendix of the paper, in the section Full-space interpretation and the role of TIE, provides a further decomposition of the full-space gradient:

∥∇F(θD)∥2=∥∇F(θd)∥2+∥(I−PP⊤)∇F(θD)∥2.\|\nabla F(\theta_D)\|^2 = \|\nabla F(\theta_d)\|^2 + \|(I - PP^\top)\nabla F(\theta_D)\|^2.

This shows that the full-space gradient consists of two components: the gradient within the selected subspace and the gradient along the orthogonal complement (i.e., the discarded directions). If the residual gradient component along the orthogonal complement can be further controlled, the convergence result within the subspace can be extended to a full-space stationarity guarantee. Therefore, our theoretical analysis does not overlook full-space optimization. Rather, it first establishes a convergence guarantee for the constrained optimization procedure actually implemented by FedTIE, and further clarifies its relationship to full-space convergence.

AuthorsOpenReview ↗

Response to Question 1

1. Regarding the Inconsistency in the Number of Warm-up Rounds. The two configurations are adopted for different experimental purposes and do not involve any substantive contradiction. In the main performance evaluation experiments, we set the number of warm-up rounds to RW=20R_W = 20, whereas in the communication cost analysis (which primarily considers Rm=130R_m = 130 rounds), we use RW=15R_W = 15 to evaluate the overall communication cost under different warm-up phase lengths. It is important to emphasize that the number of warm-up rounds RWR_W is a tunable hyperparameter rather than a fixed algorithmic setting. Its purpose is to collect client-side statistics for constructing the TIE embedding; therefore, it can be adjusted according to the specific requirements of each experiment. For example, in the ablation study where we replace TIE with a randomly generated uniform embedding, the warm-up phase is no longer needed to learn the embedding, and thus the number of warm-up rounds can be set to RW=0R_W = 0. Naturally, the privacy budget allocation and noise multiplier corresponding to different numbers of warm-up rounds are also appropriately calibrated to maintain the same end-to-end privacy budget.

2. Regarding Whether Table 3 Uses Differential Privacy. The description of the experiments in Table 3 as non-private experiments in Section 4.2 is indeed a typographical error. The actual experimental configuration should be based on Appendix E.2: the experiments do employ a differential privacy mechanism, with a privacy budget of ϵ=6\epsilon = 6 and δ=10−5\delta = 10^{-5}. We will correct this description in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.