acceptodds
Withdrawn from ICLR 2027

Privacy-Preserving Federated Fine-Tuning of Language Models under Non-IID Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentially private federated fine-tuning enables collaborative adaptation of language models while protecting sensitive training records, but its utility is challenged by client heterogeneity and privacy-induced noise. Control variates mitigate client drift, yet noisy updates also perturb local trajectories and subsequent control states. We propose LSSCA, a federated fine-tuning framework that combines stochastic controlled averaging with fixed random subspace filtering. Constructed independently of private data without requiring public reference data, the filter preserves reference-subspace components and attenuates orthogonal components of the complete control-corrected noisy update. An exact signal–noise decomposition characterizes the trade-off between noise reduction and signal distortion, without assuming that the random subspace aligns with useful gradients. Under the stated assumptions, including inactive clipping, we establish an non-convex stationarity rate at a fixed noise scale without an additional gradient-dissimilarity assumption. The analysis explicitly accounts for noise propagation through local trajectories and control states, retaining the costs of filtering in the convergence constants. Experiments across NLP benchmarks, privacy budgets, and client heterogeneity levels demonstrate improved privacy–utility trade-offs over the evaluated DP-FL baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mcdLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LSSCA, a unified privacy-preserving federated fine-tuning framework that addresses performance degradation caused by the interaction between client heterogeneity and privacy noise in differentially private federated learning. The method combines Stochastic Controlled Averaging with fixed random Subspace Filtering, correcting client drift while preserving update components within a reference subspace and attenuating privacy noise in orthogonal directions. The authors establish an O(T−1/2)\mathcal{O}(T^{-1/2}) non-convex stationarity rate at a fixed noise scale and characterize the trade-off between noise attenuation and signal distortion through analyses of the residual control dynamics and noise propagation. Experiments on multiple NLP benchmarks using RoBERTa-Base and GPT-2 XL demonstrate improved privacy–utility trade-offs across different privacy budgets and levels of data heterogeneity. The paper identifies the tension arising from the interaction between client-drift correction and privacy-noise attenuation, introduces a lightweight random subspace filter, and supports the framework with rigorous RDP accounting and non-convex convergence proofs.

Critical strengths

1.Addresses a relevant problem: the interaction between client heterogeneity and privacy noise in federated fine-tuning. Combines control-based drift correction with soft subspace filtering in a unified framework. Explicitly characterizes the trade-off between noise attenuation and signal distortion.

2.Provides RDP privacy accounting and non-convex convergence analysis under clearly stated assumptions.

3.Evaluates multiple NLP tasks and model scales, with comparisons across privacy budgets, heterogeneity levels, and component ablations. Uses exact control-increment reconstruction to reduce the uplink increment-vector payload.

Critical weaknesses

1.The benefits of random directional filtering over appropriately tuned scalar update scaling are not sufficiently isolated.

2.The empirical relevance of the inactive-clipping assumption is not established through clipping statistics. Joint sensitivity to qq and rr, including variation across random basis initializations, remains insufficiently explored.

  1. Blockwise implementation details, LoRA configurations, and large-model memory and communication costs are insufficiently documented. The correspondence between reported privacy budgets and actual training and accounting configurations is incomplete.

4.State synchronization and recovery procedures under upload or acknowledgment failures are not specified.

Questions for authors
  1. Equation (5) characterizes the signal–noise trade-off, but a random subspace does not guarantee alignment with useful gradients. What evidence distinguishes the benefit of directional filtering from ordinary update scaling under comparable learning-rate tuning, and how stable is this benefit across random basis initializations?

  2. The convergence theorem requires inactive clipping, while active clipping introduces bias outside its current scope. How frequently is clipping active in the reported experiments, and how would the resulting bias affect the convergence bound?

  3. The default q=0.5q=0.5 and r=32r=32 are not sufficiently supported by the component ablations, since these parameters jointly affect signal retention, noise attenuation, and control dynamics. How robust is performance to their joint variation across privacy and heterogeneity settings, including q=1q=1, and does r=32r=32 refer to each block or the total rank?

  4. The reported privacy budgets ϵ=1,3,6,10\epsilon=1,3,6,10 are not clearly linked to the default σ=10\sigma=10 or the training configurations. What values of TT, σ\sigma, nmin⁡n_{\min}, and accountant settings produce each budget, and are the accounting assumptions consistent with the sampling schemes, adjacency relations, and observable information of all compared methods?

  5. The blockwise filtering and LoRA configurations are insufficiently specified, making the large-model resource costs difficult to assess. What block partitions, ranks, basis-generation and storage strategies, and LoRA trainable scope are used, and how much additional GPU memory and runtime do filtering and control variates require?

  6. Ordinary partial participation is supported, but communication failures can leave client and server control states inconsistent. How does the protocol maintain or restore consistency when an upload fails after a local control update, or when the server receives an update but its acknowledgment is lost?

Reviewer_sRQDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LSSCA, a differentially private federated fine-tuning method that combines SCAFFOLD-style control variates with a fixed soft subspace filter. A random orthonormal basis is generated independently of private data. Components inside this reference subspace are preserved, while orthogonal components of the complete control-corrected noisy update are multiplied by an attenuation factor q. The paper gives an exact decomposition of signal distortion and residual Gaussian noise, a conservative record-level RDP analysis, and a non-convex convergence result under inactive clipping and a fixed noise scale. Experiments with RoBERTa-Base and GPT-2 XL report improvements over DP-FedAvg, DP-SCAFFOLD, DP-FedSAM, and DP-FedAvg-LS across several privacy budgets and heterogeneity levels.

Critical strengths
  1. The privacy model and accounting protocol are described more explicitly than in many empirical DP-FL papers. The analysis specifies record-level add/remove adjacency, fixed-size local sampling, visible client selection, adaptive composition, and the released transcript. It also correctly treats the filter and control updates as post-processing and clearly separates the privacy result, which permits active clipping, from the convergence result, which does not.

  2. The paper does not simply claim that filtering reduces noise. Equation (5) explicitly retains the signal distortion outside the reference subspace, and the convergence analysis tracks the effect of filtering on local trajectories and control states. The single-increment reconstruction is also a useful implementation observation.

Critical weaknesses
  1. The central empirical claim of selective subspace filtering is not established. The basis U is random, independent of both data and gradients, and has rank 32. In a parameter space with dimension d much larger than 32, the expected fraction of any fixed signal captured by this subspace is only r/d. The useful update should therefore lie almost entirely in the orthogonal complement, making Pq behave approximately like qI on the signal. With q = 0.5, the proposed filter may largely amount to halving both the gradient and its noise, which is closely related to changing the effective local learning rate rather than exploiting meaningful directional structure. The component ablation compares against removing the filter, but there is no qI baseline, learning-rate-matched baseline, measurement of gradient/subspace alignment, or sensitivity analysis over q, r, and random bases. Consequently, the reported gains cannot currently be attributed to selective subspace noise attenuation.

  2. The experimental privacy guarantees and implementation are not reproducible from the paper. The privacy theorem requires T, n_min, the local dataset sizes, and the noise multiplier used for each target epsilon. The experimental section reports a “default” sigma of 10 while presenting several epsilon values, but does not state the corresponding noise multipliers, total communication rounds, n_min, or the RDP order grid. The definition delta = 1/(NR) is also ambiguous when clients have heterogeneous sample counts. In addition, the paper does not specify which parameters are trained and filtered for RoBERTa, the precise LoRA configuration for GPT-2 XL, or how the blockwise random bases are partitioned and stored. These details are important because an explicit d-by-r basis can have substantial memory and computation costs. Finally, the convergence theorem assumes inactive clipping, but no clipping rates are reported, and the fixed-noise convergence rate is not a fixed-total-privacy result. Code and experiment scripts are deferred until acceptance, so these issues cannot presently be checked.

Questions for authors
  1. What fraction of the actual control-corrected gradients lies in span(U)? Please report d, r/d, the measured alignment ratio, results across several random bases, and comparisons against P = qI and an unfiltered DP-SCAFFOLD baseline with its learning rate retuned to match the effective step size. These comparisons would materially affect whether the gains support the proposed subspace interpretation.

  2. For every reported epsilon, what are T, n_min, delta, C, sigma, the RDP order grid, and the resulting accountant output? How is R in delta = 1/(NR) defined when the Dirichlet partition produces unequal client dataset sizes, and how is the statement that the default sigma is 10 reconciled with the multiple privacy budgets in Table 1?

  3. Which parameters are actually fine-tuned and filtered in each model? For the blockwise implementation, please give the blocks, rank assigned to each block, total stored basis size, projection cost, and whether the reported runtime includes these operations. Please also report the fraction of per-example gradients that are clipped, since this determines whether the convergence theorem describes the experimental regime.

Reviewer_rmTLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LSSCA, which combines SCAFFOLD-style control variates with a fixed, data-independent soft subspace filter for record-level differentially private federated fine-tuning. The filter preserves a random rank-r subspace and attenuates its orthogonal complement. It acts on the complete noisy, control-corrected update, which also changes the control-state recursion. The paper derives an exact signal-distortion/noise trade-off, a conservative privacy guarantee for the released transcript, and a nonconvex convergence result under inactive clipping. Experiments on RoBERTa-Base and GPT-2 XL report improvements over four DP-FL baselines.

Critical strengths
  1. The privacy statement has a clearly specified scope. The theorem addresses record-level visible client participation and persistent control states.

  2. This paper focus on an interesting and important problem.

Critical weaknesses
  1. It is unclear why a random subspace is a good choice for filtering. Since the subspace is chosen without using gradient information, it may suppress useful signals along with the noise.

  2. The paper reports several target ϵ values but does not provide enough information to reconstruct them, including the training horizon and noise multiplier for each setting.

  3. The claimed uplink reduction appears to follow from a control-update identity also available to SCAFFOLD-style methods.

Questions for authors
  1. Were hyperparameters selected using public data? If private data were used, how was the selection cost accounted for?

  2. What precise algorithm was implemented for the “without SCA” ablation, and how does it differ from DP-FedAvg-LS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.