- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LSSCA, a unified privacy-preserving federated fine-tuning framework that addresses performance degradation caused by the interaction between client heterogeneity and privacy noise in differentially private federated learning. The method combines Stochastic Controlled Averaging with fixed random Subspace Filtering, correcting client drift while preserving update components within a reference subspace and attenuating privacy noise in orthogonal directions. The authors establish an non-convex stationarity rate at a fixed noise scale and characterize the trade-off between noise attenuation and signal distortion through analyses of the residual control dynamics and noise propagation. Experiments on multiple NLP benchmarks using RoBERTa-Base and GPT-2 XL demonstrate improved privacy–utility trade-offs across different privacy budgets and levels of data heterogeneity. The paper identifies the tension arising from the interaction between client-drift correction and privacy-noise attenuation, introduces a lightweight random subspace filter, and supports the framework with rigorous RDP accounting and non-convex convergence proofs.
Critical strengths
1.Addresses a relevant problem: the interaction between client heterogeneity and privacy noise in federated fine-tuning. Combines control-based drift correction with soft subspace filtering in a unified framework. Explicitly characterizes the trade-off between noise attenuation and signal distortion.
2.Provides RDP privacy accounting and non-convex convergence analysis under clearly stated assumptions.
3.Evaluates multiple NLP tasks and model scales, with comparisons across privacy budgets, heterogeneity levels, and component ablations. Uses exact control-increment reconstruction to reduce the uplink increment-vector payload.
Critical weaknesses
1.The benefits of random directional filtering over appropriately tuned scalar update scaling are not sufficiently isolated.
2.The empirical relevance of the inactive-clipping assumption is not established through clipping statistics. Joint sensitivity to and , including variation across random basis initializations, remains insufficiently explored.
- Blockwise implementation details, LoRA configurations, and large-model memory and communication costs are insufficiently documented. The correspondence between reported privacy budgets and actual training and accounting configurations is incomplete.
4.State synchronization and recovery procedures under upload or acknowledgment failures are not specified.
Questions for authors
-
Equation (5) characterizes the signal–noise trade-off, but a random subspace does not guarantee alignment with useful gradients. What evidence distinguishes the benefit of directional filtering from ordinary update scaling under comparable learning-rate tuning, and how stable is this benefit across random basis initializations?
-
The convergence theorem requires inactive clipping, while active clipping introduces bias outside its current scope. How frequently is clipping active in the reported experiments, and how would the resulting bias affect the convergence bound?
-
The default and are not sufficiently supported by the component ablations, since these parameters jointly affect signal retention, noise attenuation, and control dynamics. How robust is performance to their joint variation across privacy and heterogeneity settings, including , and does refer to each block or the total rank?
-
The reported privacy budgets are not clearly linked to the default or the training configurations. What values of , , , and accountant settings produce each budget, and are the accounting assumptions consistent with the sampling schemes, adjacency relations, and observable information of all compared methods?
-
The blockwise filtering and LoRA configurations are insufficiently specified, making the large-model resource costs difficult to assess. What block partitions, ranks, basis-generation and storage strategies, and LoRA trainable scope are used, and how much additional GPU memory and runtime do filtering and control variates require?
-
Ordinary partial participation is supported, but communication failures can leave client and server control states inconsistent. How does the protocol maintain or restore consistency when an upload fails after a local control update, or when the server receives an update but its acknowledgment is lost?