acceptodds
Under review as a conference paper at ICLR 2027

Beyond Coordinate Sparsity: Controlling Module Exposure Against Gradient Reconstruction in Federated LoRA

Abstract

Federated Low-Rank Adaptation (LoRA) reduces the communication cost of language-model fine-tuning, yet shared LoRA updates can still expose private training text to gradient-based reconstruction attacks. Existing sparsification methods primarily control how many update coordinates remain visible, but this does not capture which attack-relevant LoRA modules are exposed. We introduce module exposure to characterize this distinction and propose MU-BEC, a module-level upload-control method that uses public-data training dynamics to select a fixed subset of LoRA modules for communication. Under a matched MineGrad evaluation with the same audited attack base and 3,072 nominal retained slots, Top-k yields 95% unconditional content-token recovery, whereas MU-BEC yields 0% under both Fixed Attack and Known-Mask Attack. Mechanism analyses further show that aggressive Top-k sparsification can preserve nearly all target-gradient energy, indicating that coordinate count alone does not characterize MineGrad recovery in the evaluated setting. In normal federated training, MU-BEC reduces upload communication by 36.0% relative to Top-k and Random-k, while introducing a measurable utility trade-off. These results motivate module-level observability as a complementary perspective for analyzing privacy leakage in Federated LoRA.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
parameter-efficient fine-tuning, LoRA, gradient inversion, privacy, federated learning, gradient sparsification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jkKuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a privacy analysis under LoRA fine-tuning in federated learning. It claims that under the same uploading budget, sharing different sets of LoRA module updates can lead to substantially different level of data leakage. To this end, this work leverages public data training dynamics to obtain a selection criterion, based on which a subset of LoRA module updates would be communicated. The results indicate that such selection of LoRA modules enhances privacy, reducing the token recovery rate to 0%. However, the reviewer has several critical concerns regarding the underlying assumptions and the working principles.

Critical strengths
  1. This paper studies an important problem regarding how can a selective uploading of LoRA module updates can prevent inversion attacks and preserve privacy.
Critical weaknesses

Major weaknesses:

  1. The paper lacks clarity. The threat model is not clearly stated. Does it assume an honest-but-curious adversary like Yin et al., 2021, Petrov et al., 2024 or a malicious adversary like Sami et al., 2025, Sami et al., 2026?

  2. It is stated that MineGrad can recover tokens under coordinate level sparsification, but fails under MU-BEC top-k module selection mechanism. At the same time, Fig. 3 demonstrates that MU-BEC retains utility in terms of validation accuracy. Note that MineGrad maliciously modifies the pretrained model and LoRA parameters to ensure that LoRA gradients would lead to data leakage, with little consideration to utility. In that regard, MineGrad sets a lots of parameters in the pretrained model and LoRA modules to 0, 1 or significantly large values, which largely deviate from the benign parameters, hence destroys the utility. Under an honest-but-curious adversary model where model parameters are benign and utility is retained, MineGrad attack fails to recover data. Hence, it is unclear whether the reduced privacy leakage and retained utility in MU-BEC is due to simply the benign model parameters. Are the utility experiments done under malicious setting or benign setting?

  3. In MineGrad, the attack utilizes only the gradients of LoRA-B matrix from the output projection layer in the attention block. Let’s call these “attack gradients”. LoRA matrices added to query, key matrices do not contribute to the reconstruction attack. In this regard, the paper lacks clarity regarding whether the coordinate based sparsification selects these attack gradients or whether MU-BEC discards these attack gradients.

  4. The intuition of leveraging the score in Eq. 3 and masking accordingly in Eq. 4 for preventing leakage is not clear.

  5. The proposed framework requires access to a public data, that is task-relevant to user data; to generate the module selection mask. This can be a restricted scenario in the sense that such data may not always be available. Further explanations/validations would be appreciated.

Minor weaknesses:

  1. Some definitions need to be more clear in the paper, for example- fixed attack.
  2. MineGrad demonstrates recovery for both image and text data. How does the proposed work perform on image modality?
Questions for authors

Please see the weaknesses above.

Reviewer_rzAKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Module-Update-Based Exposure Control (MU-BEC), a module-level upload control method for Federated LoRA that aims to reduce training-text privacy leakage caused by gradient reconstruction attacks. MU-BEC introduces the concept of Module Exposure, highlighting that conventional coordinate sparsification focuses on the number of uploaded parameters while overlooking the privacy implications of exposing different LoRA modules. The method selects LoRA modules based on parameter update magnitudes measured on public data and uses a fixed mask to control client uploads. Experimental results show that, under the same nominal parameter budget, Top-k achieves a content-token recovery rate of 95%, whereas MU-BEC achieves 0% under the evaluated attacks. MU-BEC also reduces upload communication by 36.0%, although it incurs some loss in model utility.

Critical strengths
  1. The paper introduces module exposure to investigate how the selection of LoRA modules affects privacy leakage beyond conventional coordinate sparsity.
  2. MU-BEC selects LoRA modules using public-data training dynamics and applies a fixed upload mask without modifying local training or server aggregation.
  3. The experiments provide useful insights into gradient reconstruction under different sparsification strategies and evaluate the trade-off between privacy protection, model utility, and communication cost.
Critical weaknesses
  1. It remains unclear whether MU-BEC's privacy protection mainly results from suppressing attack-relevant gradient signals. (Q1)
  2. Privacy protection is evaluated using gradients rather than actual federated parameter updates. (Q2–Q3)
  3. The effectiveness of module scoring may be confounded by the layer positions of the selected modules. (Q4–Q5)
  4. The definitions of mask-allowed module exposure and actual module exposure are not fully consistent. (Q6)
Questions for authors
  1. Does the 0% text recovery rate of MU-BEC in Figure 2 mainly result from suppressing the gradient signals required by the current attack? Would selecting other fixed modules that do not contain the attack-targeted gradients achieve similar reconstruction suppression?
  2. Could the authors clarify whether LoRA-A is frozen or synchronized through communication, and whether Table 2 accounts for all trainable parameters that need to be transmitted?
  3. The privacy experiments analyze text reconstruction from LoRA-B gradients, whereas normal federated training transmits local parameter updates. Could the authors further clarify the relationship between these two settings and the applicability of the reported privacy protection results to actual federated parameter uploads?
  4. MU-BEC Top-1 and Top-2 both select modules from the final Transformer layer. Could the effectiveness of update-magnitude-based module scoring depend on the layer positions of the selected modules?
  5. Could the authors further analyze text recovery rates, model utility, and communication costs under different module budgets K to provide a more comprehensive evaluation of the privacy–utility–communication trade-off?
  6. The paper uses different notions of module exposure in Eq. (2), Eq. (5), and Figure 2. Could the authors further distinguish between the set of modules allowed to be exposed by the mask and the set of attack-relevant modules with actual nonzero gradients, and unify the corresponding notation and terminology?
Reviewer_xRmiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much private text a server can reconstruct from the LoRA updates that clients upload in Federated LoRA. It argues that the number of uploaded values does not determine leakage, and that what matters is which LoRA modules are visible to the server, which it calls module exposure (Eq. 2). The proposed MU-BEC ranks the 48 LoRA modules of RoBERTa-base by their normalized update magnitude during LoRA training on public data, and then each client uploads only the LoRA-B updates of the top-KK modules through a fixed mask (the main experiments use K=1K = 1, i.e., encoder12.key). Against the MineGrad attack, with the same budget of 3,072 uploaded values and 20 target texts, Top-k sparsification allows 95% recovery of the content tokens while MU-BEC allows 0%. In normal federated training on Yahoo Answers Topics, MU-BEC uses 36.0% fewer upload bits than Top-k, at some loss in accuracy (Figs. 3–4).

Critical strengths
  • I appreciate that Fig. 2 reports, for each method, how many values are uploaded, how many of them are actually nonzero, and which modules they come from, instead of only the budget. This makes the privacy result easy to check, and Sec. 4.2 itself warns against attributing the gap to module identity alone.
  • The analysis in Table 1 is informative. At 99% sparsity, Top-k drops 76% of the nonzero entries relevant to the attack but keeps 99.99% of their gradient energy, and simply rescaling the entries kept by Rand-k (by d/kd/k) raises recovery from 53.18% to 99.27%. This shows that the number of uploaded values is a poor measure of leakage.
Critical weaknesses
  • I am not convinced that the privacy result shows more than the absence of signal. Recovery is reported only for MU-BEC Top-1 (encoder12.key), which is not one of the modules MineGrad attacks, so all 3,072 values it uploads are zero (Fig. 2B) and 0% recovery is guaranteed. This non-overlap is a coincidence, because the selection uses no information about the attack, and the paper never tests what happens when the selected module is one of the attacked ones.
  • I am also concerned about the threat model. Since clients send no coordinate indices, the server must know the mask to aggregate the updates, and the mask can be computed from public data anyway. A malicious server could therefore poison encoder12.key instead, which would defeat MU-BEC entirely. This attack is not evaluated: the "Known-Mask Attack" only tells the attacker which values are kept, but still poisons the original target modules, so the uploaded module carries no signal either way.
  • It is hard to tell what the server actually receives and how general the results are. All upload counts include only the LoRA-B matrices, and the paper never says whether LoRA-A or the classifier head is also uploaded. The 36.0% saving comes entirely from not sending coordinate indices, the accuracy loss is shown only as curves (Figs. 3–4), and all experiments use one model (RoBERTa-base), one dataset, one attack, and 20 target texts.
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.