- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FedDataLeak, a framework for detecting possible data leakage from federated LLM updates. The method combines Feature Reconstruction Fidelity (FRF) with two representation-based signals, activation sparsity and polysemanticity. The idea is to use the cheaper signals to identify suspicious updates before running more expensive reconstruction. The authors evaluate the method on three LLMs and several datasets and report improvements over existing baselines.
Critical strengths
The paper studies an important privacy problem in federated learning. The idea of monitoring model updates without always running expensive reconstruction attacks is useful. The evaluation also covers different models and includes both per-client and aggregate-only settings.
Critical weaknesses
-Limited novelty and unclear motivation for the proposed signals: My main concern is the novelty of the method. The paper mainly combines existing techniques into an auditing pipeline, so the contribution seems more engineering-oriented than methodological. I am also not convinced by the motivation for using polysemanticity and activation sparsity as indicators of privacy leakage. These signals come from model interpretability, but their connection to actual data leakage is not well explained. The paper provides ablations and correlation results, but these do not clearly show why an update containing a canary should cause a measurable change in these signals on a public reference set. Similar changes could also happen during normal fine-tuning.
-Missing simple baseline based on embedding updates: If the embedding layer is trainable, the updated embedding rows may provide strong information about which tokens appeared in the training data. This could make the canary detection task easier than it seems.
The authors already use embedding-row update norms to initialize their reconstruction candidates in Algorithm 2. However, the cheap baselines in Table 14 only include update norm, cosine similarity, and clipping saturation. I think a simple token-support detector based on embedding-row updates is an important missing baseline.
Without this comparison, it is difficult to know how much of the reported improvement comes from the proposed method rather than directly available token-level information.
-Evaluation setup and leakage labels: The main detection results use the presence of a synthetic canary as the positive label. However, detecting that a canary was included in training is not necessarily the same as showing that private information can be recovered. The paper reports separate reconstruction metrics, but I would like to see a clearer evaluation of detection performance using actual reconstruction success as the target. Also, the default configuration uses batch size 1, which is favorable for leakage attacks. Although the authors include some sensitivity experiments, larger batch sizes and more challenging aggregation settings would make the results more convincing.
-Some experimental results need clarification: Several results in Table 1 look unusually regular. For example, the AUPRC values for DLG, iDLG, and Inverting Gradients in the Medical Abstract experiments increase by exactly 0.10 across all three models. Similarly, Table 13 reports exactly +0.02 AUPRC improvement from aggregate-only to per-client visibility for every model. These patterns may simply be due to rounding, but it would be helpful to provide the unrounded results and individual seed values.
-The benefit of the triage stage is not fully clear: The paper claims that the representation-based triage makes auditing more efficient. However, Table 14 shows that FRF-only costs 7.1 seconds per round, while full FedDataLeak costs 8.9 seconds. The full method has better detection performance, but the benefit of the additional complexity is not completely clear. I would like to see a more direct comparison between the triage signal alone, FRF applied to every update, and the complete method.
Questions for authors
-
How well does a simple embedding-row token-support detector perform on the canary detection task? Could the authors include this baseline for embeddings-only and mixed fine-tuning?
-
What are the AUPRC and AUROC of the representation-based triage signal alone, without FRF? How many leakage-positive updates are missed before FRF is triggered?
-
Why does gating FRF improve the detection results compared with running FRF on every update? How much computation is actually saved when comparing methods under the same reconstruction budget?
-
Can the authors clearly specify whether the Table 1 results are per-client or aggregate-only for each dataset? Appendix D.2 identifies the Medical Abstract results as aggregate-only, but the main table should make this explicit.
-
Can the authors explain the regular numerical patterns in Tables 1 and 13 and provide the individual seed results?
4 comments
We have implemented the changes you requested in the revised manuscript (highlighted in yellow), and the paper is almost final.
Locations:
- Why PI and sparsity (W1): Sec. 3.3, "Role of the triage signals". They are triage proxies, not causal evidence. Matched canary-absent negatives control for ordinary fine-tuning, and FRF stays the primary evidence.
- Embedding-row baseline (Q1): App. C.4 defines a label-blind top-k embedding-row token-support detector, with a new row in Table 14. Its numbers are being added. Sec. 4.3 also notes that with frozen embeddings (LoRA-only), FedDataLeak keeps 0.77–0.86 AUPRC (Table 15).
- Leakage labels and batch size (W3): Sec. 4.1 and Table R1 in Sec. 4.2 score the method against realized reconstruction success. Sec. 4.3 now states the batch-size result: AUPRC 0.90/0.87/0.83 for B=1/2/4 (Table 12).
- Triage vs. FRF vs. full (Q2, Q3): "Gating versus fusion" in Sec. 4.3 compares them. FRF-only reaches 0.84 AUPRC against 0.90 for the full method. Running mixed auditing on every update gives 0.89 at 10.6 s/round against 0.90 at 8.9 s/round when gated. Triage-only AUPRC/AUROC and the missed-positive count (Table 14, App. B.2) are pending.
- Observability regime (Q4): The Table 1 caption now states the regime per dataset. Medical Abstract is aggregate-only; Stack Overflow and Enron are per-client / aggregate-only. Table 9's default regime is corrected.
- Regular patterns (Q5): App. D.2 explains that the entries are three-seed means rounded to two decimals. The unrounded and per-seed values for Tables 1 and 13 are being added and will be posted here.
If these changes resolve your concerns, we would appreciate it if you would reconsider your score. Thank you again for the feedback.
Follow-up on the rebuttal
Thank you for your responds. Most of my concern are still open and each round of rebuttal raises new concerns. so, I keep my score.
Thank you for the reply. We would like to briefly clarify where things stand.
Several of your original points are already answered by results in the revised manuscript:
- Batch size: AUPRC is 0.90/0.87/0.83 at B=1/2/4 (Table 12, Sec. 4.3).
- Token-level shortcut: with embeddings frozen (LoRA-only), FedDataLeak keeps 0.77–0.86 AUPRC across all nine dataset–model settings (Table 15).
- Benefit of fusion: AUPRC rises from 0.84 for FRF-only to 0.90 for the full method, and the gated policy costs 8.9 s/round versus 10.6 s/round when every update is audited (Sec. 4.3, "Gating versus fusion").
- Rationale and reporting: the role of the triage signals is now stated (Sec. 3.3), the observability regime is given per dataset (Table 1 caption), and rounding is explained (App. D.2).
There are a few entries which are still pending and we will include them in the final camera ready. Importantly we have narrowed the paper's claims to match what is currently measured.
If any specific concern still seems unresolved, we would be grateful if you named it, so we can answer it directly. Thank you again for your time.
We thank the reviewer for recognizing the importance of continuous privacy auditing and both visibility regimes. Some concerns are already answered by results in the paper (batch size, frozen-embedding training, the unconditional variant's cost). Other confusions we address directly.
1. Novelty and signal rationale. FedDataLeak introduces a budgeted update-level auditor, not another inversion attack. Candidate-input update matching produces Feature Reconstruction Fidelity (FRF, Eqs. 2–3); Algorithm 1/Eq. 7 implement three-state auditing for individual or aggregate updates (Eqs. 1, 9). Polysemanticity (Eq. 5) and Hoyer sparsity (Eq. 4) are triage proxies, not causal proof of leakage. Paired pre/post activations on the same public reference set (Eq. 6; Appendix B.4) capture feature specialization potentially associated with easier reconstruction. Ordinary fine-tuning can also cause such shifts; hence our matched canary-absent controls (Sec 4.1) and FRF as primary evidence. Table 7 reports ρ(q_rep, FRF)=0.74. Table 3 shows AUPRC improving from 0.84 (FRF-only) to 0.87 (+PI), 0.86 (+sparsity), and 0.90 (full). These results support complementary signals without claiming leakage specificity.
2. Embedding-row baseline (Q1). Gao et al. (COLING 2025) demonstrate that embedding gradients reveal token identities. Accordingly, Algorithm 2 already uses high-norm embedding rows for reconstruction initialization. However, candidate initialization is distinct from a standalone detector. We agree that Table 14's update-norm, cosine, and clipping controls do not replace a token-support baseline. A label-blind top-k embedding-row support detector is appropriate for embeddings-only and mixed fine-tuning, with frozen-embedding LoRA-only as a control. Ground-truth canary tokens must not enter the detector.
3. Leakage labels and batch effects. Table 1 evaluates canary inclusion, not confirmed extraction (Sec 4.1). However, Table 2 separately reports ECR 0.73, token F1 0.79, and membership AUC 0.89 (Medical Abstract/LLaMA-3.2-1B). Appendix C.5 further specifies independent, fixed-budget reconstruction-success labels with leave-one-method-out evaluation, although its complete numerical results require reporting.
Importantly, larger batches are already evaluated, contrary to the impression that only B=1 was studied. Table 12 reports AUPRC 0.90/0.87/0.83 and ECR 0.73/0.67/0.59 for B=1/2/4. Table 13 additionally examines aggregate-only auditing, where results represent cohort-level rather than client-level risk.
4. Triage effectiveness and computational cost (Q2 and Q3). Under Eq. 7, MONITOR updates receive q_rep; triggered updates receive FRF-fused scores. Appendix B.2 reports 31% FRF invocation, avoiding FRF calls on 69% of updates. This is a reduction in solver invocations, not necessarily total runtime.
Table 7 reports correlation with FRF, not standalone q_rep AUPRC/AUROC; pre-trigger missed positives are also not enumerated. These are useful reporting additions. Importantly, fusion not gating alone explains the detection improvement.
Table 3 reports FRF-only at 7.1 s/round versus full FedDataLeak at 8.9 s/round. The additional 1.8 seconds yields +0.06 AUPRC and +0.15 TPR@1%FPR (Tables 3, 14). Thus, the benefit is improved detection and selective deep inspection, not unconditional superiority in runtime.
Moreover, the reviewer overlooks the scheduler-matched comparison in Table 8: with the same 31% trigger rate and shared triage overhead, FET costs 9.9 s, Inverting Gradients 11.2 s, and FedDataLeak 8.9 s. A fully policy-consistent comparison should also recompute full-population rankings from Eq. 7 and compare unconditional FRF under an identical solver budget.
5. Observability and numerical consistency (Q4 and Q5). Medical Abstract in Table 1 is aggregate-only (Table 13). Stack Overflow and Enron are [per-client / aggregate-only]. We will mark the regime in every Table 1 row and correct Table 9, which wrongly lists per-client as the default. On the regular +0.10 and +0.02 steps: these are 3-seed means rounded to two decimals. We will post unrounded per-seed values for Tables 1 and 13 in this thread and add them to the appendix.
In conclusion, FedDataLeak's contribution is reconstruction-grounded, selective privacy auditing, rather than treating interpretability signals as direct proof of disclosure. The manuscript already contains substantial ablation, batch-size, observability, and scheduler-matched evidence. The remaining concerns primarily concern additional controls and transparent numerical reporting. We respectfully request that these existing results and the precise scope of the contribution be considered when reassessing the rating.