- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a generative-prior gradient inversion attack for audio federated learning, which uses a StyleGAN2-like mel-spectrogram GAN prior with intermediate layer optimisation and a BigVGAN v2 vocoder for waveform recovery, and extends to MFCC inputs via a differentiable DCT. Evaluated on Speech Commands and AudioMNIST under gradient perturbation, MFCC compression, trained models, and diverse architectures, the attack substantially outperforms a GAN-free baseline. The central finding is that speaker identity leakage is decoupled from content and fidelity degradation—even when reconstruction quality drops sharply, the generative prior preserves significant speaker identity information, while architectures with global pooling or attention (ResNet18, AST) remain considerably more resistant.
Critical strengths
- The paper identifies a genuine gap in audio gradient inversion—prior work lacks learned priors—and carefully distinguishes the multiple leakage axes (identity, content, attributes, fidelity) that the image literature does not capture.
- The GAN + ILO feature reconstruction and BigVGAN vocoding are straightforward and well documented, with Algorithm 1 and detailed hyperparameter tables making the method easy to reproduce and extend to MFCC inputs.
- The observation that speaker identity leakage persists even when content and fidelity degrade is a useful and somewhat counterintuitive message, and the negative control convincingly rules out the GAN prior itself hallucinating identity.
Critical weaknesses
- The core framework is largely inherited from the image-domain GIFD. The new contributions (mel-spectrogram prior, differentiable MFCC path, BigVGAN vocoder) are useful engineering adaptations, but the paper does not offer a distinct conceptual advance in generative modeling or optimization for the audio modality.
- The primary evaluations use an untrained AudioCNN, which is the most favorable setting for the attacker. Although Table 3 analyzes training up to 30 epochs, it is unclear whether the identity leakage advantage over the GAN-free baseline persists when the victim model is fully converged, which is the realistic regime in federated learning.
- The paper is transparent that this is a controlled perturbation rather than certified differential privacy. However, with noise-to-signal ratios ranging from 0.04 to 3.58, the setting is far from a meaningful privacy budget. The conclusion that identity leakage persists under perturbation is therefore limited to weak noise and should not be extrapolated to practical DP deployments.
- The claim that ResNet18 and AST are more resistant because of global average pooling and global attention is plausible, but it is supported only by post-hoc observation.
Questions for authors
- How does the attack perform when the victim model is trained to convergence (e.g., 50–100 epochs) or under multi-step local updates with larger batch sizes? Does the generative prior still provide a clear identity-leakage advantage over the GAN-free baseline in these realistic federated learning settings?
- Can the authors provide a direct ablation to test the architecture hypothesis, for example, replacing global average pooling in ResNet18 with local pooling, or adding positional encoding to AST, to confirm that the resistance is indeed due to the aggregation mechanism rather than other architectural factors?
3 comments
W3 -- Perturbation vs. a meaningful DP budget.
We agree that our setting is a controlled perturbation and not certified differential privacy, and we claim no . We further adjusted this for clarity and consistency in the paper. To locate where the claim stops holding for our work, we extended the noise grid to and ( and ).
| Dataset | Model Acc. | ECAPA Cos.Sim | Speaker ID | |
|---|---|---|---|---|
| AM | 3.58 | 98.3% | 0.405 | 84.0% |
| AM | 10.7 | 97.1% | 0.114 | 7.0% |
| AM | 35.8 | 92.1% | 0.040 | 0.0% |
| SC | 3.58 | 73.5% | 0.545 | 96.5% |
| SC | 10.7 | 57.6% | 0.180 | 24.5% |
| SC | 35.8 | 36.1% | 0.077 | 5.8% |
Identity leakage of the GAN-based attack holds up to (Speaker ID 84.0% on AM, 96.5% on SC) and breaks by (7.0% on AM, 24.5% on SC). On SC, Speaker ID reaches 5.8% only at , where accuracy falls from 79.9% to 36.1%. We have narrowed the claim accordingly: perturbation strong enough to defeat feature-space inversion does not remove speaker identity, and we do not extrapolate this to practical DP deployments.
Revision: Table 22 extended to ; new Table 23 (extended noise grid) and "Attack breaking point" paragraph in App. A.10; pointer in Sec. 5.1.
W2 / Q1 -- Attack performance on converged victims.
We agree that the untrained victim favours the attacker and that converged victims should also be tested. Therefore, we are running the attack on victims trained for 50 epochs, where validation accuracy has plateaued (AM: 96.6% at epoch 50, best 98.0%; SC: 79.9%, rising by less than 1 point over the last 10 epochs). On AM the identity advantage of the prior persists at convergence: Speaker ID 55.0% vs 23.0% for the GAN-free attack (ECAPA Cos.Sim 0.299 vs 0.189, STOI 0.705 vs 0.582). The SC run of the GAN-free baseline is still in progress; we will post the complete results and add the epoch-50 rows to Table 3 once it finishes. Larger batches are orthogonal to the prior: segment-wise and batched reconstruction, as in the long-audio extension of SPL (LAS), could be combined with our attack, and multi-step local updates remain open. We state both as limitations and future work in the extended related work (App. A.1, with the LAS reference) and the Discussion.
Pending: SC epoch-50 GAN-free baseline (expected before the end of the discussion period).
Follow-up to W2 / Q1 - Converged victims (epoch 50).
The SC run of the GAN-free baseline has completed. At epoch 50, the GAN-based attack retains more speaker identity than the GAN-free attack on both datasets:
| Dataset | Attack | Mel MSE | STOI | PESQ | ECAPA Cos.Sim | Speaker ID |
|---|---|---|---|---|---|---|
| AM | GAN-free | 30.597 | 0.582 | 1.142 | 0.189 | 23.0% |
| AM | GAN-based | 1.427 | 0.705 | 1.250 | 0.299 | 55.0% |
| SC | GAN-free | 26.854 | 0.554 | 1.348 | 0.402 | 75.3% |
| SC | GAN-based | 2.084 | 0.645 | 1.632 | 0.467 | 82.0% |
Undefended AudioCNN trained for 50 epochs (validation accuracy 96.6% on AM, 79.9% on SC), BVG vocoder; AM 100 / SC 400 targets.
From epoch 30 to the converged victim (epoch 50), the GAN-free attack's Speaker ID drops from 42.0% to 23.0% on AM and from 94.8% to 75.3% on SC, while the GAN-based attack degrades less (68.0% to 55.0% and 95.5% to 82.0%). The GAN-based attack is better on every metric on both datasets, so the prior retains its identity advantage at convergence.
Revision: epoch-50 rows added to Table 3; Sec. 5.3.
We thank the reviewer for the detailed review, and for highlighting the multi-axis view of audio leakage, the reproducibility of the method and the decoupling of identity from content and fidelity. We took the time to complete the new experiments so that each point below is answered with results.
W1 -- Contribution beyond GIFD.
We agree that the optimisation framework (W+ + ILO) is inherited from GIFD, and the paper acknowledges this along with the transfer to the audio domain with the improved pipeline and empirical findings. We see the contribution in what the audio domain reveals. A learnt audio prior keeps speaker identity recoverable where the inversion becomes information-limited (perturbation, MFCC compression), even as fidelity and content degrade. Also, the ILO ablation shows that this gain comes from the W+ prior rather than from the GIFD-specific ILO step. Finally, the aggregation ablation below shows that architectural resistance does not imply privacy since the gradient still carries identity even where the attack cannot yet exploit it.
Revision: new App. A.7 "Global aggregation ablation" (see W4 / Q2); new App. A.9 "ILO ablation".
W4 / Q2 -- Direct test of the architecture hypothesis.
We ran the ablation to identify the difference in attack performance. Instead of local pooling, we removed the aggregation entirely: the global readout (average pooling in ResNet18, CLS token in AST) is replaced by a flatten head, and everything else is fixed. AST already uses learnable positional encoding, so the test targets the readout only. GAN-based attack, same 100 targets per dataset:
Untrained, undefended victims, BVG vocoder. AST$_\text{tiny}$ (dim 192, depth 4) resists like the AST of Table 4. Both flatten heads add a similar number of parameters (1.18M and 1.23M).
Revision: new App. A.7 "Global aggregation ablation" (Table 14; gradient shift sensitivity and speaker separation ratio in Table 15); Table 20 (victim architectures) extended with the ablation variants; Sec. 5.4 and Discussion revised.