acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction of Private Speech using Generative-Prior-Guided Gradient Inversion

Abstract

Federated learning enables collaborative training without centralising raw audio, yet the gradients shared during training can leak sensitive information, including speech content and speaker identity. Existing audio gradient inversion attacks primarily reconstruct inputs directly in the feature space and have been studied mostly under relatively favourable inversion settings. In this work, we introduce a generative-prior gradient inversion framework for audio, which constrains reconstruction using a learnt mel-spectrogram prior and employs a neural vocoder for waveform recovery. To prevent prior memorisation of victim identities, the generator is trained using an identity-disjoint speaker split. We evaluate the attack on Speech Commands V2 and AudioMNIST under a range of increasingly challenging settings, including gradient clipping with Gaussian perturbation, compressed MFCC features, trained victim models, and different network architectures. Our results show that the main advantage of the generative prior emerges as the inversion problem becomes less informative. Under the strongest tested gradient perturbation, speaker recovery remains at 84.0% on AudioMNIST and 96.5% on Speech Commands, compared with 18.0% and 29.3% for the GAN-free baseline. These findings demonstrate that degraded speech reconstruction does not ensure speaker privacy, as identity leakage can persist through generative priors.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
gradient inversion, federated learning, audio privacy, generative priors, speaker identity leakage

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PJ7ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a generative-prior gradient inversion attack for audio federated learning, which uses a StyleGAN2-like mel-spectrogram GAN prior with intermediate layer optimisation and a BigVGAN v2 vocoder for waveform recovery, and extends to MFCC inputs via a differentiable DCT. Evaluated on Speech Commands and AudioMNIST under gradient perturbation, MFCC compression, trained models, and diverse architectures, the attack substantially outperforms a GAN-free baseline. The central finding is that speaker identity leakage is decoupled from content and fidelity degradation—even when reconstruction quality drops sharply, the generative prior preserves significant speaker identity information, while architectures with global pooling or attention (ResNet18, AST) remain considerably more resistant.

Critical strengths
  1. The paper identifies a genuine gap in audio gradient inversion—prior work lacks learned priors—and carefully distinguishes the multiple leakage axes (identity, content, attributes, fidelity) that the image literature does not capture.
  2. The GAN + ILO feature reconstruction and BigVGAN vocoding are straightforward and well documented, with Algorithm 1 and detailed hyperparameter tables making the method easy to reproduce and extend to MFCC inputs.
  3. The observation that speaker identity leakage persists even when content and fidelity degrade is a useful and somewhat counterintuitive message, and the negative control convincingly rules out the GAN prior itself hallucinating identity.
Critical weaknesses
  1. The core framework is largely inherited from the image-domain GIFD. The new contributions (mel-spectrogram prior, differentiable MFCC path, BigVGAN vocoder) are useful engineering adaptations, but the paper does not offer a distinct conceptual advance in generative modeling or optimization for the audio modality.
  2. The primary evaluations use an untrained AudioCNN, which is the most favorable setting for the attacker. Although Table 3 analyzes training up to 30 epochs, it is unclear whether the identity leakage advantage over the GAN-free baseline persists when the victim model is fully converged, which is the realistic regime in federated learning.
  3. The paper is transparent that this is a controlled perturbation rather than certified differential privacy. However, with noise-to-signal ratios ranging from 0.04 to 3.58, the setting is far from a meaningful privacy budget. The conclusion that identity leakage persists under perturbation is therefore limited to weak noise and should not be extrapolated to practical DP deployments.
  4. The claim that ResNet18 and AST are more resistant because of global average pooling and global attention is plausible, but it is supported only by post-hoc observation.
Questions for authors
  1. How does the attack perform when the victim model is trained to convergence (e.g., 50–100 epochs) or under multi-step local updates with larger batch sizes? Does the generative prior still provide a clear identity-leakage advantage over the GAN-free baseline in these realistic federated learning settings?
  2. Can the authors provide a direct ablation to test the architecture hypothesis, for example, replacing global average pooling in ResNet18 with local pooling, or adding positional encoding to AST, to confirm that the resistance is indeed due to the aggregation mechanism rather than other architectural factors?
3 comments

We thank the reviewer for the detailed review, and for highlighting the multi-axis view of audio leakage, the reproducibility of the method and the decoupling of identity from content and fidelity. We took the time to complete the new experiments so that each point below is answered with results.

W1 -- Contribution beyond GIFD.

We agree that the optimisation framework (W+W^{+} + ILO) is inherited from GIFD, and the paper acknowledges this along with the transfer to the audio domain with the improved pipeline and empirical findings. We see the contribution in what the audio domain reveals. A learnt audio prior keeps speaker identity recoverable where the inversion becomes information-limited (perturbation, MFCC compression), even as fidelity and content degrade. Also, the ILO ablation shows that this gain comes from the W+W^{+} prior rather than from the GIFD-specific ILO step. Finally, the aggregation ablation below shows that architectural resistance does not imply privacy since the gradient still carries identity even where the attack cannot yet exploit it.

Revision: new App. A.7 "Global aggregation ablation" (see W4 / Q2); new App. A.9 "ILO ablation".

W4 / Q2 -- Direct test of the architecture hypothesis.

We ran the ablation to identify the difference in attack performance. Instead of local pooling, we removed the aggregation entirely: the global readout (average pooling in ResNet18, CLS token in AST) is replaced by a flatten head, and everything else is fixed. AST already uses learnable positional encoding, so the test targets the readout only. GAN-based attack, same 100 targets per dataset:

DatasetModelReadoutMel MSESTOIECAPA Cos.SimSpeaker ID
AMResNet18avg. pool14.6340.3440.0663.0%
AMResNet18flatten2.7400.4180.0903.0%
AMAST$_\text{tiny}$CLS3.8250.4440.1157.0%
AMAST$_\text{tiny}$flatten0.1070.9240.80299.0%
SCResNet18avg. pool14.5810.2200.0564.0%
SCResNet18flatten13.8350.2560.0675.0%
SCAST$_\text{tiny}$CLS6.2690.1760.12213.0%
SCAST$_\text{tiny}$flatten0.3410.6990.743100.0%

Untrained, undefended victims, BVG vocoder. AST$_\text{tiny}$ (dim 192, depth 4) resists like the AST of Table 4. Both flatten heads add a similar number of parameters (1.18M and 1.23M).

  • AST: confirmed. Removing the CLS readout restores leakage (Speaker ID 7.0% →\to 99.0% on AM, 13.0% →\to 100.0% on SC). To show the mechanism, we measured the gradient shift sensitivity, 1−cos⁡1-\cos between the gradients of a mel-spectrogram and of its copy shifted in frequency or time. With the CLS readout it is 2.4-4.5$\times$ lower than with the flatten head, so many inputs produce nearly the same gradient.
  • ResNet18: not confirmed. Removing global average pooling restores no identity. Its gradient is already more input-sensitive than that of AST, and the flatten head raises the sensitivity by a further 1.6-1.9$\times$ without any gain in leakage. We have therefore removed the post-hoc attribution of the effect to pooling and state the ResNet18 cause as an open direction.
  • Resistance does not imply privacy. For every variant the gradient still separates speakers: the ratio of gradient distances between different speakers and between recordings of the same speaker (same word) is 1.23-1.43, where 1 means blind to identity, with or without the global readout.

Revision: new App. A.7 "Global aggregation ablation" (Table 14; gradient shift sensitivity and speaker separation ratio in Table 15); Table 20 (victim architectures) extended with the ablation variants; Sec. 5.4 and Discussion revised.

W3 -- Perturbation vs. a meaningful DP budget.

We agree that our setting is a controlled perturbation and not certified differential privacy, and we claim no (ε,δ)(\varepsilon, \delta). We further adjusted this for clarity and consistency in the paper. To locate where the claim stops holding for our work, we extended the noise grid to z=3×10−3z = 3\times10^{-3} and 10−210^{-2} (zD=10.7z\sqrt{D} = 10.7 and 35.835.8).

DatasetzDz\sqrt{D}Model Acc.ECAPA Cos.SimSpeaker ID
AM3.5898.3%0.40584.0%
AM10.797.1%0.1147.0%
AM35.892.1%0.0400.0%
SC3.5873.5%0.54596.5%
SC10.757.6%0.18024.5%
SC35.836.1%0.0775.8%

Identity leakage of the GAN-based attack holds up to zD=3.58z\sqrt{D} = 3.58 (Speaker ID 84.0% on AM, 96.5% on SC) and breaks by 10.710.7 (7.0% on AM, 24.5% on SC). On SC, Speaker ID reaches 5.8% only at zD=35.8z\sqrt{D} = 35.8, where accuracy falls from 79.9% to 36.1%. We have narrowed the claim accordingly: perturbation strong enough to defeat feature-space inversion does not remove speaker identity, and we do not extrapolate this to practical DP deployments.

Revision: Table 22 extended to zD=35.8z\sqrt{D} = 35.8; new Table 23 (extended noise grid) and "Attack breaking point" paragraph in App. A.10; pointer in Sec. 5.1.

W2 / Q1 -- Attack performance on converged victims.

We agree that the untrained victim favours the attacker and that converged victims should also be tested. Therefore, we are running the attack on victims trained for 50 epochs, where validation accuracy has plateaued (AM: 96.6% at epoch 50, best 98.0%; SC: 79.9%, rising by less than 1 point over the last 10 epochs). On AM the identity advantage of the prior persists at convergence: Speaker ID 55.0% vs 23.0% for the GAN-free attack (ECAPA Cos.Sim 0.299 vs 0.189, STOI 0.705 vs 0.582). The SC run of the GAN-free baseline is still in progress; we will post the complete results and add the epoch-50 rows to Table 3 once it finishes. Larger batches are orthogonal to the prior: segment-wise and batched reconstruction, as in the long-audio extension of SPL (LAS), could be combined with our attack, and multi-step local updates remain open. We state both as limitations and future work in the extended related work (App. A.1, with the LAS reference) and the Discussion.

Pending: SC epoch-50 GAN-free baseline (expected before the end of the discussion period).

Follow-up to W2 / Q1 - Converged victims (epoch 50).

The SC run of the GAN-free baseline has completed. At epoch 50, the GAN-based attack retains more speaker identity than the GAN-free attack on both datasets:

DatasetAttackMel MSESTOIPESQECAPA Cos.SimSpeaker ID
AMGAN-free30.5970.5821.1420.18923.0%
AMGAN-based1.4270.7051.2500.29955.0%
SCGAN-free26.8540.5541.3480.40275.3%
SCGAN-based2.0840.6451.6320.46782.0%

Undefended AudioCNN trained for 50 epochs (validation accuracy 96.6% on AM, 79.9% on SC), BVG vocoder; AM 100 / SC 400 targets.

From epoch 30 to the converged victim (epoch 50), the GAN-free attack's Speaker ID drops from 42.0% to 23.0% on AM and from 94.8% to 75.3% on SC, while the GAN-based attack degrades less (68.0% to 55.0% and 95.5% to 82.0%). The GAN-based attack is better on every metric on both datasets, so the prior retains its identity advantage at convergence.

Revision: epoch-50 rows added to Table 3; Sec. 5.3.

Reviewer_Gc4FOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a gradient inversion method to reconstruct speech from gradients shared during federated learning. The method uses a GAN trained on mel-spectrograms to guide reconstruction. It first optimizes the generator’s latent variables and intermediate activations to match the observed gradient, then converts the recovered spectrogram into audio using a pretrained BigVGAN vocoder.

The evaluation includes gradient perturbation, MFCC compression, trained victim models, and different architectures. The proposed method shows substantial improvements over direct feature optimization under several challenging conditions. Additional experiments examine the vocoder’s contribution, cross-dataset transfer, and speaker leakage. The results suggest that reduced reconstruction quality does not necessarily remove speaker information.

Critical strengths
  • The proposed method shows substantial improvements in reconstruction and speaker similarity under gradient perturbation and MFCC compression. Although the optimization approach builds on GIFD, its application to speech provides useful evidence that learned priors can strengthen audio gradient inversion attacks.

  • The evaluation includes several useful controls. The appendix also includes random-generation controls and speaker identification using separate enrollment recordings. The experiments across trained models and different architectures help show where the method works and where it remains limited.

Critical weaknesses
  • The main “Speaker ID %” measures whether similarity between a reconstruction and its original recording exceeds a fixed threshold. This does not directly establish how reliably the reconstructed speech can be linked to a speaker using separate recordings. The appendix provides useful same-speaker and different-speaker reference scores and identification results for LeNet, but this evaluation does not cover the main noise and compression conditions. Verification EER against separate enrollment recordings would strengthen these claims. Further, PESQ and spectral error do not directly measure linguistic content, so WER or independent keyword accuracy is needed to support claims about content degradation.

  • The model accuracy and attack results in Table 1 come from different settings. The attacks use individual gradients from untrained models, while model accuracy is measured using separate batch-training runs with a different clipping bound. The results therefore do not establish the privacy-utility trade-off for the same defended system. The authors should make this distinction clearer and limit the interpretation accordingly.

Questions for authors
  • In Fig. 4, are the genuine score distributions and reported AUROC computed against the original target recording or separate enrollment recordings? Please clarify how enrollment recordings are selected and which comparisons exclude the target recording.

  • Can you evaluate the existing reconstructions under the strongest noise and MFCC-compression settings using verification EER against separate enrollment recordings? Please include original-audio and GAN-free results as references.

  • Can you report WER or independent keyword recognition accuracy on the reconstructed audio? Since the attacker already knows the word label, this would measure whether the reconstruction preserves the word rather than whether the attack discovers unknown content.

  • How should Table 1’s model accuracy be interpreted alongside its attack results, given the different update and clipping settings? Are results available where utility and leakage are measured under the same defended update configuration?

4 comments

We thank the reviewer for the careful review, and for noting the controls in our evaluation and where the method remains limited. The questions on identity linkage, content and Table 1 helped us tighten our claims. We took the time to complete the new experiments so that each point below is answered with results.

W1 (identity) / Q2 -- Verification EER against independent enrollment recordings.

We agree that the comparison with the original recording does not establish identification through separate recordings. Therefore we added the enrollment based evaluation for all noise perturbation and MFCC settings, with the original audio and the GAN-free attack as references. Each held-out speaker's template is the mean ECAPA embedding of up to 10 recordings of that speaker that are not attack targets, so the attacked clip is never included; 11 of the 200 SC speakers have no such recording and are excluded. Thresholds are calibrated on speakers disjoint from the held-out set.

DatasetSettingEER GAN-freeEER GAN-basedTop-1 GAN-freeTop-1 GAN-based
AMOriginal audio5.0%97.0%
AMNo Def.11.7%18.0%92.0%86.0%
AMz=10−3z=10^{-3}37.7%22.0%28.0%62.0%
AMnmfcc=15n_\mathrm{mfcc}=1537.0%32.0%33.0%53.0%
SCOriginal audio7.7%73.8%
SCNo Def.25.2%11.9%17.4%52.7%
SCz=10−3z=10^{-3}38.7%16.9%1.6%37.9%
SCnmfcc=15n_\mathrm{mfcc}=1540.0%27.3%1.8%17.1%

Untrained AudioCNN, BVG vocoder. Top-1: identification over the held-out speakers (chance 10% on AM, 0.5% on SC).

On SC, the GAN-based attack has a lower EER and a higher Top-1 accuracy in every setting. On AM, both attacks identify the speakers well without a defence, and under the strongest perturbation and compression the GAN-based attack keeps the speaker identifiable where the GAN-free attack degrades. Because there are too few recordings per word and speaker, the templates mix different words (text-independent enrollment on 1 s clips), so these EERs are conservative for both attacks.

Revision: new App. A.4 "Verification against independent enrollment" (Table 10: all Table 1 and MFCC settings, with CIs and the leak rate at 1% FAR); Sec. 4.3, 5.1 and 5.2. Speaker ID % at τ=0.25\tau=0.25 is kept for comparability and described as a comparison with the original recording.

W2 / Q4 -- Table 1 model accuracy vs. attack settings interpretation.

We agree and clarify the difference as follows. The two columns come from different settings and do not establish a privacy--utility trade-off for one defended system. The attack inverts a single-sample gradient of the untrained AudioCNN. Model Acc. comes from separate practical 50-epoch training runs with the same zz applied to each mini-batch gradient (batch 64, CC set to the median batch-gradient norm). Model Acc. is meant only as a practical reference for the accuracy observed when training at each noise level. It is not a privacy guarantee, and no (ε,δ)(\varepsilon, \delta) is claimed.

The interpretation is that the noise which is affordable in training and defeats the GAN-free attack does not remove identity. At z=10−3z = 10^{-3}, GAN-free Speaker ID falls to 18.0% on AM and 29.3% on SC, while the GAN-based attack retains 84.0% and 96.5%. Training at this zz costs no practical accuracy on AM and 6.4 points on SC.

A matched configuration would still differ: The attack inverts the gradient of the first update, whereas Model Acc. reflects 50 epochs of noisy batch updates in which the gradient changes at every step. A fully matched setting would require inverting the batched, multi-step training update. We instead study the single-sample update on an untrained network as a controlled setting, because it sets the worst case for the client.

Revision: the Table 1 caption and Sec. 5.1 state the distinction; App. A.10 describes both settings and states that no (ε,δ)(\varepsilon, \delta) is claimed.

W1 (content) / Q3 -- Keyword accuracy and WER on the reconstructions.

We agree that PESQ and spectral error do not directly measure linguistic content. Therefore, we transcribed every saved reconstruction with Whisper-small (English, unconstrained decoding), which is independent of the victim and of the GAN prior. Since the attacker knows the label, we report keyword accuracy (KW acc.: the label word appears in the normalised transcript) as a measure of preservation, with WER against the label as a secondary measure. As reference, the original mel-spectrogram is passed through the same vocoder and transcribed (vocoded original), which separates errors of the vocoder and the ASR from those of the attack.

DatasetSettingKW acc. GAN-freeKW acc. GAN-basedWER GAN-freeWER GAN-based
AMVocoded original98.0%0.02
AMNo Def.97.0%94.0%0.040.07
AMz=10−3z=10^{-3}40.0%72.0%0.710.54
AMnmfcc=15n_\mathrm{mfcc}=1556.0%89.0%0.480.14
SCVocoded original79.8%0.27
SCNo Def.52.0%71.2%0.650.39
SCz=10−3z=10^{-3}19.0%58.8%1.290.56
SCnmfcc=15n_\mathrm{mfcc}=1523.2%63.7%0.890.48

Untrained AudioCNN, BVG vocoder, Whisper-small. After the MFCC round trip (nmfcc=15n_\mathrm{mfcc}=15) the vocoded original reaches 86.0% (AM) and 61.8% (SC). WER is against the one-word label and can exceed 1 when words are inserted.

Undefended, both attacks preserve the word close to the reference on AM, and the GAN-based attack preserves more on SC. Under the strongest perturbation and compression, content degrades for both attacks, but the GAN-based attack retains more of it. For the GAN-based attack, content degrades faster than identity: from undefended to z=10−3z=10^{-3}, KW acc. falls from 94.0% to 72.0% on AM and from 71.2% to 58.8% on SC, while Speaker ID remains at 84.0% and 96.5%. This supports our claim that degraded content (and fidelity) is not evidence of speaker privacy.

Revision: new App. A.5 "Linguistic content preservation" (Table 11: all Table 1 and MFCC settings, with 95% CIs); KW acc. added as a content metric in Sec. 4.3; one sentence in Sec. 5.1.

Q1 -- Fig. 4: reference recording clarification and new analysis.

The caption was ambiguous, and we have corrected it. In Fig. 4 the genuine scores compare each reconstruction with its own original recording, as in Speaker ID %, and the impostor scores compare it with recordings of the other held-out speakers. The AUROC values in App. A.3 (0.98 vs 0.55--0.61) use the same pairs. Only Top-1 excludes the original recording: it ranks speakers by similarity to templates built from their other recordings, so it does not depend on fidelity to the attacked clip.

To address the question directly, we repeated the analysis against separate enrollment recordings, selected as in our answer to W1 / Q2 (up to 10 recordings per speaker that are not attack targets, so the target recording is never included). The gap remains: AUROC 0.84 vs 0.57 on AM and 0.87 vs 0.53 on SC (GAN-based vs GAN-free), and 0.99 / 0.97 for the original audio.

Revision: the Fig. 4 caption and App. A.3 now state which comparisons use the original recording; enrollment AUROC added in App. A.3, and the new Fig. 5 repeats the analysis against independent enrollment recordings (App. A.4).

The authors address my major concerns and therefore I increased my score to 4: clear acceptance.

Thank you for the constructive feedback and for raising the score. We are glad our additional evaluations addressed your concerns.

Reviewer_GfHgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates privacy leakage from shared gradients in federated speech learning. The authors propose a generative-prior-guided gradient inversion method that uses a StyleGAN2-based model to reconstruct mel-spectrograms through gradient matching, followed by BigVGAN for waveform reconstruction. Experiments on AudioMNIST and Speech Commands evaluate the method under gradient perturbations, compressed MFCC features, trained victim models, and different network architectures. The results show that generative priors can improve speech reconstruction and preserve speaker identity information even when gradient information is degraded.

Critical strengths
  1. The paper introduces generative priors into speech gradient inversion and quantifies speaker identity leakage under perturbed gradients. This provides new empirical evidence in the speech domain.

  2. The experimental evaluation is fairly comprehensive, covering gradient perturbations, acoustic feature compression, and different model architectures. I also appreciate the speaker-disjoint setting and the additional controls used to rule out identity memorization by the generative prior.

Critical weaknesses
  1. I find the methodological contribution rather limited, as the proposed approach largely follows GIFD from the image domain. I think investigating the limitations of transferring existing image-based methods to speech could have provided more meaningful insights. For example, the attack performs poorly on ResNet18 and AST (while GIFD has shown effectiveness on ResNet-18 in image reconstruction). However, the paper provides limited analysis of why this difference occurs or how it could be addressed. I believe a deeper investigation of these limitations could lead to more meaningful insights or new methodological contributions to speech gradient inversion.
Questions for authors
  1. The main speaker identity leakage results rely on a fixed ECAPA similarity threshold (0.25). Although Appendix A.3 provides additional verification and identification analyses, could the authors clarify how the high leakage rates under strong gradient perturbations translate into actual speaker identification performance?
  2. The proposed method uses both GAN latent optimization and intermediate-layer optimization (ILO), but their respective contributions are not evaluated. How much does ILO improve reconstruction compared with optimizing only the GAN latent space?
4 comments

We thank the reviewer for the thoughtful review, and for recognizing the new empirical evidence for the speech domain, the breadth of the evaluation, and the speaker-disjoint setting. The questions on the transfer from images, identification and ILO prompted several new analyses. We took the time to complete the new experiments so that each point below is answered with results.

W1 -- Methodological contribution and difference from the image domain.

We agree that the optimisation framework (W+W^{+}, ILO) is inherited from GIFD, and the paper acknowledges this along with what transfers to the audio domain with the improved pipeline and empirical findings. Our contribution on the audio domain: an identity-disjoint mel-spectrogram prior with neural vocoding, a differentiable MFCC path, and an identity evaluation that now includes verification against independent enrollment recordings (Q1). The ILO ablation (Q2) shows that the gain over the GAN-free attack comes from the W+W^{+} audio prior rather than from the GIFD-specific ILO step.

To investigate ResNet18 and AST models, we replaced each model's global readout with a flatten head, keeping everything else fixed, and ran the GAN-based attack with 100 targets per dataset.

DatasetModelReadoutSTOIECAPA Cos.SimSpeaker ID
AMResNet18avg. pool0.3440.0663.0%
AMResNet18flatten0.4180.0903.0%
AMAST$_\text{tiny}$CLS0.4440.1157.0%
AMAST$_\text{tiny}$flatten0.9240.80299.0%
SCResNet18avg. pool0.2200.0564.0%
SCResNet18flatten0.2560.0675.0%
SCAST$_\text{tiny}$CLS0.1760.12213.0%
SCAST$_\text{tiny}$flatten0.6990.743100.0%

Untrained, undefended victims, BVG vocoder. AST$_\text{tiny}$ (dim 192, depth 4) resists similarly to the AST of Table 4 (7.0% vs 17.0% on AM, 13.0% vs 15.0% on SC).

  • AST: the CLS readout causes the resistance. Removing it restores leakage. With the CLS readout the gradient barely changes when the input is shifted (2.4-4.5$\times$ less than with the flatten head), so many inputs match the observed gradient.
  • ResNet18: global average pooling is not the cause. Removing it restores no identity, although the spectral fit improves (AM Mel MSE 14.6 →\to 2.7). Therefore, we have removed the post-hoc attribution to global pooling.
  • Why GIFD's ResNet18 result does not carry over: the prior is not the bottleneck (the same prior reaches 98% Speaker ID on AudioCNN and 91% on LeNet), and the ResNet18 gradient still carries identity (new experiment: speaker separation ratio S=1.39S = 1.39 on AM and 1.43 on SC, where S=1S = 1 means blind to identity). GIFD is evaluated by image similarity, but for speech a closer spectrogram (Mel MSE 14.6 →\to 2.7, Mel Cos.Sim 0.969 →\to 0.987) does not recover the speaker, so success measured on the spectrogram does not transfer to identity. The resistance is a harder inversion, not privacy. Isolating the ResNet18 cause is mentioned as an open direction.

Revision: new App. A.7 "Global aggregation ablation" (attack results in Table 14; gradient shift sensitivity and speaker separation ratio in Table 15); Table 20 (victim architectures) extended with the ablation variants; Sec. 5.4 and Discussion revised.

Q1 -- Speaker identification beyond the 0.25 threshold.

We added verification against independent enrollment recordings for all Table 1 and MFCC settings. Each held-out speaker's template is the mean ECAPA embedding of up to 10 recordings that are not attack targets, and the verification threshold is calibrated to 1% FAR on separate speakers. Under the strongest perturbation (z=10−3z=10^{-3}), the GAN-based reconstructions remain identifiable: Top-1 identification over the held-out speakers is 62.0% on AM and 37.9% on SC, vs 28.0% and 1.6% for the GAN-free attack. The verification EER is lower for GAN-based: 22.0% / 16.9% vs 37.7% / 38.7% (AM/SC). The full results, including the leak rate at the calibrated threshold, are in the new App. A.4 "Verification against independent enrollment" (Table 10).

Revision: new App. A.4 "Verification against independent enrollment"; Sec. 4.3, 5.1 and 5.2 point to it. Speaker ID % at τ=0.25\tau=0.25 is kept for comparability with prior work and is described as a comparison with the original recording.

Q2 -- Contribution of ILO.

We ran an equal-budget ablation (3000 iterations) of W+W^{+}-only optimisation against W+W^{+} (1500) + ILO (1500) on all AM (100) and SC (400) targets: undefended, at z=10−3z=10^{-3}, and at nmfcc=15n_\mathrm{mfcc}=15. All other settings and seeds are unchanged.

DatasetSettingVariantFeat. MSESTOIPESQECAPASpeaker ID
AMNo Def.W+W^{+} only0.1850.8952.3590.72295.0%
W+W^{+} + ILO0.1810.8952.3120.71398.0%
z=10−3z=10^{-3}W+W^{+} only0.7080.7751.3570.37179.0%
W+W^{+} + ILO0.6110.7841.4070.40584.0%
nmfcc=15n_\mathrm{mfcc}=15W+W^{+} only0.7460.8211.3630.40581.0%
W+W^{+} + ILO0.5810.8241.3730.42183.0%
SCNo Def.W+W^{+} only0.2370.7302.3410.72798.8%
W+W^{+} + ILO0.2270.7262.2990.71098.8%
z=10−3z=10^{-3}W+W^{+} only0.4160.6601.7310.53596.0%
W+W^{+} + ILO0.3870.6631.7400.54596.5%
nmfcc=15n_\mathrm{mfcc}=15W+W^{+} only0.5460.6551.3890.41890.0%
W+W^{+} + ILO0.5390.6561.3950.41484.3%

Equal 3000-iteration budget, untrained AudioCNN, AM 100 / SC 400 targets. W+W^{+} + ILO: 1500 W+W^{+} + 1500 ILO iterations (values from Tables 1, 2). Feat. MSE = Mel MSE (MFCC MSE for nmfcc=15n_\mathrm{mfcc}=15). Bold: better variant.

  • The W+W^{+}-only variant also achieves similar performance gains over the GAN-free attack; the identity leakage is carried by the W+W^{+} prior. Without ILO, Speaker ID at z=10−3z=10^{-3} is 79.0% (AM) / 96.0% (SC) vs 18.0% / 29.3% for the GAN-free baseline, and at nmfcc=15n_\mathrm{mfcc}=15 it is 81.0% / 90.0% vs 48.0% / 34.0%.

  • ILO lowers feature MSE in every setting. At z=10−3z=10^{-3} it improves all five metrics on both datasets.

  • However, the attack performance gain yielded by ILO is not consistent across the settings tested and mainly contributes as a fidelity refinement in challenging settings.

Revision: new App. A.9 "ILO ablation" (Table 21); pointers in Sec. 3.3 and Sec. 5.

Reviewer_GfHgOpenReview ↗

Response to Authors

Thanks for your response and effort. My questions about soundness (ILO/threshold/failure of AST) have been addressed, and the experiments are unusually extensive.

However, I still find it difficult to gain insights beyond transferring existing generative-prior methods (GIFD) from image to speech. The vocoder/Mel prior/differentiable MFCC path seem more like natural engineering adaptations. Although the authors found during the rebuttal that CLS/flatten readout has a substantial impact on performance, and the explaination is useful, but I think this is an empirical finding rather than a broader insight into the challenges of transferring the method to speech.

Overall, I think the paper extends an existing generative-prior inversion framework to an already-studied audio gradient inversion task, provides a thorough empirical study of a specific problem, but the methodological advance/new insights remain limited. Therefore, I'll keep my score.

Thank you for the follow-up and for confirming that the soundness questions are resolved. We respect your assessment and thank you again for the detailed review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their constructive feedback. Several points called for new experiments (verification against enrollment recordings, content metrics, aggregation and ILO ablations, an extended noise grid and converged victims), and we completed these so that each answer is backed by results.

Verification against independent enrollment recordings (GfHg Q1; Gc4F W1, Q1, Q2). Reconstructions are now also verified against templates built from separate recordings of each speaker (the attacked clip is never included), for all Table 1 and MFCC settings and for the LeNet analysis of Fig. 4. Under the strongest perturbation the GAN-based attack keeps the speaker identifiable: at z=10−3z=10^{-3} the EER is 22.0% vs 37.7% on AM and 16.9% vs 38.7% on SC. The Fig. 4 caption now states which comparisons use the original recording.

Revision: new App. A.4 "Verification against independent enrollment" (Table 10, Fig. 5); Fig. 4 caption and App. A.3; Sec. 4.3, 5.1 and 5.2.

Utility and attack settings separated in Table 1 (Gc4F W2, Q4). Model Acc. is now described as the accuracy observed when training at each zz, not a privacy-utility trade-off, and no (ε,δ)(\varepsilon, \delta) is claimed.

Revision: Table 1 caption, Sec. 5.1 and App. A.10.

Breaking point of the attack under stronger noise (PJ7Z W3). An extended noise grid up to zD=35.8z\sqrt{D} = 35.8 shows identity leakage holds up to 3.583.58 and breaks by 10.710.7; the claim is narrowed to perturbation that defeats feature-space inversion.

Revision: Table 22 extended to zD=35.8z\sqrt{D} = 35.8; new Table 23 and "Attack breaking point" paragraph in App. A.10; Sec. 5.1. reference to A.10.

Direct content metric (Gc4F W1, Q3). We added Whisper-small keyword accuracy and WER for all Table 1 and MFCC settings. Under the strongest perturbation and compression the GAN-based attack retains more content than the GAN-free baseline (at z=10−3z=10^{-3}: 72.0% vs 40.0% on AM, 58.8% vs 19.0% on SC), and its content degrades faster than its identity leakage.

Revision: new App. A.5 "Linguistic content preservation" (Table 11); Sec. 4.3 and 5.1.

Testing the aggregation hypothesis (GfHg W1; PJ7Z W1, W4, Q2). We replaced the global readout of each model with a flatten head, keeping everything else fixed. For AST this restores leakage (Speaker ID 7% →\to 99% on AM, 13% →\to 100% on SC): its CLS readout makes the gradient nearly input-invariant. For ResNet18 it does not (3% →\to 3%, 4% →\to 5%), so we removed the global-pooling post-hoc hypothesis. In all variants the gradient still separates speakers (new gradient speaker sensitivity test), so the resistance reflects a harder inversion problem, not the absence of identity.

Revision: new App. A.7 "Global aggregation ablation" (Tables 14, 15); Table 20 (victim architectures) extended; Sec. 5.4 and Discussion revised.

Contribution of ILO (GfHg Q2). An equal-budget ablation (3000 iterations) of W+W^{+}-only optimisation against W+W^{+} + ILO shows that the advantage over the GAN-free attack is carried by the W+W^{+} prior: without ILO, Speaker ID at z=10−3z=10^{-3} is 79.0% (AM) / 96.0% (SC) vs 18.0% / 29.3% for the GAN-free baseline. ILO lowers feature MSE in every setting and improves all five metrics at z=10−3z=10^{-3}, acting as a fidelity refinement.

Revision: new App. A.9 "ILO ablation" (Table 21); pointers in Sec. 3.3 and Sec. 5.

Converged victims (PJ7Z W2, Q1). We attacked AudioCNN victims trained for 50 epochs, where validation accuracy has plateaued (96.6% on AM, 79.9% on SC). The GAN-based attack retains more speaker identity than the GAN-free attack on both datasets: Speaker ID 55.0% vs 23.0% on AM and 82.0% vs 75.3% on SC, with higher STOI, PESQ and ECAPA Cos.Sim. From epoch 30 to 50, the GAN-free attack's Speaker ID drops by about 19 points on both datasets, while the GAN-based attack drops by about 13.

Revision: epoch-50 rows in Table 3; Sec. 5.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.