- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VoxGuard, a defense method that perturbs only the reference audio to disrupt the learning of audio2motion correspondences during personalized talking face generation while preserving visual frames. Experiments demonstrate reduced synchronization, improved transferability across audio encoders, and better perceptual acceptability.
Critical strengths
- The paper reframes proactive TFG defense as disrupting audio2motion correspondence, enabling protection without modifying facial pixels.
- The proposed feature redirection in audio embedding space provides a conceptually clean and model agnostic approach under constrained perturbation budgets and psychoacoustic masking.
- The paper is well motivated and clearly written, with a logical structure and coherent presentation.
Critical weaknesses
-
VoxGuard perturbs audio to achieve defense, whereas most compared methods operate in the visual domain. These methods have fundamentally different mechanisms and perceptual trade-offs, making direct comparisons potentially less informative. A comparison with audio-based protection methods, such as audio watermarking, would provide a more appropriate evaluation. Relevant works include: [1] P. O’Reilly, Z. Jin, J. Su, and B. Pardo, Deep Audio Watermarks Are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech, in Proceedings of the ICLR 2025 Workshop on GenAI Watermarking, 2025. [2] W. Liu, Y. Li, D. Lin, H. Tian, and H. Li, GROOT: Generating Robust Watermark for Diffusion-Model-Based Audio Synthesis, in Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3294–3302, 2024, doi: 10.1145/3664647.3680596.
-
An experienced adversary may detect anomalies in the perturbed audio and replace it with speech regenerated by a text2audio model before personalization. The paper does not sufficiently investigate whether VoxGuard remains effective under such adaptive countermeasures.
-
The impact of audio perturbations on benign downstream tasks and the usability of the original videos for ordinary distribution and consumption remains unclear. Further evaluation is needed to better understand the practical trade-offs of the proposed defense.
-
The approach relies on a fixed target utterance to guide representation redirection, but the sensitivity to target selection and potential linguistic mismatches is not thoroughly investigated.
Questions for authors
To address the weaknesses mentioned above, please clarify the following points in your rebuttal:
-
Could the authors clarify the rationale for comparing VoxGuard with visual domain defenses, given that VoxGuard operates in the audio domain? Would comparisons with relevant audio-based protection methods provide a fairer evaluation, or could the authors explain the limitations of such comparisons?
-
How robust is VoxGuard against adaptive adversaries who detect anomalies in the perturbed audio and replace it with speech regenerated by a text2audio model before personalization?
-
Could the authors evaluate the impact of VoxGuard on benign downstream audio tasks and the usability of protected videos in ordinary distribution and consumption?
-
How sensitive is VoxGuard to the choice of target utterance? Could different target utterances or linguistic mismatches affect the effectiveness of representation redirection?
3 comments
Response to Reviewer vQ3R: Baseline Fairness, Adaptive Attacks, and Practical Robustness (Q3 & Q4)
Q3: Benign Downstream Utility and Perceptual Usability
R3: We thank the reviewer for this valuable suggestion. Our additional evaluations indicate that VoxGuard largely preserves benign downstream ASR utility on the tested clips.
We evaluated OpenAI Whisper large-v3 on nine randomly selected clips from our evaluation datasets, comparing clean and VoxGuard-protected audio against manually transcribed references. Under identical decoding settings, WER increased from 1.80% to 3.15% (+1.35 percentage points), while CER remained unchanged at 4.55%. Identical evaluation intervals were used for both conditions.
Regarding practical usability, our human evaluation (Table 3(c)) shows that VoxGuard reduces overall acceptability from 4.61 to 3.63 and audio quality from 4.54 to 2.98 compared with Clean. Nevertheless, VoxGuard achieves numerically higher overall acceptability than Silencer-I (3.63 vs. 3.30), with higher visual-quality ratings (4.74 vs. 3.05), despite lower audio quality (2.98 vs. 4.62). These results suggest a potential perceptual benefit of shifting protection from facial pixels to audio under the evaluated settings.
We acknowledge that preserving downstream ASR utility does not eliminate perceptible audio distortion. Nevertheless, our additional experiments show that increasing the psychoacoustic masking weight (from 5 to 10) reduces CDPAM from 0.2858 to 0.2627, an 8.1% reduction, while preserving originally silent intervals reduces it to 0.1365, a 52.2% reduction. The corresponding Sync-C/D scores are 2.0970/11.0553 and 2.1448/11.0640, respectively, compared with 1.9214/11.1923 for the original VoxGuard. These results suggest promising directions for reducing audio perceptual distortion while largely maintaining defense effectiveness. We will clarify these practical trade-offs in the revised manuscript.
Q4: Sensitivity to Target Utterance Selection and Cross-Lingual Mismatch
R4: We thank the reviewer for this valuable suggestion. Our additional experiments demonstrate relatively stable defense effectiveness across different target utterances, speakers, and language settings, suggesting that VoxGuard does not critically depend on the original fixed target.
We evaluated four alternative target utterance configurations for each source video: two same-language targets (A/B) and two cross-language targets (A/B), with A and B representing distinct target speakers. The source videos cover English, Mandarin Chinese, Korean, and French. For English source videos, the cross-language targets are in Mandarin Chinese; for the other three source languages, they are in English. All experiments used InsTaG with the AVE audio encoder, and the same clean evaluation audio was maintained across target configurations for each source video.
| Target Configuration | Sync-C ↓ | Sync-D ↑ | CDPAM ↓ |
|---|---|---|---|
| Original fixed target | 1.9214 | 11.1923 | 0.2858 |
| Same-language A | 1.7722 | 11.4672 | 0.2816 |
| Same-language B | 1.8015 | 11.4353 | 0.2957 |
| Cross-language A | 1.7183 | 11.5805 | 0.2902 |
| Cross-language B | 1.9674 | 11.3238 | 0.2793 |
Each table row reports the average results across the evaluated source videos spanning four languages.
Across alternative targets, Sync-C ranges from 1.718 to 1.967 and Sync-D from 11.324 to 11.581. Same-language and cross-language targets yield comparable average Sync-C/D scores (1.787/11.451 vs. 1.843/11.452), suggesting limited sensitivity to the evaluated cross-language target configurations.
The original fixed target was used for experimental consistency rather than being a requirement of the method. These results support limited sensitivity to the evaluated target variations, although they do not establish complete target or language independence. We will incorporate this analysis and clarify its evaluation scope in the revised manuscript.
Follow-up on Our Rebuttal
Follow-up on Our Rebuttal
Dear Reviewer vQ3R,
Thank you again for your valuable feedback and constructive suggestions.
We have carefully addressed your concerns in our rebuttal, supported by additional experiments and detailed clarifications.
As the discussion period is approaching its end, we would sincerely appreciate it if you could review our responses and share any further feedback, particularly regarding whether your original concerns have been adequately addressed.
We would be happy to provide any additional clarifications if needed.
Thank you very much for your time and consideration.
Best regards,
The Authors
Response to Reviewer vQ3R: Baseline Fairness, Adaptive Attacks, and Practical Robustness (Q1 & Q2)
Q1: Baseline Selection and Comparison Fairness
R1: We thank the reviewer for suggesting these studies. We select defense baselines based on their protection objectives rather than perturbation modalities. The cited audio watermarking studies focus on watermark robustness (O'Reilly et al.) or provenance tracking (GROOT), rather than disrupting unauthorized TFG personalization. Our additional experiments further confirm that audio watermarking provides negligible protection against personalized TFG, even when introducing perceptual distortion approaching that of VoxGuard.
Despite a CDPAM of 0.2354, approaching VoxGuard's 0.2858, watermarked GROOT achieves synchronization results comparable to clean training (Sync-C/D: 6.261/7.948 vs. 6.260/7.815). Its watermark-free variant performs similarly, indicating that watermark embedding itself provides negligible protection. AudioSeal likewise shows minimal disruption.
We compare with visual-domain proactive defenses because they address the same underlying threat of unauthorized portrait generation. We selected Silencer-I because both methods aim to disrupt lip–audio synchronization. However, Silencer-I targets generalized TFG, whereas VoxGuard targets personalized TFG. To avoid potentially unfair cross-paradigm adaptation, we evaluate Silencer-I on its original Hallo pipeline and additionally report its clean baseline. For personalized TFG, we further introduce VideoFacePoison, a visual-domain defense targeting personalized learning, and compare it with VoxGuard using the same InsTaG generator.
“—” denotes clean, undefended reference videos. Human-rated overall acceptability is assessed on the five-point scale.
Under the same InsTaG pipeline, VoxGuard achieves stronger synchronization disruption than VideoFacePoison (Sync-C/D: 1.9214/11.1923 vs. 4.8218/9.1139), alongside higher human-rated overall acceptability (3.63 vs. 2.12).
We conduct these comparisons to demonstrate the feasibility of audio-domain protection and highlight its potential advantages in overall audiovisual acceptability under the evaluated settings. We will clarify the rationale for baseline selection and the scope of these comparisons in the revised manuscript.
Q2: Robustness against TTS-Based Audio Replacement
R2: We thank the reviewer for raising this important concern. Our additional experiments suggest that ASR–TTS-based audio replacement may not provide a straightforward bypass of VoxGuard, as regenerated speech can introduce substantial audio–visual misalignment that compromises personalization.
Specifically, we transcribed protected audio using OpenAI Whisper large-v3 and regenerated speech using MeloTTS. Since the regenerated speech does not naturally match the original video's timing, we evaluated two alignment strategies: truncation and time stretching.
The following table reports audio–visual synchronization between the original reference video and its corresponding audio tracks.
Neither strategy restored synchronization with the original video in our experiments. We further evaluated downstream personalization using time-stretched MeloTTS audio. The following table reports synchronization of generated videos after personalization with the corresponding training audio.
MeloTTS-based replacement yields even lower synchronization than VoxGuard-protected training, suggesting that the resulting audio–visual misalignment itself may compromise personalization. These findings indicate that simply regenerating and replacing the audio may not reliably recover downstream generation quality under the evaluated settings.