acceptodds
Under review as a conference paper at ICLR 2027

Defending Faces without Touching Pixels: Audio-Only Protection against Personalized Talking Face Generation

Abstract

The rapid development of talking face generation (TFG) has raised growing concerns about identity misuse. In particular, audio-driven 3D TFG can reconstruct a reusable personalized 3D portrait from a reference video and animate it with arbitrary audio, making identity impersonation increasingly accessible. To counter such unauthorized reuse, existing proactive defenses predominantly inject protective perturbations into visual content to disrupt generative identity acquisition. However, these defenses face a sharp tradeoff between defense effectiveness and perceptual quality. Perturbing facial regions can noticeably degrade portrait appearance due to the high perceptual sensitivity of human faces, whereas shifting perturbations toward less visually sensitive regions or the background makes them more vulnerable to resizing, compression, and background replacement. To mitigate these limitations, we propose VoxGuard, a perceptually constrained audio-only defense that disrupts the learning of correspondences between audio and identity-specific facial motion during personalization. Instead of perturbing visual content, VoxGuard modifies only the audio track of the reference video, leaving visual frames unchanged and thereby avoiding modification of visually sensitive facial regions. To further reduce perceptual distortion, we employ psychoacoustic masking to guide perturbations toward perceptually masked regions. We further optimize across heterogeneous audio encoders to enhance black-box transferability and introduce continual optimization to mitigate interference among encoder objectives while preserving previously established representation shifts. Extensive experiments demonstrate that VoxGuard effectively disrupts unauthorized 3D TFG while limiting perceptual degradation, exhibits black-box transferability, and remains effective after common audio transformations, including MP3 and AAC compression and audio resampling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Talking Face Generation, Proactive Defense, Audio Visual Learning, Identity Privacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vQ3ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VoxGuard, a defense method that perturbs only the reference audio to disrupt the learning of audio2motion correspondences during personalized talking face generation while preserving visual frames. Experiments demonstrate reduced synchronization, improved transferability across audio encoders, and better perceptual acceptability.

Critical strengths
  1. The paper reframes proactive TFG defense as disrupting audio2motion correspondence, enabling protection without modifying facial pixels.
  2. The proposed feature redirection in audio embedding space provides a conceptually clean and model agnostic approach under constrained perturbation budgets and psychoacoustic masking.
  3. The paper is well motivated and clearly written, with a logical structure and coherent presentation.
Critical weaknesses
  1. VoxGuard perturbs audio to achieve defense, whereas most compared methods operate in the visual domain. These methods have fundamentally different mechanisms and perceptual trade-offs, making direct comparisons potentially less informative. A comparison with audio-based protection methods, such as audio watermarking, would provide a more appropriate evaluation. Relevant works include: [1] P. O’Reilly, Z. Jin, J. Su, and B. Pardo, Deep Audio Watermarks Are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech, in Proceedings of the ICLR 2025 Workshop on GenAI Watermarking, 2025. [2] W. Liu, Y. Li, D. Lin, H. Tian, and H. Li, GROOT: Generating Robust Watermark for Diffusion-Model-Based Audio Synthesis, in Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3294–3302, 2024, doi: 10.1145/3664647.3680596.

  2. An experienced adversary may detect anomalies in the perturbed audio and replace it with speech regenerated by a text2audio model before personalization. The paper does not sufficiently investigate whether VoxGuard remains effective under such adaptive countermeasures.

  3. The impact of audio perturbations on benign downstream tasks and the usability of the original videos for ordinary distribution and consumption remains unclear. Further evaluation is needed to better understand the practical trade-offs of the proposed defense.

  4. The approach relies on a fixed target utterance to guide representation redirection, but the sensitivity to target selection and potential linguistic mismatches is not thoroughly investigated.

Questions for authors

To address the weaknesses mentioned above, please clarify the following points in your rebuttal:

  1. Could the authors clarify the rationale for comparing VoxGuard with visual domain defenses, given that VoxGuard operates in the audio domain? Would comparisons with relevant audio-based protection methods provide a fairer evaluation, or could the authors explain the limitations of such comparisons?

  2. How robust is VoxGuard against adaptive adversaries who detect anomalies in the perturbed audio and replace it with speech regenerated by a text2audio model before personalization?

  3. Could the authors evaluate the impact of VoxGuard on benign downstream audio tasks and the usability of protected videos in ordinary distribution and consumption?

  4. How sensitive is VoxGuard to the choice of target utterance? Could different target utterances or linguistic mismatches affect the effectiveness of representation redirection?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vQ3R: Baseline Fairness, Adaptive Attacks, and Practical Robustness (Q1 & Q2)

Q1: Baseline Selection and Comparison Fairness

 

R1: We thank the reviewer for suggesting these studies. We select defense baselines based on their protection objectives rather than perturbation modalities. The cited audio watermarking studies focus on watermark robustness (O'Reilly et al.) or provenance tracking (GROOT), rather than disrupting unauthorized TFG personalization. Our additional experiments further confirm that audio watermarking provides negligible protection against personalized TFG, even when introducing perceptual distortion approaching that of VoxGuard.

Training AudioSync-C ↓Sync-D ↑CDPAM ↓
Clean6.2607.815—
AudioSeal6.2177.8270.0064
GROOT, watermark-free resynthesis6.1917.9200.2336
GROOT, watermarked resynthesis6.2617.9480.2354
VoxGuard (Ours)1.92111.1920.2858

Despite a CDPAM of 0.2354, approaching VoxGuard's 0.2858, watermarked GROOT achieves synchronization results comparable to clean training (Sync-C/D: 6.261/7.948 vs. 6.260/7.815). Its watermark-free variant performs similarly, indicating that watermark embedding itself provides negligible protection. AudioSeal likewise shows minimal disruption.

We compare with visual-domain proactive defenses because they address the same underlying threat of unauthorized portrait generation. We selected Silencer-I because both methods aim to disrupt lip–audio synchronization. However, Silencer-I targets generalized TFG, whereas VoxGuard targets personalized TFG. To avoid potentially unfair cross-paradigm adaptation, we evaluate Silencer-I on its original Hallo pipeline and additionally report its clean baseline. For personalized TFG, we further introduce VideoFacePoison, a visual-domain defense targeting personalized learning, and compare it with VoxGuard using the same InsTaG generator.

TFG ParadigmGeneratorMethodHuman-rated Overall Acceptability ↑Sync-C ↓Sync-D ↑
GeneralizedHallo—4.616.45567.9322
GeneralizedHalloSilencer-I3.303.315011.1310
PersonalizedInsTaG—4.616.24907.8406
PersonalizedInsTaGVideoFacePoison2.12364.82189.1139
PersonalizedInsTaGVoxGuard (Ours)3.631.921411.1923

“—” denotes clean, undefended reference videos. Human-rated overall acceptability is assessed on the five-point scale.

Under the same InsTaG pipeline, VoxGuard achieves stronger synchronization disruption than VideoFacePoison (Sync-C/D: 1.9214/11.1923 vs. 4.8218/9.1139), alongside higher human-rated overall acceptability (3.63 vs. 2.12).

We conduct these comparisons to demonstrate the feasibility of audio-domain protection and highlight its potential advantages in overall audiovisual acceptability under the evaluated settings. We will clarify the rationale for baseline selection and the scope of these comparisons in the revised manuscript.

 

Q2: Robustness against TTS-Based Audio Replacement

R2: We thank the reviewer for raising this important concern. Our additional experiments suggest that ASR–TTS-based audio replacement may not provide a straightforward bypass of VoxGuard, as regenerated speech can introduce substantial audio–visual misalignment that compromises personalization.

Specifically, we transcribed protected audio using OpenAI Whisper large-v3 and regenerated speech using MeloTTS. Since the regenerated speech does not naturally match the original video's timing, we evaluated two alignment strategies: truncation and time stretching.

The following table reports audio–visual synchronization between the original reference video and its corresponding audio tracks.

Audio ConditionSync-C ↑Sync-D ↓
Original Clean Audio6.79957.5254
MeloTTS (Truncation)0.439814.5591
MeloTTS (Time-Stretching)0.593014.0270

Neither strategy restored synchronization with the original video in our experiments. We further evaluated downstream personalization using time-stretched MeloTTS audio. The following table reports synchronization of generated videos after personalization with the corresponding training audio.

Personalization AudioSync-C ↑Sync-D ↓
Clean6.2497.841
VoxGuard1.921411.192
MeloTTS (Time-Stretching)1.56511.466

MeloTTS-based replacement yields even lower synchronization than VoxGuard-protected training, suggesting that the resulting audio–visual misalignment itself may compromise personalization. These findings indicate that simply regenerating and replacing the audio may not reliably recover downstream generation quality under the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer vQ3R: Baseline Fairness, Adaptive Attacks, and Practical Robustness (Q3 & Q4)

Q3: Benign Downstream Utility and Perceptual Usability

R3: We thank the reviewer for this valuable suggestion. Our additional evaluations indicate that VoxGuard largely preserves benign downstream ASR utility on the tested clips.

We evaluated OpenAI Whisper large-v3 on nine randomly selected clips from our evaluation datasets, comparing clean and VoxGuard-protected audio against manually transcribed references. Under identical decoding settings, WER increased from 1.80% to 3.15% (+1.35 percentage points), while CER remained unchanged at 4.55%. Identical evaluation intervals were used for both conditions.

Regarding practical usability, our human evaluation (Table 3(c)) shows that VoxGuard reduces overall acceptability from 4.61 to 3.63 and audio quality from 4.54 to 2.98 compared with Clean. Nevertheless, VoxGuard achieves numerically higher overall acceptability than Silencer-I (3.63 vs. 3.30), with higher visual-quality ratings (4.74 vs. 3.05), despite lower audio quality (2.98 vs. 4.62). These results suggest a potential perceptual benefit of shifting protection from facial pixels to audio under the evaluated settings.

We acknowledge that preserving downstream ASR utility does not eliminate perceptible audio distortion. Nevertheless, our additional experiments show that increasing the psychoacoustic masking weight (from 5 to 10) reduces CDPAM from 0.2858 to 0.2627, an 8.1% reduction, while preserving originally silent intervals reduces it to 0.1365, a 52.2% reduction. The corresponding Sync-C/D scores are 2.0970/11.0553 and 2.1448/11.0640, respectively, compared with 1.9214/11.1923 for the original VoxGuard. These results suggest promising directions for reducing audio perceptual distortion while largely maintaining defense effectiveness. We will clarify these practical trade-offs in the revised manuscript.

 

Q4: Sensitivity to Target Utterance Selection and Cross-Lingual Mismatch

R4: We thank the reviewer for this valuable suggestion. Our additional experiments demonstrate relatively stable defense effectiveness across different target utterances, speakers, and language settings, suggesting that VoxGuard does not critically depend on the original fixed target.

We evaluated four alternative target utterance configurations for each source video: two same-language targets (A/B) and two cross-language targets (A/B), with A and B representing distinct target speakers. The source videos cover English, Mandarin Chinese, Korean, and French. For English source videos, the cross-language targets are in Mandarin Chinese; for the other three source languages, they are in English. All experiments used InsTaG with the AVE audio encoder, and the same clean evaluation audio was maintained across target configurations for each source video.

Target ConfigurationSync-C ↓Sync-D ↑CDPAM ↓
Original fixed target1.921411.19230.2858
Same-language A1.772211.46720.2816
Same-language B1.801511.43530.2957
Cross-language A1.718311.58050.2902
Cross-language B1.967411.32380.2793

Each table row reports the average results across the evaluated source videos spanning four languages.

Across alternative targets, Sync-C ranges from 1.718 to 1.967 and Sync-D from 11.324 to 11.581. Same-language and cross-language targets yield comparable average Sync-C/D scores (1.787/11.451 vs. 1.843/11.452), suggesting limited sensitivity to the evaluated cross-language target configurations.

The original fixed target was used for experimental consistency rather than being a requirement of the method. These results support limited sensitivity to the evaluated target variations, although they do not establish complete target or language independence. We will incorporate this analysis and clarify its evaluation scope in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Follow-up on Our Rebuttal

 

Dear Reviewer vQ3R,

Thank you again for your valuable feedback and constructive suggestions.

We have carefully addressed your concerns in our rebuttal, supported by additional experiments and detailed clarifications.

As the discussion period is approaching its end, we would sincerely appreciate it if you could review our responses and share any further feedback, particularly regarding whether your original concerns have been adequately addressed.

We would be happy to provide any additional clarifications if needed.

Thank you very much for your time and consideration.

 

Best regards,
The Authors

Reviewer_fFEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VoxGuard, an audio-only defense against unauthorized personalization of talking-face models. It modifies the reference video’s audio while leaving its frames unchanged. The perturbation redirects audio features toward an unrelated utterance, using multiple audio encoders to improve transfer. Psychoacoustic masking limits audible distortion, while sequential optimization preserves progress across encoders. Experiments evaluate disruption of personalized talking-face generation, transfer to unseen encoders, robustness to audio compression and resampling, and perceptual quality.

Critical strengths

The audio-only intervention is a useful alternative to visual protection. Targeting audio-to-motion learning allows the method to preserve the released video’s facial appearance exactly. This offers a meaningful alternative when modifying facial pixels is undesirable.

The evaluation covers several forms of transfer. The experiments include NeR and Gaussian-based personalization, a held-out DeepSpeech encoder, additional HDTF identities, and MuseTalk adaptation. These results support an effect beyond a single encoder or generation pipeline.

Critical weaknesses

It is unclear whether the method still works after someone tries to remove the added audio noise. The authors test audio compression and changes in sampling rate, but not tools designed to clean noisy speech. Please test whether these tools weaken the protection. The method also leaves the images unchanged, so its protection is limited to generation methods that learn from the reference video’s audio.

The improvement in audio quality is not clearly demonstrated. Participants rated the modified audio noticeably worse than the original audio. The authors should also ask listeners to compare results with and without psychoacoustic masking to show whether this component makes the added noise less noticeable.

The comparison with existing methods is also difficult to interpret because they are tested on different talking-face generators. Please compare the methods on the same generator to show how much they disrupt generation and how much they reduce the original video’s image or audio quality.

Questions for authors

How effective is VoxGuard after deliberate speech denoising or enhancement? Please evaluate whether protection survives audio cleaning that preserves intelligibility and usable speech quality.

Is clean driving audio used at inference and for SyncNet evaluation in every experiment? Please specify the personalization duration, training settings, and repeated run variability so the reported changes can be attributed to learning from protected reference audio.

How much does psychoacoustic masking improve human-rated audio quality at comparable defense strength? Please include the no-masking variant and uncertainty estimates in this comparison.

Can visual and audio defenses be evaluated on the same personalization pipeline, comparing perceptual quality at similar levels of generation disruption? If this is not possible, please qualify the cross-method superiority claims.

After personalization on protected videos, does synchronization recover when the driving audio receives the same perturbation procedure? This would help distinguish a damaged mapping from a mismatch between training and inference inputs.

Please compare it with joint optimization using the same loss normalization, gradient balancing, and number of encoder evaluations. Also test different encoder orders. Otherwise, it is unclear whether the improvement comes from preserving earlier objectives, additional computation, or the chosen order.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fFEB: Robustness, Perceptual Quality, and Optimization Analysis

Q1: Robustness against Audio Purification and Threat Model Scope

R1: We thank the reviewer for this suggestion. Our additional experiments demonstrate that VoxGuard retains substantial defense effectiveness after conventional speech enhancement using DeepFilterNet3, although the protection is partially weakened.

Training AudioCDPAM ↓Sync-C ↓Sync-D ↑
Clean0.00006.24907.8406
VoxGuard0.28581.921411.1923
Clean + DeepFilterNet30.07706.42137.6722
VoxGuard + DeepFilterNet30.16722.889210.5636

DeepFilterNet3 reduces the perceptual audio distance (CDPAM: 0.2858 → 0.1672), partially weakening the protection. Nevertheless, VoxGuard still substantially disrupts audio-visual synchronization, achieving Sync-C/D of 2.8892/10.5636 compared with 6.4213/7.6722 for the enhanced clean baseline. These results demonstrate that VoxGuard retains considerable protection against this representative speech enhancement method, although robustness against stronger purification remains an open challenge.

Threat Model Scope. As defined in Sec. 3.3, VoxGuard specifically targets unauthorized personalization methods that learn audio-to-facial-motion correspondences from reference videos, rather than all TFG paradigms.

 

Q2: Evaluation Protocol and Attribution of Defense Effectiveness

R2: All main experiments use clean driving audio during inference and clean reference audio for SyncNet evaluation. The perturbed-driving-audio setting in Q5 is an additional diagnostic experiment specifically designed to investigate potential train–inference mismatch, rather than part of the main evaluation protocol. Importantly, all SyncNet evaluations, including those in Q5, use clean reference audio. Our controlled experiments and repeated runs confirm that the synchronization degradation observed under clean inference is attributable to personalization with protected reference audio, rather than inference-time perturbations, evaluation-time perturbations, or random training fluctuations. Specifically, for each paired comparison, clean and protected models are personalized using identical visual frames, personalization duration, and training configurations, with the reference audio being the only intended difference.

Training settings. Following the video-split protocol used in SyncTalk and GeneFace, we use the first 10/11 of each video for personalization and reserve the remaining 1/11 for testing. The test portion is strictly held out and never seen during training. All training settings follow the official InsTaG implementation, with identical configurations for the clean and protected conditions.

Reproducibility Across Random Seeds. To further verify the reproducibility of our results, we conducted three independent runs with different random seeds. The mean and standard deviation were calculated across the three seed-level averages.

ConditionSync-C ↓Sync-D ↑
Clean6.2155 ± 0.09757.8371 ± 0.0949
VoxGuard1.9169 ± 0.032611.2281 ± 0.0729

The results consistently demonstrate substantial synchronization degradation across all three random seeds, with relatively small variations, and also support the attribution of the observed degradation to protected reference audio rather than inference-time perturbations or random training fluctuations.

We will clarify the personalization duration, training configurations, and evaluation protocol in the revised manuscript.

 

Q3: Human Perceptual Validation of Psychoacoustic Masking

R3: Following the reviewer’s suggestion, we conducted an additional human listening study, which, together with Table 3(a), shows that psychoacoustic masking improves perceived audio quality while maintaining similar defense effectiveness.

We conducted a blinded, randomized, pairwise A/B listening study comparing protected audio generated with and without masking under otherwise identical settings. Nine participants each evaluated 10 randomly selected audio pairs, with balanced A/B presentation positions and concealed condition labels. All 90 judgments favored the masked variant (100% preference). At the participant level, all nine listeners preferred the masked variant in all ten comparisons (9/9 participants), yielding a 95% Clopper–Pearson exact confidence interval of [66.4%, 100%] for the participant-level unanimous preference rate.

We acknowledge that protected audio remains perceptibly degraded relative to clean audio: masking mitigates, but does not eliminate, perturbation-induced distortion. We will include the listening protocol and results in the revised manuscript

AuthorsOpenReview ↗

Response to Reviewer fFEB: Robustness, Perceptual Quality, and Optimization Analysis(Q4 & Q5)

Q4: Fairness of Cross-Method Comparisons

R4: We thank the reviewer for this valuable suggestion. We selected Silencer-I because both methods aim to disrupt lip–audio synchronization. However, Silencer-I targets generalized TFG, whereas VoxGuard targets personalized TFG. To avoid potentially unfair cross-paradigm adaptation, we evaluate Silencer-I on its original Hallo pipeline and additionally report its clean baseline.

For personalized TFG, we further introduce VideoFacePoison, a visual-domain defense targeting personalized learning, and compare it with VoxGuard using the same InsTaG generator. The evaluations follow the data and settings of our original paper. Our additional comparisons demonstrate that audio-domain protection can effectively disrupt lip–audio synchronization while preserving facial appearance, achieving even stronger synchronization disruption than the evaluated visual-domain baseline under the same personalized TFG pipeline.

TFG ParadigmGeneratorMethodHuman-rated Overall Acceptability ↑Sync-C ↓Sync-D ↑
GeneralizedHallo—4.616.45567.9322
GeneralizedHalloSilencer-I3.303.315011.1310
PersonalizedInsTaG—4.616.24907.8406
PersonalizedInsTaGVideoFacePoison2.12364.82189.1139
PersonalizedInsTaGVoxGuard (Ours)3.631.921411.1923

“—” denotes clean, undefended reference videos. Human-rated overall acceptability is assessed on the original five-point scale.

Under the same InsTaG pipeline, VoxGuard achieves stronger synchronization disruption than VideoFacePoison (Sync-C/D: 1.9214/11.1923 vs. 4.822/9.114), alongside higher human-rated overall acceptability (3.63 vs. 2.12). Compared with Silencer-I, VoxGuard also achieves numerically higher overall acceptability (3.63 vs. 3.30), with higher visual-quality ratings (4.74 vs. 3.05), despite lower audio quality (2.98 vs. 4.62). These findings highlight the potential perceptual benefit of shifting protection from facial pixels to reference audio.

These results highlight VoxGuard's favorable performance in both synchronization disruption and overall acceptability under the evaluated settings. We will clarify the scope of these comparisons, including the absence of matched defense strengths, in the revised manuscript.

 

Q5: Synchronization Recovery under Perturbed Driving Audio

R5: We thank the reviewer for this insightful question. Our additional experiments demonstrate that applying the same perturbation procedure at inference does not restore synchronization, suggesting that the defense effect is not merely caused by a train–inference mismatch. The results are summarized below.

Training AudioDriving AudioSyncNet Reference AudioSync-C ↑Sync-D ↓
CleanCleanClean6.2497.841
ProtectedCleanClean1.92111.192
CleanPerturbedClean1.06213.532
ProtectedPerturbedClean1.01213.253

Personalization with protected audio substantially degrades synchronization even under clean inference (Sync-C: 6.249 → 1.921; Sync-D: 7.841 → 11.192). Applying the same perturbation procedure at inference does not recover synchronization (Sync-C/D: 1.012/13.253). Moreover, perturbed driving audio also degrades synchronization in clean-trained models (1.062/13.532), indicating an additional inference-time effect.

All evaluations use clean reference audio, avoiding direct perturbations to the evaluator's audio input. These findings are consistent with disrupted audio-to-motion correspondence learning during personalization.

AuthorsOpenReview ↗

Response to Reviewer fFEB: Robustness, Perceptual Quality, and Optimization Analysis (Q6)

Q6: Attribution of Continual Optimization Gains and Encoder-Order Sensitivity

R6: We thank the reviewer for this insightful suggestion.

Continual Optimization Gains. Our additional experiments demonstrate that the proposed continual optimization outperforms joint optimization under a matched total encoder-evaluation budget.

MethodLoss Norm. / Grad. BalancingRetentionEvaluations (A/E/H; Total)Sync-C ↓Sync-D ↑
VoxGuardYes / YesYes480/320/160 (960)1.921411.1923
Joint-320Yes / YesNo320/320/320 (960)2.574110.7102
Joint-480Yes / YesNo480/480/480 (1440)2.613810.6434
Equal-Weight JointNo / NoNo480/480/480 (1440)3.047110.2507

Under the matched total budget, VoxGuard outperforms Joint-320. Notably, Joint-480 matches VoxGuard's 480 AVE evaluations and uses 50% more encoder evaluations overall, yet still achieves weaker defense performance. These results support the effectiveness of our continual optimization strategy beyond differences in computational budget or AVE-specific optimization effort.

Encoder-order sensitivity. The default A→E→H order was motivated by the talking-face generation performance of the audio encoders under undefended settings. In particular, AVE achieved the highest generation quality and was therefore prioritized first in defense optimization.

To further investigate the impact of encoder ordering, we additionally evaluated two alternative orders, H→E→A and E→A→H, under the same total optimization budget, with the encoder-specific optimization budgets reassigned according to their positions in the sequence (480/320/160 evaluations for the first/second/third encoder).

OrderAVEEsperantoHuBERTMean Across Encoders
A→E→H1.9214 / 11.19233.3531 / 10.45142.9176 / 10.77592.7307 / 10.8065
H→E→A2.9689 / 10.37702.9005 / 10.71472.4104 / 11.13592.7599 / 10.7425
E→A→H2.6796 / 10.48432.9858 / 10.70483.3447 / 10.42013.0034 / 10.5364

Each pair reports Sync-C / Sync-D. The default order achieves the best average performance, while alternative orders also maintain substantial synchronization disruption. These results suggest that the effectiveness of continual optimization does not critically depend on a particular encoder order.

We will incorporate these ablations and clarify the contributions of continual optimization and the rationale for encoder-order selection in the revised manuscript.

Reviewer_rUPAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VoxGuard, a perturbation-based defense framework against identity misuse of talking face generation (TFG). The motivation is that existing methods predominantly focus on the visual space, which often noticeably degrades the portrait appearance of the protected video. The paper proposes an alternative using psychoacoustic masking to perturb the audio track while constraining the distortion. A few optimization techniques are also introduced to enhance black-box transferability and mitigate interference between different encoder objectives. Experiments show that VoxGuard disrupts unauthorized 3D TFG with visual content unchanged.

Critical strengths
  • The motivation is well-presented. It is interesting to see an optimization framework generating transferable, audio-only perturbations.

  • The paper has a nice structure and is easy to follow.

  • The experimental studies are comprehensive, testing multiple dimensions. The results support the defense effectiveness, black-box transferability, transformation robustness, and human perceptual evaluations.

Critical weaknesses
  • The perturbation is injected only in the audio space, so in terms of perceptual evaluation, the paper should focus more on the audio side. The results of Table 1(b) are not surprising, since nothing has changed in the visual space. According to the human perceptual evaluation results presented in Table 3(c), the audio degradation in the VoxGuard-protected reference video seems evident (4.54 -> 2.98). The paper could be strengthened by providing more quantitative evaluations of audio perceptual quality and by discussing how to reduce audio distortion.

  • The proposed defense framework is fairly complex, consisting of psychoacoustic masking, heterogeneous encoder optimization, continual optimization, and defense-preserving refinement. The paper should present ablations demonstrating the role of each component.

Questions for authors

Is there a way to reduce the audio quality degradation while preserving the defense performance?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rUPA: Audio Quality and Ablation Studies

Q1: Concerns About Audio Degradation

R1: We thank the reviewer for this concern. While we acknowledge the noticeable audio degradation, our motivation for shifting protection to the audio domain is its potential advantage in overall audiovisual acceptability. As shown in Table 3(c), VoxGuard achieves a numerically higher overall acceptability score (3.63) than Silencer (3.30), despite receiving a lower audio quality score. These findings suggest that preserving facial appearance may offer a favorable cross-modal perceptual trade-off, while further reducing audio distortion remains important. We will add relevant content to the discussion and limitations section.

 

Q2: How to Reduce Audio Distortion

R2: We thank the reviewer for this valuable suggestion. There are indeed methods to further improve the perceptual quality of audio, such as increasing the weight of the psychoacoustic masking loss or avoiding the introduction of perturbations into the silent segments of the original audio.

ConfigurationCDPAM ↓Sync-C ↓Sync-D ↑
VoxGuard (Masking Weight = 5)0.28581.921411.1923
Increased Masking Weight (5 → 10)0.26272.097011.0553
VoxGuard + Silence Preservation0.13652.144811.0640

Increasing the psychoacoustic masking weight reduces CDPAM by 8.1%. More notably, preserving originally silent intervals reduces CDPAM by 52.2%, with relatively modest changes in Sync-C and Sync-D. These results demonstrate that audio perceptual distortion can be substantially reduced while largely retaining the original defense effectiveness.

 

Q3: Component Contributions and Ablation Studies

R3: We thank the reviewer for this suggestion. Our original Table 3(a) already evaluates psychoacoustic masking and retention constraints, while Table 3(b) examines heterogeneous encoder optimization and continual optimization. We further conducted additional experiments to evaluate Stage-II refinement and compare continual optimization with joint optimization.

ConfigurationSync-C ↓Sync-D ↑CDPAM ↓
Stage-I only1.919611.25560.3270
VoxGuard (Stage-I + II)1.921411.19230.2858
Joint Optimization (w/o Continual Optimization)2.613810.64340.2347
Equal-weight Joint3.047110.25070.1224

Stage-II refinement reduces CDPAM by 12.6% while maintaining nearly unchanged Sync-C/D, demonstrating its contribution to perceptual quality.

Continual optimization achieves stronger defense effectiveness than both joint optimization baselines, although the latter achieve lower CDPAM, highlighting the trade-off between defense strength and audio quality.

Together with the existing ablations in Table 3(a)(b), these results further clarify the contributions of the proposed components. We will incorporate these additional experiments into the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

Summary of Manuscript Revisions

 

We sincerely thank the reviewers for their constructive feedback. We have uploaded a revised manuscript incorporating the additional experiments and clarifications discussed during the rebuttal. The main revisions are summarized below.

  1. Threat model and evaluation protocol (Sec. 3.3 and Sec. 5): We clarify the scope and limitations of audio-only protection and explicitly distinguish protected reference audio used during personalization from clean driving audio used during inference and evaluation.

  2. Defense effectiveness, fairness, and generalization (Sec. 5.1–5.2 and Appendices D–G): We incorporate same-generator comparisons with visual-domain defenses, fair-budget comparisons with joint optimization and random-seed evaluations results.

  3. Audio robustness and perceptual trade-offs (Sec. 5.1, Sec. 5.3, and Appendices H–M): We add evaluations of speech enhancement, adaptive audio replacement, subjective listening preferences, downstream speech recognition, and audio quality–defense trade-offs.

  4. Presentation and technical clarity: We refine the Abstract, Introduction, and Conclusion to better distinguish the benefits and limitations of audio-only protection, and improve the consistency of optimization descriptions and experimental details in the appendices.

We hope these revisions provide a clearer and more comprehensive basis for the final assessment. We appreciate the reviewers' and Area Chair's time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.