acceptodds
Under review as a conference paper at ICLR 2027

ASNet: Energy-Conditioned Reliability for Robust Audio Deepfake Detection

Abstract

Audio deepfake detectors that are accurate in-domain can fail under unseen genera- tors, codec/channel shifts, and adversarial perturbations while remaining confident, making confidence-based rejection unreliable. Our key insight is that a detector’s own confidence can serve two reliability roles: allocating robustness pressure across training samples and deciding which test inputs to reject. If this confidence drifts under codec or channel changes, however, it can misdirect training. We therefore introduce ASNet, which stabilizes a bounded, class-symmetric, detector- derived energy score before it controls training. Spectral and prosodic views provide complementary evidence; energy anchoring and OOD consistency stabilize the score’s scale and limit its drift; Energy-Conditioned Robustness Modulation then sets per-sample robustness weights, and the same score is reused unchanged for validation-fixed abstention. Under single-source training and adaptation-free evaluation on four external corpora, ASNet raises external Robust AUC from 0.59 to 0.65 over a compute-matched baseline and yields lower source-domain ECE before post-hoc recalibration. The reused score rejects 85% of PGD-10 errors but only 48% under a matched-budget energy-aware attack, exposing the shared score as an attack surface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio deepfake detection, adversarial robustness, selective prediction, calibration, domain generalization, energy-based uncertainty, audio forensics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ckv3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASNet, an audio deepfake detection framework designed to improve reliability under distribution shifts and adversarial perturbations. The central idea is to derive a class-symmetric energy score from the detector, stabilize this score before using it as a training signal, and then reuse the same score for inference-time abstention. Experiments under single-source training show improvements in cross-corpus robustness and calibration over compute-matched baselines, while adaptive attacks also reveal limitations of the shared energy-based reliability mechanism.

Critical strengths
  • Deepfake detectors can remain highly confident despite substantial degradation under unseen generators, codec/channel shifts, or adversarial perturbations. The attempt to jointly consider generalization, adversarial robustness, calibration, and abstention is important.
  • The experimental evaluation is relatively thorough.
  • I appreciate that the authors explicitly evaluate an attacker targeting the proposed energy-based rejection mechanism. The substantial degradation under this attack is an important negative result and gives a more realistic picture of the method’s security limitations.
Critical weaknesses
  • The novelty is somewhat incremental and distributed across several existing ideas. Energy/confidence scoring, adversarial training, uncertainty-based abstention, instance-dependent robustness weighting, multi-view representations, and consistency regularization are all established techniques.

  • The shared energy score creates a single point of failure. The same quantity controls robustness allocation and deployment-time rejection. Once an adaptive attacker explicitly targets this score, error catch drops dramatically (to 48%, and 35% under the stronger joint attack). This raises an important question about whether coupling training and inference through the same reliability signal is ultimately preferable to using separate or independently learned signals.

  • External robustness remains fairly limited despite the improvement over baselines. In particular, CodecFake remains difficult, with a Robust AUC of only 0.56, and a nontrivial fraction of errors remain below the fixed abstention threshold. Thus, the relative improvements are encouraging, but the practical reliability of the resulting detector under substantial distribution shift remains unclear.

  • The baselines considered in the paper are not the state-of-the-art detector for audio deepfake detection thus the comparison is not that fair.

  • The paper only evaluates the robustness against adversarial perturbation. For audio deepfake detection, however, the robustness against real-world corruption[1] is equally important.

[1] Li, Xiang, Pin-Yu Chen, and Wenqi Wei. "Measuring the Robustness of Audio Deepfake Detection under Real-World Corruption." Proceedings of the Sixteenth ACM Conference on Data and Application Security and Privacy. 2026.

Questions for authors
  1. Can the author compare with more advanced audio deepfake detectors using foundation models?
  2. Can the authors provide more evidences to support the proposed method's robustness against real-world perturbations?
  3. Given the relatively low absolute robustness on CodecFake, could the authors provide more analysis of which types of codec/generator shifts cause ASNet to fail and whether those failures correlate with instability of the energy score? This would help establish whether the proposed stabilization mechanism addresses the intended failure mode rather than primarily improving average performance.
5 comments

We thank the reviewer for highlighting the adaptive-attack result. The revised manuscript evaluates foundation-model baselines, non-adversarial corruptions, CodecFake failure conditions, and alternative rejection rules. All corresponding revisions are highlighted in violet, and overlapping methodological controls are cross-referenced where relevant.

W1 — Incremental novelty. Individual ingredients are established, and binary energy has the same uncertainty ordering as entropy and direction-corrected margin on a fixed detector (Eqs. 4, 29). The claim is the tested stabilization–allocation dependency, not a new score: consistency bounds gate drift (Eqs. 7, 21); conditioning adds +0.07 source Robust AUC after stabilization versus +0.04 without it (descriptive interaction +0.03; Table 28). Uniform / fixed-positive / learned gating gives 0.830 / 0.872 / 0.900 (Table 29), and matched nominal exposure yields stabilization-first 0.880 versus allocation-first 0.840 (Table 31). Controller-only transfer on the Ren backbone adds +0.045 external Robust AUC (Table 40). These controls identify a meaningful complete-system and allocation-rule effect, without claiming unique optimality or independently novel primitives (Secs. 1, 5.3; Apps. E.2–E.3/E.9).

W2 — Shared-score vulnerability. Matched attacks reduce induced-error catch 61% to 48% (energy-aware PGD) and 56% to 35% (joint APGD), with UASR reaching 17.7%/31.2% (Table 15). Under rule-aware APGD and fixed ECRM training, auxiliary-only and Mahalanobis rejection yield UASR 33.8%/37.4%, versus 31.2% for shared energy; energy+auxiliary reduces it to 26.1% (Table 16). ECRM improves both shared (36.8% to 31.2%) and combined (31.9% to 26.1%) rejection relative to Uniform. The 5.1/4.9-point selector gains do not establish an interaction. In separate reruns, substituting an auxiliary or Mahalanobis score for both allocation and abstention yields source Robust AUC 0.88/0.86, versus 0.90 for energy (Table 39). These variants still share their selected signal across both roles. The tested separate rejection signals increase rule-aware attack success, whereas combining energy with the auxiliary signal reduces it.Clean source coverage is 79.2–80.5%; on CodecFake, coverage decreases from 70.8% with shared-energy rejection to 68.6% with energy–auxiliary rejection. The finite-budget evidence does not establish secure abstention, and an independently trained rejection model is not evaluated (Secs. 5.2, 6; Apps. D.2, E.8).

W3/Q3 — CodecFake failure modes and reliability. CodecFake is hard for every detector evaluated: ASNet reaches 0.560 PGD-10 Robust AUC against 0.496 (compute-matched Ren) and 0.512–0.534 (standalone SSL) (Tables 18–19). Comparable degradation across the tested detectors indicates a challenging transfer setting beyond ASNet alone. Condition-level analysis localizes it: benchmark-held-out DAC/C7 and AudioGen/A3, a non-speech shift, give Robust AUC 0.389/0.441 and fake-miss rates 46.3%/41.7% (Table 23). Failures track energy instability: across ten conditions, drift and fake-miss rank-correlate at ρ=+0.939 (+0.24 per utterance). On identical pairs, removing scheduled consistency raises prediction flips 8.3%→11.5% and correct-to-wrong transitions 6.2%→8.8% (Table 24); stratified by within-model drift, transitions rise 2.1%→12.9% (ASNet) and 3.8%→15.9% without it, ASNet lower at every rank and the difference larger in the upper quintiles (Table 25). Quintiles are defined per model, so rank-wise differences are descriptive. Reliability stays bounded: 34% of clean CodecFake errors fall below the source-fixed threshold (Table 14). These are associations, not causal mediation.

W4/Q1 — Are stronger pretrained detectors evaluated fairly? Wav2Vec2-BERT, WavLM-Large+AASIST, and XLS-R+AASIST are fine-tuned on LibriSeVoc and evaluated with the same external splits and waveform PGD-10 protocol (Tables 18–19; App. D.4). They exceed original ASNet on external clean AUC (0.901/0.913/0.924 versus 0.881). Original ASNet reaches external Robust AUC 0.650 versus 0.604/0.618/0.631, the XLS-R margin being within reported dispersion, and source ECE 0.101 versus 0.172/0.159/0.151. WavLM+ASNet reaches 0.686 external Robust AUC but changes the system. SSL pretraining and resource budgets differ; SSL rows have one downstream fine-tuning run, so these are evaluation-matched system comparisons, not compute-matched tests or significance claims.

W5/Q2 — Non-adversarial corruption robustness. We evaluate 18 corruption types—noise (3), modification (8), compression (7)—on LibriSeVoc and WaveFake under a matched protocol (Sec. 5.1; Table 26; App. D.10). Against compute-matched Ren, ASNet has smaller clean-relative AUC drops in every family: equal-family means 0.043 vs 0.062 (LibriSeVoc) and 0.073 vs 0.103 (WaveFake). Noise, absent from the specified consistency augmentation, also favors ASNet (0.030 vs 0.043; 0.057 vs 0.076). Weighting by operator counts preserves the advantage (0.047 vs 0.067; 0.077 vs 0.110, derived from rounded family means). We use the three corruption families of Li et al., who also study 18 types. Although severities and corpora differ, our family-level results are consistent with their broader observation that noise is generally less damaging than modification and compression; this is not a reproduction of their evaluation. Per-severity and physical-playback robustness are not established.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to strengthen the paper, particularly through the analysis of Codecfake failure modes and comparisons with stronger pretrained detectors.

However, my main concerns remain. First, the novelty is still largely incremental. The additional ablations demonstrate the effectiveness of combining and ordering established components, but do not establish a sufficiently distinct methodological contribution. Second, while the alternative rejection experiments are informative, they do not directly evaluate decoupling training-time robustness allocation from inference-time rejection. The shared-score vulnerability under adaptive attacks therefore remains a concern, and the benefit of coupling these mechanisms is not sufficiently established.

I therefore keep my original score.

Reviewer_R5HaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ASNet for robust audio deepfake detection. ASNet stabilizes a bounded, class-symmetric, detector-derived energy score before it controls training. It combines spectral and prosodic views, and uses energy anchoring and OOD consistency stabilize. the score’s scale and limit its drift. Experiments on four datasets show improved detection accuracy and robustness.

Critical strengths
  1. The paper addresses an important problem of robustness and reliability in audio deepfake detection.
  2. The evaluation include several external datasets, and the comparisons and extensive ablation studies verify the effectiveness.
  3. The authors also evaluate adaptive attacks against the rejection mechanism.
Critical weaknesses
  1. The technical novelty seems limited. Most components are based on existing techniques.
  2. The effectiveness may depend on the specific architecture and training.
  3. The performance on CodecFake is relatively weak.
Questions for authors
  1. Can the proposed energy-conditioned training be applied to more architectures? This should be discussed.
  2. It seems that the energy score is equivalent to a confidence-based score. Then why energy score is utilized?
  3. How can the method improve its resistance to energy-aware adaptive attacks?
2 comments

We thank the reviewer for highlighting methodological novelty, architecture dependence, CodecFake robustness, and adaptive-attack resistance. The corresponding analyses are reported in the revised manuscript, with additions marked in blue.

W2 / Q1 — Architecture and training dependence

The allocation mechanism is evaluated independently on the Ren et al. backbone. Under identical stabilization objectives, robustness losses, optimizer, and training protocol, changing only Uniform weighting to Learned ECRM improves external Robust AUC from 0.582 to 0.627 (+0.045) (Table 40, Appendix E.9). A capacity-matched spectral-only ASNet configuration also operates with ECRM, while fixed-hyperparameter double- and triple-domain evaluations assess training-mixture sensitivity. The controlled Ren experiment demonstrates transfer of the allocation mechanism beyond the original dual-view architecture; broader architecture independence is not assumed.

Q2 — Why use energy?

For the binary detector, energy equals negative log maximum-class confidence. Energy, entropy, and direction-corrected margin therefore induce identical uncertainty rankings on a fixed model (Eqs. 4, 29). Energy is used because its bounded score scale is directly regularized by the consistency objective, providing explicit raw-gate and normalized-weight drift bounds (Eqs. 7, 21). Thus, the justification concerns its role as a stabilized training controller, not an intrinsically superior confidence ranking.

W3 — CodecFake robustness

CodecFake remains the most difficult external corpus. ASNet obtains Robust AUC 0.560, compared with 0.520 for Uniform allocation and 0.490 without the robustness block (Table 32). Condition-level analysis identifies benchmark-held-out DAC/C7 and AudioGen/A3 as difficult failure cases, with Robust AUC 0.389 and 0.441, respectively (Table 23). Paired evaluations further associate energy drift with prediction failures (Tables 24–25). These results identify the principal limitations while preserving the measured improvement over matched controls.

Q3 — Resistance to energy-aware adaptive attacks

At matched nominal attack budgets, energy-aware PGD reduces error catch from 61% to 48%, and stronger joint APGD reduces it from 56% to 35% (Table 15). An energy–auxiliary rejection rule reduces rule-aware APGD unaccepted-error attack success from 31.2% to 26.1% at near-matched source coverage (Table 16). Energy-conditioned training also improves resistance over Uniform allocation under both tested rejection rules. The combined selector mitigates, but does not eliminate, adaptive accepted-error attacks.

W1 — Technical Novelty

We thank the reviewer for highlighting the distinction between methodological novelty and established techniques. While confidence scoring, consistency regularization, and adversarial reweighting are individually known, ASNet's methodological contribution is stabilization-conditioned robustness allocation, which connects detector-score stability to the reliability of sample-wise robustness-training weights.

The novelty lies in the training-controller formulation. ASNet regularizes detector-derived reliability against codec/channel transformations before using its continuous values to allocate adversarial and consistency losses (Eq. 8). Thus, score stabilization directly constrains the signal governing robustness-training pressure, rather than serving only as an additional regularization objective.

Why energy (also addressing Q2). For binary detection, energy is a bounded, class-symmetric confidence score with the same uncertainty ranking as entropy and direction-corrected margin (Eq. 4). For the ECRM gate q(E)=σ(wgE+bg)q(E)=\sigma(w_gE+b_g), sigmoid Lipschitz continuity gives

\leq \frac{w_g^2}{16}\mathcal{L}_{\mathrm{ood}}.$$ This connects energy consistency directly to allocation-gate stability (Eq. 7). Moreover, energy's bounded range $E\in[0,\log 2]$ provides a positive lower bound on the gate, enabling the normalized-weight stability analysis (Eq. 21; App. B.6). We do not claim these properties are exclusive to energy or that the Lipschitz inequality itself is novel. **The methodological contribution is using this relationship to formulate and evaluate a stabilized robustness-training controller.** **Controlled evidence and architecture transfer (also addressing W2 and Q1).** Controller-only transfer to the Ren backbone, with stabilization and robustness objectives fixed and only the weighting rule changed, improves external Robust AUC **0.582→0.627** (Tab. 40). This demonstrates that the allocation benefit extends beyond ASNet's original architecture. Uniform/fixed-positive/learned weighting achieves source Robust AUC **0.830/0.872/0.900** (Tab. 29), distinguishing energy-dependent weighting from gate optimization. Matched-exposure scheduling yields **0.880 versus 0.840** (Tab. 31), while stabilization increases the observed allocation gain from **+0.04 to +0.07** (Tab. 28). The latter interaction is descriptive across three seeds. At matched compute, the full system further achieves external Robust AUC **0.650±0.020 versus 0.591±0.007** (Tab. 1). This provides complementary system-level evidence without attributing the entire improvement solely to allocation. **Together, these results support a distinct training-time stabilization–allocation mechanism**, investigated through analytical sensitivity control, weighting isolation, scheduling interventions, and backbone transfer. We claim neither a new uncertainty ranking, a mathematically required training order, nor an additional advantage from train–deploy score reuse. Further analytical details and limitations are provided in our response to Reviewer ckv3, **"W1. Methodological Novelty Beyond Established Components."** The corresponding manuscript evidence appears in Secs. 3 and 5.3 and Apps. B.6, D.8, E.2, E.3, and E.9.
Reviewer_79xzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASNet for robust audio deepfake detection under distribution shifts and adversarial perturbations. It combines spectral and prosodic representations, stabilizes a detector-derived energy score through energy anchoring and augmentation consistency, and uses the score to allocate robustness-training weights and perform inference-time abstention. Experiments on four external corpora show improvements over a compute-matched baseline, while adaptive attacks reveal limitations of the shared rejection mechanism.

Critical strengths
  1. Relevant problem and encouraging empirical results. The paper connects training-time robustness allocation with inference-time abstention. Under single-source training, ASNet improves external Robust AUC from 0.591 to 0.650 over a compute-matched baseline. Cross-corpus evaluations and component ablations provide useful evidence for the complete system.
  2. Reasonable disclosure of limitations. The authors acknowledge the ranking equivalence of energy, entropy, and margin for a fixed binary detector, distinguish schedule-level evidence from a pure ordering effect, and evaluate adaptive attacks that expose weaknesses in the rejection mechanism.
Critical weaknesses
  1. The central methodological contribution is insufficiently isolated. The stabilization–allocation factorial study shows a descriptive interaction, but the schedule comparisons change both activation order and cumulative objective exposure. Moreover, the Full–Uniform comparison does not establish the benefit of learning ECRM over a fixed positive energy-conditioned weighting rule. The respective contributions of gate learning, score parameterization, and consistency-loss gradient paths therefore remain unclear. The results support the complete configuration more strongly than the specific stabilize-before-control principle.
  2. The mechanism-level evidence is incomplete. Energy consistency constrains confidence magnitude but does not guarantee prediction consistency: because E(z) = E(−z), a prediction can flip without changing the energy score. The paper does not directly establish that reduced energy drift corresponds to improved prediction reliability under transformations. Similarly, penalizing cross-view cosine similarity does not by itself demonstrate complementary task information, and the grouped ablations do not isolate CMRA's contribution. These gaps weaken the explanation of why the proposed system works.
Questions for authors
  1. Can the authors isolate the benefit of stabilization order while controlling cumulative objective exposure?
  2. How does learned ECRM compare with a fixed positive energy-conditioned gate under otherwise identical training? Which gradient paths remain active in the alternative-signal experiments?
  3. On paired clean and transformed samples, does reduced energy drift correspond to fewer prediction flips or errors?
  4. Can CMRA be evaluated independently of learned fusion, or supported by a direct diagnostic of complementary detection evidence?
2 comments

We thank the reviewer for identifying the need to distinguish stabilization order, objective exposure, learned weighting, and task-relevant mechanism evidence. The revised manuscript reports targeted controls with their interpretation and limits; manuscript changes are marked in red.

W1 / Q1 — Stabilization order versus cumulative exposure

We agree that the original cumulative schedules change both order and exposure. We now report a distinct 80-epoch control: 16 base epochs, three consecutive 16-epoch single-security-block phases, and 16 joint epochs. Energy anchoring, stage-level OOD consistency, and ECRM robustness are each active for 32 epochs in both compared orderings. This also matches the scheduled duration of the gate-weighted consistency route inside ECRM. Stabilization-first reaches source Robust AUC/ECE 0.880/0.120, versus 0.840/0.160 for ECRM-first (Table 31, Appendix E.3). The default cumulative 0.900 is from a different schedule and is not used in this comparison (Table 30). The controlled contrast eliminates unequal nominal block exposure, but optimization trajectories and gradient magnitudes legitimately differ when activation order differs. The complementary stabilization–allocation factorial gives conditioning gains +0.07 with scheduled stabilization and +0.04 without it (descriptive interaction +0.03; Table 28, Appendix E.2); the latter retains consistency within the ECRM robustness block.

W1 / Q2 — Fixed-positive gating, score choice, and active gradients

Under the same architecture, losses, optimizer, seeds, schedule, and budget, Uniform, Fixed-positive (frozen gate parameters (1,0)), and Learned ECRM attain source Robust AUC 0.830/0.872/0.900 and ECE 0.190/0.150/0.101 (Table 29). Their external Robust AUC means are 0.610/0.625/0.650 (Table 33). The +0.042 Fixed-positive–Uniform contrast isolates positive energy weighting as a training intervention; the additional +0.028 Learned–Fixed-positive contrast tests parameter learning in separately trained models, not a terminal-map-only effect. The raw final sigmoid maps differ by at most approximately 0.020 over the score range.

Stop-gradient blocks score-to-weight derivatives but does not block detector gradients through the weighted adversarial-logit and OOD-consistency losses. The Learned gate receives gradients from both weighted robustness components, whereas stage-level OOD consistency does not directly differentiate gate parameters (Appendix B.6, Eqs. 13–18). These identities clarify active routes without claiming separately measured performance effects for each route. Binary energy, entropy, and direction-corrected margin give equivalent rankings on a fixed detector; energy is used as a bounded, directly regularized controller. Alternative-score experiments compare complete retrained controllers, not isolated score formulas (Table 39, Appendix E.8).

W2 / Q3 — Paired energy drift and prediction reliability

We explicitly acknowledge that E(z)=E(−z), so energy stability cannot guarantee a stable predicted class. On held-out clean/transformed LibriSeVoc pairs, Full ASNet versus removal of only stage-level OOD consistency yields energy-drift MSE 0.015 versus 0.045, prediction flips 4.2% versus 7.2%, correct-to-wrong transitions 4.1% versus 5.8%, and transformed AUC 0.990 versus 0.977 (Table 7, Appendix B.8). Both configurations retain the gate-weighted consistency term. Within Full ASNet, low-drift pairs have a 2.2% conditional flip rate versus approximately 4.3% for the higher-drift complement; corresponding rates without stage-level OOD are 4.3% and approximately 7.3%. The higher-drift rates are derived arithmetically from displayed rounded means, not independently estimated seed-wise results. Clean errors are 0.2%/4.5% and transformed errors 4.2%/8.9%, so transformation-added errors are 4.0/4.4 percentage points, not the entire transformed-error gap (Eq. 23). The evaluations establish within-model association and end-to-end correspondence, not causal mediation; the transformations come from training-augmentation families.

W2 / Q4 — Independent CMRA loss and task-based fusion evidence

The new no-CMRA control sets λ_cmra=0 while retaining both encoders, learned fusion, all other objectives, and the training protocol. Relative to Full ASNet, source Robust AUC declines 0.900→0.881, ECE rises 0.101→0.146, and clean AUC falls 0.892→0.862 on ASVspoof21 and 0.739→0.709 on CodecFake (Table 35, Appendix E.5.1). Matching-capacity frozen probes show fused-over-best-single-view ASVspoof21 AUC gains of +0.031 with CMRA and +0.020 without it; on source LibriSeVoc, the gains are +0.022/+0.034 (Table 36, Appendix E.5.2). The first experiment isolates CMRA's end-to-end training contribution, and the probes establish task-based fusion utility without claiming that cosine dissimilarity proves unique modality information or that CMRA improves the incremental fusion gain on every corpus.

Together, these controls support the narrower, testable contribution: stabilizing detector-derived energy before it allocates robustness training improves the evaluated end-to-end reliability profile. We do not claim a new uncertainty ranking, causal identification of individual gradient-path effects, guaranteed prediction invariance, or universal independence from architecture or distribution shift.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Consideration of Author Responses — Submission 48092

Dear Area Chair,

Thank you for overseeing the review process for our submission.

We have provided detailed responses to all three reviewers and incorporated corresponding manuscript revisions, including methodological clarifications, additional controlled experiments, and analyses addressing the reviewers' concerns.

While we appreciate the feedback and engagement received so far, we have not yet received follow-up comments from Reviewers 79xz and R5Ha regarding our responses and revisions.

We would be grateful if you could kindly encourage these reviewers to consider the additional evidence and clarifications, if feasible, before the final assessment.

We also respectfully request that the rebuttal discussions and manuscript revisions be taken into account when evaluating the submission, particularly where new evidence directly addresses the original concerns.

Thank you for your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.