acceptodds
Withdrawn from ICLR 2027

XLSR-EMA: A 3D Joint Forgery Manifold for Universal Speech Deepfake Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pre-trained self-supervised learning (SSL) models have achieved remarkable success in speech deepfake detection. However, existing methods typically rely on the final hidden layer or simple layer aggregation, limiting their ability to exploit the hierarchical information encoded across SSL representations. This limitation becomes more pronounced for speech generated by modern Audio Language Models (ALMs), whose artifacts span multiple acoustic and semantic levels. We propose XLSR-EMA, a universal speech deepfake detection framework that organizes the hidden representations of a pre-trained XLS-R model into a unified Time-Layer-Feature Joint Forgery Manifold. An Efficient Multi-scale Attention (EMA) module models interactions among temporal, feature, and layer dimensions, while a Dual-Purpose Perceptual Augmentation (DPPA) strategy improves channel robustness and sensitivity to subtle synthetic artifacts. Experiments on multiple benchmarks demonstrate the effectiveness and cross-domain generalization of the proposed framework. Anonymous code is https://anonymous.4open.science/r/XLSR-EMA-for-UADD-2B19available online.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio deepfake detection, Speech anti-spoofing, Self-supervised speech representations, Audio language models, Cross-domain generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9wXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes XLSR-EMA, a speech deepfake detection framework designed to detect speech generated by modern Audio Language Models (ALMs) and codec-based generative models, whose forgery artifacts are distributed across multiple acoustic and semantic levels. The framework consists of three stages. Stage A applies a Dual-Purpose Perceptual Augmentation (DPPA) strategy that adds continuous-scale noise to the raw waveform, aiming to improve robustness to channel degradation while enhancing sensitivity to subtle synthetic artifacts. Stage B constructs a joint forgery manifold by preserving the hidden representations of all layers of a frozen pre-trained XLS-R backbone, and stacking them into a unified Time-Layer-Feature representation, so that both low-level acoustic cues and high-level semantic cues are retained. Stage C applies an Efficient Multi-scale Attention (EMA) classifier that models interactions across temporal, layer, and feature dimensions through a tri-branch structure and outputs the real/fake decision. The method is evaluated on the Codecfake, ASVspoof 2021 DF, and In-The-Wild benchmarks, reporting 4.64% EER on Codecfake and 1.24% EER on ASVspoof 2021 DF, and the lowest EER in cross-dataset transfer settings. The experiments suggest the proposed framework is effective for detecting codec-based and ALM-generated speech.

Critical strengths
  1. The paper targets detection of codec/ALM-generated speech, which is a known weakness of existing detectors, and reports competitive results across multiple benchmarks, including robustness on lossy-codec conditions in ASVspoof 2021 DF.
  2. In the cross-dataset evaluation (Table 3), where all methods are trained on the same single source, XLSR-EMA reports the lowest EER in all four transfer settings, supporting the claim that preserving the full SSL representation helps transfer to unseen ALM/codec forgeries.
Critical weaknesses
  1. EMA is adopted from Ouyang et al. (2023) with limited modification, while DPPA is SNR-scheduled additive noise overlapping with existing augmentations. The main novelty—using all SSL layers—is insufficiently analyzed. Removing DPPA only slightly increases EER: 4.64→4.86 on Codecfake and 1.24→1.32 on 21DF.
  2. “Ours + Add. Data” uses additional Codecfake-mini data, while baseline training data are unspecified, potentially confounding the Codecfake gains. Codecfake-mini comes from the training split, but generator or codec overlap with the evaluation split is not discussed.
  3. Results lack run-to-run variance or statistical tests; min t-DCF is designated a primary metric but omitted; and the computational overhead of using 24 layers is unreported. The choice of g=2 also appears test-driven, with no held-out validation reported.
Questions for authors
  1. What training data did each baseline in Table 2 use? If the baselines were trained only on ASVspoof 2019 LA while XLSR-EMA additionally used Codecfake-mini, can you provide a comparison where all methods (including XLSR-EMA) are trained on exactly the same data?
  2. Codecfake-mini is drawn from the Codecfake training split and evaluation is on the Codecfake eval split. Do these splits share generators, speakers, or codec configurations? If so, how can the Codecfake result be considered independent of in-domain overlap?
  3. What concrete modifications did you make to the EMA module (Ouyang et al., 2023) beyond treating layers as channels and using a 4D input? Please point to the specific architectural differences.
  4. Is there quantitative evidence that high-SNR DPPA noise amplifies codec quantization artifacts specifically, rather than acting as generic regularization? The w/o-DPPA ablation shows a small gain (0.22 and 0.08 EER) — how do you interpret this?
Reviewer_1DeLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates speech deepfake detection, focusing on the insufficient utilization of hierarchical representations from self-supervised learning (SSL) models. The paper proposes XLSR-EMA, which consists of three main components:

  1. Time-Layer-Feature Joint Forgery Manifold: The hidden representations from all 24 layers of a pretrained XLS-R model are stacked along the layer dimension to construct a three-dimensional representation spanning time, layer, and feature dimensions, preserving acoustic and semantic information at different levels.
  2. Efficient Multi-scale Attention (EMA): Grouped attention is applied to the three-dimensional representation, extracting multi-scale forgery features through temporal pooling, feature-dimensional pooling, and local 2D convolution.
  3. Dual-Purpose Perceptual Augmentation (DPPA): Gaussian noise at different SNR levels is used for data augmentation, aiming to improve the model's robustness to channel noise and sensitivity to subtle synthetic artifacts.

The experiments use ASVspoof 2019 LA and Codecfake-mini as training data and evaluate the proposed method on Codecfake, ASVspoof 2021 DF, and In-The-Wild. In Table 2, XLSR-EMA achieves EERs of 4.64%, 1.24%, and 5.25% on these three datasets, respectively, outperforming the compared methods reported in the paper. Cross-dataset evaluations and ablation studies further demonstrate the potential value of hierarchical representations and the EMA module.

However, the reasoning and empirical evidence supporting several contributions remain incomplete. The paper lacks necessary comparative experiments, contains inconsistencies or ambiguities in some reported results, and exhibits limited novelty. Figure 2 is not explicitly referenced in the main text, although Section 3.3 describes the corresponding EMA architecture. This is a minor presentation issue that affects the connection between the architectural description and its illustration.

Critical strengths
  1. This paper focuses on the complementary information encoded across different SSL layers and aims to jointly exploit acoustic and semantic representations, providing a well-motivated research direction.
  2. This paper preserves the hidden representations from all XLS-R layers and combines directional pooling with local convolution to model information across different dimensions. Compared with classification methods that rely solely on the final hidden layer, this approach can leverage richer intermediate representations.
  3. This paper reports better performance than the compared methods on Codecfake, ASVspoof 2021 DF, and In-The-Wild, and evaluates the model's transferability across different speech synthesis methods through cross-dataset experiments. In Table 3, XLSR-EMA achieves the lowest EER across all four cross-dataset settings.
  4. This paper investigates the effects of different EMA group configurations, feature extraction branches, and DPPA. Table 5 shows that removing individual EMA branches leads to performance degradation, providing preliminary evidence for the effectiveness of the multi-branch architecture.
Critical weaknesses

Several claims made in the paper are not sufficiently supported by the reported experimental evidence, particularly in the following aspects:

  1. The novelty and necessity of the core design are insufficiently demonstrated. The Time-Layer-Feature Joint Forgery Manifold is primarily constructed by stacking hidden representations from multiple XLS-R layers, while EMA is an existing attention module. Although Table 5 compares a final-layer classifier with the full model, the paper lacks comparisons with simple multi-layer fusion, weighted fusion, and alternative cross-layer attention mechanisms. Furthermore, the paper does not quantitatively or visually demonstrate which useful layer-wise information is preserved compared with compressed representations, despite claiming that prior aggregation methods may discard such information.
  2. The motivation and effectiveness of DPPA are insufficiently supported. The paper claims that adding Gaussian noise at high SNR can amplify quantization artifacts, but provides no direct evidence for this claim. In Table 5, removing DPPA only increases the EER from 4.64% to 4.86% on Codecfake and from 1.24% to 1.32% on ASVspoof 2021 DF. Furthermore, the paper lacks separate ablations for the high- and low-SNR augmentation strategies, as well as comparisons with conventional data augmentation methods.
  3. The interpretations of Figure 3 lack sufficient experimental details and supporting evidence. Figure 3(a) does not specify the dataset, sample selection, temporal alignment, or spectral residual computation procedure, making it difficult to determine whether the observed time-frequency differences reflect codec-related artifacts, transient phase distortion, or amplification caused by DPPA. Figure 3(b) does not clarify which feature representations are used for t-SNE, how they are extracted and processed, or which samples are visualized. The substantial overlap between real and fake speech representations also makes the claimed clear separation difficult to assess.
  4. The generalization claims in Section 4.4 are insufficiently supported by the reported results. The authors state that the cross-dataset results indicate robustness to unseen synthesis algorithms, codec configurations, speaker distributions, and recording environments. However, Table 3 only reports aggregate EERs under different cross-dataset settings, without providing separate analyses for these distribution shifts. Consequently, the reported results do not sufficiently establish robustness to each of these factors independently.

The paper also has the following limitations regarding comparative experiments, result consistency, training protocols, and generalization performance:

  1. The fairness and completeness of the comparative experiments are insufficient. In Table 2, the proposed method is labeled as "Ours + Add. Data", but it is unclear whether the other baselines use the same training data. Furthermore, the paper does not discuss previously reported stronger results from works such as IB-CAAN, SATYAM, and VGGish-CBAM, or the differences in their evaluation protocols.
  2. The experimental results contain inconsistencies. Tables 4 and 5 report different EERs for the same EMA group configurations on ASVspoof 2021 DF. The reported EERs are 1.35% and 1.88% for g=1, and 1.55% and 1.24% for g=2, respectively. The paper does not explain these discrepancies.
  3. The training and checkpoint selection protocols are insufficiently documented. Section 4.1 states that the validation set is excluded, while Section 4.2 reports early stopping with patience=3. The released training implementation appears to perform early stopping based on training loss rather than an independent validation metric. Although checkpoints are saved after each epoch, the procedure used to select and produce the final best_model.pth checkpoint is not specified, affecting the reproducibility of the reported results.
  4. The claim of universal speech deepfake detection is insufficiently supported. Although the paper reports strong overall performance, Table 4 shows that the g=2 configuration still achieves an EER of 17.15% on the unseen C7 codec, substantially higher than the average EER of 3.57% on Codecfake reported in the same table.
Questions for authors
  1. Could the authors conduct comparative experiments with simple multi-layer concatenation, weighted layer fusion, and alternative attention mechanisms under the same XLS-R backbone and training data to clarify the respective contributions of the Time-Layer-Feature Joint Forgery Manifold and EMA?
  2. Why do Tables 4 and 5 report different EERs for identical EMA grouping configurations on ASVspoof 2021 DF?
  3. Why would adding Gaussian noise at high SNR amplify quantization artifacts rather than simply serve as conventional data augmentation?
Reviewer_TQMzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes XLSR-EMA for speech deepfake detection by combining multi-layer XLS-R representations, an EMA attention module, and perceptual data augmentation. The reported results are strong, particularly on Codecfake and cross-dataset evaluations. However, I have concerns about the methodological novelty and experimental fairness.

Critical strengths

The paper addresses an important problem in detecting speech generated by modern audio models.

The cross-dataset experiments are valuable, and the reported results are promising.

Critical weaknesses
  • The proposed framework mainly combines stacked XLS-R representations with an existing EMA module and noise augmentation. The 3D manifold is essentially a rearrangement of hidden features. It is unclear what new modeling capability this provides beyond standard multi-layer feature fusion. Stronger comparisons with simple stacking, CNN-based fusion, and existing attention mechanisms are needed.

  • The proposed method uses additional Codecfake-mini training data, while Table 2 does not clearly indicate whether the baselines use the same data and training settings. The large performance gap on Codecfake (4.64% versus 25.65% EER) may partly result from this difference. A comparison under identical training conditions is necessary.

Questions for authors
  • The authors claim that high-SNR Gaussian noise enhances subtle forgery artifacts, but this explanation is not sufficiently supported. In Table 5, removing DPPA only increases the Codecfake EER from 4.64% to 4.86%. Comparisons with standard Gaussian noise augmentation and RawBoost, along with separate evaluations of different noise regimes, would strengthen this claim.

  • The paper states that the validation set is excluded but also reports early stopping with a patience of 3. The model selection procedure should therefore be clarified. In addition, results are reported using only one random seed. Multiple runs with mean and standard deviation would improve reliability.

  • The experimental results are promising, but the current evidence does not sufficiently establish that the improvements mainly come from the proposed architecture. The effects of additional training data, the individual contributions of the proposed components, and the reproducibility of the experiments require further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.