acceptodds
Under review as a conference paper at ICLR 2027

One Detector Can Catch Them All: Unified On-Device AI Audio Detection

Abstract

Recent advancements in generative AI technologies have lowered the bar to generate human-like audio ranging from voice, music, to sound effects. Today, AI audio can be used maliciously at low cost, from deepfake calls (voice), to streaming fraud (music), and even to fake emergency/disaster scenarios (sound effects). In response, we need the ability to detect AI-generated audio content. However, adoption of AI audio detectors has been sparse largely due to three limitations: 1) existing detectors are specialized for certain types of AI audio (e.g., voice-only), which do not scale when end-users must employ one detector per use case; 2) detectors are vulnerable to audio transformations, like resampling and compression, that are commonly applied before audio reaches the end-user; and 3) more robust detectors require multiple GB of VRAM that device manufacturers are reluctant to give up for AI audio detection, forcing audio data to be sent to third party cloud services. In this paper, we argue that the "ideal" design for AI audio detection should address all three limitations: a _single_, robust, on-device model, that is capable of simultaneously detecting all three kinds of AI audio (voice, music, and sound effect). Our work presents the first step toward realizing a _unified_ AI audio detector. In order to accomplish this, we conduct a large-scale measurement study by first collecting and aggregating 15 AI audio benchmarks across voice, music, and sound effect, including AI samples generated by over 230 unique AI audio generators. Our initial results show that unified AI audio detection is possible (AUROC 0.91), however achieving low false positive rates sacrifices detection performance (30% TPR at 0.5% FPR). To improve existing detectors, we propose a lightweight, model-agnostic, and data-free training harness called Rubato that leverages previously unused signals (codec and sampling rate) in audio metadata as additional inputs. Experiments show that Rubato improves detection rate in low false positive regimes by over 13%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI generated audio, AI music, deepfake detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7fbXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates unified on-device AI-generated audio detection across voice, music, and sound effects. The authors construct a benchmark with over 3 million samples from more than 230 generators and evaluate existing detectors under various audio transformations and low false-positive-rate (FPR) settings. Based on the observed performance degradation, they propose RUBATO, a lightweight training framework that preserves high-frequency audio information and incorporates sampling-rate and codec embeddings. Experiments show improvements in overall detection performance at low FPR, particularly for lightweight models such as LCNN.

Critical strengths
  1. Practical motivation and emphasis on low-FPR detection. The paper highlights the importance of evaluating AI audio detectors under strict false-positive constraints, which is relevant to real-world deployment. The substantial gap between overall AUROC and TPR at 0.5% FPR provides a useful perspective on current detector limitations.

  2. Large-scale benchmark and comprehensive robustness evaluation. The benchmark covers diverse audio types and a large number of generators. Evaluating compression, resampling, audio editing, and multi-step transcoding provides useful insights into detector robustness under practical audio processing conditions.

  3. Lightweight design with measurable performance improvements. RUBATO is relatively simple and can be integrated into several existing detection architectures. The improvements in low-FPR detection, together with the reported model size and inference efficiency, demonstrate its potential for resource-constrained applications.

Critical weaknesses
  1. Insufficient related-work coverage and unclear novelty of unified audio detection.

The paper positions unified AI audio detection as a largely unexplored problem. However, relevant prior efforts are overlooked. In particular, AT-ADD (ACM MM 2026) explicitly investigates all-type audio deepfake detection across speech, environmental sounds, singing voices, and music. ESDD2 (ICME 2026) also explores joint forensic analysis of speech and environmental sounds, although under a different component-level detection setting.

These omissions weaken the paper's novelty claims and problem positioning. The authors should more clearly distinguish their contributions from existing multi-type audio forensic research, particularly regarding the unified detection setting and its associated challenges.

  1. The research objectives are insufficiently unified, and the proposed method does not consistently address the identified challenges.

The paper simultaneously emphasizes unified detection, robustness to audio transformations, low-FPR performance, and on-device deployment. Although these objectives are individually meaningful, their relationships are not sufficiently established, making the central research problem somewhat unclear.

More importantly, RUBATO's empirical improvements are not consistent across audio types. For example, LCNN improves from 26% to 39% overall TPR at 0.5% FPR, but its voice detection TPR decreases from 10% to 8%. Similar degradation or lack of improvement is observed for voice detection across all evaluated architectures. Thus, the overall gains are primarily driven by music and sound effects, while the most challenging voice detection setting remains unresolved.

Furthermore, although low-FPR detection is presented as a central motivation, RUBATO does not explicitly optimize the low-FPR operating region. The paper would benefit from a clearer explanation of which specific limitation RUBATO is intended to address and how its design directly targets that limitation.

  1. Insufficient isolation of performance gains and limited benchmark validation.

The contributions of high-frequency preservation, metadata conditioning, and increased model capacity are not sufficiently disentangled. In particular, RUBATO increases LCNN's parameter count, while the component ablations are not reported separately for voice, music, and SFX. It therefore remains unclear whether the improvements reflect benefits specific to unified detection or more general advantages of retaining high-frequency information.

Furthermore, the benchmark aggregates bona fide and synthetic audio from different source collections, potentially introducing dataset-specific correlations. Although randomized codec and resampling augmentation may mitigate simple metadata shortcuts, the paper does not sufficiently validate their absence. Additional controlled experiments with matched audio conditions, model capacity, and unseen codecs would strengthen the conclusions about RUBATO's effectiveness and generalizability.

Questions for authors

Q1. How does the proposed benchmark and framework differ from existing efforts such as AT-ADD and ESDD2? Could the authors clarify what new challenges their unified detection setting introduces beyond these prior studies?

Q2. Could the authors report separate ablation results for high-frequency preservation, sampling-rate embeddings, and codec embeddings across voice, music, and SFX? Why does RUBATO fail to improve voice detection despite its overall gains?

Q3. Could the authors provide audio-type-specific ablations that isolate high-frequency preservation, metadata conditioning, and model capacity? Additionally, how do they ensure that the observed metadata benefits are not influenced by dataset-source correlations, particularly when bona fide and synthetic audio originate from different collections?

Q4. Can the authors evaluate performance using a single threshold calibrated on the development set and fixed across audio types and transformation conditions? Would the claimed low-FPR improvements remain consistent under such a deployment-oriented evaluation?

Reviewer_U8K3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RUBATO, a system for detecting AI-generated speech, music, and sound effects using sampling-rate and codec metadata while preserving high-frequency audio information. Evaluated on approximately 3 million samples from over 230 generators, RUBATO improves low-FPR detection for several architectures. For LCNN, TPR at 0.5% FPR increases from 0.264891 to 0.388558. However, it remains unclear whether these gains generalize beyond dataset-specific encoding and provenance patterns.

Critical strengths
  1. The benchmark covers three audio modalities, over 230 generators, and multiple audio transformations, making the empirical investigation practically relevant.
  2. RUBATO improves LCNN's TPR at 0.5% FPR from 0.264891 to 0.388558, with reported on-device deployment measurements.
Critical weaknesses
  1. Codec and sampling-rate metadata may be correlated with class labels or dataset sources. The experiments do not show whether RUBATO’s gains hold when these patterns are controlled or differ between training and testing.
  2. Voice-detection performance declines for some models. For example, TPR drops from 0.10 to 0.08 for LCNN and from 0.09 to 0.04 for AASIST, raising questions about whether RUBATO consistently improves detection across all audio types.
Questions for authors
  1. How were the data splits created across generators, datasets, and processing pipelines, and what steps were taken to prevent leakage through codec and sampling-rate patterns?
  2. Do RUBATO’s gains hold when genuine and synthetic audio are re-encoded using the same codecs and sampling rates, or when those patterns differ between training and testing?
  3. How consistent are the improvements across audio types, random seeds, and test samples, especially given the declines in voice-detection performance?
Reviewer_LGeHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies unified on-device AI audio detection across speech, music, and sound effects. The authors construct a large-scale benchmark of over three million audio samples from more than 230 generators, and evaluate existing detectors under various audio transformations including multiple transcoding. Experiments reveal substantial detection performance degradation under audio transformations and at low false positive rates. To address these limitations, the paper proposes RUBATO, a lightweight training framework that preserves high-frequency information and incorporates learnable embeddings of sampling rate and codec metadata into existing detectors. Experiments across four lightweight architectures show improved overall detection performance, with LCNN achieving a 13-percentage-point increase in TPR at 0.5% FPR. The work provides useful insights into unified audio detection and practical deployment.

Critical strengths
  1. Comprehensive benchmark and robustness evaluation. The paper constructs a large-scale benchmark covering speech, music, and sound effects. Its evaluation of various audio transformations, especially multiple transcoding, provides useful insights into detector robustness under realistic conditions.
  2. Lightweight approach with encouraging results. RUBATO improves overall detection performance with limited modifications to existing architectures. Experiments across multiple models and audio categories demonstrate its effectiveness, while the deployment results show its potential for efficient on-device detection.
Critical weaknesses
  1. Insufficient analysis of speech detection performance. Table 4 shows that RUBATO does not improve speech detection across the four evaluated models, despite improving overall performance. The paper lacks an explanation for this result, particularly why the proposed components benefit music and sound effects but not speech.
  2. Potential reliance on dataset-specific shortcuts. RUBATO uses sampling rate and codec metadata as additional inputs. Since real and AI-generated music are collected from different sources, these features may capture dataset-specific differences rather than generalizable detection cues. Although data augmentation is applied, the current experiments do not fully rule out this possibility.
Questions for authors
  1. Why does RUBATO fail to improve speech detection across all four models? Could the authors analyze the contribution of each component to speech detection performance?
  2. Does RUBATO still improve detection when real and AI-generated audio have matched codec and sampling-rate distributions? Could the authors provide a metadata-only baseline to investigate potential shortcut learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.