acceptodds
Under review as a conference paper at ICLR 2027

Samvad-DF: Unmasking Partial Audio Deepfakes in Multispeaker Conversational Settings

Abstract

Audio deepfake detection is entering a new regime. Rather than synthesizing entire utterances, emerging attacks manipulate authentic recordings by seamlessly replacing a sentence or even a single word. While visual deepfake research has progressively expanded from isolated face swaps to complex multi-person scenarios, audio detection has largely remained focused on single-speaker English speech and binary utterance-level classification, with limited emphasis on localizing manipulated content. To address this gap, we introduce SAMVAD-DF, the first large-scale benchmark for partial audio deepfakes in multispeaker Hindi conversations. Built from 107 hours of in-the-wild debates and podcasts, the dataset captures substantial acoustic diversity, containing recordings with up to 76 speakers. SAMVAD-DF defines three granularities of manipulation, ranging from speaker-level manipulations to highly challenging sub-second word-level replacements. Constructing this benchmark also reveals a fundamental limitation of conventional training. Existing pipelines assign file-level labels to every short audio window, causing clean regions far from the splice to be treated as manipulated during supervision. We instead introduce a validated, strictly bounded temporal annotation protocol that labels only the true manipulated span, enabling faithful localization learning. Experiments demonstrate a clear trade-off between detection and localization. Training with precise temporal labels improves boundary-overlap localization by approximately one-third, while increasing conventional file-level error by around 10 percentage points. These findings highlight that strong utterance-level performance does not necessarily imply accurate localization. SAMVAD-DF provides both evaluation protocols and diagnostic tools to facilitate rigorous research on the next generation of audio deepfake detection.

Primary area:
datasets and benchmarks
Keywords:
partial audio deepfake, word-splice, multispeaker audio, omnivoice

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_s9gQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SAMVAD-DF, a benchmark for detecting and localizing partial audio deepfakes in multispeaker Hindi conversations. It contains three manipulation granularities, standard and refined temporal annotations, and experiments with several detection and localization models. The paper argues that precise temporal labels improve localization while making file-level detection more difficult.

Critical strengths
  1. The paper addresses an important gap by studying partial manipulation in multilingual, multispeaker conversational audio.
  2. The distinction between file-level detection and temporal localization is useful, and the comparison of original and refined labels gives a helpful diagnostic perspective.
Critical weaknesses
  1. The benchmark relies mainly on one TTS system and one text-generation pipeline, so transfer to other generators is not yet clear.
  2. Fake text is restricted to pure Devanagari, while real recordings contain Hinglish. The 93.98% transcript-only AUC suggests a strong label-correlated cue.
  3. File-level and localization experiments use different windows, labels, and loaders, so the effect of refined labels is not fully isolated.
  4. The human-assisted annotation procedure and reproducibility details are brief, especially for the relatively small Verified Track.
  5. The writing should be revised for clarity, and the figures should be redrawn with clearer, higher-resolution technical graphics. Figure 1 is currently more decorative than explanatory.
Questions for authors
  1. Could the authors provide a generator-held-out result and a control in which fake transcripts preserve the script and code-switching distribution of bona fide speech?
  2. Could they clarify annotation agreement and report the main comparison under one consistent evaluation protocol?
Reviewer_WkQGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SAMVAD-DF, a benchmark for detecting and localizing partial audio deepfakes in multispeaker Hindi conversations. The dataset is constructed from approximately 107 hours of real-world debates, panel discussions, and podcasts, and includes three manipulation settings: full-speaker replacement, partial-segment substitution, and intra-segment word splicing. The authors employ voice cloning and controlled audio splicing to generate manipulated samples while preserving the surrounding authentic conversational context. Beyond dataset construction, the paper investigates the impact of temporal labeling conventions on deepfake detection and localization. The authors propose refined temporal annotations that label only windows overlapping the manipulated regions, rather than assigning spoof labels to all windows from a manipulated recording. They evaluate multiple existing detection and localization architectures using file-level EER/AUC and temporal localization metrics, including Seg-AUC and mIoU. The experiments show that refined labeling improves segment-level discrimination and slightly improves boundary overlap for AASIST, while substantially degrading its file-level detection performance. Additional analyses examine codec-related shortcuts, acoustic and lexical confounds, performance across manipulation granularities, and cross-dataset generalization.

Critical strengths

None

Critical weaknesses

Main argument

  1. There is a very clear numerical error in this work's core conclusion: The paper reports in Sections 4.1, 4.4, and Table 4 that Global mIoU improved from 0.2233 to 0.2337 after refining the labels. Based on that, the authors explicitly claim that mIoU achieved a relative improvement of approximately 38% and describe it in the abstract as an improvement of about one-third of the localization. However, the actual calculation is: 0.2337−0.22330.2233×100=4.66\frac{0.2337-0.2233}{0.2233}\times100 = 4.66%, mIoU only increased by 0.0104 in absolute value and 4.66% in relative value.

  2. And mIoU is not the only numerical error. The abstract claims that refined temporal labels increase conventional file-level error by approximately 10 percentage points. However, the AASIST data in Table 3 shows Standard and Verified EER from around 25% to more than 45%, which is 20 percentage points, not 10. More importantly, the detection performance of refined AASIST is close to that of random guessing. 50% is random guessing for EER. There is no explanation for this extreme performance degradation.

  3. The result cannot convince me that the performance change is actually due to more accurate time labeling or stamping. The original labels treat all training windows in the spoof file as fake; the corrected labels only treat windows that overlap with the real edited regions as fake. This means that the category of a large number of training windows will change from fake to bona fide. This is not an experiment that can completely isolate the impact of annotation accuracy. In particular, the EER of AASIST decreased from 26.43% to 45.70%, which may reflect insufficient training of the classifier under extreme class imbalance conditions, rather than the inherent detection-localization trade-off claimed by the authors.

  4. I suppose the most important experiment, the word-splice experiment, was not completed. Does the improvement in localization brought about by refined labels occur in the easiest full-speaker replacement, or does it actually improve the most difficult word-splice localization? The word-splice results in Table 5 still come from the original-label AASIST. The existing Global Seg-AUC in Table 4 combines different types of attacks and also cannot answer this question.

Questions for authors
  1. Refined labels substantially change the positive-to-negative training window ratio. How do the authors distinguish the effect of improved temporal supervision from class imbalance or changes in effective training sample counts?

  2. Could the authors provide per-attack-type EER, Seg-AUC, and mIoU for the refined-label AASIST model? This is particularly important for verifying whether the localization improvements extend to the most challenging word-splice attacks.

  3. Given the 93.98% transcript-only AUC and 12.44 dB ambient-noise dropout, what evidence rules out lexical and acoustic shortcuts as major contributors to the reported detection and localization performance?

  4. How are prediction thresholds selected and continuous window scores converted into manipulation intervals? In particular, how do the authors explain BAM's high Seg-AUC (0.9029) but near-zero mIoU (0.0047)?

Reviewer_u3o9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Samvad-DF, a partial audio deepfake benchmark built using 107.4 hours of Hindi, English Youtube audio such as podcasts and debates. The fakes are generated via Hindi paraphrasing (through Qwen 2.5-7B) which are synthesised through Omni-Voice zero-shot cloning. These are spliced into 5 minute chunks with RMS matching. Three manipulation modes are employed, Full speaker replacement, 1-3 utterance substitution, and intra-segment word splice. Pyannote/NeMo check outlines the evaluation subset. The paper evaluates eight baseline detectors using 4-second sliding windows, max-pooled to a file-level decision. The authors also argue that assigning the file-level label to every window inflates detection performance while hurting localization. To show this, they retrain AASIST with labels restricted to the manipulated span. Seg-AUC improves from 0.80 to 0.90, but file-level EER worsens from 26% to 46%. Additional analyses cover a codec-signature shortcut found during development, ambient-noise dropout at splice points, a lexical confound (Devanagari-only fakes vs. code-switched real audio), and cross-dataset transfer to LENS-DF.

Critical strengths
  1. New data setting: No existing partial-deepfake corpus covers multispeaker, in-the-wild, code-switched Hindi audio. Prior partial-spoof datasets are mostly clean read speech in English or Mandarin. This benchmark is more representative of conditions more likely to be faced in the real-world, especially for Hindi audio.
Critical weaknesses
  1. The main localization claim is misreported: The abstract and Section 4.4 say refined labels improve boundary localization by “approximately one-third,” but Table 4 shows mIoU going from 0.2233 to 0.2337, a gain of under 5%. The refined-label AASIST is also reported at 0.94 Seg-AUC alongside 45.7% file-level EER under max-pooling, which is hard to reconcile unless Seg-AUC ignores false alarms in bona fide files. The paper does not explain this. The refined labels are described as human-verified in Section 4.1 but as a purely automatic overlap rule in the Reproducibility Statement. Since the paper’s methodological contribution rests on this comparison, these inconsistencies weaken it considerably.
  2. Evaluation and Results sections: Several key results are missing, including the per-attack breakdown and word-splice analysis for the corrected-label model, which the authors keep deferring until later. Some reported numbers also suggest evaluation errors. SLS-XLSR has a Seg-AUC of 0.22, well below chance. BAM has the highest Seg-AUC (0.90) but near-zero mIoU and worse-than-chance file-level EER. As submitted, the baseline results are not usable as reference numbers for future work.
Questions for authors

Clarifications

  1. Table 4 shows mIoU going from 0.2233 to 0.2337. How was the “approximately one-third” improvement computed?
  2. How does refined-label AASIST reach 0.94 Seg-AUC while its file-level EER is 45.7%? Are bona fide files included in the Seg-AUC computation?
  3. Are bona fide files repeated across manipulation modes? If so, how many unique bona fide files are in the test set?

Suggestion

  1. A listening test on the spliced samples would establish how convincing the fakes are.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.