acceptodds
Under review as a conference paper at ICLR 2027

From Decodability to Utility: Speech Deepfake Detection in Audio Language Models

Abstract

Audio language models have emerged as general-purpose speech interfaces, yet how their speech deepfake detection capability should be characterized remains unclear. Existing evaluations often rely on a single model endpoint, obscuring whether observed behavior reflects information supported by the underlying audio-language system or its effective use. We present Representation Accessibility Versus Exploitation by the Language Pathway (RAVEL), a controlled evaluation framework that separates native behavior, supervised decodability of frozen representations, and downstream exploitation through task adaptation, together with encoder-depth analysis. We evaluate thirteen Audio-LLMs across five datasets, with supervised comparisons on ten open-weight models and source-protocol and depth analyses. Native outputs are benchmark-dependent and often fail in decision validity or correctness, whereas frozen audio- and language-side representations frequently support strong supervised detection. This coexistence shows that weak native behavior need not imply the absence of accessible authenticity information. Native adaptation consistently improves in-domain detection over language-side frozen readout. Depth-wise adaptation further shows that intermediate representations can outperform final blocks in cross-domain discrimination, while encoder regions favored by frozen readouts can reverse under adaptation and shift across target domains. Under 19LA → AT-ADD transfer, Native-SFT outperforms both frozen readouts in EER for 9/10 models but in Macro-F1 for only 1/10, revealing metric-dependent transfer gains. Overall, supervised decodability and adaptation utility are distinct: stronger frozen-readout performance does not determine which representations are most effective for encoder-to-language adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speech deepfake detection, Audio-LLMs, Representation Probing.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ACfSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evaluation framework for deepfake detection through speech deepfake detection using Audio-LLMs. The authors perform this evaluation via zero-shot generation of the AudioLLM, learning linear probes across the representations of the audio encoder and fine-tuning the model using SFT across the different layers of the audio encoder. Experiments are performed using 13 audio LLMs across 5 datasets.

Critical strengths
  1. Scope and rigour of evaluation. The paper evaluates 13 models on 5 datasets, which makes the analysis broad and the conclusions generalizable.

  2. Unified and reproducible detection techniques. The paper judges the quality of detection via different methods such as zero-shot generation, linear probing and depth-wise SFT which can give a good overview of the representational evolution across different layers and components of an Audio LLM.

Critical weaknesses
  1. Paper has very less technical novelty and the insights from the work are covered in prior work. This paper copies the analysis done in prior work [1] for paralinguistic speech understanding to speech deepfake detection. Moreover, the findings of this work are similar to those of the prior works on similar Audio-LLM probing [1].

  2. Depth-wise findings for the encoder are very noisy. Figure 4 reports the encoder-depth findings under 19LA source supervision and similar results are expanded in Appendix F. The layer-wise results in Figure 4 seems very noisy and adjacent layers seem to be performing very different. This can be an artefact of number of seeds that the authors run their experiment for. It is recommended to perform the experiment over 5 seeds and then report the average across those to see whether the layer-wise trends hold.

  3. Depth-wise findings are limited to the audio-encoder. The paper shows the variation and contribution of using different layers of the audio encoder for downstream LLM processing, but it does not report how those representations transform within the LLM through probing (or any other similar analysis).

  4. Formatting issues, AI Use and Ethics

    • Overly long appendix without any index: A lot of results and additional information are present in the 29 page appendix which is difficult to navigate through without a content page for the appendix.
    • AI USE: The code provided in the supplementary material is almost entirely AI written but the authors do not disclose this in their AI Use statement.
    • Ethics Statement: The paper misses an Ethics Statement. Given that this work deals with speech deepfake detection, the authors should include an Ethics statement in the revision/camera ready to note the unethical uses of their approach.
    • Table 2 is never referenced in the text.

[1] J. Pang, A. Chaubey, and M. Soleymani, “Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox,” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=v7rYbRR9Zw

Questions for authors

In addition to answering to the above weaknesses, I request the authors to respond to the below minor questions.

  1. For Depth-SFT, was the adapter or the connector also finetuned for training each of the depth-wise encoder variants?

  2. Why was the Audio-Linear config done on 4 seconds of audio as opposed to LM-Linear/Native-SFT which was done on original audio duration (L240-242)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ACfS — Part 1: Beyond Frozen Probing

W1. Relationship to VoxParadox and the contribution of depth-wise adaptation

Thank you for highlighting this connection and recognizing the scope of our evaluation. We agree that applying established probing to another task alone would not establish a distinct contribution. RAVEL goes further by independently training and evaluating adaptation at every encoder depth, rather than inferring adaptation utility from frozen-probe performance.

VoxParadox [1] studies paralinguistic understanding under linguistic–acoustic contradiction. It includes encoder/LLM probing and uses Prompt-Conditioned Layer Mixer (PCLM), SFT, and DPO to improve acoustic-information utilization. We recognize these contributions. Unlike PCLM, which learns to combine selected encoder layers, our study directly compares frozen readouts with independently trained adaptation across the complete encoder-depth range.

What independent depth-wise adaptation adds. At each depth, we separately fit a Linear readout, train an AASIST readout on frozen encoder sequences, and train a Depth-SFT configuration through the model-specific language pathway. Encoder blocks remain frozen; language-side adapters and, where applicable, connection parameters are adapted. Across six models—five with 32 blocks and Phi-4 with 24—this gives 184 independently trained Depth-SFT configurations, alongside the corresponding frozen readouts. These are separate adaptation runs, not layer features extracted from one fine-tuned model or adaptation restricted to probe-selected layers. Full sweeps are reported in Sections 3.2/5.4 and Appendix F.

The following examples use 19LA supervision and the original single-seed depth results. All metrics are percentages; Macro-F1/accuracy use the fixed real–fake decision rule.

1. A weak frozen readout can underestimate a layer's adaptation utility. At Qwen L14 on ITW:

Qwen2.5-Omni-3B, L14EER ↓Macro-F1 ↑Accuracy ↑
Frozen Linear41.4727.237.22
Frozen AASIST42.1849.049.35
Depth-SFT12.2474.974.96

Thus, weak transfer with either frozen readout does not establish that L14 is unsuitable for adaptation (Tables 9–11). This is not merely an isolated gain: every Qwen Depth-SFT connection from L08 through L23 improves on L31 in all three metrics on both 21LA and ITW. The probe and adaptation pipelines differ in capacity and input processing; these results test probing as a predictor rather than isolate a causal benefit of the LLM alone.

2. A probe-preferred layer need not remain preferable after adaptation. The StepAudio comparison reports absolute EER / Macro-F1 / accuracy; bold marks the better layer within each procedure and target.

StepAudio procedureLayer21LAITW
Frozen AASISTL138.30 / 76.7 / 87.7514.39 / 83.0 / 83.45
Frozen AASISTL3126.20 / 55.5 / 66.7427.16 / 71.4 / 72.07
Depth-SFTL1310.58 / 87.1 / 95.9926.84 / 64.3 / 64.35
Depth-SFTL3119.95 / 80.5 / 93.0419.81 / 73.0 / 73.03

On ITW, AASIST favors L13, whereas Depth-SFT favors L31 in all three metrics; on 21LA, both favor L13 (Tables 13–14). AASIST at L13 remains the stronger ITW detector in this example—the finding is a change in preferred depth, not universal superiority of SFT. Our five-run follow-up preserves these ITW EER preference directions for both StepAudio and MOSS, as detailed in our response to W2.

Together, these results show why frozen decodability is not sufficient to determine adaptation utility. Qwen demonstrates that a layer with weak frozen readouts can still support effective adaptation; StepAudio shows that stronger frozen performance at one depth does not determine the better adapted connection, and that this relationship changes across targets. The contribution is not simply a larger probing benchmark or the generic observation that fine-tuning can help, but the direct evaluation of these relationships through complete, independently trained depth sweeps using both discrimination and fixed-rule decisions.

We have foregrounded this distinction from VoxParadox in Section 2, clarified the per-depth adaptation design in Section 3.2, and incorporated these comparisons into Section 5.4, with full profiles provided in Appendix F. We hope these revisions clarify the additional question addressed by our study while acknowledging the important foundations provided by prior work.

[1] J. Pang, A. Chaubey, and M. Soleymani. Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox. ICML, 2026.

AuthorsOpenReview ↗

Response to Reviewer ACfS — Part 2: Five-Run Validation

W2. Reliability of the encoder-depth findings

Thank you for pointing out that the single-seed depth profiles may be sensitive to training variability. We agree that the local fluctuations should not be interpreted as stable layer-by-layer differences without repeated runs. Following your suggestion, we retained the original seed 42 and added four runs (123, 2026, 2027, and 2028) for StepAudio L13/L31 and MOSS L05/L31, using both Frozen AASIST and Depth-SFT. This gives five runs for each of eight fixed configurations: 40 training instances, including 32 additional runs, evaluated on 19LA Eval, 21LA, and ITW. The layer pairs were fixed after the initial exploratory sweeps; these are targeted repetitions, not full-depth five-seed sweeps.

Absolute results. Each cell below reports EER / Macro-F1 / accuracy as mean ± sample SD across the five runs, in percent. Lower EER and higher F1/accuracy are better. Macro-F1 and accuracy retain the original fixed decision rule, P(real) ≥ 0.5. The evaluation uses the complete 21LA set (181,566 utterances) and ITW set (31,779 utterances).

ModelProcedureLayer21LA: EER / F1 / AccITW: EER / F1 / Acc
StepAudioFrozen AASISTL138.88 ± 0.58 / 74.30 ± 4.97 / 85.50 ± 4.4616.53 ± 2.07 / 76.40 ± 7.94 / 76.67 ± 8.05
StepAudioFrozen AASISTL3128.91 ± 1.94 / 55.68 ± 3.46 / 68.70 ± 6.5233.73 ± 4.82 / 53.83 ± 14.55 / 56.02 ± 12.68
StepAudioDepth-SFTL1310.77 ± 0.64 / 85.04 ± 7.10 / 95.60 ± 1.4426.68 ± 1.70 / 63.40 ± 10.63 / 64.17 ± 9.15
StepAudioDepth-SFTL3119.90 ± 0.52 / 81.15 ± 1.15 / 93.75 ± 0.8320.56 ± 0.65 / 68.66 ± 6.58 / 68.88 ± 6.22
MOSSFrozen AASISTL058.93 ± 0.99 / 86.88 ± 8.19 / 94.07 ± 5.7722.40 ± 4.39 / 62.92 ± 6.90 / 65.66 ± 8.34
MOSSFrozen AASISTL3119.02 ± 1.95 / 72.64 ± 3.15 / 87.16 ± 2.7627.72 ± 5.07 / 68.58 ± 9.43 / 69.26 ± 9.13
MOSSDepth-SFTL0510.47 ± 0.30 / 91.00 ± 0.37 / 97.08 ± 0.0926.40 ± 1.40 / 48.07 ± 3.21 / 50.89 ± 2.38
MOSSDepth-SFTL3120.65 ± 1.36 / 79.85 ± 0.74 / 94.25 ± 0.2321.31 ± 0.92 / 65.35 ± 3.78 / 65.60 ± 3.54

The key EER comparisons repeat despite variation in absolute performance. On ITW, Frozen AASIST favors the earlier anchor and Depth-SFT favors L31 in all five runs for each model. On 21LA, both procedures favor the earlier anchor in all five runs for each model. For example, StepAudio AASIST at L31 has 27.16% ITW EER in the original run, compared with 31.59%–39.83% in the four additional runs. Its absolute performance is therefore more variable than the original result alone suggests, but L13 has lower EER in every corresponding run. StepAudio Depth-SFT instead favors L31 in every run. Thus, the repeated evidence supports these specific procedure- and target-dependent preferences, not a claim that absolute performance is insensitive to training randomness.

Fixed-rule classification is not uniformly as stable as EER, and we report it separately. On ITW, StepAudio AASIST favors L13 in F1 and accuracy in 5/5 runs; StepAudio Depth-SFT favors L31 in F1 in 5/5 and accuracy in 4/5. The remaining accuracy comparison differs by only 0.01 percentage points (68.21% versus 68.22%). MOSS Depth-SFT favors L31 in all three metrics in 5/5 runs. However, MOSS AASIST does not reproduce its EER preference in fixed-rule classification: although L05 has lower EER in 5/5 runs, L31 has higher F1 in 4/5 and accuracy in 3/5. Its mean F1/accuracy are 62.92%/65.66% at L05 versus 68.58%/69.26% at L31. On 21LA, the earlier anchors win F1 and accuracy in either 4/5 or 5/5 runs, depending on the configuration. We therefore do not generalize the EER preference reversal to every metric.

The revised Section 5.4 summarizes the repeated comparisons, while Appendix F.8 provides the complete per-seed results across all three evaluation sets and their five-run mean and sample SD. The original full-depth curves will remain identified as single-run exploratory profiles; their individual local peaks and troughs have not been validated by these repetitions. In particular, the Qwen example in our W1 response remains an original single-run result. The supported conclusion is that the specified cross-domain EER comparisons repeat across five runs, while fixed-rule decision performance requires its own variability assessment. Thank you for encouraging this check; it lets us distinguish the repeatable observations from the fluctuations more clearly.

AuthorsOpenReview ↗

Response to Reviewer ACfS — Part 3: Language-Side Evaluation and Internal Probing

W3. Probing representations within the LLM

Thank you for this suggestion. Probing internal LLM blocks would provide a useful complementary view of where authenticity information remains decodable. Our study evaluates the language pathway, but the depth-wise axis is the audio-encoder connection point, not the internal LLM block. We should make this distinction clearer.

The existing evaluation goes beyond encoder-side readouts. LM-Linear extracts the final post-normalization representation supplied to the language-model output head at the position predicting the first answer token. The frozen model processes the audio–instruction context without a reference answer, and a supervised linear classifier is fitted to this language-side vector (Appendices C.1–C.2). Native-SFT instead adapts language-side parameters while keeping the released audio pathway, connection, and output head frozen, and evaluates the resulting real/fake answer-token scores (Appendix C.3). These settings examine information accessible at the language-side endpoint and detection performance after language-side adaptation, respectively.

For example, Table 1 already reports the following MOSS results under 19LA-source supervision. Each cell shows EER ↓ / Macro-F1 ↑ / accuracy ↑, all in percent; classification uses the fixed real–fake comparison.

MOSS-Audio-4B procedure19LA EvalITW
LM-Linear3.58 / 91.37 / 96.4229.04 / 32.52 / 40.30
Native-SFT1.77 / 92.38 / 96.8128.33 / 43.88 / 47.61

These existing results show that the final language-side representation supports supervised detection and that language-side adaptation can improve the resulting detection performance in this example. They do not identify which internal LLM block is responsible. The two procedures also differ in trainable capacity, so this is a comparison of their practical detection performance, not a capacity-matched attribution to a particular internal transformation.

Depth-SFT addresses a different, complementary question: which encoder depth provides a useful input to the language pathway after adaptation? Each encoder-depth configuration is adapted separately and evaluated through its final authenticity output. This directly measures the downstream utility of the selected connection, but does not reveal the sequence of representational changes inside the LLM. Conversely, probing internal LLM blocks would describe decodability at intermediate locations; it would complement, rather than replace, the independently trained connection-depth comparison.

We have not added an internal LLM-depth sweep, nor do we present the endpoint analyses as a substitute for this analysis. The revised manuscript clarifies this distinction and explicitly states the corresponding limitation:

Our depth-wise analysis varies the audio-encoder connection depth and evaluates downstream detection utility. LM-Linear probes only the final language-side representation; we do not trace authenticity information through individual LLM blocks. Accordingly, the results characterize interface and adaptation utility rather than localize information changes within the language backbone.

Reviewer_mG6iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework, RAVEL, for evaluating audio LMs for audio deepfake detection in three settings, including zero-shot native output, linear probes on frozen representations, and LoRA fine-tuning. It tests 14 models across four datasets. The authors find that native detection is weak, frozen representations support strong in-domain detection, and fine-tuning further improves performance. However, improvements in cross-domain generalisation are not observed. They also find that intermediate layers can transfer better and that different layers have different suitability for probing and fine-tuning.

Critical strengths
  1. The problem of using audio LMs for detection is timely and interesting.
  2. A wide range of audio LMs are included in the experiments.
Critical weaknesses
  1. The paper does not compare Audio-LLMs with dedicated detectors, such as SSL-AASIST or XLS-R Conformer, trained under the same 19LA protocol. The best cross-domain results also appear to be well below the published results of dedicated detectors.

  2. The motivation for using Audio-LLMs is unclear. Why should they be used as the backbone when they are considerably heavier than pretrained audio encoders? Is the trade-off really worth it? Audio-LLMs are not specifically trained to preserve low-level acoustic artefacts during pretraining. Is using them for this task overkill? Also, does Audio-LLM pretraining provide a better foundation for developing open-set detectors?

  3. What is the practical use case of the proposed evaluation framework? Will it be used to identify which Audio-LLM backbone is more suitable as the base architecture for building Audio-LLM-based detection models? Given that the paper reports rankings that change across source datasets, target datasets and metrics, it is unclear whether the framework can reliably guide such a choice.

  4. Using intermediate transformer layers for audio deepfake detection is not new, as previous work has explored layer weighting, fusion and selection. The depth-wise analysis offers limited practical guidance, as the optimal layers vary across target domains and require exhaustive experiments to identify. No source-based layer selection strategy is provided. This limits the novelty and practical value of the analysis.

Questions for authors

Please see my weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mG6i — Part 1: Dedicated-Detector Comparison

W1. Comparison with dedicated detectors

Thank you for pointing out that the Audio-LLM results need a clearer reference to established detectors. We agree, and have added evaluations of released AASIST, AASIST-L, RawGAT-ST, and TSSDNet checkpoints, the official w2v-AASIST model, and the authors' released AMSDF ensemble.

For a more direct comparison, we include Depth-SFT configurations from our encoder-depth analysis rather than only Native-SFT. All Audio-LLM rows use 19LA-source supervision and report the original seed-42 results. We evaluate both systems on the complete 21LA list (181,566 utterances) and ITW (31,779 utterances). Each entry reports EER / Macro-F1 / accuracy, in percentages.

SystemConfiguration21LA: EER / F1 / AccITW: EER / F1 / Acc
AASISTReleased checkpoint11.88 / 88.49 / 95.9343.51 / 55.91 / 55.99
AASIST-LReleased checkpoint12.99 / 86.74 / 95.3944.62 / 51.13 / 51.57
RawGAT-STReleased checkpoint13.62 / 86.57 / 95.3252.52 / 45.22 / 45.34
TSSDNetReleased checkpoint19.82 / 49.03 / 89.8842.29 / 29.64 / 38.51
AMSDFOfficial ensemble7.63 / 94.19 / 97.999.03 / 63.43 / 63.76
w2v-AASIST19LA-trained, epoch 485.92 / 95.19 / 98.338.51 / 56.45 / 57.64
QwenDepth-SFT, L31 (final, n=1)21.78 / 75.90 / 93.1746.15 / 27.50 / 37.38
QwenDepth-SFT, L14 (source-Dev selected, n=1)8.11 / 91.80 / 97.3312.24 / 74.90 / 74.96
StepAudioDepth-SFT, L13 (fixed anchor, seed 42)10.58 / 87.07 / 95.9926.84 / 64.31 / 64.35
StepAudioDepth-SFT, L31 (final, seed 42)19.95 / 80.54 / 93.0419.81 / 72.98 / 73.03
MOSSDepth-SFT, L05 (fixed anchor, seed 42)10.00 / 91.07 / 97.1126.60 / 48.89 / 51.47
MOSSDepth-SFT, L31 (final, seed 42)21.80 / 80.30 / 94.4122.05 / 69.99 / 70.00

Qwen L14 follows the source-only selection rule in Part 4: choose the lowest unrounded 19LA Dev EER, resolving exact ties toward the shallower layer. Its tied candidates are L14/L17/L20. StepAudio L13/L31 and MOSS L05/L31 are fixed anchor pairs selected after the exploratory sweeps and subsequently repeated over five seeds; they were not selected through the source-only policy. L31 denotes the final encoder block. The complete depth profiles are provided in Appendices F.1--F.6, source-selection records in Appendix F.7, and repeated-run results in Appendix F.8.

For AASIST, AASIST-L, RawGAT-ST, and TSSDNet, both class logits are used to obtain the real/fake decision; w2v-AASIST probabilities are obtained through a two-class softmax. AMSDF retains the authors' official ensemble rule with a fixed 0.5 decision threshold. No target-domain labels are used to adjust decision thresholds. EER is computed from continuous bona fide scores, while Macro-F1 and accuracy use the fixed decision rule P(real) ≥ 0.5. Audio-LLM EERs retain the original estimator, whereas the additional detector EERs use tied-score ROC interpolation.

These are released-system references rather than the requested strictly matched SSL-AASIST/XLS-R-Conformer retraining. Training protocols, checkpoint selection, and computational budgets differ across systems. The comparison provides useful performance context: dedicated detectors achieve strong cross-domain score discrimination, while selected Audio-LLM configurations can provide higher fixed-rule Macro-F1 and accuracy. In particular, w2v-AASIST achieves the lowest ITW EER (8.51%), whereas source-selected Qwen L14 obtains higher ITW Macro-F1 and accuracy (74.90% and 74.96%). This illustrates why score discrimination and fixed-threshold classification should be considered separately. Part 2 further discusses the motivation and practical trade-offs of Audio-LLM-based detection.

AuthorsOpenReview ↗

Response to Reviewer mG6i (Part 2/4): Motivation and Practical Trade-off

W2. Why evaluate Audio-LLMs for speech deepfake detection?

Thank you for asking us to make this motivation more explicit. We agree that being larger, or expressing an answer in natural language, is not by itself a reason to prefer an Audio-LLM. Our original presentation should have separated two questions more clearly: how well a system separates real and fake scores, and how reliably it classifies an utterance using a decision rule fixed before target evaluation.

EER addresses the equal-error point of a threshold sweep. In cross-domain use without target labels, however, the system cannot choose its operating threshold from the target set. A lower EER therefore does not necessarily identify the system with better decisions under the prescribed rule. This is not a distinction between models that can and cannot output labels: dedicated detectors and our score-based Audio-LLM interfaces both produce binary decisions by comparing class scores.

The added comparison makes this distinction concrete. On ITW, w2v-AASIST achieves 8.51% EER but only 56.45%/57.64% Macro-F1/accuracy, while the AMSDF ensemble achieves 9.03% EER and 63.43%/63.76%, respectively. In comparison, source-selected Qwen Depth-SFT at L14 achieves 12.24% EER but higher Macro-F1/accuracy (74.90%/74.96%). StepAudio and MOSS at their final Depth-SFT connections also achieve 73.03% and 70.00% accuracy, respectively. These comparisons illustrate that stronger score discrimination does not necessarily translate into better fixed-rule classification. They do not establish a universal performance advantage of Audio-LLMs over dedicated detectors.

The Audio-LLM-specific question is whether its existing audio representations and language interface can support useful authenticity decisions, and how that depends on the way they are accessed and adapted. A motivating use case is evaluating authenticity as an additional task in a system already using an Audio-LLM. Native outputs test the released interface, frozen readouts test supervised decodability, and task adaptation tests whether the audio-language pathway can be made useful for this task. The Qwen L31/L14 contrast also shows why evaluating only one connection can give an incomplete picture of that potential.

We do not assume that general-purpose pretraining preserves all low-level artifacts or guarantees open-set detection. These are reasons to evaluate Audio-LLM representations rather than assume their suitability. We also acknowledge the computational trade-off: our experiments do not establish a matched latency or memory advantage over dedicated detectors, nor do they show that adding a language model is the most economical solution. Our contribution is instead to characterize when and how existing Audio-LLM interfaces can support speech authenticity detection, particularly when authenticity assessment is needed as an additional capability in an existing Audio-LLM system.

AuthorsOpenReview ↗

Response to Reviewer mG6i (Part 3/4): Practical Use of RAVEL

W3. What practical guidance does RAVEL provide?

We understand the concern: if rankings change, how can the evaluation guide a choice? RAVEL is not intended to identify a universally optimal backbone. Instead, it helps assess a model under the interface, adaptation procedure, and decision rule that will actually be used, rather than transferring a ranking obtained under a different evaluation setting.

One practical example is threshold transfer. On ITW, Kimi Native-SFT improves EER over LM-Linear from 24.23% to 13.02%, while Macro-F1 drops from 66.39% to 38.08%. To examine whether this discrepancy reflects the fixed decision threshold rather than poor score discrimination, we keep the Native-SFT checkpoint unchanged, select a minimum-BER threshold using only 19LA Dev, and apply it directly to ITW. Macro-F1 increases from 38.08% to 78.39% for Kimi and from 28.69% to 59.81% for StepAudio. This adjustment changes the operating decisions, not the underlying scores or their discrimination. Importantly, the same transferred thresholds reduce Macro-F1 on 19LA Eval for all four examined models, showing that threshold transfer is not universally beneficial.

A second example concerns encoder-depth selection. As reported in Part 4, we select layers exclusively using 19LA Dev EER and evaluate the independently trained Depth-SFT configuration at each selected depth. The results show that source-selected layers can improve over final-layer references, but the benefit depends on both the selection interface and target domain. In particular, a layer preferred by a frozen probe does not necessarily provide the best input for subsequent adaptation.

Together, these examples illustrate how RAVEL distinguishes score discrimination from fixed-rule decision performance and frozen-readout preferences from downstream adaptation utility. Its practical value lies in providing evidence for evaluating specific model configurations and deployment conditions, rather than promising a universally best backbone. The revised results and discussion incorporate both analyses to clarify this model- and setting-dependent guidance.

AuthorsOpenReview ↗

Response to Reviewer mG6i (Part 4/4): Depth-Wise Analysis and Source-Only Selection

W4. Contribution beyond intermediate-layer use and practical layer selection

Thank you for pointing out the connection to previous layer weighting, fusion, and selection studies. We agree that finding an intermediate layer better than the final layer is not, by itself, our distinctive contribution. The question here is whether layers favored by frozen readouts also make suitable inputs to language-pathway adaptation, and whether that choice transfers across domains.

Following your suggestion, we evaluated source-only layer selection using the existing complete sweeps. For each model and selection interface (Linear, AASIST, or Depth-SFT), we select the layer with the lowest 19LA Dev EER and fix it before evaluating 21LA or ITW; exact ties are resolved toward the shallower layer. When a frozen probe selects the layer for Depth-SFT, only the layer index is transferred: the reported test result comes from the independently trained Depth-SFT configuration at that depth, not from the probe itself.

To make this comparison directly checkable, the table below gives representative selected layers and their absolute test EERs. Every row executes Depth-SFT. Values reproduce the original single-seed sweeps at the precision shown in Tables 11 and 14; they are not five-seed means.

ModelLayer-selection ruleLayer21LA EER (%)ITW EER (%)
QwenFinal-layer referenceL3121.7846.15
QwenLowest Linear EER on 19LA DevL0213.9842.36
QwenLowest AASIST EER on 19LA DevL0017.7652.31
QwenLowest Depth-SFT EER on 19LA DevL148.1112.24
StepAudioFinal-layer referenceL3119.9519.81
StepAudioLowest Depth-SFT EER on 19LA DevL1510.8621.66

For Qwen, source-only Depth-SFT selection reduces ITW EER from 46.15% to 12.24%. Selecting through Linear instead gives 42.36%, while AASIST selection gives 52.31%. Thus, a layer preferred by a frozen probe does not necessarily provide the most useful adaptation input. Qwen L14 is selected from the exact Dev-optimal set L14/L17/L20 using the shallower-layer tie rule, without accessing ITW labels or scores. The displayed Dev EER values are rounded; selection uses the stored unrounded values.

StepAudio illustrates the separate domain-transfer issue: the source-selected L15 improves 21LA EER from 19.95% to 10.86%, but not ITW EER (19.81% to 21.66%). This contrast shows that successful source-domain selection does not guarantee improvement across target domains.

Across all six models, same-procedure source selection improves EER over the final-layer reference in 6/6 cases on 21LA for each of Linear, AASIST, and Depth-SFT; the corresponding ITW counts are 4/6, 4/6, and 3/6. When Linear or AASIST selects for Depth-SFT, the corresponding counts are 6/6 and 5/6 on 21LA, but 3/6 and 1/6 on ITW. These counts use the original depth-sweep EER estimator. With tied-score EER and selected layers held fixed, the AASIST-to-Depth-SFT ITW count becomes 2/6; all other counts remain unchanged. Complete selection records and aggregate counts are provided in Appendix F.7.

The practical lesson is to distinguish selecting a layer for a frozen readout from selecting an input for adaptation. Source-only selection can improve on the final layer, but its benefit should be checked under the intended procedure and transfer setting. It provides useful guidance without eliminating the need to evaluate candidate configurations.

We further repeated the fixed StepAudio L13/L31 and MOSS L05/L31 comparisons, retaining seed 42 and adding four seeds (123, 2026, 2027, and 2028). The following table reports paired per-seed EER differences, Δ\DeltaEER = EER(L31) −- EER(earlier anchor), as mean ±\pm sample SD in percentage points:

Model and fixed anchorsFrozen AASIST ΔEER (pp)Depth-SFT ΔEER (pp)
StepAudio: L13/L31+17.20 ± 3.07−6.12 ± 2.10
MOSS: L05/L31+5.32 ± 5.03−5.10 ± 2.07

For both models, Frozen AASIST favors the earlier anchor and Depth-SFT favors L31 in EER on ITW in all five runs; tied-score EER preserves these directions. On 21LA, both procedures instead favor the earlier anchors in all five runs. These are targeted repetitions under the original pipelines, not full-depth five-seed sweeps or matched-preprocessing controls. Complete per-seed results and five-run summaries across all three evaluation sets are provided in Appendix F.8.

Reviewer_cidUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically evaluates speech deepfake detection across a range of open-weight and closed-source audio-language models. It examines three aspects: native detection performance, the information accessible from frozen representations, and detection performance after supervised fine-tuning.

The evaluation finds that cross-domain performance often degrades, intermediate encoder layers can outperform the final layer for adaptation, and the most useful layers can vary across models and datasets.

Critical strengths

The paper addresses a relevant question: how well existing audio-language models can detect speech deepfakes. The evaluation is comprehensive and considers native outputs, frozen representations, and supervised adaptation. It also provides useful observations about cross-domain transfer and the role of encoder depth.

Critical weaknesses

The paper’s methodological novelty and explanatory analysis are limited. Although it presents extensive results, it provides limited insight into how the findings could guide the development of better deepfake detection models. In particular, it does not sufficiently explain why performance varies across the evaluated models or why different encoder layers are preferred across datasets.

Questions for authors

Have the authors analyzed whether differences in model architecture, pretraining, or adaptation methods are associated with the observed differences in detection performance?

Why do intermediate encoder layers sometimes outperform the final layer for deepfake detection?

What might explain why the most effective encoder layers differ across models and datasets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer cidU — Part 1: Model Differences

Thank you for your thoughtful review and for recognizing the breadth of the evaluation. We understand your main concern: the paper should help readers interpret the performance differences and use the findings when developing detectors. Our original presentation required readers to piece together model information and results across sections. We summarize the relevant configurations and evidence directly below.

Q1. Architecture, pretraining, and adaptation

Yes. Following your suggestion, we examined the documented model backgrounds alongside the encoder, connector, and adaptation implementations. The table below summarizes the six models in our encoder-depth study.

ModelEncoder and audio-to-language connectionChecked Depth-SFT adaptation configuration
Qwen2.5-Omni-3B32-block Whisper/Qwen-related encoder; pooling + linear projectionTerminal normalization and audio projection; language attention/MLP LoRA
Step-Audio-2-mini32-block Qwen2-Audio-family encoder; convolutional/MLP adapterAudio adapter; language attention/MLP LoRA
Voxtral-Mini-3B32-block Whisper-derived encoder; frame-concatenation MLPMultimodal projector; language attention LoRA
Phi-4-multimodal24-block ASR Conformer; MLP connectorReleased native speech LoRA, including attention and MLP; connector frozen
MOSS-Audio-4B32-block from-scratch audio Transformer; gated MLP + DeepStackAudio adapter and DeepStack mergers; language attention LoRA
Audio-Flamingo-Next32-block AF-Whisper encoder; pooling/RoTE/MLP connectionMultimodal projector; language attention LoRA

Encoder blocks remain frozen in these depth configurations. The table describes Depth-SFT, not Native-SFT: Native-SFT keeps the original audio pathway and connection frozen while adapting language-side parameters; LM-Linear fits a readout without adapting the pretrained model. In MOSS, truncation also changes the available DeepStack connections. These distinctions matter when interpreting what is being compared.

Documented task exposure is another important difference. SALMONN-2, included in the ten-model evaluation, has explicit HoliAntiSpoof exposure during instruction tuning with its audio encoder frozen. We therefore flag it separately rather than attribute its results to architecture or general audio pretraining. For the other models, lack of disclosed spoofing supervision is not evidence of its absence.

To examine the relationship with performance, the following Table 1 results compare two models with related encoder ancestry under 19LA supervision. All entries are EER (%):

ModelProcedure19LA EvalITW
Qwen2.5-Omni-3BLM-Linear6.0640.75
Qwen2.5-Omni-3BNative-SFT1.5744.14
Step-Audio-2-miniLM-Linear3.4838.91
Step-Audio-2-miniNative-SFT1.5120.24

Both improve in-domain after adaptation and reach similar source EERs, but transfer differs: Qwen's ITW EER increases, whereas StepAudio's decreases. Across all ten models under 19LA supervision, plus Qwen/Kimi/MOSS under the two additional source protocols, Native-SFT improves in-domain EER, Macro-F1, and accuracy over LM-Linear in all 16 model–source comparisons. Adaptation therefore has a consistent source-domain benefit in these experiments, but this is not a sufficient predictor of transfer.

The practical implication is to evaluate the intended adaptation procedure and its transfer behavior, rather than infer suitability from encoder family or source performance alone. Architecture, training exposure, and adaptation scope provide relevant context, but these complete-pipeline comparisons do not isolate their individual causal contributions. The depth comparisons below further examine how preferences change within a backbone.

The revised manuscript summarizes the six model architectures in Section 4, retains the full model information and implementation details in Appendices B/C, and incorporates the corresponding interpretation into Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer cidU — Part 2: Intermediate Encoder Layers

Q2. Why do intermediate layers sometimes outperform the final layer?

A possible explanation is that representations useful for the native audio-language training tasks need not be equally useful for authenticity detection. However, this alone does not establish that final layers discard spoofing artifacts. Our additional evidence points to a more specific distinction: usefulness for a frozen readout is not the same as usefulness as an input to language-pathway adaptation.

First, the intermediate-depth advantage is not confined to one selected layer. In the original Qwen Depth-SFT sweep, every connection from L08 through L23 has lower EER than L31 on both targets: 6.38–12.96% versus 21.78% on 21LA, and 12.24–39.42% versus 46.15% on ITW. Every layer in this interval also improves fixed-rule Macro-F1 over L31 (Table 9). This establishes a contiguous region of useful alternatives in that sweep, rather than only an isolated minimum.

Second, we examined whether the same preference holds when encoder outputs are used differently. We fixed the StepAudio L13/L31 and MOSS L05/L31 comparisons after the initial sweeps and added four runs to the original seed-42 results, using seeds 123, 2026, 2027, and 2028. ITW EER (%) is reported below as mean ± sample SD over five runs:

ModelProcedureEarlier anchorFinal layer L31
StepAudioFrozen AASIST16.532 ± 2.065 (L13)33.730 ± 4.818
StepAudioDepth-SFT26.683 ± 1.702 (L13)20.559 ± 0.649
MOSSFrozen AASIST22.397 ± 4.391 (L05)27.716 ± 5.074
MOSSDepth-SFT26.405 ± 1.397 (L05)21.306 ± 0.921

For each model, the earlier anchor is better with AASIST, while L31 is better with Depth-SFT, in all five runs. The directions also remain unchanged when EER groups tied scores before interpolation. These repetitions validate the fixed comparisons, not every point of the original full-depth curves.

This helps interpret the intermediate-layer gains. Frozen AASIST trains a separate detector on encoder sequences and bypasses the language backbone. Depth-SFT instead trains the model-specific connection and language-side adapters while keeping the encoder blocks frozen. A low EER with the first procedure therefore measures what that readout can exploit; it does not establish that the same layer is the best input to the second procedure. The branches also differ in input processing and optimization, so we do not attribute the reversal solely to the LLM or to the loss of a particular acoustic cue.

For detector development, connection depth should therefore be evaluated through the intended adaptation pipeline, rather than chosen only from a frozen-probe peak. Intermediate layers are useful candidates, but the final layer should remain a reference rather than be discarded on the assumption that it contains less authenticity information. The repeated comparisons are reported in Section 5.4 and Appendix F.8, with the model-specific protocols documented in Appendix C.

Thank you for your detailed explaination regarding the architecture, and difference between intermediate laters and final layers for deepfake detection.

As speech language models and audio language models continue to evolve, what insights can your current evaluation provide for training these models to better support deepfake detection? I think this should be one of the key questions that the evaluation paper aims to answer.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Summary

We thank all reviewers for their constructive feedback. In response, we have revised the manuscript and conducted additional experiments to strengthen the evidence for RAVEL and clarify its contributions.

The main revisions address four aspects:

1. Independent depth-wise adaptation (all reviewers)

Clarification: We distinguish RAVEL from frozen layer probing and layer fusion. Across six Audio-LLMs, we independently train 184 Depth-SFT configurations, comparing their adaptation utility with frozen Linear and AASIST readouts.

Finding: Frozen-readout preferences do not reliably predict adaptation utility. A layer with weak frozen readouts can still provide effective inputs for Depth-SFT, while preferred depths can differ across procedures and target domains.

2. Repeated-run validation (cidU; ACfS)

Added: Four additional seeds (123, 2026, 2027, and 2028) for fixed StepAudio L13/L31 and MOSS L05/L31 comparisons, retaining the original seed 42.

Finding: The five-run results confirm procedure- and target-dependent EER preferences for these fixed configurations. We also report metric variability and distinguish targeted repetitions from the original single-seed full-depth sweeps (Section 5.4 and Appendix F.8).

3. Practical layer selection and detector comparisons (mG6i; ACfS)

Added: Source-only layer selection using 19LA Dev EER, with selected layers fixed before target evaluation. Complete selection records and six-model summary statistics are provided in Appendix F.7. We also compare representative Depth-SFT configurations with released dedicated detectors, including AASIST, AASIST-L, RawGAT-ST, TSSDNet, w2v-AASIST, and the official AMSDF ensemble (Table 3).

Finding: Source-selected layers provide useful but procedure- and target-dependent guidance. Detector rankings also differ between EER and fixed-threshold classification, highlighting the distinction between score discrimination and operating-point performance.

4. Methodological clarification and reproducibility (all reviewers)

Clarification: We detail model-specific adaptation components, distinguish audio-encoder connection depth from internal LLM-block probing, and explain the duration-handling differences between evaluation interfaces. We also add appendix navigation, expand reproducibility documentation, and complete the AI-use and ethics statements.

Overall, the revised manuscript clarifies that native detection, frozen representation accessibility, and downstream adaptation measure distinct capabilities. Our findings do not establish a universally optimal Audio-LLM, guarantee open-set generalization, or demonstrate a computational advantage over dedicated detectors. Instead, RAVEL provides a controlled framework for examining how these capabilities relate across interfaces, adaptation procedures, and target domains.

New experiments, updated results, and substantive revisions are highlighted in blue in the revised manuscript. Detailed responses to individual reviewers follow below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.