- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an evaluation framework for deepfake detection through speech deepfake detection using Audio-LLMs. The authors perform this evaluation via zero-shot generation of the AudioLLM, learning linear probes across the representations of the audio encoder and fine-tuning the model using SFT across the different layers of the audio encoder. Experiments are performed using 13 audio LLMs across 5 datasets.
Critical strengths
-
Scope and rigour of evaluation. The paper evaluates 13 models on 5 datasets, which makes the analysis broad and the conclusions generalizable.
-
Unified and reproducible detection techniques. The paper judges the quality of detection via different methods such as zero-shot generation, linear probing and depth-wise SFT which can give a good overview of the representational evolution across different layers and components of an Audio LLM.
Critical weaknesses
-
Paper has very less technical novelty and the insights from the work are covered in prior work. This paper copies the analysis done in prior work [1] for paralinguistic speech understanding to speech deepfake detection. Moreover, the findings of this work are similar to those of the prior works on similar Audio-LLM probing [1].
-
Depth-wise findings for the encoder are very noisy. Figure 4 reports the encoder-depth findings under 19LA source supervision and similar results are expanded in Appendix F. The layer-wise results in Figure 4 seems very noisy and adjacent layers seem to be performing very different. This can be an artefact of number of seeds that the authors run their experiment for. It is recommended to perform the experiment over 5 seeds and then report the average across those to see whether the layer-wise trends hold.
-
Depth-wise findings are limited to the audio-encoder. The paper shows the variation and contribution of using different layers of the audio encoder for downstream LLM processing, but it does not report how those representations transform within the LLM through probing (or any other similar analysis).
-
Formatting issues, AI Use and Ethics
- Overly long appendix without any index: A lot of results and additional information are present in the 29 page appendix which is difficult to navigate through without a content page for the appendix.
- AI USE: The code provided in the supplementary material is almost entirely AI written but the authors do not disclose this in their AI Use statement.
- Ethics Statement: The paper misses an Ethics Statement. Given that this work deals with speech deepfake detection, the authors should include an Ethics statement in the revision/camera ready to note the unethical uses of their approach.
- Table 2 is never referenced in the text.
[1] J. Pang, A. Chaubey, and M. Soleymani, “Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox,” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=v7rYbRR9Zw
Questions for authors
In addition to answering to the above weaknesses, I request the authors to respond to the below minor questions.
-
For Depth-SFT, was the adapter or the connector also finetuned for training each of the depth-wise encoder variants?
-
Why was the Audio-Linear config done on 4 seconds of audio as opposed to LM-Linear/Native-SFT which was done on original audio duration (L240-242)?
5 comments
Response to Reviewer ACfS — Part 2: Five-Run Validation
W2. Reliability of the encoder-depth findings
Thank you for pointing out that the single-seed depth profiles may be sensitive to training variability. We agree that the local fluctuations should not be interpreted as stable layer-by-layer differences without repeated runs. Following your suggestion, we retained the original seed 42 and added four runs (123, 2026, 2027, and 2028) for StepAudio L13/L31 and MOSS L05/L31, using both Frozen AASIST and Depth-SFT. This gives five runs for each of eight fixed configurations: 40 training instances, including 32 additional runs, evaluated on 19LA Eval, 21LA, and ITW. The layer pairs were fixed after the initial exploratory sweeps; these are targeted repetitions, not full-depth five-seed sweeps.
Absolute results. Each cell below reports EER / Macro-F1 / accuracy as mean ± sample SD across the five runs, in percent. Lower EER and higher F1/accuracy are better. Macro-F1 and accuracy retain the original fixed decision rule, P(real) ≥ 0.5. The evaluation uses the complete 21LA set (181,566 utterances) and ITW set (31,779 utterances).
| Model | Procedure | Layer | 21LA: EER / F1 / Acc | ITW: EER / F1 / Acc |
|---|---|---|---|---|
| StepAudio | Frozen AASIST | L13 | 8.88 ± 0.58 / 74.30 ± 4.97 / 85.50 ± 4.46 | 16.53 ± 2.07 / 76.40 ± 7.94 / 76.67 ± 8.05 |
| StepAudio | Frozen AASIST | L31 | 28.91 ± 1.94 / 55.68 ± 3.46 / 68.70 ± 6.52 | 33.73 ± 4.82 / 53.83 ± 14.55 / 56.02 ± 12.68 |
| StepAudio | Depth-SFT | L13 | 10.77 ± 0.64 / 85.04 ± 7.10 / 95.60 ± 1.44 | 26.68 ± 1.70 / 63.40 ± 10.63 / 64.17 ± 9.15 |
| StepAudio | Depth-SFT | L31 | 19.90 ± 0.52 / 81.15 ± 1.15 / 93.75 ± 0.83 | 20.56 ± 0.65 / 68.66 ± 6.58 / 68.88 ± 6.22 |
| MOSS | Frozen AASIST | L05 | 8.93 ± 0.99 / 86.88 ± 8.19 / 94.07 ± 5.77 | 22.40 ± 4.39 / 62.92 ± 6.90 / 65.66 ± 8.34 |
| MOSS | Frozen AASIST | L31 | 19.02 ± 1.95 / 72.64 ± 3.15 / 87.16 ± 2.76 | 27.72 ± 5.07 / 68.58 ± 9.43 / 69.26 ± 9.13 |
| MOSS | Depth-SFT | L05 | 10.47 ± 0.30 / 91.00 ± 0.37 / 97.08 ± 0.09 | 26.40 ± 1.40 / 48.07 ± 3.21 / 50.89 ± 2.38 |
| MOSS | Depth-SFT | L31 | 20.65 ± 1.36 / 79.85 ± 0.74 / 94.25 ± 0.23 | 21.31 ± 0.92 / 65.35 ± 3.78 / 65.60 ± 3.54 |
The key EER comparisons repeat despite variation in absolute performance. On ITW, Frozen AASIST favors the earlier anchor and Depth-SFT favors L31 in all five runs for each model. On 21LA, both procedures favor the earlier anchor in all five runs for each model. For example, StepAudio AASIST at L31 has 27.16% ITW EER in the original run, compared with 31.59%–39.83% in the four additional runs. Its absolute performance is therefore more variable than the original result alone suggests, but L13 has lower EER in every corresponding run. StepAudio Depth-SFT instead favors L31 in every run. Thus, the repeated evidence supports these specific procedure- and target-dependent preferences, not a claim that absolute performance is insensitive to training randomness.
Fixed-rule classification is not uniformly as stable as EER, and we report it separately. On ITW, StepAudio AASIST favors L13 in F1 and accuracy in 5/5 runs; StepAudio Depth-SFT favors L31 in F1 in 5/5 and accuracy in 4/5. The remaining accuracy comparison differs by only 0.01 percentage points (68.21% versus 68.22%). MOSS Depth-SFT favors L31 in all three metrics in 5/5 runs. However, MOSS AASIST does not reproduce its EER preference in fixed-rule classification: although L05 has lower EER in 5/5 runs, L31 has higher F1 in 4/5 and accuracy in 3/5. Its mean F1/accuracy are 62.92%/65.66% at L05 versus 68.58%/69.26% at L31. On 21LA, the earlier anchors win F1 and accuracy in either 4/5 or 5/5 runs, depending on the configuration. We therefore do not generalize the EER preference reversal to every metric.
The revised Section 5.4 summarizes the repeated comparisons, while Appendix F.8 provides the complete per-seed results across all three evaluation sets and their five-run mean and sample SD. The original full-depth curves will remain identified as single-run exploratory profiles; their individual local peaks and troughs have not been validated by these repetitions. In particular, the Qwen example in our W1 response remains an original single-run result. The supported conclusion is that the specified cross-domain EER comparisons repeat across five runs, while fixed-rule decision performance requires its own variability assessment. Thank you for encouraging this check; it lets us distinguish the repeatable observations from the fluctuations more clearly.
Response to Reviewer ACfS — Part 3: Language-Side Evaluation and Internal Probing
W3. Probing representations within the LLM
Thank you for this suggestion. Probing internal LLM blocks would provide a useful complementary view of where authenticity information remains decodable. Our study evaluates the language pathway, but the depth-wise axis is the audio-encoder connection point, not the internal LLM block. We should make this distinction clearer.
The existing evaluation goes beyond encoder-side readouts. LM-Linear extracts the final post-normalization representation supplied to the language-model output head at the position predicting the first answer token. The frozen model processes the audio–instruction context without a reference answer, and a supervised linear classifier is fitted to this language-side vector (Appendices C.1–C.2). Native-SFT instead adapts language-side parameters while keeping the released audio pathway, connection, and output head frozen, and evaluates the resulting real/fake answer-token scores (Appendix C.3). These settings examine information accessible at the language-side endpoint and detection performance after language-side adaptation, respectively.
For example, Table 1 already reports the following MOSS results under 19LA-source supervision. Each cell shows EER ↓ / Macro-F1 ↑ / accuracy ↑, all in percent; classification uses the fixed real–fake comparison.
| MOSS-Audio-4B procedure | 19LA Eval | ITW |
|---|---|---|
| LM-Linear | 3.58 / 91.37 / 96.42 | 29.04 / 32.52 / 40.30 |
| Native-SFT | 1.77 / 92.38 / 96.81 | 28.33 / 43.88 / 47.61 |
These existing results show that the final language-side representation supports supervised detection and that language-side adaptation can improve the resulting detection performance in this example. They do not identify which internal LLM block is responsible. The two procedures also differ in trainable capacity, so this is a comparison of their practical detection performance, not a capacity-matched attribution to a particular internal transformation.
Depth-SFT addresses a different, complementary question: which encoder depth provides a useful input to the language pathway after adaptation? Each encoder-depth configuration is adapted separately and evaluated through its final authenticity output. This directly measures the downstream utility of the selected connection, but does not reveal the sequence of representational changes inside the LLM. Conversely, probing internal LLM blocks would describe decodability at intermediate locations; it would complement, rather than replace, the independently trained connection-depth comparison.
We have not added an internal LLM-depth sweep, nor do we present the endpoint analyses as a substitute for this analysis. The revised manuscript clarifies this distinction and explicitly states the corresponding limitation:
Our depth-wise analysis varies the audio-encoder connection depth and evaluates downstream detection utility. LM-Linear probes only the final language-side representation; we do not trace authenticity information through individual LLM blocks. Accordingly, the results characterize interface and adaptation utility rather than localize information changes within the language backbone.
Response to Reviewer ACfS — Part 1: Beyond Frozen Probing
W1. Relationship to VoxParadox and the contribution of depth-wise adaptation
Thank you for highlighting this connection and recognizing the scope of our evaluation. We agree that applying established probing to another task alone would not establish a distinct contribution. RAVEL goes further by independently training and evaluating adaptation at every encoder depth, rather than inferring adaptation utility from frozen-probe performance.
VoxParadox [1] studies paralinguistic understanding under linguistic–acoustic contradiction. It includes encoder/LLM probing and uses Prompt-Conditioned Layer Mixer (PCLM), SFT, and DPO to improve acoustic-information utilization. We recognize these contributions. Unlike PCLM, which learns to combine selected encoder layers, our study directly compares frozen readouts with independently trained adaptation across the complete encoder-depth range.
What independent depth-wise adaptation adds. At each depth, we separately fit a Linear readout, train an AASIST readout on frozen encoder sequences, and train a Depth-SFT configuration through the model-specific language pathway. Encoder blocks remain frozen; language-side adapters and, where applicable, connection parameters are adapted. Across six models—five with 32 blocks and Phi-4 with 24—this gives 184 independently trained Depth-SFT configurations, alongside the corresponding frozen readouts. These are separate adaptation runs, not layer features extracted from one fine-tuned model or adaptation restricted to probe-selected layers. Full sweeps are reported in Sections 3.2/5.4 and Appendix F.
The following examples use 19LA supervision and the original single-seed depth results. All metrics are percentages; Macro-F1/accuracy use the fixed real–fake decision rule.
1. A weak frozen readout can underestimate a layer's adaptation utility. At Qwen L14 on ITW:
Thus, weak transfer with either frozen readout does not establish that L14 is unsuitable for adaptation (Tables 9–11). This is not merely an isolated gain: every Qwen Depth-SFT connection from L08 through L23 improves on L31 in all three metrics on both 21LA and ITW. The probe and adaptation pipelines differ in capacity and input processing; these results test probing as a predictor rather than isolate a causal benefit of the LLM alone.
2. A probe-preferred layer need not remain preferable after adaptation. The StepAudio comparison reports absolute EER / Macro-F1 / accuracy; bold marks the better layer within each procedure and target.
On ITW, AASIST favors L13, whereas Depth-SFT favors L31 in all three metrics; on 21LA, both favor L13 (Tables 13–14). AASIST at L13 remains the stronger ITW detector in this example—the finding is a change in preferred depth, not universal superiority of SFT. Our five-run follow-up preserves these ITW EER preference directions for both StepAudio and MOSS, as detailed in our response to W2.
Together, these results show why frozen decodability is not sufficient to determine adaptation utility. Qwen demonstrates that a layer with weak frozen readouts can still support effective adaptation; StepAudio shows that stronger frozen performance at one depth does not determine the better adapted connection, and that this relationship changes across targets. The contribution is not simply a larger probing benchmark or the generic observation that fine-tuning can help, but the direct evaluation of these relationships through complete, independently trained depth sweeps using both discrimination and fixed-rule decisions.
We have foregrounded this distinction from VoxParadox in Section 2, clarified the per-depth adaptation design in Section 3.2, and incorporated these comparisons into Section 5.4, with full profiles provided in Appendix F. We hope these revisions clarify the additional question addressed by our study while acknowledging the important foundations provided by prior work.
[1] J. Pang, A. Chaubey, and M. Soleymani. Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox. ICML, 2026.