acceptodds
Under review as a conference paper at ICLR 2027

Learning Along Depth: Depth-Weighted Layer Supervision for Speech Anti-Spoofing

Abstract

Advances in neural speech synthesis have made audio deepfakes increasingly hard to detect, and self-supervised Speech Foundation Models (SFMs) have become the standard front-end for anti-spoofing. Standard fine-tuning, however, supervises only the final representation. Intermediate layers therefore remain weakly adapted to the task, and the burden of discrimination shifts to ever more expressive back-end classifiers. In this work, we ask how supervision can instead reshape the internal representation hierarchy of the SFM encoder. Our layer-wise analysis of fine-tuned SFM shows that anti-spoofing discriminability emerges late in the network, and later still as model size grows, so most intermediate layers remain poorly aligned with the task. Motivated by this, we propose Depth-Weighted Layer Supervision (DWLS), an effective multi-layer supervision framework that progressively aligns representations across the encoder depth with the anti-spoofing objective. Across diverse deepfake benchmarks, DWLS paired with a lightweight classifier achieves competitive in-domain performance and strong out-of-domain generalization, suggesting that improving internal feature alignment can be a more efficient route to adaptation than increasing back-end complexity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Deepfake Detection, Speech Anti-Spoofing, Speech Foundation Models, Layer Supervision, Representation Learning, Cross-Domain Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HnV6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how deepfake-related information can be spread across the layers of the fine-tuned pretrained self-supervised speech encoders. It applies the final linear classifier to every intermediate layer. From this logit-lens style probe, it argues that discriminability of bonafide and spoof appears only in the last layer. It then proposes the method DWLS, which alternates final-layer fine-tuning with phases where a shared pooling and classifier head is applied to all layers. The per-layer losses are weighted by a cosine schedule that moves from deep to shallow layers. Experiments use XLS-R trained on ASVspoof 2019 LA and evaluated on ten in-domain and out-of-domain sets, with linear, MLP, and KAN heads. The paper also includes neural collapse (NC1/NC3) and CKA analyses.

Critical strengths

None

Critical weaknesses

Main argument

  1. The layer-wise analysis does not support the claim that "discriminability emerges late."

In Section 3, the author applies the final-layer classifier WW to every layer. WW was trained only on hL−1h^{L−1}. Fig. 2 and Fig. 4(d–e) also show that intermediate layers are nearly orthogonal to the final layer (AngDist ≈ 0.5, cosine ≤ 0.05). So, high EER at early layers mainly shows that WW does not fit those layers' coordinate systems. It does not show that the information is missing.

The paper's own numbers point to this. XLS-R-2B reaches 82.62% EER at layer 11. Flipping the sign of the score would give about 17% EER, which means the layer carries a strong discriminative signal. If it is really "uninformative", the EER should be around 50% and close to random selection. A per-layer trained linear probe is the standard tool for this question, and prior layer-wise studies cited by the paper (El Kheir et al., 2025; Pasad et al.) use such probes. Without it, the motivation for the method is not established, and this is a very serious issue.

  1. The headline layer-wise improvement is close to being true by construction.

In Fig. 4(c), DWLS trains the shared head on all layers, while the baseline head never sees intermediate layers. A lower logit-lens EER at layer 7 under DWLS (4.39% versus the baseline, described as "43.5% improvement") is therefore expected. It is not evidence that the representations themselves became better. A minimal example makes this issue clear. Suppose the baseline is:

h1→h2→⋯→h24→Wh_1\rightarrow h_2\rightarrow \cdots \rightarrow h_{24}\rightarrow W

and it is trained only with L=CE(Wh24,y)\mathcal L = CE(W h_{24},y). There is therefore no direct training constraint requiring Wh7W h_7 to be discriminative. By contrast, DWLS optimizes an objective of the form CE(Wh7,y)+CE(Wh8,y)+⋯+CE(Wh24,y)CE(W h_7,y) + CE(W h_8,y) + \cdots + CE(W h_{24},y). DWLS has an inherent advantage because Wh7W h_7 was explicitly part of its training objective, whereas the baseline was never trained to make the same classifier WW work at layer 7. A fair comparison would train a probe per layer for both models, or evaluate both with a head trained the same way.

  1. Novelty and positioning.

The core novelty of the paper is not new. Applying a shared or auxiliary classifier to intermediate layers is deep supervision. There are a lot of references about deep supervision. Examples include Deeply-Supervised Nets [1], GoogLeNet auxiliary heads [2], intermediate CTC in speech [3], and early-exit networks [4]. None of these are discussed. The claim in Section 2 that existing methods "all address how to aggregate intermediate features; none changes how the layers are trained" clearly overstates the gap. I suppose the new parts are the cosine depth schedule and the alternation, but neither is ablated against simpler options:

  • a constant uniform multi-layer loss,
  • progressive phases only, with no alternation,
  • a SUPERB-style learned weighted layer sum with the same linear head,
  • a per-layer linear probe baseline.
  1. The empirical evidence is weak.
  • In Table 2, Linear-DWLS is worse than its own baseline on 19LA, 21DF, 25DF, and LSV. KAN-DWLS is worse on 21LA (+1.41) and 25DF (+0.86). The average gains (0.26–1.17 EER) only come from a few sets.

  • The Linear-DWLS numbers in Table 2 match the D→S, N=300 row of Table 1. That row was the best on ITW. This suggests the direction and N were chosen using out-of-domain test results, even though the paper says checkpoints are selected on the 19LA dev set. Please clarify how the final configuration was chosen.

  1. Some internal contradictions.
  • Eq. 11 and Fig. 1 define a deep-to-shallow schedule. Section 6.1 then concludes that feature collapse "requires progressive shallow-to-deep optimization," while Section 6.3 says deep-to-shallow generalizes better. Which schedule is the proposed method?
  • Section 6.1 says NC3 is "rapidly established," which reads as good alignment. Section 7 says "NC3 remains persistently high … the linear classifier does not align with the class-mean axis." These two statements conflict.
  • The conclusion says DWLS is "a more efficient route … than increasing back-end complexity." But back-end parameters are tiny next to the ~315M backbone, and all systems have about the same total size. The extra training cost of DWLS (pooling and a head on 25 layers) is not reported.

I tend to reject the paper at the current stage for two main reasons: (1) The motivating analysis and the layer-wise results are confounded. (2) The method is a form of deep supervision with auxiliary losses. That idea is well known, and the paper does not cite or compare against it. On top of that, the empirical gains are mixed.

[1] Lee, Chen-Yu, et al. "Deeply-supervised nets." Artificial intelligence and statistics. PMLR, 2015.

[2] Szegedy, Christian, et al. "Going deeper with convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[3] Lee, Jaesong, and Shinji Watanabe. "Intermediate loss regularization for CTC-based speech recognition." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.

[4] Teerapittayanon, Surat, Bradley McDanel, and Hsiang-Tsung Kung. "Branchynet: Fast inference via early exiting from deep neural networks." 2016 23rd international conference on pattern recognition (ICPR). IEEE, 2016.

Questions for authors
  1. Can you report per-layer trained linear probe EER for the baseline and DWLS, both in Section 3 and in Fig. 4(c)?
  2. Why are mean-pooled intermediate features almost exactly orthogonal to the final layer (cosine ≤ 0.05)? Could a few large-norm dimensions dominate the cosine and angular metrics?
  3. Table 1 reports AUC, but Table 2 does not. Please be consistent.
  4. Fig. 2 and Fig. 4 axis labels are too small to read.
  5. The limitations section should include single-seed results and a single backbone.
Reviewer_6v2XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks where, along the depth of a speech foundation model, anti-spoofing information appears, and proposes Depth-Weighted Layer Supervision (DWLS). Training alternates between ordinary fine-tuning of the last layer and phases in which every layer is supervised through one shared classifier, with per-layer weights that follow a cosine schedule from deep to shallow layers (or the reverse). Experiments fine-tune XLS-R-300M with linear, MLP and KAN classification heads and evaluate on in-domain and out-of-domain spoofing datasets. The paper reports lower average EER for all three heads and explains the gains with layer-wise EER curves, Centered Kernel Alignment (CKA, a representation-similarity measure) and neural-collapse metrics, arguing that task-relevant structure forms in earlier layers.

Critical strengths
  1. The method is simple and improves detection without changing the deployed model. Each baseline/DWLS pair in Table 2 has the same architecture and parameter count. With the linear head, EER falls from 4.69% to 3.38% on In-the-Wild (ITW), from 17.51% to 14.44% on DFADD, and from 19.56% to 17.75% on SpoofCeleb. This makes intermediate supervision a practical training option, which is the main reason the paper is not a clear reject.

  2. The evaluation covers several distribution shifts and three classifier heads. Linear, MLP and KAN heads are compared on in-domain and out-of-domain benchmarks, and the average EER drops for all three (7.62 to 6.87, 7.28 to 6.11, 5.87 to 5.61). The appendix gives training settings, dataset descriptions and pseudocode. Some individual datasets get slightly worse (with the linear head, four of the eight columns rise, including 19LA and 21DF), and the tables show this honestly.

Critical weaknesses
  1. The representation analysis does not support the paper's central explanation. Three separate issues:

    (a) The layer-wise probe is confounded. Equations 4–5 apply the final-layer classifier WW directly to every intermediate layer. A high EER at layer ℓ\ell under this readout can mean that layer ℓ\ell lacks spoofing information, or only that its representation is not in the form WW expects. DWLS trains the intermediate layers against exactly this shared classifier, so a lower EER under the same readout cannot tell the two apart. Evaluating each frozen layer with its own classifier would.

    (b) The printed CKA formula is not standard CKA. Equation 7 defines Z∈Rd×NZ \in \mathbb{R}^{d \times N} (features × utterances) and uses the numerator ∥Zf⊤Zℓ∥F2\|Z_f^{\top} Z_\ell\|_F^2, an N×NN \times N product. Standard linear CKA (Kornblith et al., 2019, https://proceedings.mlr.press/v97/kornblith19a/kornblith19a.pdf) requires the d×dd \times d product ∥ZℓZf⊤∥F2\|Z_\ell Z_f^{\top}\|_F^2 in this layout; the denominators are equivalent either way. The printed quantity changes when the feature space is rotated, which CKA should not. This is probably a transposition typo, but the authors need to state which expression produced Figures 4(a, b, f).

    (c) NC3 is interpreted in two opposite ways. NC3 measures how well the classifier direction aligns with the direction between the two class means (0 means aligned). Section 6.1 says NC3 "is rapidly established" and "stabilizes rapidly", treating early stabilization as alignment achieved. Section 7 says NC3 "remains persistently high", meaning the classifier does not align. Both cannot be true, and the paper's account of how representations form depends on which one is.

  2. The ablations do not show what the cosine schedule and the alternation add. Table 1 varies the weighting direction and the value of NN; Table 2 compares full DWLS with standard fine-tuning. Nothing compares DWLS with plain multi-layer supervision under the same training budget, for example uniform layer weights with the same alternation. Direct supervision of hidden layers has precedents such as Deeply-Supervised Nets (Lee et al., 2015, https://proceedings.mlr.press/v38/lee15a.html), which the paper does not discuss, so the specific contribution of the schedule is not established. The ambiguity of NN (see the presentation note below) makes Table 1 harder to interpret. In addition, Appendix A.2 fixes a single training seed (1234); the ± intervals in the tables come from bootstrapping evaluation utterances (Section 5) and say nothing about variation across training runs.

Questions for authors
  1. Layer-wise probes and diagnostics (clarification).

    • (a) If each frozen layer is given its own classifier, fitted and selected only on source training and development data, does the conclusion that discriminability appears only in late layers still hold?
    • (b) Which expression was used to compute the CKA plots: the one printed in Equation 7, or the standard d×dd \times d form?
    • (c) How do you reconcile the NC3 statements in Section 6.1 and Section 7?
  2. Matched control for the schedule. Under the same training budget and checkpoint-selection rule, how much does full DWLS gain over (a) uniform multi-layer supervision without alternation and (b) the same alternation with fixed uniform weights? This comparison would directly change my assessment of the schedule's contribution.

  3. Training details.

    • (a) Are NN = 300–1200 checkpoints along one training run, or separate runs with different phase lengths?
    • (b) How many optimization steps were run in total, and which development-set criterion selected the configuration reported in Table 2?
    • (c) What MLP hidden size was actually trained? Appendix A.4 describes a 1024→128→2 head with about 131.5k parameters, but Table 2 shows the MLP model with about 1.06M more parameters than the linear model.
Reviewer_KbN8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how anti-spoofing information is distributed across the layers of self-supervised speech foundation models (SFMs). It first analyzes post-trained wav2vec 2.0 and XLS-R models, using layer-wise EER, angular distance and CKA relative to the final layer. Based on this analysis, the authors argue that discriminative capacity emerges only in the last few layers, and later in larger models. Motivated by this, they propose Depth-Weighted Layer Supervision (DWLS), which alternates between standard final-layer fine-tuning and a multi-layer objective.

Critical strengths

The method is simple, adds no inference-time cost, and can be applied to any SFM-based detector. Training details are well documented.

Critical weaknesses
  1. The terminology and analysis methods are often unfamiliar to the anti-spoofing community. For example, Sec. 3 aims “to disentangle task-discriminative capacity (Eq. 5) from global geometric alignment”. “Global geometric alignment” is not an established term in this field and is not clearly defined in the paper; and Sec. 6 analyzes the representations through Neural Collapse (NC1/NC3), which is rarely used in anti-spoofing. The standard way to examine layer-wise behavior in this field is per-layer EER from independently trained probes.

  2. Section 3 is the main motivation for DWLS, but I do not think it supports the claim that anti-spoofing discriminability emerges only in late layers based only on the experiments provided. The reasons are:

    a). How W is trained is not described. The paper does not say how W is trained. If W is trained only on the last layer, a high EER at intermediate layers may simply reflect a mismatch between representation spaces, not a lack of discriminative information.

    b). Angular distance and CKA measure how similar each layer is to the last layer, not whether it can separate bona fide from spoofed speech. Also these two metrics are rarely used in anti-spoofing. So their relevance to the claims in Sec. 3 should be justified.

    c). Few models and no consistent trend. Only four models are analyzed, all from the wav2vec 2.0 and XLS-R family. And the four curves in Figure 2 (top) do not show a consistent pattern. Other SFM such like WavLM or HuBERT should also be considered for generalization.

Questions for authors

The notation is hard to follow. For example, the W below Eq. (3), the W above Eq. (4), and the W in Eq. (8) seem to have different meanings right? and the same issue occurs with the symbol N. Also, the Multi-Layer Auxiliary Classification subsection mentions “a shared classification head fθf_\theta to produce per-layer prediction logits”, but fθf_\theta does not appear in the equations that follow; instead g is used, and it is unclear whether gg is the same as fθf_\theta.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.