acceptodds
Withdrawn from ICLR 2027

Mixture Probing in Frozen Audio Encoders: Shortcut Bounds and a Suppression Law

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Linearly mixed audio is an attractive testbed for representation analysis because the ground-truth sources are known exactly. The standard instrument is a probe: train a classifier on a frozen encoder's embedding of a mixture to name a source inside it, and read its accuracy as evidence about what the encoder retains. We show that this instrument is biased in two ways that have not been reported, quantify what survives once both are removed, and identify the mechanism of what survives. First, if mixtures are built by drawing *distinct* classes, which is the obvious construction, the masker labels alone determine an AUC of with no audio at all. On a ten-class corpus at this reaches , exceeding the value our own probe measured there; drawing classes i.i.d. drives the bound to exactly . Second, when sources are paired by natural co-occurrence, a predictor given only the masker's class reaches AUC against the audio probe's , so the probe is reading the loud source and inferring. Third, with both shortcuts closed, weak-source decodability still falls with the number of sources at a matched conditional ceiling, AUC per (95% CI ; 63 cells, three encoders, three corpora). Fourth, corrupting an oracle time-frequency mask at a controlled rate shows this capacity law is entirely a suppression effect: a reader that can remove dB of masker energy pays only of the AUC that costs at dB, and at dB it pays . Naming one class among many is not intrinsically harder; separating it is. Part of the remaining loss is architectural rather than informational: pooling over the encoder's own frame tokens recovers up to AUC at from the same frozen features, and this margin widens with training budget while the clip vector saturates. We release the diagnostic suite with a full account of the hypotheses we falsified and of the controls we withdrew, so that the retractions can be audited beside the results.

Primary area:
interpretability and explainable AI
Keywords:
audio representation learning, probing, shortcut learning, evaluation methodology, frozen encoders, sound event recognition, source separation, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_gjCmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how well frozen audio encoders (CLAP, AST, and AudioMAE) retain information about a weaker audio source when probed on synthetic mixtures of k∈2,3,5k \in {2, 3, 5} sources from ESC-50, UrbanSound8K, and FSD50K. The authors point out two shortcuts in mixture probing benchmarks (sampling kk distinct classes without replacement and pairing sources by natural co-occurrence), and show that after closing these shortcuts, linear and shallow probes on frozen clip embeddings degrade as the number of sources kk increases. Using an oracle Ideal Binary Mask (IBM) corrupted by random bin flipping, they argue that this degradation is driven by the lack of masker suppression in the frozen encoder, and show that pooling over frame tokens recovers part of the lost accuracy compared to clip-level pooling.

Critical strengths

None

Critical weaknesses
  1. The main finding of the paper (that audio classifiers fail to capture the weakest source in a mixture as interfering sources are added, and that source separation resolves this) is an already known property of audio classifiers that has been shown in the literature, specifically by Tzinis et al. [A]. Evaluating this known property on newer encoders like CLAP does not make the core claim novel, and the authors completely neglect to compare against existing audio source separation methods used as a front-end before CLAP or the other encoders [A] to show what the actual capability of these models is when paired with real source separation.

  2. Several claims in the paper are incomplete and reflect a weak grounding in the audio separation literature. First, what actually determines whether a classifier captures multiple sources is not the raw number of sources kk in the mixture, but the actual overlap of those sources in the time-frequency (Fourier) domain, as was also shown in [A] since models can easily capture multiple sources when they do not overlap heavily in the Fourier domain. Second, the authors only use an Ideal Binary Mask (IBM) rather than standard separation masks such as the Ideal Ratio Mask (IRM) or Ideal Phase-Sensitive Mask (PSM) [B]. Third, the paper is poorly written (for example, putting hyper-specific details like β=−0.098\beta = -0.098 AUC per log⁡k\log k with 95% CIs and 63 cells directly in the abstract) and relies on trivial synthetic experiments and filler math like Proposition 2, which assumes uniform energy retention in expectation across frequencies even though audio representations are sparse in the time-frequency domain.

[A] Tzinis, E., Wisdom, S., Hershey, J.R., Jansen, A. and Ellis, D.P., 2020, May. Improving universal sound separation using sound classification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 96-100). IEEE.

[B] Erdogan, H., Hershey, J.R., Watanabe, S. and Le Roux, J., 2015, April. Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 708-712). IEEE.

Questions for authors

Can you provide results where an actual learned audio source separation model is used as a front-end before CLAP, AST, and AudioMAE, as well as oracle evaluations using ideal ratio mask or phase-aware masks (IRM or PSM [B]) instead of random bit-flipping of a hard IBM?

[B] Erdogan, H., Hershey, J.R., Watanabe, S. and Le Roux, J., 2015, April. Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 708-712). IEEE.

Reviewer_6qgvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work investigates (linear) probing of frozen audio encoders on mixes of concurrent sound sources. It identifies two biases in the conventional probing setup that make results look better than they really are.

Critical strengths
  • the idea of the paper and the investigation itself is interesting and invokes interest.
  • investigating "hidden" bias in probing and using mixes in audio as the testbed is genuinely interesting
Critical weaknesses

The writing reads mostly like unedited LLM output, very incoherent writing. While the general idea of the paper is interesting and the results (although somehwat burried) could be valuable, the paper is in general very strangely written, very very 'hard to read for a human. This problem already starts in the abstract, which suddenly becomes a very specific number dump with no narrative arc. This can also be seen in the introduction and throughout the paper. For example, I dont really understand the dictionary learning/interpretability motivation with no real explanations to the reader. It also seems to jump all over the place (also very little related work to better position this paper). Im not even sure is the paper is really building upon this motivation. Additionally, terms are introduced without real explanation: "LUFS", "Maskers" (definition?), "arms" (arms is a very very obvious LLM term that comes straight from the agent, used quite a lot at different places and not really understandable). These are only some examples. In general, the writing needs to be overhauled completely to make it digestible for a human reader and not read like an LLM-only text.

Outdated methodology. The used encoders are all very outdated. Because the claim is about frozen encoders generally, the evidence only provides a very tiny scope. Newer SSL models are absent (e.g.,very recent state-of-the-art from this years ICML: BAT (Ghaffari et al.) and AudioMosaic (Huang et al.)). Additionally, only linear probing is investigated. Audio encoders could have richer information in their representations where more advanced probing methods are required (e.g. very recent from the last ICLR: Unmute the patch tokens (Rauch et al. that seems to investigate probing methods in audio). These seem to be current developments that are not investigated at all. These additions would strengthen the contributions of this paper.

Questions for authors

See weaknesses. In general, I think this paper does deliver some useful and interesting insights but the presentation needs a complete overhaul

Reviewer_7q7hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this paper the authors perform different controlled studies on probes (classifiers) acting on representations obained via (frozen) audio encoders. The probes are trained on the embeddings to detect the classes of the sources contained in the original audio mixtures.The authors first use a label-only ideal model that has access to the labels of the maskers in a mixture (the complementary sources to the candidate source that is being probed). The authors show that such a model (that never sees the embedded audio), would give a source detection AUC > 0.5 if drawing audio is without replacement. They argue that the original probe could exploit a similar mechanism of using the maskers to detect the candidate, without representing it. Second, they argue that, when sources share a context (e.g., engine sound, car sound), the model can skip detecting the target and based their decision on the co-occuring sources. After resolving these shortcuts, they see that another factor that can decrease the classification strength is the increase in maskers in a track.They try to understand if this issue is because of data itself or if it's the encoder / probe that drops the information. To do this they compare with a version of the mixture with an IBM (ideal binary mask) on the target source, showing good behavior on the later. Finally they study how operating at a frame level can help increase classification strength.

Critical strengths
  • Studying possible shortcuts that afflict the training of source-level probing methods over representations given by audio encoders is a relevant topic, and the authors discuss two important cases.
Critical weaknesses
  • Main issue is that the paper is an example of AI slop: superficially there aren't (many) grammatical errors, but the majority of the text is very difficult to follow, not being clear and comprehensible (and becomes partially intelligible only through AI assistence). Often many terms / expressions are un-natural / strange for an formal paper (e.g., the use of "cell", the use of "arm", "The bound was live in our own design", "the reader enjoys") and many terms are introduced without a definition. In the questions for authors I have listed many examples of such issues.

  • The main methodological problem is that, while the authors propose some useful considerations that could hint at shortcuts when probing sources in a mixture, the way the authors frame their observation is in a very general / overarching term, for example implying upper bounds on AUC when there isn't proof for such a bound except experimental correlation:

    • For example, in Section 4, the AUC bound derived on an ideal probe that has access to masker labels, it's used to show the probe has a shortcut but such use is not justified deeply: an actual probe could not work like an ideal device that has access to the masker labels. It is not shown that the encoder fails to encode the target, or that the probe actually uses the shortcut.
    • In Section 5, “Which no accoustic account predicts” is an overstatement. The flat behavior is suggestive evidence for a shortcut but not a proof of the mechanism. In general, it could be that the target track contains some cueus that are still visible by the probe even when its volume is relatively low.
    • Saying “The probe is not reading the weak source; it is reading the loud one and inferring.“ (Section 5) is stronger than what these comparisons show. While there is evidence shown, it does not conclusively identify the probe’s strategy. Also, it should not be called an upper bound, there isn’t a mathematical proof for this. It is only an indicator.
    • Isn’t trivial that probe would work pretty well if you apply the IBM on it (in Section 6)? It is supplied privileged information. In other words, this comparison does not establish that the unmasked embedding lacks target information. The information might remain encoded in a form the probe cannot extract.
  • There is a strange focus on withdrawals (for example the Remark 2). Why authors talk about withdrawing things if they have not even been published? One can do an applendix about negative results but there is too much focus in the main text on such aspects.

  • Contributions are poorly described and justified, the authors don’t put clearly what novelties have added to the literature. It’s just a listing of the different sections.

Questions for authors
  • Line 039: "of them that vector still describes.", I would say "which of them are still described by that vector”
  • Line 044: "Unlike the synthetic ground truth used in dictionary-learning benchmarks,": Not all dictionary-learning benchmarks use synthetic ground truth. The authors could use “the majority”.
  • Line 045: Usless use of “here”
  • Line 047: "This paper is about how that instrument fails, what survives its repair, and why.", I would rephrase this phrase.
  • Line 050: "maskers" is not defined at this point. One has a definition in Section 3 but it is too far away, one should introduce what the term means already here.
  • Line 052: "cell", this term appears repeatedly in the text, not being defined anywhere: i guess it means "experimental setting" or something like that
  • Line 056: "arms", another recurrent term, it is an out-of-place word, it should be changed by "cases" or something similar
  • Line 057: "target-to-rest level", please explain that level means volume / volume level
  • Line 058: "faster than a conditional ceiling measured on the same mixtures declines": This phrase is written in a awkward way. Also what type of conditional ceiling? It is not introduced.
  • Line 060: "We then ask what causes it, and measure rather than interpret.", this phrase sounds very colloquial and not scientific
  • Line 061: "Corrupting an oracle ideal binary mask at a controlled rate sweeps the target-to-masker ratio the reader enjoys": This phrase is not only grammatically incorrect: it also contains the part “reader enjoys”. What is a “reader” and what “enjoying” means here. At a first read, it would seem that this might be refering to the reader of the article and the experiment would be towards their enjoyment. Quite ironic.
  • Line 062: "the number of sources is almost free at the favourable end of that axis and costs the entire effect at the unfavourable end.": What does “free” mean? What are the “favourable / unfavorable ends”?
  • Line 063: "one axis rather than two,": what are these axes?
  • Line 070: what metric is the 0.5? I imply it’s the AUC, but authors have have to say that
  • Line 070: "The bound was live in our own design": A bound is not live, one maybe could say was contained
  • Line 075: "shortcuts closed": maybe shortcuts resolved
  • Line 075: what are the matched units?
  • Line 078: "A mechanism, measured rather than argued", This unatural phrasing recurs here and could be replaced with a clearer, more natural phrase.
  • Line 100: "Ours what is physical,": when mixing sources with iid, the mixing is not necessarily physical
  • Line 104: Other precedent works investigates the relation between sum of mixtures and sum of embeddings such as "Postolache, E., Mariani, G., Mancusi, M., Santilli, A., Cosmo, L., & Rodolà, E. (2023, June). Latent autoregressive source separation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 8, pp. 9444-9452)."
  • Line 128: "Readers": authors should state clearly that a reader is a classifier on f(m)f(m).
  • Line 128: What is a "tower"?
  • Line 130: "which uses no encoder and exists so that claims about the encoder are not claims about probes": convoluted phrase
  • Paragraph "Encoders and corpora." (Section 3): The datasets and baselines should be cited.
  • Line 199: "The possibility is nevertheless now removed rather than argued away.": another example of strange sounding phrase
  • Table 2:
    • Table 2 is very confusing because the main values discussed in the prose of the paragraph (0.968 AUC for the label predictor and 0.620 in the control branch) are not present in the table.
    • It is very confusing that the IBM coverage measures a different quantity from the others (which are AUC). It is easily confused with an AUC value.
  • Line 211 (methodological question): Why the control “arm” has 0.620 AUC if in the previous section the iid scenario was 0.5?
  • Line 213: The oracle ceiling / mask coverage with the IBM (ideal binary mask) is not introduced in a sufficient way in this section. Like what is the difference between the oracle ceiling and the IBM coverage. Link at least to Section 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.