acceptodds
Withdrawn from ICLR 2027

ECHO: Embedding Convergence and Hidden Organization in Audio Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained models trained once on large-scale datasets are now routinely repurposed for entirely new tasks, but there is no reliable way to predict, in advance, how much a given model will benefit from adaptation without first fine-tuning it. This is a real cost in domains where labeled data is scarce, forcing researchers to spend weeks and compute on trial-and-error. We propose ECHO: a concrete, falsifiable geometric signature of adaptation headroom, together with a precise map of the tasks where that signature holds and where it does not. We show that a single measurable property of a model's internal representation; independent of its training data, architecture, or task performance predicts how much a model stands to gain from adapting to a new task, before any adaptation is performed. We study numerous independently trained audio models spanning speech, music, environmental sound, bioacoustics, and machine acoustics. We measure how each model internally arranges sound relative to every other model, and find that agreement between models is real but only partially explained by what they were trained on, no single factor accounts for it fully. On tasks with real learnable signal, models with more clustered representations gain roughly 3.7 times more from adaptation than already-spread models. Directly measuring changes in each model's representational geometry during fine-tuning shows that more collapsed frozen representations tend to reorganize more. This reorganization does not consistently lead to higher accuracy, and the relationship varies across the three adaptation mechanisms. Domain-matched pretraining is therefore not sufficient for model selection: general-purpose models with no domain-specific training outperform domain-specialist models on their own home tasks by up to 7%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, transfer learning, representational similarity analysis, embedding geometry, audio representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yq7TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work studies the representational geometry in 19 pretrained audio models (encoders) trained with different training regimes (supervised, SSL etc.) with different data domains (speech, music, sounds, birds etc.). Study reports several results which among which the most important are that more uniform embeddings work better without training and that discriminative models tend to have them. On the other hand when finetuning, we can gain more from clustered embeddings although those are also models with the weaker initial performance.

Critical strengths
  • A very interesting, fresh, and potentially practically important topic
  • The general idea for the experiments is insightful and when properly performed this could be an important study. I appreciate nevertheless the statistical evaluations performed to compare the results
Critical weaknesses
  • The writing of the paper is unnecessarily convoluted, stretched, quite far from the proper academic style - see Introduction where the same information (e.g. evaluation on downstream tasks only and not geometry) is repeated several times. The core question the paper asks is finally defined in the second page (around line 68)
  • The clarity of main contributions list can be improved. It’s hard to extract the true core contributions from those convoluted descriptions
  • Given the recent advances with follow-up works related to The Platonic Representation Hypothesis, limiting the experiments to just a single domain is a bit disappointing. I’m curious how the main claims of this work would transfer even to the image domain. Nevertheless I don’t consider this as a significant limitation
  • “To our knowledge, no prior work has systematically measured convergence across a broad, paradigmdiverse set of audio foundation models spanning multiple training paradigms, architectures, and domains.” - I recently came across “What Do Language Models Hear? Probing for Auditory Representations in Language Models” by Ngo & KIM ACL 2024 - probably worth taking a look, might be relevant, although authors consider rather correspondence between audio and language models
  • Figure 1 is of the poor quality - some texts are barely readable
  • The results presented in Figure 2 are not very convincing, with quite small differences, and the r=0 value is within the confidence intervals for almost all evaluated setups
  • How does “Finding 4.2 showed that embedding uniformity governs whether a model’s frozen distances are already usable, without any downstream training.” - I think this is an overstatement or am I missing something? We can see from the results that kNN accuracy is higher for discriminative models, what is rather expected? It’s hard to clearly see those results, I’d suggest plotting them as e.g. a scatter plot of uniformity vs. frozen kNN accuracy, with points colored by domain and per-domain correlations.
  • The main claim of Tab 2. is that models with more clustered embeddings have higher gains, but they also have lower initial scores, what is not taken into account

Small:

  • There is an issue with hyperrref
Questions for authors

See weaknesses

Reviewer_B3xvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper conducts a systematic study on a wide range of pretrained audio models across different domains to investigate whether frozen representations geometry can predict the adaptation headroom. The geometry may predict how much a model benefits from finetuning on new downstream tasks. The paper identifies the uniformity of embeddings as a geometric predictor of adaption gain that holds when the frozen probe performs above chance and vanishes on near-chance tasks (task difficulty). It also introduces a falsifiable scope showing where the diagnostic works and where not.

Critical strengths

Comprehensive model and dataset coverage. This paper covers a wide range of 19 pretrained models with different training paradigms across a diverse set of acoustic domains. This is a very good coverage compared to previous representation geometry work, especially in audio. This provides investigating the dependence of geometric factors on the underlying sound structure.

Bounded and falsifiable diagnostic and well-motivated task. Their embedding uniformity metric predicts adaptation headroom only when the frozen probe performs above chance and fails on near-chance tasks. This defines a very nice and precise scope. The authors do not claim a universal predictor and show limitations. The metric requires no labels and is directly calculated from the embeddings, a well-motivated problem of choosing a respective audio model for a given downstream task.

Critical weaknesses

Exclusive reliance on linear probe limits geometric insights. The paper exclusively relies on linear probes for frozen geometry evaluations. This limits insights to linear decision boundaries and does not consider non-linear structures of more advanced probes. Attentive probing ["ATTENTION, PLEASE! REVISITING ATTENTIVE PROBING THROUGH THE LENS OF EFFICIENCY", Psomas et al. and "Cluster and Predict Latents Patches for Improved Masked Image Modeling", Darcet et al. and "Keep It SimPool", Psomas et al.] is widely used in frozen adaptions in vision models. This is seems to be investigated in detail in pretrained audio models ["Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification", Rauch et al. and "BAT: Better Audio Transformer Guided by Convex Gated Probing", Ghaffari et al.] with prototype probing that seem to utilize frozen representations of audio models better than simple linear probes. This could better reveal which model parts contribute to representational similarity without finetuning bias. More advanced readouts for the different architectures could clarify whether uniformity is truly architecture-agnostic or confounded inductive bias. Additionally, no alternative advanced probes are used to evaluate performance difference across architecture or understand which representation components/layers matter most for a downstream task. Understanding which representations contribute to performance across architectures would provide more actionable insights.

Terminology, readility, structure. Some exemplary issues that impact the paper's clarity and readability that should be fixed throughout the complete work:

  • inconsistent/confusing usage of "frozen": used for pretraiend encoder geometry as frozen representations and features probes with a linear probe, e.g. L069
  • general polishing: formating problems with blank spaces for brackets (L.043), terms like "mixed" for architectures are unclear, abbreviations are capitalized inconsistently (Embedding in L.088)
  • strange phrasing: "acoustic stimuli", "texture-like", "raw-separable", "head-exploitable" are used once but never really explained. This wordiness appears freuqently and disrupts reading flow.
  • llm-style phrasing: "BirdNET-family inference" does not make sense at all
  • Task difficulty seems to be the most important quantity but the exact calculation is introduced too late around L.373. Additionally, the paper does not address whether task difficulty is stable across architectures. Task difficulty is a very important topic but also a very hard metric to define properly. It suddenly becomes important too late.

Actionable Lessons. The conclusion only summarizes findings without translating them into practical guidance, even though it's more or less promised in the introduction. There should be a dedicated section/discussion answering: What should we actually do with these insights? For example:

  • How to choose between models for new tasks based on frozen geometry without adaption?
  • When is uniformity vs. task difficulty the right metric to check first? (without investigations before)
Questions for authors

Please provide answers to the weaknesses listed above. I think this paper could be a good contribution but needs some rework/additions as I outlined above.

Reviewer_xnCJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a framework for analyzing the geometrical properties of pre-trained audio model representations and investigates whether these properties can predict adaptation gain. The authors compare a wide range of models across multiple domains using established metrics such as RSA and embedding uniformity. The main finding is that a model with concentrated embeddings tend to benefit mode from fine-tuning, although this relationship is not consistent across all domains and adaptation methods.

Critical strengths

The paper considers a wide range of pre-trained audio models with different architectures, training objectives, and data exposure, and evaluates them across multiple audio domains. This provides an interesting comparison of how different models organize their latent representations. The approach relies on established and well-defined metrics such as RSA and embedding uniformity, which provide a reasonable basis for comparing some properties of latent representations. The observed relationship between uniformity and adaptation gain is interesting, particularly because uniformity can be measured without labeled data or fine-tuning. The authors also clarify the limitation of their approach as they acknowledge that this relationship between uniformity and adaptation gain does not hold consistently across all domains.

Critical weaknesses

While the paper is framed as a study of latent representation geometry, it primarily analyzes a small set of geometric and distributional statistics computed from embeddings of pre-selected audio datasets. These metrics are well established and provide a reasonable basis for empirical comparisons, which I see as a strength. However, I am not convinced that they provide a systematic characterization of the underlying representation geometry. The results are inherently dependent on the selected input distributions, and the paper does not sufficiently justify whether these inputs adequately probe the models or whether the measured properties are stable across different probe sets. It is also unclear how the effects of architecture, pre-training objectives, and data exposure can be separated. I therefore see the approach more as a useful heuristic analysis of pre-trained representations than a systematic framework for understanding their geometry or predicting adaptation capacity.

While embedding uniformity is a well-established metric, I find the motivation for using it to predict adaptation gain relatively weak. Uniformity measures how concentrated the embeddings are, but does not directly capture the task-relevant information learned by the model or its ability to adapt. In particular, it is unclear whether the observed relationship reflects actual adaptation capacity or simply differences in frozen model performance, since models with lower initial accuracy may have more room for improvement.

The paper provides empirical correlations, but does not explain why these geometric properties should predict adaptation gains or establish that the relationship generalizes across tasks and models. This limits the practical significance of the proposed framework as a reliable model-selection tool.

Questions for authors
  1. What justifies the selected audio datasets as representative probes of each pre-trained model's geometry? Since uniformity is a property of the embedding distribution induced by a specific input distribution, how stable are the model rankings under changes in probe-set size, class composition, acoustic conditions, and dataset source? Is there evidence that the measured geometry reflects task-relevant structure rather than nuisance acoustic variations?
  2. The paper uses the Wang-Isola uniformity statistic across models with different latent dimensions and pre-training objectives. How do the authors account for the dependence of uniformity on embedding dimensionality, kernel bandwidth, and representation extraction? Does the relationship with adaptation gain persist after normalizing against dimension-matched reference distributions and controlling for frozen task performance? What evidence establishes that uniformity measures adaptation capacity rather than general embedding dispersion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.