acceptodds
Withdrawn from ICLR 2027

Quantifying Separation in Representations for Speech Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised speech modelling aims to learn useful general-purpose speech representations for a wide variety of downstream tasks. Despite their superior capability to capture complex data properties, one of the drawbacks is that information relevant for different tasks is typically entangled in the representation space, thereby limiting general task performance and imposing extra difficulty for universal speech representation learning. A formal measure of the degree of separation of task-relevant information in representations has the potential to mitigate this but is under-investigated in the literature. To address this gap, this work introduces the concept of Task Separation, which extends the concept of disentanglement from latent factors to downstream task use. It aims to quantify the degree of separation of representation content between tasks. Using Self-supervised Learning (SSL) features linear models are added to probing tasks. An information-theoretic measure of Task Separation is derived from the parameter space of such linear probers. Experiments examine the degree of separation between four speech tasks and different SSL presentations: speaker identification (SID), speech emotion recognition (SER), phone recognition (PR) and automatic speech recognition (ASR). Experiments show that: 1) the separation score for tasks between SID and SER, PR and ASR, respectively, is at a moderate level, ranging from 0.32 to 0.67, suggesting the difficulty of separating task-relevant information of SID from the other three tasks; 2) with a relatively high separation score, the linear model parameters from SER, PR and ASR can filter out the relevant information for SID effectively; 3) the distribution derived from the parameter space of linear probers can be used to interpret the optimal design of a minimum hierarchical model to combine two speech tasks. These findings enrich the scientific understanding of speech representations from the view of downstream tasks, with a formal evaluation protocol contributing to the goal of universal speech representation learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Task-level disentanglement, probing task, hierarchical model design, universal speech representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5ZncOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors of the article propose a method to measure the disantenglement of speech tasks by a given SSL network. This measure is operated at the frame level for ASR, SER and SID tasks, on the IEMOCAP dataset, by leveraging a Linear layer classifier head from a single layer. Networks leveraged include Wav2Vec2.0 and WavLM. Experiments show the frame-level separability is low for all tasks, showing a non-independence between the tasks. Additional experiments show the proposed separability metric can be improved with finetuning, but no experiment show the degradation on the base performances of the foundational models.

Critical strengths

The experimental process is explained in enough details to be reproducible, using a known dataset and known models.

Critical weaknesses

Some claims of the authors should be verified and provided adequate citations, like: "Most real-world data is believed to be generated based on a mixture of multiple latent factors. One of the working assumptions is that the changes among them are independent." Does this imply variations in emotions are independent from the linguistic content? or that the speaker identifying information would be independent from the dialect of a given speaker? Citations or clarification are needed.

The evaluation of a given metric on a dataset as small as IEMOCAP (limited set of utterances, limited set of speakers) critically affects the results presented.

The frame-wise evaluation of the separability is limited for the selected tasks. At least for the SID task, the speaker prosody plays an important part in speaker recognition tasks, and cannot be guessed from a single frame, which is why in standard benchmarks studying SSL models capabilities (like SUPERB), not all classifier heads are linear layers.

In general, the novelty of the technique is limited, but as I didn't find a specific reference showing a paper doing the exact same thing, I would put that as a minor weakness.

Questions for authors

Could you please adress the main weaknesses mentionned early?

Reviewer_nV3POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Task Separation, a task-level alternative to latent-factor disentanglement for analyzing speech representations. Instead of assuming known independent latent factors, it treats downstream tasks as the relevant factors and quantifies how separately task-relevant information is encoded in representations. The measure is derived from the parameter space of linear probes: task-specific weight magnitudes are averaged into a relative importance vector, and an entropy-based score is computed over how exclusively each representation dimension is used by different tasks.

Critical strengths

Task Separation is derived from linear-probe weights and an entropy-based score. It does not require ground-truth latent factors, making it practical for real-world speech data.

Critical weaknesses
  1. Insufficient theoretical justification and cross-task comparability 2.Linear probes are too weak and may distort task-relevant information 3.Claims exceed the experimental scope: With one corpus, four tasks, and two models, the claim of establishing a “formal evaluation protocol for universal speech representation learning” is too strong. The claim should be softened or supported by more corpora and tasks.
Questions for authors
  1. Why should the elementwise magnitude of linear-probe weights be interpreted as “task-relevant information”?
  2. Can you report confidence intervals across seeds and folds to show that the observed high/low separation patterns are statistically meaningful?
  3. PR/ASR linear probes perform poorly. How do you rule out that the learned W mainly reflects underfitting or optimization difficulty rather than task-relevant information? Can you provide robustness results with stronger probes, different regularization, or different learning rates?
  4. Table 3 contains mixed and sometimes very small improvements. Are these gains significant across runs? Are they compared against parameter-matched two-layer baselines? How do you justify the hierarchical-design conclusions from these results?
  5. With only IEMOCAP, four tasks, and two SSL models, what evidence supports the claim of establishing a “formal evaluation protocol for universal speech representation learning”? Would you consider softening this claim or adding more tasks?
Reviewer_LWDDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Task Separation, a measure of how much different speech tasks rely on different dimensions of a frozen self-supervised representation. The authors train a linear probe for each task, use the magnitude of its weights to estimate how much the task depends on each dimension, and compare these estimates across tasks using a disentanglement score from the Disentanglement-Completeness-Informativeness (DCI) framework.

The paper compares the separation of information for four tasks, speaker identification, emotion recognition, phone recognition and ASR. The authors find that speaker identification is separated from the other three tasks while emotion and content tasks overlap. They argue that the score can predict which tasks interfere when one is trained on the other’s output, and can guide the design of hierarchical models that combine two tasks

Critical strengths

The paper studies a useful question for universal speech representation learning, which is how different speech tasks share information in SSL representations. Applying a disentanglement metric to downstream tasks, not only to latent factors, is also an interesting direction.

Critical weaknesses
  • I think the separation score mostly reflects differences in weight scale between probes, not differences in which dimensions each task uses. In Eq. 2, the importance values are normalized across tasks for each dimension, but they are never normalized within each task, so a probe with uniformly larger weights dominates every dimension. SID is also the only task trained with a learning rate of 0.01, ten times higher than the other tasks, while SER, which is also an utterance-level classification task, has weights on the same scale as PR and ASR. This suggests that the high SID scores come from the training setup and not from how SID uses the representation.

  • Averaging the weights over labels makes the importance vectors the same length, but it does not make them comparable across tasks. A dimension that is important for a single label is diluted more for tasks with many labels, such as PR with 49 outputs, than for tasks with few labels, such as SER with 4. The weight scale also depends on the loss and the number of training steps.

  • The metric depends on the coordinate basis of the features. A rotation of the representation keeps the same information but changes the score, and SSL features have no reason to align task information with individual dimensions. Taking absolute values also removes the direction each task uses, so two tasks that read the same dimension in different directions count as overlapping.

  • Several probes barely solve their task, so their weights may not reflect task information.

  • The experiments used to support the metric give weak evidence that it predicts task behavior. The first experiment tests the hypothesis that when two tasks have a high separation score, one task’s probe output filters out the information needed for the other. The results only partially support this. Task pairs that include SID drop more than pairs of SER with PR or ASR at both layers, but at layer 12, PR and ASR have the lowest separation score and still drop by up to 67%.

Questions for authors
  • Can you report results with variance over several seeds for Table 3, and compare the hierarchical models with a standard multi-task model with shared input features?

  • Can you show the findings on datasets with more speakers, such as MSP-Podcast or LibriSpeech, on other SSL models, and in a multilingual setup? It would also help to test pairs of tasks with a known relation. For example, gender or age prediction should overlap with SID, and accent or language identification should overlap with PR, so the score should be low for these pairs and higher for unrelated pairs. This would show whether the score is reliable.

  • Can you clarify how task performance affects the score? If a probe is close to chance for a task, the information needed to solve it is not available, so the score computed from its weights may not be meaningful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.