acceptodds
Under review as a conference paper at ICLR 2027

A Study on the Geometry of Accent in Speech Foundation Models

Abstract

Speech foundation models are known to encode phonetic, speaker, and higher-level linguistic information, as measured by decoding probes. Within the representation space, how these factors are geometrically organized is comparatively underexplored. We study this organization through accent, an attribute that self-supervised learning (SSL) models are not trained to represent. We ask how accent is arranged in representation space, how that arrangement changes across network depth, and whether it is shared across independently trained models, using three corpora (L2-ARCTIC, VCTK, GLOBE) and three encoders (WavLM, HuBERT, Whisper). We report three properties. First, accent variation forms a continuous, low-dimensional structure rather than isolated clusters, with softer cluster boundaries than speaker identity, populated interpolation paths, a connected accent graph, and a low-rank set of accent displacement directions (effective rank 5 on the parallel corpora). Removing the top five accent directions reduces a speaker-disjoint accent probe to near chance while leaving speaker identity largely intact, and ablating random subspaces of equal rank does neither. Second, because accent is deterministic given speaker, a probe whose training and test folds share speakers can classify accent by memorizing speaker identity. Under speaker-disjoint evaluation, where no speaker appears in both training and test folds, accent decodability is weakest in early layers and strengthens to a mid-to-late peak, whereas speaker identity peaks in the earliest layers. The apparent early accent structure under speaker-shared evaluation is largely speaker leakage, which shrinks with depth. Third, accent geometry agrees across the two SSL encoders (–, permutation ). On GLOBE this agreement exceeds both the untrained encoder and acoustic (MFCC) baselines and reaches the split-half noise ceiling, while on the smaller corpora the baselines approach the SSL–SSL correlation, so our learned-convergence claim is based on GLOBE. One initial expectation fails, where accent centroids cannot be expressed as convex combinations of other accents, so continuity does not imply compositionality. The speaker-first depth ordering we observe matches prior layer-wise analyses of these encoders. We release and report all code and results.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
speech representation learning, self-supervised speech representations, representational geometry, accent variation, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pfjGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the latent feature geometry regarding accents in SSL models (HuBERT and WavLM) and ASR models (Whisper). It conducted probing tests using frozen features from the models on accented corpora L2-ARCTIC, VCTK and GLOBE. The probing tests include manifold continuity, rank dimension, convex decomposibility, accent vs. speaker distribution across layers and agreement across encoders. The main findings are that (1) accent-related variation occupies a smooth and low-rank subspace; (2) speaker-disjoint accent decoding improves toward intermediate or later layers, showing a different distribution compared to speaker identity information.

Critical strengths
  1. The paper focuses on a novel problem: studying the activation manifold of speech encoders regarding accents, which is underexplored.
  2. Formulating the smoothness and rank dimensions of accents provides useful insights about the accent manifold geometry.
  3. The experiments seem to be solid, using multiple speech encoders and datasets.
Critical weaknesses
  1. I find that overall, it is difficult to understand the exact physical meaning of several proposed probing tests in stage I. For example, TwoNN is used to obtain intrinsic dimension d⋆d^\star, silhouette of accent labels gives "softness". These techniques are not properly defined, described and interpreted.
  2. Similarly, the "matrix V of pairwise accent-centroid difference vectors" is not clear to me. It is shown that this matrix is low-rank. However, Section 4.4 further shows that accent centroid cannot be formulated as a linear combination of others, which is bit perplexing to me. A more in-depth analysis should be conducted here.
  3. Section 5 establishes the "speaker leakage gap" as the accent-speaker entanglement. However, speaker-disjoint decoding itself is not sufficient to obtain a speaker-invariant accent decoder. The training data of that speaker-disjoint accent classifier still only have one speaker always speaking in one accent, so theoretically the accent classifier can also learn to use speaker identity, instead of accent to perform the classification. Testing on unseen speaker alone cannot fully be understood as speaker-invariant. 3.In general, I think the paper scratches the surface with many metrics, but provides a relatively limited interpretation of the probing tests. I would perfer a deeper analysis compared to showing many results that are hard to interpret.
Questions for authors
  1. For Whisper model, do you use the Whisper Encoder or Decoder?
  2. How were the Stage I/III analysis layers selected?
3 comments
AuthorsOpenReview ↗

Response to Reviewer pfjG (Part 1/2)

Dear Reviewer,

Thank you for your detailed review. Your review lists two main findings. We have dropped our first claim, that accent occupies a smooth, low-rank subspace. The second, the depth result, holds, and we now call the probe speaker-generalizing.

W1. Physical meaning of the Stage I tests.

We added a definitions table with each statistic and its null (Appendix A, Table 4), and each test in Section 3.2 is now paired with a null or a control.

  • Silhouette measures how much closer an utterance is to its own class than to the nearest other class, from −1 to 1. We report it for accent labels, speaker labels and shuffled accent labels. The softness score was a ratio of these silhouettes. It is undefined when the speaker silhouette is at or below the shuffled one, which happens on GLOBE, so we dropped it (Appendix E.2).
  • TwoNN estimates intrinsic dimension from the ratio of each point's second to first nearest-neighbour distance. We compare it with a Gaussian sample of the same size and dimension. It is below that reference (5 to 22 against 29 to 34), but it describes the utterance cloud as a whole and is not specific to accent, so we do not use it as evidence about accent (Section 4.2).
  • Interpolation density and graph connectivity ask whether the region between accent centroids is populated. A positive control that scales between-accent spread by λ\lambda shows how far the data are from the point where these tests register separation (about λ=5\lambda = 5).

W2. The matrix V, and low rank against convexity.

Each row of V is a difference μj−μk\mu_j - \mu_k between two accent centroids, one row per pair. Its row space is the space spanned by the centroid offsets, so its rank is at most K−1K − 1, and its rank says how many directions the accent differences use.

These two properties measure different things. Rank is about how many directions the centroids spread out in. Convex decomposability is about whether each centroid sits inside the region enclosed by the others. A triangle is a simple example. Its three corners span two dimensions, and the corners cannot be written as a weighted average of the other two. So a set of centroids can be low rank with every centroid on the outer edge, and the two results still fit together.

Our revision also makes this question simpler. When we compare the rank of V against a null that shuffles whole speakers between accent labels, no cell comes out lower than chance (pp between 0.13 and 1.00), so we dropped the low-rank claim. For convexity, we now use a matched-spectrum null. It keeps the singular values of the centroid matrix and shuffles the layout, so low rank cannot give a positive result by itself. Simplex reconstruction does not beat this null on L2-ARCTIC or VCTK. On GLOBE, there is a small effect for WavLM and Whisper, but the three further checks we ran suggest it is not the mixture structure (Section 4.5).

AuthorsOpenReview ↗

Response to Reviewer pfjG (Part 2/2)

W3. Speaker-disjoint decoding and speaker invariance.

We agree. Testing on unseen speakers shows that the probe transfers to new speakers, and it does not show that the probe ignores speaker identity. We now call the probe speaker-generalizing and removed "speaker-invariant" from the paper (Section 5.1).

We also added a control for the concern you raise. We fit a within-accent LEACE eraser, which removes linear within-accent speaker variation and keeps the accent centroids fixed, on half of each accent's training speakers and apply it to the other half, where the probe is trained and tested. If the probe relied on within-accent speaker variation, accuracy should drop after erasure, and we did not see a drop. On L2-ARCTIC, accuracy after erasure is higher than the matched probe in 14 of 15 layer and encoder cells, and on VCTK it is within 0.05 in all 15 (Section 5.2). On GLOBE, there are more speakers than PCA dimensions, the eraser removes all within-accent variation, and the test is not informative. The rank ablation gives a related result. Removing the accent span leaves a within-accent speaker probe at 0.98 on L2-ARCTIC (from 0.99). This does not rule out every possible use of speaker cues, and we include it in Section 5.1.

W4. Many metrics, limited interpretation.

This is fair. We cut metrics that did not hold up or that cannot be interpreted at our sample sizes. Cluster softness and neighborhood consistency are removed (Appendix E.2). CCA is excluded because with K≤40K ≤ 40 points in 64 dimensions it returns 1.0 for every pair (Appendix E.3). The per-axis acoustic correlations are descriptive only, since at K=6K = 6 to 88 the number of large correlations matches chance (Appendix E.6). The UMAP figures are removed. The main text now gives each test, its null, and what it shows, and the paper makes four claims, listed at the end of Section 1.

Q1. Whisper encoder or decoder?

Encoder only, from Whisper-large-v3 (Section 3.1). The input is padded to 30 seconds, and we mean-pool only over the frames that cover the audio. The submission averaged over padding frames as well, and this is fixed in the revision. While checking the setup, we also found that the submission used a fine-tuned HuBERT checkpoint. The revision uses HuBERT-large pretrained on Libri-Light without fine-tuning, and all HuBERT results were recomputed.

Q2. How were the Stage I and III layers selected?

For each corpus and encoder we fix one layer before computing any geometry. At each candidate layer (6, 9, 12, 18, 21 for WavLM and HuBERT, and 8, 16, 24, 28, 31 for Whisper) we score a logistic-regression accent probe with five-fold speaker-disjoint cross-validation and keep the layer with the highest mean accuracy. This gives layers 18, 12 and 28 on L2-ARCTIC and 12, 12 and 24 on VCTK and GLOBE (WavLM, HuBERT, Whisper).

The submission selected layers with speaker-shared folds. That rule picks layer 6 for WavLM on L2-ARCTIC and GLOBE, the shallowest candidate and the one where speaker identity is easiest to read. This was a mistake on our part, and all Stage I and III results were recomputed with the speaker-disjoint rule. The choice of layer is less stable on regions where the probe curve is flat. On VCTK the WavLM and HuBERT scores at layers 12 to 21 are within about 0.05 of each other, which is why the Stage II peaks in Table 2 differ slightly from the analysis layers there. We note this in Section 7.

Thank you once again for your in-depth review and pointing our the flaws to help improve our paper.

Thanks the authors for providing thorough analyses, revising the original claims and previously problematic experimental settings. I genuinely appreciate the honesty in acknowledging these flaws. However, I think this response has pretty major revisions that may have already exceeded the scope of a "rebuttal". After dropping the first claim (low-rank and smooth accent subspace), and reinterpreting "speaker-generalizing" instead of "speaker-invariant", the remaining findings are thus more about the layerwise information distribution inside SSL/Whisper encoder models. The claimed "geometry" of the accent subspace has limited insight. Hence, I will keep my score and incline to reject this paper.

Reviewer_kU7NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the properties of the accent distributions in commonly used embedding models such as WavLM, Hubert and Whisper.

The paper looks at the aspects such as cluster formation, effective dimensionality, and whether the observed geometry is consistent across different encoder models.

Critical strengths

The paper clearly lists its contributions. The experimental findings seemed to have followed a good methodology.

Critical weaknesses
  • I am not very sure about the novelty of these findings. Most of the findings seem to have precedents in the literature as well. For instance: https://arxiv.org/pdf/2011.12649v3 seems to reach similar conclusions regarding the continuity in the manifold. Then, https://arxiv.org/html/2306.06524v1 talks about the fact that accent information is more crisp in the later layers.
Questions for authors
  • From Figure 2, it's a bit hard to deduce that the accents form a continuous manifold. Within a same accent group though it looks as it there are clear clusters of few data points.

  • The analysis, and the conclusions seem interesting. But what are the broader implication of these results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kU7N

Dear Reviewer,

Thank you for your review and for the two references.

W1. Novelty relative to prior work.

Both papers are relevant and we have added them to our discussion in Section 2 Related Work. Bartelds et al. (2022), your first link, compute pronunciation distances between non-native and native speakers from SSL features and find that middle layers match human accentedness ratings best. Yang et al. (2023) probe a wav2vec 2.0 model fine-tuned for accent identification on L2-ARCTIC and find accent-specific phoneme information in layer 9.

Our study differs in a few ways in that for ours, we freeze all three encoders and none of the encoders are trained or fine-tuned on accent. We look at how accent categories are arranged relative to each other (overlap, rank of centroid differences, mixture structure) and if that arrangement is the same across three independently trained encoders. Both papers do not test cross-model agreement. Every accent probe in our study, including the one used for layer selection, is speaker-disjoint, and every geometric statistic is compared against a null that reassigns whole speakers to accent labels.

On the layer result, both papers fit with our Stage II finding that accent information peaks in the middle to late layers. Our addition is that under speaker-shared evaluation accent seems to be present early, and this early signal is mostly speaker leakage. Speaker-shared selection picks the shallowest layer for WavLM on L2-ARCTIC and GLOBE, so the evaluation setup changes which layer is to be used.

Q1. Figure 2 and the clusters within accents.

You are right, and this led to one of the larger changes in our paper. The small clusters within an accent group are individual speakers. With three to four speakers per accent on the parallel corpora, utterances from the same speaker sit together. We tested this directly. For the tests that we used for continuity like interpolation density and graph connectivity, they also pass even when the classes are shuffled labels or speakers, so they cannot tell accents apart from either (Section 4.1). A positive control shows they only register separation at about five times the observed between-accent spread. We removed the UMAP figure and the continuous manifold claim and have changed it to say accent classes overlap heavily.

Q2. Broader implications.

There are four that we can potentially see:

  1. For any attribute that is constant within a speaker (accent, dialect, L1 background and similar), speaker-disjoint evaluation is needed at every step, including layer selection. Without it, the selected layer and the apparent formation order both shift toward the layers where speaker identity is easiest to read.
  2. Continuity and rank statistics need nulls and positive controls matched to the label structure, since heavily overlapping classes and low-rank continuous structure can give the same summary numbers.
  3. On GLOBE, three encoders trained with different objectives and data have similar arrangement of 40 accent centroids, to within their own measurement noise and beyond an MFCC baseline. This supports the representational convergence view (Huh et al., 2024) for an attribute that none of the models is trained on.
  4. For downstream use, the results point to two tests could be future work. One removes the accent span at the speaker-disjoint layer and checks WER on accented speech. The other checks whether accent directions fit on one encoder transfer to another, which the GLOBE agreement result suggests might work. Both are in Section 7 and in our response to Reviewer 1.

In the updated paper, we have included it in Section 7. On presentation, we added a definitions table (Appendix A), paired each test with its null or control in Section 3.2, and moved metrics that did not hold up to a diagnostics appendix with the reason for each (Appendix E).

Thank you once again for your review and helping to strengthen our paper.

Thank you for your rebuttal, but my concerns on novelty and broader impacts still remain. I will keep my score.

Dear Reviewer,

Thank you for reading our rebuttal. Could we get some feedback on which parts of the novelty and broader implications in our rebuttal that still have issues that could help us improve in our future work? Thank you.

Reviewer_tYTBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how accent information is geometrically organized within speech foundation models, focusing on its continuity, dimensionality, and evolution across network layers. The authors analyze three speech encoders (WavLM, HuBERT, and Whisper) across three datasets (L2-ARCTIC, VCTK, and GLOBE) using geometric analysis and probing techniques. First, the authors report that accent representations form a continuous, low-dimensional structure, with approximately five accent-discriminative directions on the parallel corpora. Next, speaker-disjoint probing reveals that speaker identity is strongest in early layers, whereas speaker-invariant accent information becomes more distinguishable in middle-to-late layers. Finally, representational similarity analysis suggests strong agreement between WavLM and HuBERT accent geometries, particularly on GLOBE, although the proposed convex compositionality hypothesis is not supported.

Critical strengths

S1. The paper investigates an interesting and relatively underexplored problem of understanding the geometric organization of accent representations in speech foundation models.

S2. The experimental analysis is reasonably comprehensive, covering three datasets, three encoders, and multiple geometric metrics with appropriate speaker-disjoint probing controls.

S3. The paper is generally well written and easy to follow, with a structured presentation of hypotheses, experimental results, and limitations.

Critical weaknesses

W1. Insufficient Evidence for the Claimed Continuous, Low-Dimensional Manifold: The evidence for continuity primarily relies on cluster softness, interpolation density, etc. which do not necessarily establish that accent representations form a meaningful continuous manifold. These metrics can be sensitive to the PCA projection, neighborhood size, and density thresholds, but corresponding sensitivity analyses are limited. Furthermore, the matrix of pairwise differences between K accent centroids has rank at most K−1 by construction, making the reported effective rank of five for six accents less compelling. The random-direction baseline should preserve this centroid-difference structure to establish whether the observed low rank is genuinely meaningful.

W2. Dataset Confounding and Limited Generalizability: The conclusions are heavily dependent on a small number of speakers per accent in L2-ARCTIC and VCTK, while GLOBE introduces uncontrolled recording conditions, linguistic content, and self-reported accent labels. Although speaker-disjoint probing addresses speaker memorization, it does not eliminate other potential confounding factors such as recording channel, lexical content, and demographic correlations. The cross-model convergence claim also relies primarily on GLOBE, where the MFCC baseline already achieves a relatively high correlation of 0.83 compared to 0.97 for WavLM–HuBERT. More controlled experiments are necessary to determine whether the observed geometry captures accent-specific linguistic structure rather than shared acoustic or dataset characteristics.

W3. Limited Functional Validation and Unclear Practical Significance: The paper primarily relies on geometric measurements and linear probing to establish its findings, without demonstrating whether the identified accent subspaces have a meaningful effect on downstream speech tasks. Although removing the top accent directions reduces accent classification accuracy while preserving speaker identity on the parallel corpora, the same separation is not convincingly demonstrated on GLOBE. Furthermore, the paper does not evaluate whether manipulating these directions affects speech recognition accuracy, accent robustness, or other relevant downstream behaviors. Without such experiments, it remains unclear whether the identified geometric properties provide actionable insights beyond descriptive representation analysis.

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer tYTB (Part 1/2)

Dear Reviewer.

Thank you for your constructive review. Your points on continuity and rank are valid, and with that, we updated several claims in the paper. We reran all experiments with speaker-disjoint layer selection and added nulls and controls for each test. We go through each point below.

W1. Continuity and low rank.

We agree the original evidence did not fully support a continuous manifold. As such, we dropped cluster softness, which is undefined whenever the speaker silhouette is at or below the shuffled-label silhouette (Appendix E.2), and we just report the silhouettes.

For interpolation density and graph connectivity, we did three things.

  1. Sensitivity sweep over four values of kk crossed with five density floors (interpolation) or five edge thresholds (graph), plus PCA sizes of 16, 32, 64 and 128 (Section 4.1, Table 6). The result is the same in all nine corpus and encoder cells.
  2. We ran both tests on shuffled labels and with speakers used as classes. Both tests pass, so they do not tell accents apart from either.
  3. Positive control that scales between-accent spread by λ\lambda with within-accent spread fixed (Section 4.1, Figure 2). Graph connectivity first drops at λ=5\lambda = 5 (λ=3\lambda = 3 for Whisper on VCTK), and interpolation density drops only at λ=20\lambda = 20 on the parallel corpora and not on GLOBE. So the tests do work, and they only start to pick up separation once accents are about five times further apart than they are in the data.

We updated it to describe accent classes as heavily overlapping instead of continuous manifold.

On rank, you are right that V has rank at most K−1K − 1 by construction, so an effective rank of 4 or 5 for six accents does not value-add much. We added the kind of baseline you suggested. The null reassigns whole speakers to accent labels at random, keeps the number of speakers per label, and rebuilds the centroids and VV from those groups (1000 draws). This keeps the centroid-difference construction and the speaker sampling. No cell has a stable rank below this null (pp between 0.13 and 1.00), and this holds at every PCA size. We removed the low-rank claim (Section 4.2).

W2. Confounds and the convergence claim.

We agree that speaker-disjoint probing may not fully remove channel, content or demographic confounds, so we made three changes.

  1. The MFCC baseline. The 0.83 figure in the submission came from an earlier version of this baseline that we could not reproduce. The rebuilt MFCC RDM (per-utterance MFCC mean and standard deviation, standardized and averaged per accent, Appendix D) correlates 0.41 to 0.43 with the three encoders on GLOBE.
  2. Raw RSA does not account for measurement noise, so we added a speaker split-half analysis. We split the speakers of each accent into two halves 200 times, compare encoder A on one half with encoder B on the other, and divide by the geometric mean of each encoder's own split-half reliability. On GLOBE, this ratio is 0.93 to 1.00 for the encoder pairs and 0.56 to 0.74 for encoder against MFCC. A partial Mantel correlation controlling for the MFCC RDM is 0.85 for every pair, and an untrained encoder correlates 0.20 to 0.30 with the trained ones (Table 3).
  3. On VCTK, the MFCC baseline reaches the same ratio as the encoder pairs (0.93 to 0.99), and on L2-ARCTIC the MFCC RDM has a split-half reliability of 0.04. We update it that we cannot separate learned agreement from shared acoustics on the two controlled corpora, and the convergence claim is based on GLOBE only (Section 6.3).

We still do not control channel or gender on GLOBE, and part of the shared arrangement could come from recording conditions that correlate with accent labels. This is listed in the threats to validity (Section 7). We also report all GLOBE results with and without merging labels that name the same variety (Appendix C).

AuthorsOpenReview ↗

Response to Reviewer tYTB (Part 2/2)

W3. Functional validation.

We rebuilt the rank ablation with controls below full span, since removing all K−1K − 1 directions removes linear decodability by construction. We compare against random directions inside the accent span, the top-variance directions, and random directions anywhere (Section 4.3, Figure 3). On L2-ARCTIC with WavLM, removing four of five accent directions takes accent accuracy from 0.68 to 0.22 (chance 0.17), against 0.37, 0.47 and 0.68 for the three controls. A speaker probe trained within each accent goes from 0.99 to 0.98 with the full span removed.

On GLOBE, the result is weaker, as you noted. The accent probe starts at 0.28 (chance 0.07), and the accent directions remove 0.02 to 0.05 more accuracy than random directions in the span. With WavLM, removing 12 directions takes accent accuracy from 0.28 to 0.19 and the within-accent speaker probe from 0.55 to 0.50. We describe this as accent information being concentrated in the leading directions, and we do not claim a clean separation on GLOBE.

We did not test ASR or other downstream tasks (Section 7). Parekh (2026) report that attenuating an accent subspace in a fine-tuned ASR model increases WER. Their subspace is estimated at an early layer where we find substantial speaker leakage, so it may also contain speaker variance. The practical result we can support is about evaluation. Speaker-shared layer selection picks the shallowest layer for WavLM on two of three corpora, so a system that chooses its layer for accent adaptation this way would read from a layer where speaker identity dominates.

There are two possible follow-ups for downstream tasks. The first is to remove the accent span at the speaker-disjoint layer and at the early layer used by Parekh (2026), and compare WER on accented test sets such as L2-ARCTIC. This would show whether the speaker variance in the early-layer subspace is what drives the WER change. The second uses the GLOBE agreement result. If the three encoders share the same arrangement of accents, accent directions or adapters fit on one encoder might transfer to another, and that can be tested on accent identification and ASR. We have not run these, and we added a sentence on them to Section 7.

Thank you once again for your review. Our updated paper makes fewer claims, and we think the ones left with the updated experiments are better supported now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.