- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the latent feature geometry regarding accents in SSL models (HuBERT and WavLM) and ASR models (Whisper). It conducted probing tests using frozen features from the models on accented corpora L2-ARCTIC, VCTK and GLOBE. The probing tests include manifold continuity, rank dimension, convex decomposibility, accent vs. speaker distribution across layers and agreement across encoders. The main findings are that (1) accent-related variation occupies a smooth and low-rank subspace; (2) speaker-disjoint accent decoding improves toward intermediate or later layers, showing a different distribution compared to speaker identity information.
Critical strengths
- The paper focuses on a novel problem: studying the activation manifold of speech encoders regarding accents, which is underexplored.
- Formulating the smoothness and rank dimensions of accents provides useful insights about the accent manifold geometry.
- The experiments seem to be solid, using multiple speech encoders and datasets.
Critical weaknesses
- I find that overall, it is difficult to understand the exact physical meaning of several proposed probing tests in stage I. For example, TwoNN is used to obtain intrinsic dimension , silhouette of accent labels gives "softness". These techniques are not properly defined, described and interpreted.
- Similarly, the "matrix V of pairwise accent-centroid difference vectors" is not clear to me. It is shown that this matrix is low-rank. However, Section 4.4 further shows that accent centroid cannot be formulated as a linear combination of others, which is bit perplexing to me. A more in-depth analysis should be conducted here.
- Section 5 establishes the "speaker leakage gap" as the accent-speaker entanglement. However, speaker-disjoint decoding itself is not sufficient to obtain a speaker-invariant accent decoder. The training data of that speaker-disjoint accent classifier still only have one speaker always speaking in one accent, so theoretically the accent classifier can also learn to use speaker identity, instead of accent to perform the classification. Testing on unseen speaker alone cannot fully be understood as speaker-invariant. 3.In general, I think the paper scratches the surface with many metrics, but provides a relatively limited interpretation of the probing tests. I would perfer a deeper analysis compared to showing many results that are hard to interpret.
Questions for authors
- For Whisper model, do you use the Whisper Encoder or Decoder?
- How were the Stage I/III analysis layers selected?
3 comments
Response to Reviewer pfjG (Part 2/2)
W3. Speaker-disjoint decoding and speaker invariance.
We agree. Testing on unseen speakers shows that the probe transfers to new speakers, and it does not show that the probe ignores speaker identity. We now call the probe speaker-generalizing and removed "speaker-invariant" from the paper (Section 5.1).
We also added a control for the concern you raise. We fit a within-accent LEACE eraser, which removes linear within-accent speaker variation and keeps the accent centroids fixed, on half of each accent's training speakers and apply it to the other half, where the probe is trained and tested. If the probe relied on within-accent speaker variation, accuracy should drop after erasure, and we did not see a drop. On L2-ARCTIC, accuracy after erasure is higher than the matched probe in 14 of 15 layer and encoder cells, and on VCTK it is within 0.05 in all 15 (Section 5.2). On GLOBE, there are more speakers than PCA dimensions, the eraser removes all within-accent variation, and the test is not informative. The rank ablation gives a related result. Removing the accent span leaves a within-accent speaker probe at 0.98 on L2-ARCTIC (from 0.99). This does not rule out every possible use of speaker cues, and we include it in Section 5.1.
W4. Many metrics, limited interpretation.
This is fair. We cut metrics that did not hold up or that cannot be interpreted at our sample sizes. Cluster softness and neighborhood consistency are removed (Appendix E.2). CCA is excluded because with points in 64 dimensions it returns 1.0 for every pair (Appendix E.3). The per-axis acoustic correlations are descriptive only, since at to the number of large correlations matches chance (Appendix E.6). The UMAP figures are removed. The main text now gives each test, its null, and what it shows, and the paper makes four claims, listed at the end of Section 1.
Q1. Whisper encoder or decoder?
Encoder only, from Whisper-large-v3 (Section 3.1). The input is padded to 30 seconds, and we mean-pool only over the frames that cover the audio. The submission averaged over padding frames as well, and this is fixed in the revision. While checking the setup, we also found that the submission used a fine-tuned HuBERT checkpoint. The revision uses HuBERT-large pretrained on Libri-Light without fine-tuning, and all HuBERT results were recomputed.
Q2. How were the Stage I and III layers selected?
For each corpus and encoder we fix one layer before computing any geometry. At each candidate layer (6, 9, 12, 18, 21 for WavLM and HuBERT, and 8, 16, 24, 28, 31 for Whisper) we score a logistic-regression accent probe with five-fold speaker-disjoint cross-validation and keep the layer with the highest mean accuracy. This gives layers 18, 12 and 28 on L2-ARCTIC and 12, 12 and 24 on VCTK and GLOBE (WavLM, HuBERT, Whisper).
The submission selected layers with speaker-shared folds. That rule picks layer 6 for WavLM on L2-ARCTIC and GLOBE, the shallowest candidate and the one where speaker identity is easiest to read. This was a mistake on our part, and all Stage I and III results were recomputed with the speaker-disjoint rule. The choice of layer is less stable on regions where the probe curve is flat. On VCTK the WavLM and HuBERT scores at layers 12 to 21 are within about 0.05 of each other, which is why the Stage II peaks in Table 2 differ slightly from the analysis layers there. We note this in Section 7.
Thank you once again for your in-depth review and pointing our the flaws to help improve our paper.
Thanks the authors for providing thorough analyses, revising the original claims and previously problematic experimental settings. I genuinely appreciate the honesty in acknowledging these flaws. However, I think this response has pretty major revisions that may have already exceeded the scope of a "rebuttal". After dropping the first claim (low-rank and smooth accent subspace), and reinterpreting "speaker-generalizing" instead of "speaker-invariant", the remaining findings are thus more about the layerwise information distribution inside SSL/Whisper encoder models. The claimed "geometry" of the accent subspace has limited insight. Hence, I will keep my score and incline to reject this paper.
Response to Reviewer pfjG (Part 1/2)
Dear Reviewer,
Thank you for your detailed review. Your review lists two main findings. We have dropped our first claim, that accent occupies a smooth, low-rank subspace. The second, the depth result, holds, and we now call the probe speaker-generalizing.
W1. Physical meaning of the Stage I tests.
We added a definitions table with each statistic and its null (Appendix A, Table 4), and each test in Section 3.2 is now paired with a null or a control.
W2. The matrix V, and low rank against convexity.
Each row of V is a difference μj−μk between two accent centroids, one row per pair. Its row space is the space spanned by the centroid offsets, so its rank is at most K−1, and its rank says how many directions the accent differences use.
These two properties measure different things. Rank is about how many directions the centroids spread out in. Convex decomposability is about whether each centroid sits inside the region enclosed by the others. A triangle is a simple example. Its three corners span two dimensions, and the corners cannot be written as a weighted average of the other two. So a set of centroids can be low rank with every centroid on the outer edge, and the two results still fit together.
Our revision also makes this question simpler. When we compare the rank of V against a null that shuffles whole speakers between accent labels, no cell comes out lower than chance (p between 0.13 and 1.00), so we dropped the low-rank claim. For convexity, we now use a matched-spectrum null. It keeps the singular values of the centroid matrix and shuffles the layout, so low rank cannot give a positive result by itself. Simplex reconstruction does not beat this null on L2-ARCTIC or VCTK. On GLOBE, there is a small effect for WavLM and Whisper, but the three further checks we ran suggest it is not the mixture structure (Section 4.5).