acceptodds
Under review as a conference paper at ICLR 2027

From Decodability to Reusable Features: Mandarin Lexical Tone in Speech SSL Models

Abstract

Speech SSL representations can make a linguistic contrast decodable without organizing it into a stable, reusable feature. Mandarin lexical tone makes this distinction testable because T1–T4 form a closed inventory whose realization varies across speakers and syllables. We fit label-blind TopK sparse autoencoders (SAEs) to Mandarin HuBERT and wav2vec 2.0 states from 3,000 AISHELL-3 utterances. Selected features are tested on unseen initial–final types and across in- dependent dictionary fits. Full-state probes, rank-matched decoder-span and PCA controls, and interventions separately assess information availability, inventory re- covery and downstream influence. Tone is decodable at every evaluated layer, and many SAE features transfer and recur, but stable sparse localization is uneven and concentrated in T3. With 64 features, sparse SAE activations retain about 80% of the full four-tone probe gain, whereas the span of the same decoder directions retains 99–102%, matching the full-state endpoint and selected PCA directions. The selected SAE geometry supports the complete inventory, but the TopK ac- tivation map exposes it incompletely. SAE and PCA directions also influence a frozen downstream tone readout. The gap between decoder-span and TopK recov- ery supports a feature-absorption account in which tone evidence is distributed across context-conditioned activations rather than assigned to stable category co- ordinates. The study separates availability, factor-general localization, category coverage and downstream access.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Speech Self-Supervised Learning, Sparse Autoencoders, Mechanistic Interpretability, Lexical Tone, Representation Learning, Feature Absorption, Concept Localization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_e7CLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the organization of Mandarin lexical tone in self-supervised speech representations using label-blind TopK sparse autoencoders. Results show that tone-selective features frequently transfer across syllable types and recur across independent fits, but stable sparse localization is uneven and strongest for T3. While the 64 selected sparse features retain ~80% of the full-state above-chance macro-recall gain, their decoder span retains ~100%. This gap suggests that tone information is present in the selected representational geometry but is incompletely exposed by sparse activations, consistent with a feature-absorption account in which tone evidence is distributed across context-dependent features rather than reliably localized in individual sparse coordinates.

Critical strengths
  • Well-constructed Research Questions: they decompose multiple claims that are often conflated.
Critical weaknesses
  • The motivation for Mandarin as the linguistic target could be better justified. Although Mandarin is one of the most widely spoken languages, other tonal languages should still be tested to demonstrate novelty.
  • The terms “reusable feature” and “feature absorption” seem to be too strong for what is demonstrated. The experiment results do not support those phenomena at this point.
  • While this paper may be valuable for analyzing Mandarin lexical tones in SSL models, its current framing and contribution are not well aligned with ICLR’s focus. It would be better suited for domain-specific venues such as Interspeech.
Questions for authors
  • See Critical Weaknesses.
  • The current version of this paper omits several details, for example, (1) F0 in the main text and in the caption of Figure 1. (2) Mandarin phonological components such as “initial–final types”. Can the authors further clarify the definitions or provide examples of those missing parts for better clarity and readability?
6 comments
AuthorsOpenReview ↗

Rebuttal: Why Mandarin tone, and fit with ICLR

Dear Reviewer e7CL,

Thank you for asking why this case matters and whether it belongs at ICLR. Answering these questions helped us state the motivation and contributions more clearly in the revised Introduction. We hope the answers below address your concerns, and we would be grateful if you would reconsider your assessment.

W1. Why Mandarin, not other tone languages?

Mandarin is chosen because it makes the paper's question testable, not because it is widely spoken. The question is whether a contrast that a model can decode is also organized as a reusable sparse feature. Testing it needs three conditions, and Mandarin meets all three:

  • a closed inventory (T1–T4), which defines what complete recovery means;
  • established decodability, from prior probing (de la Fuente & Jurafsky, 2024; Shen et al., 2024) and our own full-state probes at every layer (macro recall 0.74–0.93, chance 0.25; Sec. 4.2);
  • large nuisance variation across speakers and syllable types, so that transfer can be tested on syllable types held out from fitting.

The paper's novelty lies in separating decodability from reusable sparse features, not in the number of languages. Prior probing shows that Mandarin tone is decodable, and speech SAE studies examine features for selected concepts (Parra, 2025; Mariotte et al., 2026; Aparin et al., 2026). We ask whether the features for all four tones transfer to unseen syllables, recur after refitting and together recover the inventory. They transfer and recur, but their sparse activations recover the four tones less well than the span of their own decoder directions, in all 36 strict comparisons. So decodability, a transferable feature and complete inventory recovery are different claims that need separate evidence.

Other tone languages would show how far the result extends. They would not change what the Mandarin case already shows, namely that decodability and reusable sparse localization can come apart in the same representation.

W2. Are "reusable feature" and "feature absorption" too strong?

No. Both terms are used in a narrowly defined sense, and the revision tests the second one directly. "Reusable feature" is the property the paper sets out to test, as its title indicates, and it asks for more than decodability. A feature must also transfer to unseen contexts and recur after refitting, and Sec. 3.2 tests both. "Feature absorption" was presented in the original submission as a hypothesis consistent with the recovery gap, not as a demonstrated mechanism.

  • Reusable feature. Sec. 3.2 defines it by two tests: a selected feature transfers to initial–final types excluded from fitting, and the same tone-selective function is rediscovered after independent refitting. Of 432 evaluations, 321 exceed test AUC 0.55, and 37 of 48 model–layer–tone cells stay above chance in all nine refits (Sec. 4.2). The paper also reports where the term stops applying. Only seven cells show an SAE-specific advantage over dense controls in every refit, five of them T3.

  • Feature absorption. It is the hypothesis that category evidence is carried by more specific, context-conditioned features rather than by one category-selective feature, as proposed for language-model SAEs (Chanin et al., 2025). The revision tests it directly (Sec. 4.5; App. Q). Tone-probe evidence is spread over many latents whose identity depends on syllable type, but these latents fire alongside the broad tone latent rather than replacing it. The result is closer to feature splitting than to absorption proper, and the Disscussion states it (p.9).

W3. Fit with ICLR

The paper is a representation-learning paper that uses Mandarin tone as a controlled test case. ICLR has published the core studies of what SAE features reveal about learned representations (Huben et al., ICLR 2024; Gao et al., ICLR 2025; Leask et al., ICLR 2025). We ask two of their questions for speech SSL models: whether SAE features are stable across refits, and what sparse activations, as opposed to decoder geometry, reveal about a representation.

The central finding concerns SAEs, not Mandarin phonology. The same SAE features expose less information through their activations than their own decoder directions contain (Sec. 4.2), which matters for how SAE results are read in any modality. The paper's primary area is unsupervised and self-supervised representation learning, a core ICLR topic.

Q2. Definitions and examples

The paper defines these terms with examples. Figure 1 shows the four tone contours, with an example syllable for each tone (妈 mā, 麻 má, 马 mǎ, 骂 mà). The Introduction defines F0 as fundamental frequency, the acoustic basis of pitch, and an initial–final type as the pinyin syllable without its tone digit. Methodology terms are defined in Sec. 3.2 (transfer, recurrence, selectivity advantage) and Sec. 3.3 (decoder span, retained above-chance gain).

Dear Reviewer e7CL,

We would appreciate it if you could take another look at our rebuttal, which we hope has clarified some of the misunderstandings underlying your concerns. We welcome your further feedback and hope you will reconsider your evaluation in light of our clarifications and additional evidence.

Thank you for your rebuttal.

W1 & W3. I respectfully disagree with the authors’ stance that “other tone languages would show how far the result extends.” Evidence from another tonal language could help clarify the interpretation of the findings, not just to test whether they generalize. Figure 3(b) shows that consistent SAE-specific selectivity is limited and concentrated in T3: 5 of the 7 model–layer–tone cells with recurrent selectivity advantage over dense controls across 9 refits correspond to T3. The newly provided Appendix Q further shows substantially higher hit rates for the selected broad T3 latent than for the corresponding latents of other tones at the two evaluated model–layer settings.

Therefore, I wonder if the strongest sparse-localization results reflect general properties of tone representations or the combination of acoustic cues associated with Mandarin T3. For example, in Cantonese or Vietnamese, where the relationships between pitch contours, phonation, and tone categories differ, would consistent sparse selectivity extend to categories without comparable phonation cues, or remain concentrated in particular acoustically distinctive categories? Separately, would the gap between sparse activations and their decoder span persist?

W2 & W3. I appreciate the additional diagnostics and the authors’ willingness to revise their interpretation. However, Section 4.5 and Appendices M and Q do not support feature absorption as the explanation for the recovery gap, and Appendix Q explicitly states that the analysis does not establish feature splitting as its mechanism either. The stage-wise analysis usefully locates approximately two-thirds of the gap between the selected encoder and decoder spans and approximately one-third at TopK gating, but it still leaves open why this allocation arises. Although transfer and recurrence support the authors’ operational definition of “reusable,” they should be distinguished from broad category coverage and consistent SAE-specific selectivity. The concentration of the latter in T3 limits the breadth of the positive findings, as discussed in W1 & W3.

Despite these clarifications and additional experiments, my main concerns remain unresolved. Therefore, I will retain my original score.

Reviewer_jtE5OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether Mandarin tone information in speech SSL models forms reusable sparse features. Using 108 TopK SAE fits on HuBERT and wav2vec 2.0, it evaluates transfer to unseen initial–final types, functional recurrence across refits, four-tone recovery, and effects on frozen downstream readouts. Tone is broadly decodable, but consistent SAE-specific selectivity is concentrated in T3. With 64 selected features, sparse activations retain approximately 80% of the full-state above-chance classification gain, whereas their decoder span retains approximately 99–102%. Both SAE and PCA directions influence downstream tone readouts.

Critical strengths

A meaningful evaluation protocol for reusable features. Held-out initial–final types, independent SAE refits, and dense controls move beyond illustrative examples to test generalization and functional recurrence systematically.

An informative decoder-span control. Comparing sparse activations with the span of the same selected decoder directions helps distinguish limitations of the selected geometry from limitations of the activation mapping. This is the strongest methodological contribution.

Critical weaknesses
  1. The mechanism underlying the recovery gap remains underdetermined. The decoder-span comparison simultaneously bypasses the encoder, ReLU, and TopK gating. Moreover, Table 4 reports better linear-probe performance on reconstruction than on the full SAE code, although reconstruction is a linear transformation of that code. Optimization, conditioning, and regularization therefore warrant closer examination before attributing the gap to feature absorption.
  2. The central recovery result has limited coverage. It is evaluated at two focused model–layer settings under a different partition protocol from the strict refit matrix. Its robustness across layers and SAE configurations remains less established than the broad transfer results.
Questions for authors

Clarification: What explains reconstruction outperforming the full SAE code under linear probing? Does the main recovery gap persist after representation-specific regularization tuning and convergence checks?

Mechanistic clarification: What evidence directly supports context-conditioned feature absorption beyond the observed activation-versus-subspace performance gap?

Robustness evidence: Does the recovery gap persist under strict dictionary-segment holdout and at layers with stronger transfer across all four tones?

2 comments
AuthorsOpenReview ↗

Rebuttal: Probe training, where the gap arises, and the strict-holdout gap

Dear Reviewer jtE5,

Thank you for calling the decoder-span control the paper's strongest methodological contribution. The control matters because it shows that the decoder directions of the selected features carry tone information, their own sparse activations do not expose. Your two questions go to where this gap comes from, and we added a stage-wise reading of the SAE and a direct test of absorption. We hope the answers below address your concerns and inform your final assessment.

W1 / Q1. Why reconstruction beats the full code, and whether probe training explains the gap?

The reconstruction contains no tone information that the code lacks; the code probe is harder to train. The code is the activations of all 4,096 features (at most 128 non-zero), and the reconstruction is the code passed through the decoder, a linear map. So any probe on the reconstruction can be rewritten as a probe on the code. Rewriting each trained reconstruction probe in this way reproduces its predictions exactly (logit differences < 8.6 × 10⁻⁵; App. N). The code can reach 0.849 at HuBERT L8 and 0.808 at wav2vec 2.0 L4, but a probe trained directly on it stops at 0.801 and 0.782. A likely reason is that the same weight decay penalizes a readout differently on 4,096 sparse inputs than on 2,304 dense ones.

Probe optimization, conditioning and regularization do not explain the main gap. Every readout in the main comparison reads 64 standardized inputs, so the size difference that handicaps the full-code probe is absent, and two views with the same information score the same (preactivations 0.821, encoder span 0.822; App. M). We also tuned weight decay (0, 0.001 or 0.01) and training length separately for each representation on validation data, then evaluated on held-out data with three readout seeds. The decoder span still beats the sparse activations in all 54 paired evaluations, with a mean gap of 0.104 at both layers, against 0.103 and 0.105 under the shared setting (App. O). Validation macro recall changed by at most 0.002 on average between 160 and 320 epochs, although validation loss was still improving at 320 epochs in 9 of 18 decoder-span fits.

W1 / Q2. What evidence supports feature absorption beyond the activation-vs-span gap?

The revised version now tests absorption directly (Sec. 4.5; App. Q). The test supports a context-dependent allocation of tone evidence and qualifies its mechanism. We first locate where the tone information is lost, keeping the SAE and its 64 selected features fixed and reading them out after each encoder step (Sec. 4.5; App. M).

Readout of the same 64 featuresHuBERT L8wav2vec 2.0 L4
Decoder directions (span)0.8920.870
Encoder directions (span)0.8220.805
Activations before ReLU0.8210.805
Activations after ReLU0.8170.800
Activations after TopK (as used)0.7890.765

The encoder directions already recover much less than the decoder directions, and the activations before ReLU match them. So about two thirds of the loss arises in the encoder's choice of directions. ReLU costs almost nothing, and TopK, which keeps only the 128 strongest features per syllable, costs the remaining third.

Feature absorption is a candidate explanation for this encoder-side loss, because a general tone feature stays silent in contexts where more specific features fire (Chanin et al., 2025). One result fits this. At HuBERT L8,a T2-versus-T4 contrast works for new speakers (AUC about 0.70) but falls to chance on new syllable types (about 0.51; App. I). The direct test examines the syllables on which the broad tone latent is silent. There, about ten latents, whose identity varies with syllable type, share half of the tone-probe evidence. Absorption predicts that these latents replace the broad latent. Instead, they fire about as often when it is active, and are no more often negatively aligned with its encoder than matched random latents. The pattern is closer to context-conditioned feature splitting than to consistent single-latent absorption, but it does not establish splitting as the mechanism behind the encoder–decoder gap (Discussion).

W2 / Q3. Does the gap hold under strict holdout and at all layers?

The gap holds at all six tested layers, including the one where all four tones transfer. Under strict holdout, the syllable types used for testing never enter SAE training, and features are reselected in every refit (Sec. 4.2; App. O).

Layers 4–9, both models (36 comparisons)Result
Decoder span beats the activations of the same 64 features36 / 36 (mean +0.113 macro recall)
PCA beats the activations36 / 36
Decoder span beats PCA34 / 36

The six layers include HuBERT L6, where all four tones exceed AUC 0.55 in every refit (Sec. 4.2).

Reviewer jtE5 - please indicate whether rebuttal addressed your concerns.

Reviewer_a9uzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether Mandarin lexical tone information in speech self-supervised learning models is represented by stable and reusable sparse features. The authors use sparse autoencoders (SAEs) to analyze HuBERT and wav2vec 2.0 representations, evaluating feature transfer, recurrence, tone-inventory recovery, and downstream interventions. The results suggest that tone information is broadly decodable, but its localization into reusable sparse features is limited, particularly outside T3.

Critical strengths
  1. The paper addresses an interesting question by distinguishing feature decodability from sparse localization and reusability. The research motivation is clear and provides useful insights into the interpretability of speech representations.

  2. The experimental design is relatively rigorous, including independent SAE refits, held-out syllable types, rank-matched PCA and decoder-span controls, and intervention experiments. The results support the main observations and provide a useful analysis of sparse versus distributed representations.

Critical weaknesses
  1. Limited generalizability. The main experiments focus on a single Mandarin corpus and two closely related pretrained speech models. The additional WavLM evaluation is limited, making it unclear whether the findings generalize to other architectures, languages, or speech conditions.

  2. Incomplete explanation of feature absorption. The decoder-span results are interesting, but the observed gap between sparse activations and dense subspaces may also arise from TopK gating, nonlinear activation, or encoder-decoder mismatch. The underlying mechanism is not fully isolated.

  3. Limited causal evidence. The intervention experiments mainly evaluate changes in frozen downstream readouts. Moreover, PCA directions often produce stronger effects than SAE directions, leaving the unique benefit of sparse features for causal control unclear.

Questions for authors
  1. Could the authors further distinguish feature absorption from information loss caused by TopK gating or SAE encoder-decoder mismatch?

  2. Would the main findings remain consistent under a combined speaker-disjoint and syllable-type-disjoint evaluation protocol?

  3. Could the authors extend the analysis to additional speech models or corpora to better establish the generality of the observed sparse-feature patterns?

  4. Given that PCA directions often produce stronger intervention effects, what additional advantages do SAE features provide for downstream interpretability or causal control?

1 comment
AuthorsOpenReview ↗

Stage-wise gap, joint holdout and SAE features beyond PCA

Dear Reviewer,

We appreciate your constructive suggestions on the joint speaker/syllable-type holdout and disentangling absorption from TopK gating. We have incorporated these new analyses into the revised manuscript and hope our responses below address your concerns.

W2 / Q1. Feature absorption vs. TopK gating and encoder–decoder mismatch

They sit at different levels of evidence and explanation. TopK, ReLU and the encoder–decoder difference show where the tone information is lost; absorption is a hypothesis about why. The submitted decoder-span control established the gap. To locate it, the revised version reads the same SAE at each stage, with its weights and the 64 selected features fixed (Sec. 4.5; App. M):

Readout of the same 64 featuresHuBERT L8wav2vec 2.0 L4
Decoder span0.8920.870
Encoder span0.8220.805
Activations before ReLU0.8210.805
Activations after ReLU0.8170.800
Activations after TopK (as used)0.7890.765

About two thirds of the gap lies between the decoder and encoder directions, one third in TopK, and almost none in ReLU. Absorption would explain the larger encoder-side share: a broad tone latent stays silent where more specific latents fire. Imperfect sparse inference by a linear encoder could produce the same asymmetry (O'Neill et al., 2025), so the revised version tests absorption directly (Sec. 4.5; App. Q). Tone-probe evidence is spread over many latents whose identity depends on syllable type; these latents fire at nearly the same rate whether the broad latent is active or silent and are no more often negatively aligned with its encoder than random latents, closer to feature splitting than to absorption.

W1 / Q3. Additional speech models and corpora

The paper already includes a third model, WavLM Base, which differs from the two primary encoders in objective and pretraining language (Sec. 4.4). Following your question, the revised version adds inventory recovery there, and the same gap appears (App. L). The decoder span reaches 0.673 and the full state 0.670, against 0.526 for the sparse activations and 0.623 for PCA. In the two primary models, the gap also holds at all six layers under strict refits (36/36; Sec. 4.2). The Limitations call for replication across further architectures, corpora and speech factors.

Q2. joint speaker and syllable-type-disjoint evaluation

Transfer persists in most cells when speakers and initial–final types are held out together, and it remains uneven across tones. The original submission tests the two factors separately (Table 4). Following your suggestion, the revised version adds the joint holdout (Sec. 4.4; App. P). Its test pool contains only speakers and initial–final types unseen in SAE fitting, feature selection and readout training (1,867 tokens, 45 speakers, 94 types). With three fresh SAEs per model, AUC exceeds 0.55 in 20 of 24 seed–tone evaluations, and in all three refits for five of eight model–tone cells. Mean test AUC across refits:

Joint holdoutT1T2T3T4
HuBERT L80.6900.6800.7200.527
wav2vec 2.0 L40.5590.6660.6270.603

The picture matches the submitted screen at these layers (App. F). Transfer exceeds chance in most cells, and HuBERT L8 T4 stays weak. This is a single joint split with fresh SAEs on smaller pools, so it supports the factor-specific conclusion rather than a fixed ranking of tones.

W3 / Q4. Limited causal evidence, and what SAE features add beyond PCA

The interventions test a deliberately narrow claim. We edit a direction at one layer and check whether the change reaches a frozen tone readout at layer 12 (Sec. 3.4). Both SAE and PCA directions change the tone readout, while the initial and final margins stay within the equivalence bounds (Sec. 4.3). This shows that the directions affect what later layers pass on. It does not show that listeners would hear a different tone, as the Limitations note.

PCA's larger effect is part of the finding. We do not claim that SAE directions are stronger. The paper treats downstream influence as a separate property, and this result shows that sparse features do not have it alone. Because PCA directions also move the tone readout, an intervention effect by itself cannot show that a sparse feature is the unit the model uses.

What SAE features add is a unit of analysis, not a stronger lever, and this is the paper's first contribution. Prior speech SAE work studies selected concepts; this paper tests every category of a closed inventory. A single SAE feature can be tested on its own for tone selectivity, transfer to unseen syllable types and recurrence after refitting. 321 of 432 such evaluations exceed AUC 0.55, and 37 of 48 cells stay above chance in all nine refits (Sec. 4.2). SAE features show where tone is localized; PCA shows how much a dense basis of the same size retains.

Reviewer_Rq32OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether Mandarin lexical tones are represented as stable and reusable sparse features in speech SSL models. Using SAEs on HuBERT and wav2vec 2.0, the authors evaluate feature transfer, recurrence, inventory recovery, and causal interventions. Results show that tone information is highly decodable, but sparse features do not consistently capture all four tones. The findings suggest that tone information is distributed across context-dependent activations.

Critical strengths
  1. The choice of Mandarin lexical tone, which contains only 4 ground-truth labels, is a good fit to track the general representations that are used for the actual compute process causally.
  2. The decoder-span comparison reveals that information loss may arise from sparse activation allocation rather than insufficient representation capacity.
Critical weaknesses
  1. In lines 64- 66, the paper says, ... Mandarin tone provides such a test. T1–T4 form a closed inventory with external labels, so failure to find a sparse coordinate cannot simply be attributed to missing tone information. ... I believe it is hard to derive such a conclusion from the given condition in this sentence. Decodability is not definitely guaranteed by the existence of ground-truth labels, but rather by actual results from probing. This sentence has leap in logic and thus somewhat misleading.
  2. The experiments do not clearly distinguish feature absorption from TopK gating or encoder–decoder mismatch.
  3. Functional recurrence does not necessarily imply that independently trained SAEs learn the same underlying representations.
  4. The experiments focus primarily on Mandarin speech and a specific SAE architecture.
Questions for authors
  1. Did the authors consider multilingual models to test generalization across languages? Although Mandarin is a good setting, which makes experiments more straighforward, but as the core question here is the general (causal) representations, I think testing this in the other language (e.g., if the adjacent represenation is also shown in the similar tone in the other languages) is desirable.
  2. Can the authors distinguish feature absorption from information loss caused by TopK gating, ReLU, or encoder–decoder mismatch?
  3. Do recurrent tone-selective features exhibit similar decoder directions and activation patterns across independently trained SAEs?
4 comments
AuthorsOpenReview ↗

Decodability premise, where the recovery gap arises, and a direct absorption test

Dear Reviewer Rq32,

Thank you for noting that Mandarin tone suits this question and that the decoder-span comparison separates how sparse activations allocate information from what the representation contains. Your questions led to two new analyses: a stage-wise reading of the SAE and a direct test of feature absorption. We hope the answers below address your concerns, and would be grateful if you would reconsider your rating.

W1. The sentence (lines 64–66): labels do not guarantee decodability

The premise of this sentence is decodability, not the labels. In the orginal submission, the preceding sentence sets that premise :"a setting in which category information is already known to be decodable", and the first paragraph cites prior probing evidence (de la Fuente & Jurafsky, 2024; Shen et al., 2024). Our full-state probes confirm it at every layer (macro recall 0.74–0.93, chance 0.25, Sec. 4.2). That sentence thus can read correctly on its own. In the revision, the sentence states the premise more clearly:"Because tone is already known to be decodable, a failure to find a sparse coordinate reflects how the information is encoded, not whether it is present."(p.2) The labels have a separate role, given in the next sentence: "The externally labeled T1–T4 inventory defines what complete recovery means."

W2 / Q2. Feature absorption vs. TopK, ReLU and encoder–decoder mismatch

They can be separated, because they work at different levels. TopK, ReLU and the encoder–decoder difference are where the tone information is lost, and absorption is a candidate account of why. To find where in the SAE the tone information is lost, the Revision reads the same SAE after each step, with its weights and the 64 selected features fixed (Sec. 4.5; App. M).

Readout (strict partitions)HuBERT L8wav2vec 2.0 L4
Selected 64, preactivation0.8210.805
Selected 64, after ReLU (no TopK)0.8170.800
Selected 64, after TopK0.7890.765
Encoder span (rank 64)0.8220.805
Decoder span (rank 64)0.8920.870

Since the preactivations match the encoder span, the gap splits into three parts. About two thirds lies between the encoder and decoder directions, about one third in TopK, and almost none in ReLU.

Absorption could explain the encoder–decoder loss, about two thirds of the gap, because a broad latent stays silent where more specific latents fire (Chanin et al., 2025). Tone features do depend on syllable context. At HuBERT L8, a T2-versus-T4 contrast separates the tones across speakers (AUC 0.70) but not on held-out initial–final types (0.51; App. I). Our direct test (Sec. 4.5; App. Q) finds that where the broad latent is silent, about ten latents, whose identity varies with syllable type, share half of the tone-probe evidence. Absorption predicts that they replace the broad latent. Instead, they fire about as often when it is active, and are no more often negatively aligned with its encoder than matched random latents. The pattern is closer to context-conditioned feature splitting than to consistent single-latent absorption, but it does not establish splitting as the mechanism behind the encoder–decoder gap (Discussion, p. 9).

W3 / Q3. Do recurrent features share decoder directions and activation patterns?

Yes, closely, though not identically. For the recurrent T3 feature, mean decoder cosine across independently fitted dictionaries is 0.62 (HuBERT) and 0.68 (wav2vec 2.0), where two random directions give about 0, and activation correlation is 0.72 and 0.76 (App. I). The paper does not claim that refitted SAEs learn the same representation. Recurrence is defined functionally as rediscovery of the same tone-selective function after refitting (Sec. 3.2). SAEs refitted on the same activations learn different features across seeds, so identical directions would be the wrong test.

W4 / Q1. Other languages, multilingual models and SAE architecture

Our paper already includes a model with a different pretraining language: WavLM Base, pretrained on English only and evaluated on Mandarin. The revised version adds inventory recovery at the same layer (App. L). As in the two Chinese checkpoints, the decoder span (0.673) matches the full state (0.670), while the sparse activations fall short (0.526, below PCA at 0.623). Other tone languages are future work.

On SAE architecture, all dictionaries are TopK SAEs, which fix the number of active features and allow matched comparisons across all 108 dictionaries. Other common architectures differ mainly in how they enforce sparsity. The stage-wise analysis shows that about two thirds of the gap arises in the encoder directions, before the sparsity step acts, and most of the gap is not produced by TopK itself, so changing the sparsity mechanism alone is unlikely to remove most of the gap.

Reviewer Rq32,

We respectfully encourage you to revisit our rebuttal, as some of your concerns may stem from misunderstandings that we have attempted to clarify. We would greatly appreciate your feedback on our responses and reconsideration of your assessment in light of the additional evidence provided.

Reviewer Rq32 - can you please confirm whether you read the rebuttal and indicate whether it addressed your concerns or not?

I appreciate the authors' addressing the concerns well, so I raise the score.

Reviewer_4x9TOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how Mandarin lexical tones are represented inside self-supervised speech models, asking whether the four tones are encoded as distinct, reusable features or distributed across many internal representations. To examine this, the authors use sparse autoencoders (SAEs) to identify individual features associated with tone and test whether these features remain consistent across different speech contexts, model layers, and independent training runs.

The study finds that Mandarin tones can be reliably decoded from the models' internal representations, and that some sparse features consistently carry tone-related information. However, these features do not provide a clear advantage over conventional dense representations, such as principal component analysis (PCA). A small collection of sparse features preserves much of the tone information, but the corresponding dense feature directions preserve substantially more. Experiments that manipulate these representations also show that sparse features do not have a unique causal advantage.

The paper's central contribution is a systematic distinction between information being decodable and being represented as a stable, reusable sparse feature. Its findings suggest that Mandarin tone is encoded primarily through distributed, context-dependent representations rather than through dedicated internal features for each tone category. This challenges the assumption that linguistically meaningful categories necessarily correspond to distinct, interpretable features within speech models.

Critical strengths
  1. The decoder-span vs TopK-activation dissociation (Table 3; Fig 4b) is the paper's real payload and it is consequential: it shows that a negative SAE interpretability result ("tone doesn't live in clean sparse features") can coexist with the information being fully present in the SAE's own direction geometry. That cleanly separates "the dictionary lacks the directions" from "the sparse gate mis-allocates," which matters for anyone drawing representational conclusions from SAE activations in speech.

  2. Rare methodological discipline for an interpretability paper: the dense PCA/random controls, factor-held-out transfer, and the finding that PCA steering matches or exceeds SAE steering (Fig 5) actively argue against the authors' own tools being special. This credibility makes the "distinct claims need distinct evidence" principle land.

Critical weaknesses
  1. Narrow empirical base for a general "evaluation principle." The primary matrix is two checkpoints that share language, pretraining data, depth and width (differing mainly in objective), one tone language, and one read-speech corpus (AISHELL-3). The only architecture-diversifying evidence is a single-layer, single-split WavLM control, and it already shifts allocation (T2 near chance; 4/12 SAE-advantage cells). So the paper's own generalization evidence is fragile, and whether the central span-vs-activation dissociation even replicates on WavLM is never shown. This caps how strongly the methodological lesson can be claimed as general rather than a controlled Mandarin case (conceded in Limitations, to their credit, but the limitation stands).

  2. The headline dissociation is demonstrated under a relaxed holdout. Inventory-recovery and intervention case studies drop the strict dictionary-segment refit (Table 6), so the result most readers will cite rests on a protocol where test segments can appear in label-free fitting, a weaker guarantee than the paper's own strict matrix, precisely where robustness matters most.

Questions for authors
  1. Does the 80%-activation / 99%-span dissociation (Table 3) survive under the strict dictionary-segment refit rather than the focused segment-selection protocol? Could label-free fitting on test segments inflate decoder-span recovery?

  2. Does that same dissociation replicate on the WavLM control, or only on the two matched Tencent checkpoints? If it does not, the feature-absorption conclusion's generality is materially weaker.

  3. Syllable-level 3-bin pooling discards fine F0-contour dynamics that define contour tones; does a higher-temporal-resolution SAE change sparse localization for T2/T4 (the categories nearest chance), or is the 5-bin control (App. J) sufficient to rule this out?

Future-work (suggestion, not required here): additional tone languages/corpora and architectures under the strict refit; a direct mechanistic test (rather than an interpretation) of absorption.

2 comments
AuthorsOpenReview ↗

Narrowed claim, strict refit, WavLM and a direct absorption test

Dear Reviewer 4x9T,

Thank you for singling out the decoder-span dissociation and for asking whether it survives stricter refits. They are now in the revised version. We hope the answers address your concerns and inform your assessment.

W1. A narrow empirical base for a general evaluation principle

The revision narrows contribution (iii) to a claim about how to read SAE evidence, and one well-controlled case is enough to support that claim. The submission called it "a general evaluation principle for speech SSL". The revision states it more narrowly, as "Interpretation of SAE evidence": "Decodability, transfer, recurrence, inventory recovery and downstream influence answer different questions; evidence for one does not establish the others." It does not claim that these properties always come apart, only that having one does not guarantee the others. For example, tone is decodable at every evaluated layer (macro recall 0.74–0.93; Sec. 4.2), but the sparse activations of the selected features recover less of the four tones than the span of their own decoder directions. Decodability does not guarantee inventory recovery.

The case itself also rests on more than one model pair. The dissociation holds in all three encoders (App. L) and, in the two Chinese checkpoints, at all six layers under the strict refit (36/36; App. O). WavLM shifts which tones receive reusable features but keeps the gap. The allocation varies across models while the dissociation holds, which is why localization has to be tested in each model. We do not claim that the Mandarin findings generalize. The Limitations call for replication across further architectures, corpora and speech factors.

W2 / Q1. Does the dissociation survive the strict dictionary-segment refit?

It does, and the decoder-span scores barely change. In the focused analysis of the original submission, test segment types were already excluded from labeled feature selection and readout training; they could appear only in label-free SAE fitting. To rule out any inflation of the span, we repeated the comparison under the strict refit, which also excludes test segment types from SAE fitting and reselects features in each refit (Sec. 4.2; App. O). Under the strict refit, the span still reaches 0.892 at HuBERT L8 and 0.870 at wav2vec 2.0 L4, so label-free fitting on test segments did not inflate it. The same pattern holds at every layer from 4 to 9.

Strict refit, layers 4–9, both models (36 comparisons)Result
Decoder span > sparse activations of the same 64 features36 / 36 (mean +0.113 macro recall)
PCA > sparse activations36 / 36
Decoder span > PCA34 / 36

The retention figures (about 80% of the full-state gain for the sparse activations, 99% for the decoder span) come from the focused analysis. Since the strict refit leaves the span unchanged, label-free fitting did not inflate them. The revised Sec. 3.3 also describes the focused partition explicitly. The intervention result does not depend on this partition either. SAE and PCA are fitted on the same label-free pool, so any overlap would affect both alike, and PCA still has the larger effect for every tested tone (Sec. 4.3).

Q2. Does the same dissociation replicate on WavLM?

It replicates, with a larger gap than in the two primary encoders. WavLM Base, pretrained on English LibriSpeech, was in the submission as a transfer control (Sec. 4.4). The revised version adds the inventory-recovery comparison at the same layer (App. L). The decoder span matches the full state, while the sparse activations fall 0.147 below the span. So activation–span gap appears in all three models.

Q3. Is the five-bin control sufficient for T2/T4?

For pooled syllable representations, it is, and both relevant controls were in the submission. Across three-bin, five-bin and voiced-span pooling, T3 AUC is 0.797/0.777/0.764 for HuBERT and 0.747/0.719/0.733 for wav2vec 2.0, while T2 and T4 stay between 0.503 and 0.521 (App. R). The same three-bin pooling supports all four tones at HuBERT L6, where every tone exceeds AUC 0.55 in all nine refits (Sec. 4.2), so pooling is not what limits T2/T4. A frame-level SAE would change the unit of analysis from syllables to frames and would test a different question.

Future-work: a direct mechanistic test

The revision tests absorption directly (Sec. 4.5; App. Q). Where the broad tone latent is silent, about ten latents that vary with syllable type share half of the tone-probe evidence. Absorption predicts that they replace it, but they fire about as often when it is active and are no more often negatively aligned with its encoder than matched random latents. The pattern is closer to context-conditioned feature splitting than to consistent single-latent absorption, but splitting is not established as the mechanism. The decoder-span dissociation does not depend on either account.

Reviewer 4x9T - could you please confirm that you read the authors rebuttal and indicate whether it addressed your concerns?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview for the AC and reviewers

We thank the AC and all five reviewers for their constructive feedback, which has strengthened the paper. We have uploaded the revised PDF, with all revisions highlighted in blue.

Summary of the paper.

The paper addresses an open question in SAE interpretability: what can be inferred about a model from the features an SAE exposes, or fails to expose? SAE features need not be canonical units of analysis, and audio SAE work is still establishing their stability and use. Using the closed inventory of four Mandarin tones, the paper shows in a controlled comparison that the same selected SAE features recover the tones clearly better through their decoder directions than through their sparse activations, in all 36 strict comparisons, with most of the loss arising in the encoder. So an apparent negative result, a tone without a clean sparse feature, becomes informative. The information is present in the dictionary's directions, and the activations fail to expose it. The paper states which claims the evidence supports and where the usual reading of SAE activations breaks down.

The reviewers credit the methodological rigor of the study

4x9T calls the decoder-span dissociation "the paper's real payload" and notes "rare methodological discipline for an interpretability paper." jtE5 calls the decoder-span control "the strongest methodological contribution" within a "meaningful evaluation protocol." Rq32 finds Mandarin tones "a good fit" for causal tracking and notes the decoder-span comparison "isolates sparse activation allocation." a9uz finds the design "relatively rigorous", and e7CL notes that the research questions "decompose multiple claims that are often conflated."

Main concerns and new evidence

  1. Does the gap survive stricter evaluation? (4x9T, jtE5) Under the strict refit, which keeps test syllable types out of SAE fitting, the decoder span beats the sparse activations in all 36 comparisons across layers 4–9 in both models (Sec. 4.2; App. O). With weight decay and training length tuned separately for each readout, the gap persists in all 54 paired evaluations (App. O, Table 24).
  2. Where does the gap arise? (Rq32, a9uz, jtE5) Reading the same SAE after each step places about two thirds of the gap between the encoder and decoder directions and about one third in TopK, with almost none in ReLU (Sec. 4.5; App. M).
  3. Is it feature absorption? (all reviewers) A direct test finds that where the broad tone latent is silent, about ten latents that vary with syllable type share half of the tone-probe evidence, but they fire about as often when it is active. The pattern is closer to context-conditioned feature splitting than to consistent single-latent absorption (Sec. 4.5; App. Q; Discussion).
  4. Does it hold beyond two similar models? (4x9T, Rq32, a9uz) WavLM Base, pretrained on English, shows the same gap. Its decoder span matches the full state while its sparse activations fall 0.147 below the span (Sec. 4.4; App. L).
  5. Does transfer survive a joint holdout? (a9uz) With speakers and syllable types held out together, AUC exceeds 0.55 in 20 of 24 seed–tone evaluations (Sec. 4.4; App. P).

Clarifications by section

  • Abstract. Reports the decoder-span comparison and where the gap arises.
  • Introduction. States the decodability premise directly (Rq32), explains why Mandarin offers a direct test and defines F0 and initial–final types with examples (e7CL), and restates contribution (iii) as "Interpretation of SAE evidence" (4x9T).
  • Method (Sec. 3.3). Explains how the decoder-span comparison holds features and dimensionality fixed, states that test syllable types can enter label-free SAE fitting under the focused protocol, and adds the strict-holdout comparison across layers 4–9 and the affine-equivalence check for reconstruction and code probes (4x9T, jtE5).
  • Discussion. Treats absorption and feature splitting as tested accounts (all reviewers).
  • Appendices. New App. L–Q report the WavLM, stage-wise, affine-equivalence, strict-holdout, joint-holdout and absorption analyses.

Replies to each reviewer follow in their threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.