acceptodds
Under review as a conference paper at ICLR 2027

Phonological Interference in Multilingual Speech Models

29 views

Abstract

Phoneme-level speech models recognize or generate speech as a sequence of phonemes, the smallest sound units that distinguish words. These models enable fine-grained pronunciation control and understanding, yet they often fail on input that does not match any single training language, such as speech alternating between two languages, known as code-switching, or speech in low-resource languages absent from training. We identify a systematic failure mode behind this, *phonological interference*: models assume the input is in a single language and impose its phonology, overriding local phoneme-level decisions that conflict with it. We measure interference by how often a model retains phonemes that one language has but the other lacks. On code-switched input, two phone recognizers (speech-to-phoneme models) and a phoneme-conditioned text-to-speech model lose 32%-79% of these phonemes, but lose far fewer of the phonemes both languages share. On unseen languages, recognizers impose the phonology of the training language they assign to the speech: the more confident the assignment, the more they lose phonemes the unseen language has but the assigned language lacks. We probe the models' language estimate from their internal activations, and trace interference to a low dimensional subspace. On monolingual speech, steering this subspace toward a different language makes the model lose the phonemes that only the original language uses and produce those that only the target language has. We introduce *windowed language estimation* (WLE), an inference time repair that replaces the model's language estimate in this subspace with one computed from a short window around each position. On code-switched input, WLE removes 34%-58% of the interference in all three models, and in the phone recognizers it leaves monolingual performance essentially unchanged.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, phone recognition, text-to-speech

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HZCyOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies phonological interference: multilingual phoneme-level models infer one language from the whole input and impose its phonology, dropping phonemes the other language lacks. The authors measure this via unshared phoneme recall on code-switched and unseen-language speech, trace the cause to a low-dimensional language subspace whose causal editing reproduces the failure, and propose WLE, an inference-time repair that replaces subspace coordinates with estimates from a short local window. WLE removes 34–58% of interference (69% for TTS with span windows) at almost no cost to monolingual performance.

Critical strengths
  1. Complete causal story: the subspace is built from training centroids, editing it reproduces the failure, and three controls (random subspace, erasure, third-language edit) rule out trivial explanations. Careful interpretability work.
  2. WLE is practical—no language labels, no external LID, no retraining—and beats the naive sliding window, which costs 1–3 pp PER while WLE stays within ±0.35 pp.
  3. The unshared/shared recall contrast cleanly separates interference from general accuracy drops, and the phenomenon is shown in recognition, synthesis, and linked to Whisper's language confusion, broadening impact.
Critical weaknesses
  1. The unseen-language evidence is purely observational: recordings are compared by probe confidence, with no intervention, and WLE is not applied there—precisely the setting the paper motivates as central for universal phone recognizers.
  2. Narrow code-switching setting: all utterances switch exactly once, both languages are seen in training, SwitchLingua has only 288 utterances, and Spliced FLEURS uses spliced audio with unnatural boundaries.
  3. References are derived from orthographic text via eSpeak NG; G2P noise may contaminate the unshared-phoneme judgments the metric relies on. This is not analyzed.
  4. Hand-picked hyperparameters (edit layers 6–8/14–18/14–22, 2 s window) with little sensitivity guidance; WLE's double encoding cost is never quantified.
Questions for authors
  1. What is the runtime/memory overhead of the second windowed encoding pass?
  2. Why does WLE help POWSM less (36%) than PhoneticXeus (58%)—the <unk> tag, CTC head, or encoder architecture?
  3. Can the centroids/subspace be extended to unseen languages, e.g., by interpolating from the nearest training languages? This seems the natural next step.
2 comments
AuthorsOpenReview ↗

Response to reviewer HZCy

We thank the reviewer for the careful reading and the positive evaluation of our causal argument. We address each point below.

W1. The unseen-language evidence is observational, and WLE is not applied there

We refer the reviewer to the global response.

W2. Narrow code-switching setting

We acknowledge this in the limitations and have expanded the discussion. The single-switch setting is deliberate: the SEP/TOG design needs a span that can be processed with and without the other language.

W3. G2P noise in eSpeak NG references

While G2P may include errors, our findings do not depend on G2P accuracy:

  • Some of our evaluations are reference-free, e.g., the phone-recognition steering experiment in Section 3, which does not use G2P conversion.
  • The DoReCo evaluation set has human time-aligned phonemic transcriptions.
  • SEP and TOG are scored against the same reference for the same span, so a G2P error at a position affects both conditions equally.

W4 and Q1. Hyperparameters and WLE overhead

Hyperparameters. Rank 15 is the full span of the 16 centroids. WLE reduces interference at every window width from 1 to 4 s, with small monolingual cost at every width (Appendix H), and 2 s was simply the middle choice. We now summarize this in the main text.

Overhead. WLE encodes the audio a second time in 2 s windows, which are independent and are encoded in one batch, and the decoder runs once as usual. On both recognizers this takes about 3× the unmodified encoder's time at any utterance length, and adds at most 11% to peak GPU memory.

Q2. Why does WLE help POWSM less than PhoneticXeus?

A plausible cause is that POWSM is trained to generate a language tag, which may spread language identity across more directions than the centroid span. Finding which architectural and data properties affect interference is an interesting direction for future work, which we discuss in the limitations section.

Q3. Extending the centroids to unseen languages

We agree this is the natural next step. Table 16 shows that the basis must contain the pair's languages to repair code-switching. This suggests that for an unseen language, a target centroid estimated from the nearest training centroids, weighted by the probe's scores, could be substituted.

AuthorsOpenReview ↗

Follow-up to Reviewer HZCy

We thank the reviewer again for the careful reading and positive evaluation. With a few days left in the discussion period, we would like to ask whether our response addresses the reviewer's questions, including the WLE overhead, the window width sensitivity, and G2P noise.

We are happy to answer any remaining questions.

Reviewer_ZevdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors analyze phonological interference in three multilingual speech models (two phoneme recognizers and one TTS) when the input contains code switch (utterances in two languages) and speech from a language unknown to the models (TTS model not evaluated). The authors propose a method, windowed language estimation (WLE), that estimates a language representation on short windows, and introduce that information in the recognition models/TTS during inference to reduce phonological interference (phoneme substitution, mainly) during code-switching. The idea is technically sound and justified although its application in more commonly used or advanced models, including some LLMs, would have had a higher impact.

Critical strengths
  • Many current speech models do not work well on code-switching and this idea could potentially improve their performance.
  • The hypotheses are sound, well-grounded (in general), and are supported by the results.
Critical weaknesses
  • The application of this idea in more commonly used or advanced models, including some LLMs would have had a higher impact. The authors only evaluate three systems that might not be representative of the state of the art.
  • It is unclear how this method could help decode unseen languages. A more sophisticated method that considers not only the phone set but also similarities in phonotactics between the unknown language and those already known by the models might be needed.
Questions for authors

Some sentences that are crucial to understand the methods, are ambiguous. Could you please develop the meaning of this sentence? “We apply the interference metric (Section 2) to both code-switched datasets. On SwitchLingua, SEP processes each span alone. On Spliced FLEURS, SEP replaces the other-language half with a same-language half, so SEP and TOG differ only in the other half’s language, with length and speaker change held constant.”

What is the language probe accuracy for the two recognizers? The authors say “In both recognizers, accuracy plateaus by mid-network and barely changes within one or two layers of our choice. It stays high even when the probe has never seen the target language’s corpus, so it is not picking up recording artifacts”, but they don’t specify what the accuracy is.

The usefulness of WLE when used in models that have never seen the input language is unclear to me. The main text is not clear on that. If WLE does not reduce the measured interference on DoReCo, the authors should clearly state it and propose possible solutions.

In the manuscript, the authors never mention the potential influence of language phonotactics in the results. The models evaluated, POWSM (Li et al.,2026) and PhoneticXeus (Bharadwaj et al., 2026a), and the TTS, use a relatively long context from the input to predict a certain phoneme at time t. We could say that those models have learned the phonotactics of each language, and once they decode for that language, the predicted phoneme does not only depend on the input frames at that time but also on the surrounding phonetic content (see Feng et al 2021 reference included below). Reading the manuscript, one might say that the authors suggest that once a model focuses on a language, it has a phonetic inventory preference, but they do not mention that it also has learned phonotactic expectations which could be part of the cause of the phonetic interference. This could be especially important when trying to propose an adapted method to use WLE with unknown languages.

Feng, Siyuan, et al. "How phonotactics affect multilingual and zero-shot ASR performance." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.

4 comments
AuthorsOpenReview ↗

Response to reviewer Zevd

We thank the reviewer for the positive assessment and for the Feng et al. (2021) reference, which we have added to the discussion. We address each point below.

W1. More commonly used or advanced models, including LLMs

State-of-the-art audio LLMs commonly use Whisper encoders as their audio backbone (e.g., Qwen-Audio, Qwen2-Audio, SALMONN), and LLM-based TTS models generate speech tokens with a text LLM. Our evaluated models cover both designs: POWSM is a Whisper-style model, and our TTS model, CosyVoice3, is built on a Qwen2 LLM.

While we focus on state-of-the-art phoneme-level models, since phoneme output is needed to measure interference at the fine-grained level we define, we also applied the same probe and steering to Whisper large-v3, a standard ASR model, in Appendix F. Its encoder's language estimate shifts when another language is in the context, as in the phone recognizers, and steering its language subspace moves the language Whisper picks and the language of its transcript toward the target. Since this is the encoder that audio LLMs build on, we expect interference to propagate to them, and we leave a full investigation to future work (now noted in our limitations, lines 520–522).

W2 and Q3. Unseen languages

We refer the reviewer to the global response.

Q1. The SEP/TOG sentence in Section 2.2

The SEP/TOG comparison was designed to compare the effect of having two languages processed together vs. in isolation. Original code-switched utterances consist of recordings in language A followed by language B (A–B; TOG condition).

  • On SwitchLingua, the SEP condition processes a single language's material in isolation (A).
  • On Spliced FLEURS, SEP instead processes respliced audio of the form A′–A (where A′ is from a different speaker). This controls for recording length and rules out shorter recordings being the cause of interference.

Q2. Probe accuracy for the two recognizers

We refer the reviewer to Figure 6 (Appendix D), which shows the probe score of the true language at every layer. The phone recognizers reach 0.99 at layer 8 of POWSM and layer 17 of PhoneticXeus. We now report these numbers explicitly in the main text.

Q4. Phonotactics

We thank the reviewer for the insightful point, and agree that phonotactics may be another cause of interference. Our metric measures interference through the phoneme inventory, and our experiments do not separate the two. We now state this in the limitations (Section 6), citing Feng et al. (2021), and leave disentangling the contributions of the phoneme inventory and phonotactics to future work.

AuthorsOpenReview ↗

Follow-up to Reviewer Zevd

We thank the reviewer again for the positive assessment. Since only a few days are left in the discussion period, we would like to ask whether our response addresses the reviewer's questions.

To summarize, we discussed how the evaluated models relate to audio LLMs and the Whisper large-v3 analysis, clarified the SEP/TOG design in Section 2.2, now report the probe accuracies in the main text, clarify in the main text that WLE does not reduce interference on unseen languages, and discuss phonotactics in the limitations.

We are happy to answer any remaining questions. If the response resolves the reviewer's concerns, we would appreciate it if the reviewer would consider updating the score.

Thanks for your responses, which clarify my questions. However, regarding Q1 and Q2, any reader might have the same concerns as I did. Have the authors taken any action to make sure the paper is clear about these two points?

AuthorsOpenReview ↗

Changes in the revised PDF for Q1 and Q2

We thank the reviewer for the comment and engagement. We addressed both points (Q1 and Q2) in the revised version we uploaded, with prominent revisions marked in red. For Q1, we replaced the sentence the reviewer quoted in Section 2.2 (L211–238). We now explain that the same span is scored with and without the other language, give an example in a German–English code-switched pair, and state that on Spliced FLEURS, SEP replaces the other-language half with a same-language half from a different speaker and sentence which controls for length and speaker change. For Q2, Section 2.1 (L187–189) we now report that on held-out speakers the probe assigns the correct language a score of 0.99 at layer 8 of POWSM and layer 17 of PhoneticXeus (full layer sweep is in Appendix D).

Reviewer_YcoWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates phonological interference in multilingual speech models. The authors evaluate this phenomenon with shared and unshared phoneme recall on two code-switching datasets, SwitchLingua and Spliced FLEURS (synthesized), as well as unseen languages. The authors further identify a low-dimensional language subspace and demonstrate the steering results. Given above findings, the paper proposes WLE, an inference-time intervention that uses local language representations to reduce interference. Experiments on phone recognition and speech synthesis models show that WLE mitigates interference on code-switched speech while largely preserving monolingual recognition performance.

Critical strengths
  1. The paper identifies systematic phoneme-level errors in code-switched inputs, providing a fine-grained analysis of language confusion.

  2. The evaluation complements SwitchLingua with Spliced FLEURS synthesized from monolingual clips. The latter provides a controlled setting for further analysis.

  3. The paper proposes WLE to mitigate interference at inference time. The experiments demonstrate consistent improvements on both code-switching datasets while largely preserving monolingual recognition performance.

Critical weaknesses
  1. The authors claim the existence of the language subspace. However, the interpretation of the function of the language subspace is less convincing. Although multiple text LLMs are referred in Line 085-086, the analogy to the phoneme representation in speech models seems to be not proper. According to recent findings in [1, 2], the language subspace is based on the assumption of the existence of the language-agnostic semantic space. LLMs therefore leverage a low-dimensional subspace to control the language identity. In the case of phoneme representations, I highly doubt the authors' claim. Given that different languages inherently have different phoneme sets, I am not sure that speech models actually need a dedicated subspace for language identification, as has been observed in text LLMs. The results in Figure 5(b) further support that these dimensions serve other functions. A proper explanation is that this is a subspace from which language identity can be decoded, rather than one dedicated to language identification like in text LLMs.

[1] Tang et al. Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models. ACL 2024. [2] Zhong et al. Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models. EACL 2026

  1. I'm a bit confused about the evaluation. If I understand correctly, the paper measures phonological interference through the relative recall degradation of shared and unshared phonemes. However, shared phonemes can remain correctly predicted regardless of which language the model internally favors. Therefore, a smaller recall drop for shared phonemes is expected and does not independently establish phonological interference.

  2. The effectiveness of WLE on unseen languages seems to be limited. Although unseen language is presented as another major phenomenon, Appendix C reports that WLE does not reduce interference on DoReCo. I feel that the authors need to narrow down the scope of the paper.

  3. The phone recognizers are trained on monolingual speech, making code-switching an out-of-distribution setting. I wonder whether there are already models enhanced to address the code-switching issue, and to what extent this issue has been addressed in such models.

Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer YcoW

We thank the reviewer for the detailed feedback, and address the concerns below.

W1. On the interpretation of the language subspace, and its purpose

While our empirical findings (Section 3, Figure 5(b), Table 9) show the existence of a subspace controlling the output phoneme inventory, the reviewer is concerned about our "interpretation of the function of the language subspace". We agree that speech models may not represent "language-agnostic semantic[s]" as do LLMs, and our comparison to language confusion in LLMs (lines 073–075 and 086–090) does not make this claim.

Rather, we empirically find that steering this subspace toward another language on monolingual speech, with the audio unchanged, makes the phone recognizers lose 86–88% of the phonemes only the original language uses and produce phonemes only the target language has. Shared phonemes drop only a quarter to a third as much, and the same shift through a random subspace of equal rank has almost no effect (Δ\Delta 0.03–0.06 vs. 0.35–0.78). That is, we find confusion on a phonological rather than semantic level, motivating the name of the phenomenon (phonological interference). This also goes beyond a subspace from which language identity can merely be decoded: it causally affects the output.

We agree that shared phonemes are also affected (Figure 5(b)), so we do not claim the subspace is dedicated to language identity alone. We have made this point clearer in our revision, which now cites [1, 2] in the introduction (lines 088–090) as work showing that LLM language subspaces can likewise be steered to change the output language.

[1] Tang et al. Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models. ACL 2024.

[2] Zhong et al. Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models. EACL 2026.

W2. How interference is established

The reviewer notes that "a smaller recall drop for shared phonemes is expected and does not independently establish phonological interference". We agree that it is expected under interference, and this is exactly why we use it. Shared phoneme recall is our control (lines 127–129): a general loss of accuracy on out-of-distribution input would lower shared and unshared phonemes alike, whereas interference predicts the selective drop we observe (unshared recall drops 1.9–8.2× as much as shared recall, Table 5).

We also do not rely on this contrast alone:

  1. Most lost unshared phonemes are replaced by phonemes of the other language (65–78% on SwitchLingua, Section 2.3).
  2. The same phoneme loses more recall next to a language that lacks it than next to one that has it, in both recognizers (Table 7), which rules out unshared phonemes simply being harder.
  3. Steering the language subspace on monolingual speech reproduces the effect (Section 3).

We have clarified the evaluation design in the revision (Section 2.2, lines 211–238).

W3. WLE does not help on unseen languages

We refer the reviewer to the global response.

W4. Are there models already enhanced for code-switching, and how far do they address this?

We chose POWSM and PhoneticXeus because they are the state of the art in multilingual phone recognition with public checkpoints, and we are not aware of a public phoneme-level model designed for code-switching.

At the word level, prior work addresses code-switching in two ways. One is training on real or synthesized code-switched speech [1]. The other is model design that needs no code-switched training data, such as zero-shot code-switched ASR that combines monolingual modules [3]. Even so, [2] finds that language confusion remains a key challenge in code-switched ASR. These methods are evaluated with word-level metrics, so whether they reduce phoneme-level interference is open.

We thank the reviewer for the suggestion. We have added this to the limitations (lines 520–522) and will extend the related work with this line of research.

[1] Lee et al. UniCoM: A Universal Code-Switching Speech Generator. Findings of EMNLP 2025.

[2] Liu et al. Code-Switching Speech Recognition under the Lens: Model- and Data-Centric Perspectives. IEEE TASLP 2026.

[3] Yan et al. Towards Zero-Shot Code-Switched Speech Recognition. ICASSP 2023.

AuthorsOpenReview ↗

Follow-up to Reviewer YcoW

We thank the reviewer again for the detailed feedback. Since only a few days are left in the discussion period, we would like to ask whether our response addresses the reviewer's concerns.

To summarize, the revision clarifies that we claim the language subspace contains the model's language estimate and we show in our experiments that it causally controls the phoneme inventory the model applies, not one used exclusively for language identification. Shared phoneme recall is a control for a general loss of accuracy, and interference is also supported by the replacement analysis in Section 2.3 and steering on monolingual speech in Section 3. The main text now clarifies that WLE is a repair for code-switched speech and does not reduce interference on unseen languages.

We are happy to answer any remaining questions. If the response resolves the reviewer's concerns, we would appreciate it if the reviewer would consider updating the score.

I thank the authors for their response, which has addressed many of my concerns. Although I remain unsatisfied with the explanation of the subspace analysis like Figure 5(b), I will raise my score to 3 accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response to all reviewers

Global Response

We thank the reviewers for their careful reading. We are encouraged that they found the phenomenon systematic and the analysis fine-grained (YcoW), the hypotheses sound and supported by the results (Zevd), and the causal story complete and WLE practical (HZCy). All three reviewers asked about unseen languages, so we answer that here. The remaining points are in the individual responses. Changes in the revised PDF are marked in red.

Summary of changes

  1. Scope of WLE (YcoW, Zevd, HZCy). We now state in the main text that WLE is designed to mitigate errors in code-switched speech. The unseen-language results provide a diagnostic of the same mechanism, but WLE does not reduce interference there (previously only discussed in Appendix C).
  2. Framing of the language subspace (YcoW). We clarify that we claim a low-dimensional subspace that (1) contains the model's language estimate and (2) causally controls which phoneme inventory the model applies. We do not claim the subspace is used exclusively for language identification, or that there is a semantic-agnostic language space like in LLMs.
  3. The shared-phonemes control is clarified (YcoW). We added a discussion on the role of shared phonemes as a control, and an explicit discussion on the interference evidence in our results.
  4. Other additions: probe accuracies, WLE runtime overhead, a discussion of phonotactics, a summary of hyperparameter selection, clarified wording in Section 2.2, and why we study monolingually trained phone recognizers.

A shared concern: WLE is designed for code-switching, not unseen languages

For unseen languages, our contribution is the diagnosis: we show that the same interference appears there and analyze what causes it, which is a first step towards a fix. Mitigating it is a separate problem, and we agree that the paper should state more clearly that WLE repairs code-switching and does not reduce interference on unseen languages.

The revision describes WLE as a repair for code-switched speech in the introduction, and states in Sections 2.4 and 6 that it does not repair interference on unseen languages, which was previously only in Appendix C.

The reason WLE does not resolve this is that its subspace has no centroid for an unseen language, and the local estimate is likely the same wrong language as the global estimate from the whole recording, so replacing one with the other has little effect. Future work could explore mitigation methods for unseen-language interference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.