acceptodds
Under review as a conference paper at ICLR 2027

LEAD: Models Know the Language, but Token Preferences Do Not Always Follow

Abstract

Multilingual large language models often experience *language confusion*, failing to consistently generate responses in the intended language. Existing mitigation approaches often require costly model retraining or inference-time interventions that assume access to target language labels. Moreover, they do not distinguish whether language confusion stems from a failure to encode the intended language or from a failure to utilize the encoded language information during generation. This paper introduces **Language Evidence-Aware Decoding** (LEAD), a lightweight method that exploits language-subspace logits from the final hidden state to selectively intervene only at decoding time. Our method is motivated by the v-info theory and three empirical findings: linear probes identify languages with high accuracy, the logit contributions from language identity are weak, and causal interventions on the LM head reduce language confusion. Experiments on multiple multilingual LLMs across diverse languages demonstrate that LEAD substantially reduces language confusion while preserving general task performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Language confusion, Multilingual Llms, representational learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hN3ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes LEAD, a training-free decoding method that uses language-specific subspaces to reduce language confusion.
  • LEAD consistently improves language consistency across multiple models and benchmarks while largely preserving downstream task performance.

Overall Assessment

  • While the methodological novelty is somewhat limited, the empirical improvements are substantial and consistent across models and benchmarks.
  • The training-free approach and preservation of downstream performance make the method practically valuable.
  • Despite the limitations in baseline coverage and evaluation scope, the experimental results provide sufficient evidence of a useful contribution.
Critical strengths
  • Strong and consistent improvements over LCG and other baselines, particularly in response-level language consistency.
  • The improvements are substantial in several settings and generalize across different model families.
  • No additional model training or target-language labels are required at inference time.
  • The probing and causal intervention analyses provide useful insights into the relationship between language representations and token selection.
Critical weaknesses
  • Limited novelty: Language subspace extraction and language-aware decoding have been explored in prior work. The main methodological distinction from LCG lies in the scoring mechanism.
  • Insufficient baselines: Relevant methods such as ReCoVeR, TLPO, and simple script-based filtering are not directly compared.
  • Incomplete evaluation: Semantic quality, inference overhead, and statistical significance of the main improvements are not sufficiently assessed.
Questions for authors

See the weakness section please.

5 comments

Thanks a lot for your time on our work. We'll try my best to answer your concerns!

Methodological distinctions from LCG

LEAD is motivated by a different mechanistic understanding of language confusion. Our analysis (§3) shows that language identity remains recoverable from hidden states even when its contribution to token ranking is weak. This motivates us to explicitly extract and utilize language-identity evidence during decoding, rather than learning an auxiliary language-classification gate.

  • LCG trains a four-output language-family gate through self-distillation, whereas LEAD constructs a language-identity subspace through SVD of multilingual representations, without training an auxiliary gate. Instead of making family-level masking decisions, LEAD evaluates language evidence for individual candidate tokens, avoiding the granularity limitations of LCG's four-family gate.
  • LEAD explicitly decompose logits into language-identity and residual contributions, and introduce Relative Language Advantage (RLA) to normalize token-level evidence across decoding steps. We further investigate the approximately bimodal distribution of RLA scores using a two-component Gaussian mixture (Appendix B).
  • LEAD introduces a progressive nucleus expansion strategy to handle cases where RLA masking removes all candidates from the sampling nucleus. It admits additional tokens in descending order of their original probabilities and reapplies the masking rule against a fixed top-k reference until a valid candidate remains (§4.3).

Comparison with additional baselines

We added ReCover and a rule-based token filter with top-20 majority voting as inference-time baselines for LEAD. Adding TLPO would exceed our compute budget. Choo et al. [1] report approximately 6–10 hours on eight H100 GPUs for a single epoch of PPO-based post-training per model–language pair, which means that training one model for a single epoch on six languages needs at least 3 days on eight H100 GPUs. Our evaluation already includes ORPO as a representative post-training baseline, so the expanded comparison covers both inference-time intervention and post-training.

Statistical reliability, inference overhead, and semantic quality

  1. We used two-sided paired permutation tests with 100,000 permutations, keeping the three decoding seeds for each test item together, and applied Holm correction across all 24 comparisons. LEAD improved mean scores in all 24 comparisons, with 22 remaining statistically significant after correction (Holm-adjusted (p<0.05)), including all 12 RPR comparisons and 10 of 12 WPR comparisons.

  2. On a single NVIDIA A100 GPU with a 30B model (64 requests, concurrency 8, 2,000 input tokens and 100 output tokens), LEAD increases average decoding latency from 20.6349 to 20.9865 ms/token, introducing 0.3516 ms/token (1.70%) overhead relative to standard decoding.

  3. For semantic quality, we conducted an LLM-as-Judge(M-Prometheus-14B) evaluation of semantic quality on a five-point scale across three models. On LCB and FLORES-NO-LATIN , LEAD achieves average scores closely matching the Baseline, with differences ranging from −0.02 to +0.01 points. These results suggest that LEAD largely preserves semantic quality while improving language consistency.

References

[1] Jinho Choo, JunSeung Lee, Jimyeong Kim, Yeeho Song, S. K. Hong, and Yeong-Dae Kwon. 2026. TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 42670–42690. Association for Computational Linguistics.

AuthorsOpenReview ↗

Q2. Comparison with Additional Baselines

We have added two inference-time baselines, ReCoVeR and a Unicode filter, to our evaluation. For ReCoVeR, we use the hyperparameters reported in the original paper for Llama-3.1-8B-Instruct. For Qwen3-8B, we select the steering strength on a held-out Bactrian-X set, yielding α=0.5\alpha=0.5 for both monolingual and cross-lingual tasks. The Unicode filter infers the dominant script from the model’s top-20 candidates. LEAD achieves the highest RPR and the lowest Latin and CJ confusion rates for both models across all four benchmarks.

LCB-Mono

ModelMethodRPR ↑WPR ↑Latin ↓CJ ↓
Llama-3.1-8B-InstructReCoVeR94.3197.7816.191.83
Unicode filter96.9299.1113.031.50
LEAD98.3399.427.500.25
Qwen3-8BReCoVeR96.3399.0721.360.58
Unicode filter96.4798.5522.810.33
LEAD97.4499.5116.860.17

CrossSum

ModelMethodRPR ↑WPR ↑Latin ↓CJ ↓
Llama-3.1-8B-InstructReCoVeR87.8298.1645.053.33
Unicode filter77.2597.4553.928.26
LEAD87.8799.2023.461.33
Qwen3-8BReCoVeR80.4397.9152.6211.24
Unicode filter83.6999.3651.179.40
LEAD93.1699.0532.561.69

FLORES-NO-LATIN

ModelMethodRPR ↑WPR ↑Latin ↓CJ ↓
Llama-3.1-8B-InstructReCoVeR95.2599.505.481.27
Unicode filter92.8198.187.801.66
LEAD97.5199.432.870.14
Qwen3-8BReCoVeR95.1899.114.141.41
Unicode filter96.3699.293.390.78
LEAD98.8599.860.990.14

LCB-Cross

ModelMethodRPR ↑WPR ↑Latin ↓CJ ↓
Llama-3.1-8B-InstructReCoVeR87.2197.2031.842.26
Unicode filter78.4095.1351.101.08
LEAD87.7597.4128.950.17
Qwen3-8BReCoVeR82.5593.8449.270.59
Unicode filter78.5294.2051.840.25
LEAD84.3395.1140.960.08

Thank you for the detailed response to my concerns. I will maintain my positive rating.

Reviewer_FaAVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies language confusion in multilingual LLMs and asks a well-posed question: does confusion arise because the model fails to represent the intended output language, or because it fails to exploit language-identity evidence that is already present when ranking candidate tokens? Through layer-wise linear probing, a logit-decomposition analysis (LLMR/RLDR), and causal interventions on the LM head, the authors argue for the latter: the intended language remains linearly decodable from the final hidden state—even immediately before a confusion event—yet the LM head does not assign this evidence sufficient influence over token ranking. Motivated by this diagnosis, they propose Language Evidence-Aware Decoding (LEAD), an inference-time intervention that constructs a low-rank language-identity subspace via SVD, projects the current hidden state onto it, scores candidate tokens by their relative language evidence within the top-k set, and masks candidates with insufficient support. Experiments on Llama3.1-8B, Qwen3-8B, and Qwen3-30B-A3B across LCB, CrossSum, and FLORES+ show consistent reductions in language confusion relative to standard decoding, LCG, ITLC, and ORPO, with downstream task accuracy largely preserved.

Overall, this is a practically relevant decoding strategy paired with an interesting representation-level perspective on multilingual confusion. The empirical contribution is the stronger half of the paper; the mechanistic story is suggestive but not yet established with the rigor the authors claim for it.

Critical strengths
  1. The method targets an important problem in multilingual generation without additional training or oracle target-language labels at inference time—an advantage over activation-steering approaches such as ITLC, which require the target language to be known in advance. The decoding rule is conceptually simple and plausibly integrable into existing inference pipelines.

The gains are convincing in several challenging settings. On Llama3.1-8B, RPR rises from 71.77% to 87.75% on LCB-crosslingual and from 58.52% to 87.87% on CrossSum; on Qwen3-8B, CrossSum RPR improves from 76.78% to 93.16%. The head-to-head comparisons with LCG, ITLC, and ORPO (Appendix G.1) further strengthen the practical case, and the preservation of MGSM/M-MMLU/XWinograd accuracy addresses the most obvious deployment concern. This is, in my view, the strongest argument for acceptance.

  1. Rather than treating language confusion as a purely output-level phenomenon, the paper traces language information from hidden states to logits. The combination of out-of-distribution linear probing (train monolingual/test cross-lingual and vice versa), logit decomposition, and LM-head interventions offers multiple angles on the same question. The observation that the target language remains linearly decodable immediately before a confusion point (probe probabilities of 0.937 vs. 0.008 for target vs. unintended language) is striking and, if robust, is a genuinely useful empirical finding for the community. Even where the causal story is incomplete, the attempt to link interpretability analysis to a concrete decoding mechanism is worthwhile.
Critical weaknesses
  1. The mechanistic claim about insufficient LM-head utilization outruns the evidence.

The paper's central mechanistic claim is that language confusion arises primarily because the LM head under-utilizes language information already encoded in the final hidden representation. The evidence does not yet isolate this explanation from plausible alternatives.

First, high linear-probe accuracy shows that language identity is recoverable from the hidden state. In my opionion, it does not show that the representation encodes the intended language as a dominant generation objective, nor that the model's actual decision process relies on this direction. Probe accessibility and decision-relevance are distinct properties, and the paper sometimes slides between them.

Second, low LLMR values do not by themselves demonstrate under-utilization. The magnitude of a projected logit component depends on the chosen subspace, the scale of the decomposition, and which token comparisons are made; a numerically small component can still be decisive when competing logits are close—which is precisely the regime where confusion occurs. The paper would be considerably stronger if it established a direct relationship between LLMR/RLDR and empirical confusion probability, rather than reporting aggregate averages.

Third, the LM-head intervention modifies the language-related components of token embeddings. The component-transfer results are encouraging, but such interventions also alter the model's relative lexical preferences, so the observed RPR improvements do not uniquely identify insufficient language-evidence utilization as the cause of confusion. The random-direction and norm-scaling controls are helpful but do not fully rule out this confound.

These issues matter because the mechanistic explanation is presented as a major contribution beyond the decoding algorithm; if it is to carry that weight, it needs sharper causal identification.

  1. The contribution of the language-identity subspace, and the inference-time cost, are not sufficiently isolated.

The method rests on the assumption that the extracted low-rank subspace captures language identity rather than generic statistical differences among multilingual representations. The PCA visualization (Figure 5) and rank-scaling study (Figure 10) are suggestive but do not establish that the language-specific construction is necessary for the decoding gains. More targeted controls are needed: random orthogonal subspaces of matched rank, generic PCA subspaces, and simpler token-level filtering or logit-adjustment baselines under identical decoding settings. Without these, it is difficult to tell whether LEAD works because of its language-identity geometry or simply because it imposes a low-dimensional constraint on candidate ranking.

In addition, the intervention relies on heuristic Unicode-script identification to detect candidate-language conflicts. This is a reasonable engineering choice, but script identity is not language identity—particularly for languages sharing a writing system (the authors acknowledge the Latin-script limitation) and for naturally code-switched outputs. The method's behavior in these regimes is under-characterized.

Finally, despite the "lightweight" framing, the paper reports no systematic measurement of decoding latency, throughput, or intervention frequency. Since these directly determine the practical value of a decoding-time method, their absence weakens the efficiency claim.

Questions for authors

Q1. Can you distinguish under-utilization of language evidence from alternative explanations of confusion? Linear probing shows that target-language information remains accessible in the final hidden state, but this does not establish the LM head as the primary locus of failure. Could you provide a more direct causal analysis separating the effect of strengthening target-language evidence from general changes in token preferences—for example, targeted interventions on the logit margins between confusion tokens and their non-confusion neighbors, rather than on whole token sets? A per-step analysis linking LLMR/RLDR to the probability of an actual confusion event would also substantially strengthen the mechanistic claim.

Q2. How sensitive is LEAD to the construction of the language-identity subspace? Please compare the SVD-derived subspace against (i) random orthogonal subspaces of identical rank, (ii) generic PCA subspaces of identical rank, and (iii) simpler logit-adjustment or token-filtering baselines under matched decoding settings. Such controls are needed to establish that the improvement depends specifically on extracting language identity rather than on imposing a low-dimensional constraint. Relatedly, Appendix D.2 states that the subspace is built from Bactrian-X and CrossSum representations while CrossSum is also used for evaluation—please clarify whether these subsets are strictly disjoint, including any preprocessing used to prevent leakage.

Q3. What is the actual inference-time overhead of LEAD? Please report decoding throughput or latency relative to standard sampling and LCG under identical hardware and generation settings, together with the fraction of decoding steps at which intervention and nucleus expansion are activated. These numbers are necessary to substantiate the "lightweight" claim.

4 comments

Mechanistic Analysis of Language Confusion

We thank you for the constructive and insightful comments on our mechanistic analysis!

For the "First" and "Second" experiments, you can get more detailed explanation from our revised pdf (marked with a blue background in the main text).

  1. We examined the relationship between 1/RLDR1/\mathrm{RLDR} and the observed confusion event rate across decoding steps. As shown in Figure 3 (page 5), the observed confusion event rate generally increases across per-step 1/RLDR1/\mathrm{RLDR} quintiles, with a particularly pronounced trend on LCB. This indicates that confusion becomes more likely when the residual logit advantage of confusion tokens outweighs the linguistic evidence favoring correct-language candidates.

  2. Following your suggestion, we conducted a more targeted intervention on the logit margin between a confusion token and its neighboring correct-language token, rather than modifying entire token sets. We compared language-guided adjustments with equal-magnitude random-sign perturbations. As shown in Figure 6 (page 10), the target-token rank-up rate increases with intervention strength λ\lambda and consistently exceeds the random-sign control. On 500 LCB prompts (λ=0.5\lambda=0.5), RPR improves from 82.16% to 84.77% for Llama-3.1-8B and from 83.80% to 86.20% for Qwen3-8B, whereas the random-sign control yields 79.56% and 81.80%, respectively. This targeted intervention suggests that amplifying the linguistic component of token embeddings can mitigate language confusion.

  3. We would like to further clarify our mechanistic interpretation and welcome your thoughts on this issue. We agree that linear probing establishes only that target-language information is linearly accessible from the final-layer hidden states. This motivates asking how much of an advantage the language-related logit component gives target-language tokens in the output ranking. The association between higher 1/RLDR1/\mathrm{RLDR} and more frequent confusion and local targeted intervention suggest that a competing non-target token’s residual logit advantage can outweigh this language-related advantage. Properties of the LM head may help explain this imbalance: first,tokens from high-resource languages are overrepresented among high-norm output embeddings (Appendix D.4); second, our component-transfer experiments and local targeted intervention suggest that language-related logit contributions vary across tokens even within the same language family.

Thus, our whole story is : 1.Target-language information remains linearly accessible in the final hidden states. 2.During token ranking, a confusion token’s residual logit advantage can outweigh the target-language token’s language-related advantage. 3.Properties of the LM head may contribute to this imbalance and thereby to language confusion.

Language Subspace Ablation(Appendix D.2.2) and Data Leakage

We thank the reviewer for suggesting these additional controls. We conducted ablation experiments comparing LEAD with random orthogonal subspaces, PCA-based subspaces, and simple token-filtering baselines under matched decoding settings.

For the PCA variant, we measure how much variance each principal component captures along language-identity directions and retain components that capture little of this variation.

The results show that low dimensionality alone is insufficient to explain LEAD's effectiveness. For example, on Qwen3-8B LCB(cross-lingual), LEAD achieves 84.27% RPR, compared with 77.74% for PCA, 76.45% for random orthogonal subspaces, 78.01% for the top-1 rule, and 67.67% for the top-10 majority rule.

Regarding potential data leakage

The CrossSum representations used for subspace construction are drawn exclusively from the training split, whereas evaluation uses the validation split. The two subsets are disjoint, and validation examples are excluded from subspace construction.

Inference Efficiency and Intervention Frequency

  • Inference latency. We measured decoding latency on A100 GPU under identical hardware and generation settings (Qwen3-30B, 64 requests, concurrency 8, 2,000 input tokens and 100 output tokens). Baseline achieves 20.6349 ms/token, LEAD achieves 20.9865 ms/token (+0.3516 ms/token, 1.70%), and the open-source LCG implementation incurs 22.1556 ms/token (+1.5207 ms/token) .

  • Intervention frequency. On LCB-cross, LEAD intervenes on only 4.30%, 1.39%, and 1.02% of decoding steps for Llama-3.1-8B, Qwen3-8B, and Qwen3-30B-A3B, respectively. On LCB-mono, the corresponding rates are 2.13%, 0.68%, and 0.52%. Nucleus expansion is rarer, occurring at 0.51%, 0.36%, and 0.32% of decoding steps on LCB-cross and 0.24%, 0.17%, and 0.13% on LCB-mono, in the same model order. These results demonstrate that LEAD performs sparse interventions while introducing limited inference overhead.

Dear Reviewer FaAV,

We are reaching out to follow up on our response to your concerns. We have thoroughly considered all your comments and provided the necessary clarifications. We highly value your expertise and would like to know if you have any further feedback or concerns. We are dedicated to addressing any outstanding matters you may have raised before the discussion period ends.

Best regards,

The Authors of Paper 22030

Thank you for response. After carefully reviewing the additional information, most of my concerns have been addressed, and I have decided to maintain my positive score.

Thank you for constructive feedback and positive assessment of our work!

Reviewer_SAnEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LEAD, a training-free decoding method that filters tokens from the wrong language using hidden-state information. It improves language consistency while largely preserving task performance, but its real-world benefits remain unclear.

Critical strengths
  1. Easy to apply. LEAD requires no retraining and works during decoding.
  2. Explains the problem. The authors analyze hidden states and logits to understand why models switch languages.
Critical weaknesses
  1. Unclear practical impact. It is unclear how often language confusion occurs in normal use.
  2. RPR may overstate gains. Even a few foreign-language tokens count as a failure, so better RPR does not necessarily mean better overall responses.
Questions for authors
  1. How common is this problem? What is the failure rate on real-world prompts?
  2. What errors are fixed? Does LEAD fix major language switches or just a few foreign words?
  3. Does prompting work just as well? How much does LEAD help over stronger language instructions?
  4. Does it remove valid code-switching? Can it preserve necessary foreign words and technical terms?
  5. How sensitive is it to hyperparameters? Do the results hold across different settings without model-specific tuning?
  6. How much slower is decoding? What is the added latency per token?
6 comments

Thanks a lot for your kind suggestions. We'll try our best to answer your concerns!

Practical prevalence of language confusion

Marchisio et al. [1] report a 97.6% line-level pass rate for GPT-4o on cross-lingual prompts adapted from ShareGPT user requests. On FLORES-NO-LATIN, Zhang et al. [2] report Chinese/Japanese and Latin-script confusion rates of 0.57% and 0.67% for GPT-5-Chat and 2.27% and 5.07% for Qwen3-235B-Instruct. As Mukherjee et al. [3] illustrate, if a service handling more than 2 billion requests per day had a 98% response pass rate (RPR), more than 40 million responses per day would still fail the language check.

Recovery from major language switches

LEAD can correct major language switches, not just isolated foreign words. Using langdetect, we identify baseline responses predominantly written in an unintended language and evaluate whether LEAD restores the target language. On FLORES-NO-LATIN, LEAD successfully recovers 53/70 major-switch cases (75.71%) for Llama-3.1-8B and 3/3 cases (100%) for Qwen3-8B.

Besides, Nie et al. [4] showed that replacing the first off-target token at the confusion point substantially reduced major language confusion in Llama3-8B, which provides the theoretical basis for why LEAD can correct major language switches.

An additional case study illustrates this behavior in open-ended generation:

Input: Reply in Chinese. Generate a scientific report of 900 words discussing the effects of global warming on the Arctic.
Baseline: The Arctic, often referred to as the Earth's "climate barometer," is experiencing some of the most rapid and profound changes due to global warming.
LEAD: 全球变暖已成为21世纪最严峻的环境问题之一,其对地球生态系统的影响尤为显著。北极地区作为全球气候变化的“放大器”,其变化速度远超全球平均水平。

LEAD vs prompting method

We agree that LEAD should be compared against a stronger prompting baseline. We therefore evaluated a prompt that explicitly instructs the model to avoid language confusion and provides an example of a language-confused output alongside its correction. LEAD still outperforms this baseline. The full prompt and results are provided on the next page of this rebuttal.

Preservation of natural code-switching

We actually investigate this in Appendix G.5 (Table 15).

FLORES-WITH-LATIN contains legitimate Latin-script terms in reference translations. For Qwen3-8B with LEAD, the Latin-character rate is 22.77% on FLORES-WITH-LATIN, compared with only 0.99% on FLORES-NO-LATIN. This indicates that LEAD does not indiscriminately suppress Latin-script content.

Furthermore, we compare LEAD-enhanced 8B models against vanilla Qwen3-30B and Qwen3-30B+LCG, the latter being explicitly designed to preserve natural code-switching. Despite their smaller size, Llama-3.1-8B+LEAD (22.44%) and Qwen3-8B+LEAD (22.77%) exhibit Latin-character rates close to Qwen3-30B (24.15%) and Qwen3-30B+LCG (20.30%). This provides further evidence that LEAD does not systematically eliminate Latin-script content.

This behavior is consistent with our masking workflow (§4.3), which activates intervention only when mixed-language candidates are detected, avoiding unnecessary masking otherwise.

Hyperparameters sensitivity

We use the same hyperparameters across all three models without model-specific tuning. Appendix G.6 (Figure 12) provides a rank-scaling study, showing that language confusion generally decreases as the subspace rank increases. we fix the relative-advantage threshold at 0, avoiding model-specific threshold calibration and use top-k = 20 for language-evidence scoring to reduce noise from low-probability tokens.

We provide the sensitivity analysis in comments titled "Q5. Hyperparameters Sensitivity".

Inference overhead

We benchmark LEAD on a single NVIDIA A100 GPU using a 30B model (64 requests, concurrency 8, 2,000 input tokens and 100 output tokens). The average decoding latency increases from 20.6349 ms/token for standard decoding to 20.9865 ms/token with LEAD, introducing only 0.3516 ms/token (1.70%) additional overhead.

References

[1] Marchisio et al. Understanding and Mitigating Language Confusion in LLMs. EMNLP 2024, pp. 6653–6677.

[2] Zhang et al. Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillation. ICLR 2026.

[3] Anjishnu Mukherjee, Chutong Meng, and Antonios Anastasopoulos. Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs. COLM 2026.

[4] Nie et al. Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models. Findings of EMNLP 2025, pp. 690–706.

AuthorsOpenReview ↗

Q3. Comparison with the Prompting Baseline

We evaluate two system-prompting strategies for mitigating language confusion: a generic instruction that explains and discourages language confusion, and a target-language-specific instruction that explicitly constrains the language of the response. Each system prompt is prepended to the benchmark input. Actually, these two prompt templates have no better performance than our 3-shots baseline.

(1) Generic anti-confusion prompting.

We use the following system prompt:

Language confusion in the context of large language models (LLMs) refers to the phenomenon where a model mixes or confuses elements of multiple languages inappropriately during generation. Pay attention to prevent it. For example, "곧 방호복을 입은 경찰관들이 yard에 들어와 타격 가스로 수감자들을 몰아넣었다." is incorrect. It should be "곧 방호복을 입은 경찰관들이 마당에 들어와 최루가스로 수감자들을 몰아넣었다." Pay attention to avoid language confusion during generation.

(2) Target-language-specific prompting.

We additionally evaluate a more explicit instruction that requires the response to use the benchmark’s target language while preserving task-required proper nouns, technical terms, and quotations. The following English rendering illustrates the template:

Please answer strictly in {target language}. Regardless of the language of the input, express the natural-language content of your response in {target language}, while preserving proper nouns, technical terms, and quotations from the original text that are required by the task. Complete the original task in full, and do not omit content because of the language constraint.

The results of both prompting strategies are summarized below, which show that LEAD achieves better language consistency than both generic anti-confusion prompting and target-language-specific prompting.Target-language prompting was evaluated only on LCB-Cross.

BenchmarkModelMethodRPR ↑WPR ↑Latin ↓CJ ↓
LCB-CrossLlama-3.1-8B-InstructGeneric prompting71.0692.0955.183.34
Target-language prompting78.8092.4649.883.29
LEAD87.7597.4128.950.17
Qwen3-8BGeneric prompting76.2995.1352.120.50
Target-language prompting79.5193.9853.260.64
LEAD84.3395.1140.960.08
Qwen3-30BTarget-language prompting83.2293.6947.970.22
LEAD88.3696.2536.830.00
LCB-MonoLlama-3.1-8B-InstructGeneric prompting93.9298.1515.171.67
LEAD98.3399.427.500.25
Qwen3-8BGeneric prompting93.8397.6025.110.75
LEAD97.4499.5116.860.17
CrossSumLlama-3.1-8B-InstructGeneric prompting62.8892.4262.3918.29
LEAD87.8799.2023.461.33
Qwen3-8BGeneric prompting76.0697.9853.9914.54
LEAD93.1699.0532.561.69
FLORES-NO-LATINLlama-3.1-8B-InstructGeneric prompting88.5396.3811.143.46
LEAD97.5199.432.870.14
Qwen3-8BGeneric prompting94.7198.804.471.41
LEAD98.8599.860.990.14
AuthorsOpenReview ↗

Q5. Hyperparameters Sensitivity

To evaluate the sensitivity of LEAD to hyperparameter choices, we conducted a grid search across three models with the following settings:

  • top-k ∈ {10, 15, 20, 25}
  • threshold ∈ {0, 0.02, 0.04, 0.06, 0.08, 0.10}

The results indicate that LEAD maintains relatively stable language consistency across the tested hyperparameter range. Across the three models and four evaluation datasets, the maximum variation in Response Pass Rate (RPR) is 1.34 percentage points, while the maximum variation in Word Pass Rate (WPR) is 0.72 percentage points. Qwen3-8B exhibits particularly low sensitivity, with RPR variations remaining within 0.62 percentage points across all four datasets.

The table below reports the difference between the maximum and minimum values observed across completed hyperparameter configurations for each model and dataset, measured in %.

ModelDatasetΔRPRΔWPR
Llama-3.1-8BLCB-cross0.950.72
LCB-mono0.920.50
CrossSum1.340.42
FLORES-NO-LATIN0.490.26
Qwen3-8BLCB-cross0.620.22
LCB-mono0.33<0.01
CrossSum0.470.23
FLORES-NO-LATIN0.210.12
Qwen3-30BLCB-cross0.500.31
LCB-mono0.060.06
CrossSum1.060.33
FLORES-NO-LATIN0.190.05

These findings suggest that LEAD is relatively insensitive to hyperparameter choices, maintaining comparable language consistency without requiring fine-grained tuning to a particular configuration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Rebuttal

We thank all reviewers for their comprehensive reviews and insightful suggestions. Major changes in the revised manuscript are highlighted in blue. We also upload our codes as a zip file to facilitate reproducibility.

As the discussion phase comes to an end, we would like to summarize the efforts we have made during this period.

All reviewers recognized LEAD’s simplicity and effectiveness, which is supported by our extensive empirical experiments across multiple models and benchmarks. They also recognized the value of our mechanistic analysis and the useful insights it provides into language confusion.

The discussion centered on three main points: (a) clarifying the prevalence of language confusion in practice; (b) further strengthening the rigor of our mechanistic analysis; and (c) providing additional validation of LEAD’s effectiveness and efficiency.

To address (a), we provided evidence from several studies. On FLORES-NO-LATIN, Qwen team report Latin-script and Chinese/Japanese confusion rates of 0.67% and 0.57% for GPT-5-chat. Mukherjee et al. further illustrate the potential impact at scale: if a service handling over 2 billion requests per day achieved a 98% response pass rate, more than 40 million responses would still fail the language check each day. Together, these reports show that language confusion persists even in commercial LLMs and illustrate its potential impact at deployment scale.

To address (b), we further clarified our mechanistic account. Target-language information remains linearly accessible in the final hidden states, yet during token ranking, a confusion token’s residual logit advantage can outweigh the correct-language token’s language-related advantage. Since language evidence is unevenly distributed within the same family and LM Head also exhibits a bias toward high-resource languages, LM Head may under-utilize language evidence and then cause language confusion. Following the reviewers’ suggestions, we added local causal intervention experiments and a per-step analysis of the relationship between 1/RLDR1/\mathrm{RLDR} and observed confusion rates to examine this account more directly.

To address (c), we supplemented our existing evaluation with measurements of decoding latency and intervention frequency, showing that LEAD’s selective intervention rule yields sparse interventions with little inference overhead. We also added subspace ablations at the reviewers’ suggestion, suggesting that a low-rank constraint alone cannot explain the effectiveness of LEAD’s masking. In addition, we added two further baselines (Recover and Unicode filtering) and evaluated semantic quality. These results further support LEAD’s ability to mitigate language confusion while largely preserving semantic quality and general task performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.