- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- The paper proposes LEAD, a training-free decoding method that uses language-specific subspaces to reduce language confusion.
- LEAD consistently improves language consistency across multiple models and benchmarks while largely preserving downstream task performance.
Overall Assessment
- While the methodological novelty is somewhat limited, the empirical improvements are substantial and consistent across models and benchmarks.
- The training-free approach and preservation of downstream performance make the method practically valuable.
- Despite the limitations in baseline coverage and evaluation scope, the experimental results provide sufficient evidence of a useful contribution.
Critical strengths
- Strong and consistent improvements over LCG and other baselines, particularly in response-level language consistency.
- The improvements are substantial in several settings and generalize across different model families.
- No additional model training or target-language labels are required at inference time.
- The probing and causal intervention analyses provide useful insights into the relationship between language representations and token selection.
Critical weaknesses
- Limited novelty: Language subspace extraction and language-aware decoding have been explored in prior work. The main methodological distinction from LCG lies in the scoring mechanism.
- Insufficient baselines: Relevant methods such as ReCoVeR, TLPO, and simple script-based filtering are not directly compared.
- Incomplete evaluation: Semantic quality, inference overhead, and statistical significance of the main improvements are not sufficiently assessed.
Questions for authors
See the weakness section please.
5 comments
Q2. Comparison with Additional Baselines
We have added two inference-time baselines, ReCoVeR and a Unicode filter, to our evaluation. For ReCoVeR, we use the hyperparameters reported in the original paper for Llama-3.1-8B-Instruct. For Qwen3-8B, we select the steering strength on a held-out Bactrian-X set, yielding for both monolingual and cross-lingual tasks. The Unicode filter infers the dominant script from the model’s top-20 candidates. LEAD achieves the highest RPR and the lowest Latin and CJ confusion rates for both models across all four benchmarks.
LCB-Mono
| Model | Method | RPR ↑ | WPR ↑ | Latin ↓ | CJ ↓ |
|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | ReCoVeR | 94.31 | 97.78 | 16.19 | 1.83 |
| Unicode filter | 96.92 | 99.11 | 13.03 | 1.50 | |
| LEAD | 98.33 | 99.42 | 7.50 | 0.25 | |
| Qwen3-8B | ReCoVeR | 96.33 | 99.07 | 21.36 | 0.58 |
| Unicode filter | 96.47 | 98.55 | 22.81 | 0.33 | |
| LEAD | 97.44 | 99.51 | 16.86 | 0.17 |
CrossSum
| Model | Method | RPR ↑ | WPR ↑ | Latin ↓ | CJ ↓ |
|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | ReCoVeR | 87.82 | 98.16 | 45.05 | 3.33 |
| Unicode filter | 77.25 | 97.45 | 53.92 | 8.26 | |
| LEAD | 87.87 | 99.20 | 23.46 | 1.33 | |
| Qwen3-8B | ReCoVeR | 80.43 | 97.91 | 52.62 | 11.24 |
| Unicode filter | 83.69 | 99.36 | 51.17 | 9.40 | |
| LEAD | 93.16 | 99.05 | 32.56 | 1.69 |
FLORES-NO-LATIN
| Model | Method | RPR ↑ | WPR ↑ | Latin ↓ | CJ ↓ |
|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | ReCoVeR | 95.25 | 99.50 | 5.48 | 1.27 |
| Unicode filter | 92.81 | 98.18 | 7.80 | 1.66 | |
| LEAD | 97.51 | 99.43 | 2.87 | 0.14 | |
| Qwen3-8B | ReCoVeR | 95.18 | 99.11 | 4.14 | 1.41 |
| Unicode filter | 96.36 | 99.29 | 3.39 | 0.78 | |
| LEAD | 98.85 | 99.86 | 0.99 | 0.14 |
LCB-Cross
| Model | Method | RPR ↑ | WPR ↑ | Latin ↓ | CJ ↓ |
|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | ReCoVeR | 87.21 | 97.20 | 31.84 | 2.26 |
| Unicode filter | 78.40 | 95.13 | 51.10 | 1.08 | |
| LEAD | 87.75 | 97.41 | 28.95 | 0.17 | |
| Qwen3-8B | ReCoVeR | 82.55 | 93.84 | 49.27 | 0.59 |
| Unicode filter | 78.52 | 94.20 | 51.84 | 0.25 | |
| LEAD | 84.33 | 95.11 | 40.96 | 0.08 |
Thank you for the detailed response to my concerns. I will maintain my positive rating.
Thanks a lot for your time on our work. We'll try my best to answer your concerns!
LEAD is motivated by a different mechanistic understanding of language confusion. Our analysis (§3) shows that language identity remains recoverable from hidden states even when its contribution to token ranking is weak. This motivates us to explicitly extract and utilize language-identity evidence during decoding, rather than learning an auxiliary language-classification gate.
We added ReCover and a rule-based token filter with top-20 majority voting as inference-time baselines for LEAD. Adding TLPO would exceed our compute budget. Choo et al. [1] report approximately 6–10 hours on eight H100 GPUs for a single epoch of PPO-based post-training per model–language pair, which means that training one model for a single epoch on six languages needs at least 3 days on eight H100 GPUs. Our evaluation already includes ORPO as a representative post-training baseline, so the expanded comparison covers both inference-time intervention and post-training.
We used two-sided paired permutation tests with 100,000 permutations, keeping the three decoding seeds for each test item together, and applied Holm correction across all 24 comparisons. LEAD improved mean scores in all 24 comparisons, with 22 remaining statistically significant after correction (Holm-adjusted (p<0.05)), including all 12 RPR comparisons and 10 of 12 WPR comparisons.
On a single NVIDIA A100 GPU with a 30B model (64 requests, concurrency 8, 2,000 input tokens and 100 output tokens), LEAD increases average decoding latency from 20.6349 to 20.9865 ms/token, introducing 0.3516 ms/token (1.70%) overhead relative to standard decoding.
For semantic quality, we conducted an LLM-as-Judge(M-Prometheus-14B) evaluation of semantic quality on a five-point scale across three models. On LCB and FLORES-NO-LATIN , LEAD achieves average scores closely matching the Baseline, with differences ranging from −0.02 to +0.01 points. These results suggest that LEAD largely preserves semantic quality while improving language consistency.
References
[1] Jinho Choo, JunSeung Lee, Jimyeong Kim, Yeeho Song, S. K. Hong, and Yeong-Dae Kwon. 2026. TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 42670–42690. Association for Computational Linguistics.