- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a new methodology called MELD (metric) to estimate multilingual capability from model's hidden states. Their metric achieves high correlation scores with downstream model capability, and also works across model families.
Critical strengths
The metric seems novel and performs better than their discussed baseline. The metric also has high-correlation with both in-model downstream multilingual benchmarks and across different model families as well.
Critical weaknesses
-
All the downstream benchmarks target "discriminative" tasks where the answer space is defined via MCQs. The authors did not test their methodology on generative tasks. Any reasons behind not looking into those? They can look into trying XLSum, or even use NTREX as an evaluating task where they use FLORES to calculate the initial intrinsic score.
-
Another problem regarding the downstream benchmarks: m-MMLU and MMMLU are the same benchmarks and only differ by the fact that m-MMLU is machine translated v/s MMMLU which is human-translated. Why did the authors choose both of these benchmarks? m-ARC and m-HellaSwag also come from the same pipeline or suite of translated benchmarks (Okapi) - not sure if it makes sense to agree with the high-correlation of these benchmarks. An ideal eval setup could have been MMMLU, Belebele and a couple of generative benchmarks like XLSum, Translation (WMT) etc.
-
The main results in Table 2 and 3 takes into account all languages (low, medium, high-resource languages). Separating them out (like done in Figure 1) would actually help us know if the correlations actually explain the middle-language layer neutrality or is an artifact of the amount of training data that these models have seen.
Questions for authors
-
I see in appendix that the authors did a sweep on the percentage of middle layer and come to conclusion that 10% works better. However, the graph (Figure 2) that the peak is hit between 30-40%. I don't understand how materially does the extra 20-30% affect this choice. Can the authors explain their rationale? How did you define the "middle" layer across various model architecture for your experiments in lines 354-357?
-
Any reason why did the authors choose different eval setups for pre-trained v/s instruct and reasoning models? 0-shot v/s 5-shot eval? why not do 0-shot or 5-shot all the way for all the models?
-
I am still not convinced with the motivation of this work. How does the MELD score help in future design of LLMs?
-
Suggestion: The authors would greatly benefit from using simpler phrases / sentences in their draft to improve the readability of the paper.
4 comments
4. Selection of the Central 10% of Layers. Our selection is motivated by a trade-off between predictive performance and computational efficiency, rather than the claim that 10% always yields the highest correlation. As Figure 2 shows, correlations remain relatively stable across a broad range of middle-layer fractions, with only modest improvements near 30–40%. The central 10% achieves correlations close to the observed maximum while requiring representation extraction and scoring from substantially fewer layers.
We define the middle-layer region proportionally for each architecture, selecting approximately the central 10% of its transformer layers, centered at the network midpoint. Thus, the selected region adapts to models with different depths rather than relying on fixed layer indices.
We have clarified this rationale in Section 4.6 and Appendix A.2.2, where Figure 2 presents the layer-fraction ablation.
5. Zero-Shot vs. Five-Shot Evaluation. We agree that different evaluation protocols across models could affect the fairness of model comparisons. To address this, we revised our across-model evaluation to use a uniform five-shot protocol for all 14 models, regardless of training regime. Under this setting, MELD achieves average Pearson/Spearman correlations of 0.839/0.827, compared with 0.730/0.569 for MEXA and 0.487/0.541 for Language Ranker. For within-model language ranking, we retain the original settings (zero-shot for pretrained models and five-shot for instruction-tuned and reasoning models), since languages are compared under the same protocol within each model.
We have clarified this distinction in Section 4.5 (Evaluation Metrics) and updated the across-model results in Table 4.
6. Practical Motivation and Future Usefulness. We appreciate the opportunity to clarify MELD's intended practical value. MELD is designed as an intrinsic diagnostic and evaluation tool, rather than a direct method for modifying model architectures or training procedures.
By estimating multilingual understanding-related performance from internal representations, MELD could help compare multilingual model checkpoints, identify languages with weaker representation neutrality, and monitor the effects of multilingual training strategies without requiring extensive downstream benchmarks or aligned translation corpora.
Our results establish predictive associations rather than causal mechanisms. Therefore, we do not claim that directly maximizing MELD would necessarily improve multilingual understanding. Instead, MELD provides a potentially useful measurement tool for evaluating and investigating future multilingual model designs.
Finally, we appreciate the reviewer's constructive feedback, which helped us improve the consistency of our evaluation, better characterize resource-dependent performance, and clarify MELD's motivation and limitations.
References
[1] Wendler et al. (2024). Do Llamas Work in English? On the Latent Language of Multilingual Transformers. ACL.
[2] Bandarkar et al. (2025). Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models. ICLR.
[3] Harrasse et al. (2025). Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders. arXiv.
[4] Kargaran et al. (2025). MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment. Findings of ACL.
[5] Li et al. (2025). Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages. AAAI.
[6] Tamo et al. (2026). LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them? ICLR.
[7] Zhao et al. (2026). When Less Language Is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners. NeurIPS.
[8] Lu et al. (2025). Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline. EMNLP.
[9] Wang et al. (2025). Lost in Multilinguality: Dissecting Cross-Lingual Factual Inconsistency in Transformer Language Models. ACL.
Replying to the rebuttal
Thank you for running the experiments. Like I said earlier, and the other reviewer also stated: the current metric does not bring anything new to the table. It would have been interesting if your metric was robust and task-agnostic. My current score reflects the assessment of this work.
We thank you for the time and effort devoted to reviewing our paper. We appreciate your feedback and constructive suggestions. They helped us improve the manuscript.
We believe the novelty of our proposed approach lies in replacing sentence-pair alignment with a set-based, distribution-level measure that operates on independently sampled monolingual corpora, without requiring parallel datasets as prior methods do.
Furthermore, MELD is specifically designed to measure multilingual understanding and reasoning. As explained earlier, our choice of multiple-choice benchmarks is deliberate, as they assess comprehension, knowledge access, and reasoning while reducing confounding factors associated with unrestricted generation.
Once again, we sincerely appreciate your engagement and valuable feedback.
We sincerely thank the reviewer for their encouraging and positive assessment of MELD’s novelty and empirical performance, as well as for their thoughtful and constructive suggestions. Several of these comments motivated additional analyses and revisions to the manuscript (
the revised manuscript has been uploaded). We address each concern below.1. Evaluation on Generative Tasks. We appreciate the suggestion to include generative benchmarks in the analysis. We would like to clarify that MELD primarily aims to predict multilingual understanding and reasoning performance, rather than the full range of multilingual capabilities, including unrestricted generation.
This focus is motivated by prior studies suggesting that intermediate layers support shared semantic and reasoning processes, whereas language-specific processing and output realization are more prominent in the outer layers [1–3]. Multiple-choice benchmarks allow us to assess comprehension, knowledge access, and reasoning while reducing the influence of output formatting and language-specific answer realization. This choice is also consistent with prior intrinsic evaluation methods, including MEXA and Language Ranker [4,5].
Related studies further motivate distinguishing understanding from generation. For example, LinguaMap [6] shows that models can identify correct answers while responding in an unintended language, while Zhao et al. [7] demonstrate that reducing language-specific representations can improve reasoning while affecting output fidelity. Similar distinctions have been observed in multilingual factual recall [8,9].
To clarify this scope, we revised the manuscript as follows:
We agree that multiple-choice tasks do not completely isolate understanding from generation, and extending MELD to open-ended generative evaluation remains an important direction for future work.
2. Overlap Among Downstream Benchmarks. We agree that m-MMLU and MMMLU share the same underlying benchmark and should not be considered independent task families. We include both to examine whether the relationship between intrinsic scores and downstream performance remains consistent across different translation methodologies and language coverage: m-MMLU is machine-translated and covers 31 languages, whereas MMMLU is professionally translated and covers 14.
Similarly, although m-ARC and m-HellaSwag share the Okapi translation pipeline, they evaluate different underlying capabilities: science reasoning and commonsense inference, respectively. BELEBELE provides an additional reading-comprehension evaluation outside the Okapi suite. Importantly, MELD achieves strong correlations on the independently sourced benchmarks as well.
We agree that broader benchmark diversity would provide a more comprehensive assessment. However, multilingual benchmarks with broad language coverage remain relatively scarce.
3. Language Resource-Group Analysis. We appreciate this suggestion. To examine whether the correlations differ across language resource levels, we conducted an additional analysis dividing the 41 target languages into high-resource, medium-resource, and low-resource groups.
We recomputed within-model correlations separately for each group. MELD achieves Pearson/Spearman correlations of 0.894/0.844 for high-resource languages and 0.921/0.893 for medium-resource languages, outperforming or performing comparably to the baselines. For low-resource languages, correlations are weaker (0.419/0.385), and no method consistently dominates. As documented in
Table 6, only four low-resource languages are covered by m-MMLU, m-ARC, and m-HellaSwag, while MMMLU covers just one. Such limited coverage makes subgroup correlations sensitive to individual languages.We report the resource-group assignments, effective benchmark coverage, and correlations in
Appendix A.9(Tables 14–16). These findings provide a more detailed characterization of MELD's strengths and limitations, although they cannot fully disentangle the influence of training-data availability.