MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocol

Multilingual Information Retrieval reflects real-world search settings where users issue queries over mixed-language corpora. Existing evaluations mainly reward language-agnostic semantic relevance, treating relevant passages equally regardless of language. Yet retrieval utility also depends on the language of the retrieved passages: users expect results they can read and verify in the query language, and query–passage language mismatch can complicate downstream grounding in Retrieval-Augmented Generation systems. To evaluate this aspect, we introduce **MLAIRE**, a **M**ultilingual **L**anguage-**A**ware **I**nformation **R**etrieval **E**valuation protocol that disentangles cross-lingual semantic retrieval from query-language preference. MLAIRE constructs controlled pools with parallel passages across languages, enabling measurement of whether retrievers find relevant passages and whether they prioritize query-language passages when equivalent translations are available. We further propose language-aware metrics, including Language Preference Rate (LPR) and Lang-nDCG, together with a 4-way decomposition separating semantic and language-preference failures. Evaluating 31 dense, sparse, and late-interaction retrievers, we show that standard metrics obscure distinct behaviors: semantically strong retrievers may return correct content in a non-query language, while language-preserving retrievers may retrieve less relevant passages.