acceptodds
Under review as a conference paper at ICLR 2027

Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities

11 views

Abstract

Large Language Models increasingly serve as interfaces for knowledge-intensive information seeking tasks across languages by synthesizing multilingual evidence. Prior work has shown that they often exhibit query-language preference – the tendency to favor sources written in the language of the query – but has largely examined this behavior in settings where equivalent knowledge is available across languages. However, this bias becomes consequential when sources in different languages provide incomplete or inconsistent accounts of the same fact, since the information users receive then depends on the sources a model selects to use. To characterize query-language preference under such cross-lingual knowledge disparities, we introduce Waldo, a multilingual Question-Answering (QA) benchmark constructed from Wikipedia. Waldo contains 12K QA pairs targeting knowledge gaps, where a fact is available in one language but absent in another, and knowledge conflicts, where language editions provide conflicting versions of the same fact. Evaluating eight models across five languages, we find that when one language edition merely lacks the relevant fact, models generally use evidence from the other language regardless of the query language. Under conflicting accounts, however, model responses strongly align with the document in the query language, causing semantically equivalent queries to elicit different accounts depending on the user's language. Finally, we explore two different approaches that could mitigate this preference under knowledge conflicts: a mechanistic intervention that ablates attention heads associated with query-language preference, and LoRA-based training, which reduces the preference gap by up to 61.5%.

Primary area:
datasets and benchmarks
Keywords:
query-language preference, cross-lingual knowledge disparity, multilingual information systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TZ7DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates query-language preference in multilingual question answering when documents in different languages contain missing or conflicting knowledge. The authors introduce WALDO, a benchmark constructed from multilingual Wikipedia articles, and evaluate eight LLMs across five source languages. They find that query-language preference is relatively limited under knowledge gaps but becomes pronounced under knowledge conflicts, where models tend to follow the document matching the query language. The paper further explores two mitigation approaches, attention-head ablation (QPHA) and QLoRA fine-tuning, to reduce this preference.

Critical strengths
  1. The paper studies an important and well-motivated problem. Understanding how query language influences information selection is relevant to the reliability and fairness of multilingual LLM-based systems.

  2. The WALDO benchmark is a valuable contribution. It combines naturally occurring knowledge gaps from multilingual Wikipedia with controlled knowledge conflicts, providing a systematic testbed for studying query-language preference.

Critical weaknesses
  1. The paper does not adequately define or evaluate the desired behavior under unresolved knowledge conflicts. When two documents provide conflicting information without sufficient evidence to determine which is reliable, a more appropriate response may be to explicitly report the disagreement rather than select either answer. Indeed, the Introduction advocates surfacing unresolved conflicts. However, the benchmark and its evaluation metrics do not assess whether models can recognize and report such disagreements. Moreover, both mitigation approaches encourage models to select the predefined original answer rather than explicitly acknowledge the conflict. Consequently, there is a fundamental mismatch between the paper's stated motivation and what its benchmark and mitigation experiments actually evaluate.

  2. The mitigation experiments lack a simple and important prompting baseline. For example, a system prompt could explicitly instruct models to consider evidence from all languages equally and report conflicting information when the correct answer cannot be determined, rather than favoring the document matching the query language. It is unclear whether such a straightforward approach would already substantially mitigate the observed problem without requiring attention-head interventions or fine-tuning.

Questions for authors

What exactly is QLoRA intended to teach the model? Is the goal to enable the model to identify the factually correct answer from conflicting multilingual documents, regardless of the query language? If so, it is unclear how the proposed training objectives would enable this capability to generalize to unseen conflicts.

5 comments
AuthorsOpenReview ↗

Response to Reviewer TZ7D

We sincerely thank the reviewer for their insightful feedback and for finding the benchmark well-motivated and a valuable contribution. We respond to each point below.

W1. No evaluation of the desired behavior (surfacing disagreement under knowledge conflicts); mismatch between the paper’s motivation and the benchmark and mitigation approaches.

A1. Thank you for raising this point. As suggested, we measured how often models explicitly flag the conflict between documents, using GPT-5.6 as an LLM judge. Across 100 sampled responses for each open-source model, only 2 of 300 (0.7%) flagged the conflicting accounts. Table 1 shows one such case. In general, models rarely surface conflicts and instead tend to follow the document that matches the query language, confirming the gap our work highlights: query-language preference is a key barrier to this desired behavior.

Table 1. Example of response that flags the conflict

TypeContent
QuestionWhere did New World Development say New World China's headquarters would be moved to in 2021?
Source-language document answer广州 (Guangzhou)
Conflict-injected English document answerShenzhen
ResponseNew World Development announced that New World China's headquarters would be moved to Shenzhen [Document ID: 1]. However, the text also mentions a subsequent move to Guangzhou [Document ID: 2], so it appears there may be a discrepancy or an update.

Q1. What exactly is QLoRA intended to teach the model? Is it to enable models to assess factuality of answers? If so, how can this generalize to unseen conflicts?

AQ1. Thank you for the question! Ideally, models would surface such disparities to users. While QPHA and QLORA are not designed to teach conflict resolution or assess evidence reliability, we view them as an intermediate step toward that goal: a model is unlikely to weigh evidence fairly or surface disagreement while the query language still determines which evidence it relies on. By counteracting this dependence, our mitigations make the model’s evidence selection less dependent on query language. In turn, the model assigns more comparable probability to both candidate answers, so that both can be surfaced and presented as a conflict. We see extending QLoRA toward explicit conflict surfacing as a natural direction for future work.

Regarding generalization, since QLoRA trains the preference rather than specific facts, its effect extends beyond the conflicts seen in training:

  • Swapped roles: the same adapters trained on the original configuration also reduce query-language preference when the roles of the two documents are swapped.
  • Balanced mixture: in a mixture of the original and swapped-role datasets, the original adapter reduces query-language preference to near zero (see Table 2 in our response to Reviewer wBYk).
  • Knowledge gap: in the knowledge gap setting, which contains no injected conflict, QLoRA (Promote) still significantly reduces the preference (see A1-1+AQ2 to Reviewer X3s3).

We will add these results and discussion to the revised paper.

W2. Missing a simple prompting baseline that explicitly instructs models to consider evidence from all languages equally when the correct answer cannot be determined.

A2. The prompt baseline reduces query-language preference for two of three models, but less than QPHA and QLoRA.

Thank you for raising this point. We added a prompt baseline with the system prompt "The language of the query does not necessarily indicate that either document is more reliable or complete. Evaluate both documents solely based on their content, and answer using the information you judge most reliable." As Table 2 shows, the prompt baseline significantly reduces both agreement and probability gap relative to vanilla for LLaMA-3.1 and Aya-Expanse, but not for Qwen-3. In all cases, it remains well above QPHA and QLoRA, suggesting that explicit instructions alone are insufficient to override query-language preference. We will include this baseline in the main paper of the revised manuscript.

Table 2. Comparison with the Prompt baseline (lower is better)

Significance relative to Vanilla: * p < .05, ** p < .01, *** p < .001.

Approach**LLaMA-3.1Qwen-3Aya-Expanse
MetricΔ AgreeΔ ProbΔ AgreeΔ ProbΔ AgreeΔ Prob
Vanilla63.029.959.012.158.833.5
Prompt58.7**22.3**59.614.356.0*30.8*
QPHA (Zero)56.3**19.4***54.3*10.555.7*29.7*
QPHA (Mean)55.9**9.1***55.4*10.1*55.0*28.7**
QLoRA (Promote)2.3***4.3***4.3***7.7***0.3***4.9***
QLoRA (Contrastive)1.5***6.4***1.4***5.3***0.9***1.7***

We would be happy to address any further questions or concerns you may have.

Thank you for the detailed response and additional experiments.

I appreciate the clarification that QPHA and QLoRA primarily target query-language preference rather than conflict resolution. The new prompt baseline and generalization analyses are also useful and partially address my second concern and question.

However, my main concern remains. I understand that the experiments demonstrate query-language bias when models are asked to select an answer from conflicting multilingual evidence, and that the proposed methods aim to reduce this bias.

My concern is about the practical relevance of this experimental setting and the corresponding mitigation approaches. In realistic scenarios, when conflicting evidence cannot be reliably resolved, models could simply report the disagreement rather than being required to select one answer. The added prompt baseline still instructs models to "answer using the information you judge most reliable," rather than explicitly allowing them to report unresolved conflicts, as I originally suggested.

Therefore, I remain unconvinced that the current experimental setting and mitigation approaches adequately reflect how such knowledge conflicts should be handled in practice.

AuthorsOpenReview ↗

Response to Reviewer TZ7D

We thank the reviewer for carefully reading our response and for clarifying this concern!

Following the reviewer’s original suggestion, we ran an additional prompt baseline that explicitly allows models to report unresolved conflicts: "If the documents disagree on the answer and the information does not show which is correct, report this as an unresolved conflict and describe what each document says." We evaluated it on the same knowledge-conflict set, and measured conflict flagging on the same 100 randomly sampled instances per model using GPT-5.6 as an LLM judge.

We find two results. First, even when explicitly allowed to report the conflict, models rarely do so: Prompt (report) flags the conflict in only 3 of 300 responses (1.0%), compared with 2 of 300 (0.7%) for Vanilla. Second, as Table 3 shows, query-language preference persists under this prompt, remaining above QPHA and QLoRA on both metrics.

Together, these results show that models still favor query-language evidence and rarely surface conflicts even when explicitly instructed to. We therefore believe that reducing query-language preference is a prerequisite for reporting unresolved conflicts, as it allows models to weigh both candidate answers more comparably and, in turn, to report both accounts. We will include this baseline and discussion in the revised manuscript.

Table 3. Results for the new prompt baseline

Prompt (reliable): our earlier prompt baseline; Prompt (report): new baseline that instructs reporting unresolved conflicts.

Δ Agree, Δ Prob significance is relative to Vanilla: * p < .05, ** p < .01, *** p < .001.

ApproachLLaMA-3.1Qwen-3Aya-Expanse
MetricΔ AgreeΔ ProbΔ AgreeΔ ProbΔ AgreeΔ Prob
Vanilla63.029.959.012.158.833.5
Prompt (reliable)58.7**22.3**59.614.356.0*30.8*
Prompt (report)56.5**23.6*57.9*11.958.131.0*
QPHA (Zero)56.3**19.4***54.3*10.555.7*29.7*
QPHA (Mean)55.9**9.1***55.4*10.1*55.0*28.7**
QLoRA (Promote)2.3***4.3***4.3***7.7***0.3***4.9***
QLoRA (Contrastive)1.5***6.4***1.4***5.3***0.9***1.7***

We hope this addresses the reviewer’s concern, and we would be happy to answer any further questions!

Reviewer_X3s3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether multilingual LLMs prefer evidence that’s written in the same language as the query. Previous work mostly looks at this when the same facts exist in all languages or there is disparities (in fictional documents). The paper particularly looks at cases where knowledge is uneven across languages. The main concern therefore is whether the same semantic question would produce different factual answers because of the question’s language.

The author introduces WALDO, a Wikipedia-based multilingual QA benchmark covering two settings: knowledge gaps (absent information in another language) and knowledge conflicts (documents contain contradictory facts). For the conflict setting, GPT-5.6 injects an altered version of the fact into the English article. They construct approximately 12,000 QA pairs, then investigate two mitigation methods: Query-language Preference Head Ablation (QPHA), which intervenes on selected attention heads, and QLoRA fine-tuning with objectives designed to reduce language-dependent answer selection.

Eight models (LLaMA-3.1 8B, Qwen-3 8B, Aya-Expanse 8B, GPT-4o, GPT-4.1, DeepSeek-V4-Pro, Grok-4.3, GPT-5.5) are evaluated using French, Chinese, Japanese, Korean, Bengali and English. The experiments show that there is relatively low query-language preference under knowledge gaps, especially for closed-source models, but a large average 55.7-percentage-point answer-agreement gap under conflicts. QPHA provides modest improvements, while QLoRA substantially reduces the measured gaps.

Critical strengths

Prior work either assumed equivalent knowledge across languages or used fictional documents. The counterfactual design, which holds the context fixed and varies only the query language, is really clean and very easy to interpret. Additionally, I believe the coverage is broad enough since the paper uses eight models and five typologically diverse languages. The knowledge-gap instances come from actual revision-sourced differences between Wikipedia editions, with an audited filtering pipeline and translation-quality gating. The controls (swapped roles, knowledge parity, and confounder tests) make the result credible. Together these make the central finding convincing.

Critical weaknesses
  1. The mitigation results could possibly not be showing reduced language preference. Every conflict document contains at least one LLM-inserted sentence (Sec. 3.2, and Prompt A.9 for swapped roles), and QLoRA is trained on these documents with the goal of picking the original fact. It might be the case that it’s learning to spot the inserted edit (possibly detectable as it’s LLM edited?). The paper does not test whether that’s the case. A related problem is the evaluation prompt (A.7), which forces a single-sentence answer citing one document. The paper says the ideal behavior under conflict is to surface the disagreement (p.2 lines 81-83), but the setup doesn’t leave any room for that, and the metric rewards picking the true side.

  2. The probability metric may be measuring the wrong thing, and the judge is not validated. Eq. 3 appears to score the source-language answer string under both source-language query and the English query. But the models reply in the query language 92 to 95% of the time (B.4), so the source-language string would look less probable after an English question, no matter which document the model trusts. The parity control (Table 12) seems consistent with this. Agreement gap is not significant, but probability gaps of 8.3 and 9.2 is there, which is roughly half to two thirds of what the same model shows in knowledge-gap setting (Fig. 2). That gap could be the language effect, since there is no preference to find. Also, GPT-5.6 generates the data, injects conflicts, judges agreement (Prompt A.8), and classifies topics, but there is no reported human agreement for the judge, and GPT-5.5 is among the evaluated models.

Questions for authors
  1. How much of the probability gap is answer-language fluency? Either rescore with both candidate answers in the query language, or subtract the Table 12 gap as baseline.
  2. Could QLoRA and QPHA be learning to detect the insertion rather than reducing language preference? Did you evaluate the tuned and ablated models on the knowledge-gap and parity settings or any general capability benchmark, and was there degradation? Can a simple classifier locate the inserted sentence?
  3. All agreement scores come from GPT-5.6 deciding whether an answer matches. What is its accuracy against human labels, and how does it score an answer that mentions both facts or flags the conflict?

Curious: The mitigation set keeps only conflict items “that exhibit query-language preference.” How is that defined?

3 comments
AuthorsOpenReview ↗

Response to Reviewer X3s3 #1

We sincerely thank the reviewer for recognizing the value of our experimental setup and the breadth of our analyses. We address each concern below.

W1-1+Q2. The model might be learning to spot the inserted conflict rather than reducing query-language preference. Mitigation approaches should be evaluated in the knowledge gap setting. Can a simple classifier locate the inserted conflict?

A1-1+AQ2. QLoRA still significantly reduces query-language preference in the knowledge gap setting, and an LLM-based classifier cannot locate the inserted conflict better than chance.

Thank you for this question. To address this concern, we evaluated the original QLoRA adapter (Promote) in the knowledge gap setting. Since this setting contains no injected conflict, a model that had only learned to detect insertions would show no benefit here. As Table 1 shows, QLoRA still substantially reduces both agreement and probability gap across all three models, indicating that it reduces query-language preference itself rather than exploiting cues from the inserted conflict.

Table 1. Results for knowledge gap with QLoRA (Promote)

Significance relative to Vanilla: *** p < .001.

ModelLLaMA-3.1Qwen-3Aya-Expanse
Δ AgreeΔ ProbΔ AgreeΔ ProbΔ AgreeΔ Prob
Vanilla15.918.512.18.618.212.9
QLoRA (Promote)2.9***0.5***2.9***2.4***5.3***4.4***

We further tested whether the inserted conflicts are easy to detect. On a random sample of 200 conflict-injected documents, we prompted GPT-5.5 to identify the one sentence that seemed out of place. It identified the inserted sentence in only 3 of 200 cases (1.5%), no better than chance (4.6%, computed as the average of 1/N across documents, where N is the number of sentences). This suggests that the inserted conflicts are not trivially detectable, again highlighting that the observed query-language preference reflects models’ genuine behavior rather than an artifact. We will add both results to the Appendix of the revised manuscript.

W1-2. The evaluation prompt forces the model to cite a single document, leaving no room to surface disagreement.

A1-2. Even with a neutral prompt that does not require citing a single document, models almost never flag the conflict. Thank you for raising this concern. We re-ran the evaluation for the three open-source models using a neutral version of our evaluation prompt, removing the instruction "You must cite the most relevant document by including only its Document ID in brackets at the end of the sentence (e.g. [Document ID])."

We then used GPT-5.6 as an LLM judge to measure how often models explicitly flag the conflict between documents. Across 100 sampled responses per model, only 3 of 300 (1.0%) responses flagged the conflicting accounts under the neutral prompt, compared with 2 of 300 (0.7%) under the original prompt (see Table 1 in our response to Reviewer TZ7D for an example of a flagged response). This difference is not statistically significant when tested with a two-sided Fisher’s exact test. This suggests that models rarely flag conflicts regardless of whether they are asked to cite a single document. We will include this analysis in the revised paper.

W2-1+Q1. Probability metric might be confounded by answer-language fluency; the knowledge parity gap (Table 12) should be subtracted as a baseline.

A2-1+AQ1. The probability gap remains significant after subtracting the knowledge parity baseline, even though this correction is conservative.

This is a great observation! Following the reviewer’s suggestion, we subtract the answer probability gap under knowledge parity from that under the knowledge gap and knowledge conflict settings: Δadj(s) = Δprob(conflict) - Δprob(parity), for s ∈ {gap, conflict}. As the reviewer notes, this correction assumes that any remaining parity gap comes entirely from answer-language fluency. In practice, however, part of the parity gap likely stems from other factors, such as translating the target fact into English or its placement within the document, so this subtraction over-corrects for fluency and gives a conservative estimate. Even so, the probability gap remains positive and significant across all models and settings (Table 2). We will include this analysis in the revised Appendix.

Table 2. Adjusted answer probability gap

Significance relative to their respective Vanilla: * p < .05, ** p < .01, *** p < .001.

ModelKnowledge Gap (raw → adjusted)Knowledge Conflict (raw → adjusted)
LLaMA-3.118.5 → 10.1**26.0 → 17.7***
Qwen-38.6 → 7.0**10.3 → 8.7**
Aya-Expanse12.9 → 3.6*28.3 → 19.2***
AuthorsOpenReview ↗

Response to Reviewer X3s3 #2

W2-2+Q3. Human validation is needed for using GPT-5.6 in (1) data generation, (2) LLM-judge agreement, and (3) conflict topic classification.

A2-2+AQ3. Thank you for this helpful suggestion. Human validation on sampled subsets confirms the high quality of GPT-5.6 outputs at each step:

(1) Data generation. For each language, we sampled 20 Wikipedia revisions and their associated QA pairs, and recruited 2-3 native speakers to review them. Across the five languages, annotators agreed with the GPT-5.6-generated QA pairs in 99.3% of cases and rated them highly for usefulness (mean 4.5/5) and reliability (mean 4.7/5), indicating that the pairs faithfully capture the target facts. We further ensure data quality by verifying translation quality (Appendix A.3) and retaining only conflict instances that pass an answerability check (Section 3.2).

(2) LLM-judge agreement. We sampled 50 answer-response pairs (10 per language) and asked 3 human annotators to judge whether each response agrees with the reference answer. Majority-vote labels agreed with GPT-5.6 in 98% of cases, with high inter-annotator agreement (Fleiss’ κ = 0.973).

(3) Conflict topic classification. We sampled 50 conflict sentences and asked 3 annotators to classify each into one of six categories (date, number, person, location, organization, or misc), using the same instructions given to GPT-5.6 (Prompt A.6). Majority-vote labels agreed with GPT-5.6 in 96% of cases (Fleiss’ κ = 0.952), consistent with prior work showing that LLMs, even for those weaker than GPT-5.6, perform well on such annotation tasks [1].

Together, these results confirm that GPT-5.6 annotations are reliable across all three steps.

Q4. Mitigation keeps only conflict items that “exhibit query-language preference.” How is this defined?

AQ4. For a given model, we define an instance as exhibiting query-language preference if the model’s response contains the source document’s original target fact when queried in the source language, but does not when queried in English. In terms of our answer agreement metric, the instance scores 1 under the source-language query and 0 under the English query. We will clarify this definition in the revised paper.

We would be happy to address any further questions or concerns you may have.

Reference

[1] Gilardi et al., ChatGPT matched or beat crowd workers on text-annotation tasks (PNAS 2023)

Thank you for the detailed response and additional experiments. They address several of my concerns. I am keeping my score.

Reviewer_wBYkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the query-language preference of multilingual Large Language Models (LLMs). The paper observed that the model has a tendency to favor contextual evidence written in the same language as the user's query, specifically when cross-lingual sources provide incomplete or conflicting information. While prior studies predominantly assume that equivalent facts exist across languages, this paper examines what happens when models face real-world cross-lingual knowledge disparities during test-time retrieval and generation.

Critical strengths
  1. The paper is well-written and easy to follow. The narrative flows logically from problem formulation and benchmark construction (WALDO) to diagnostic evaluation and mitigation, supported by clear figures and well-structured analyses.

  2. The idea is also intuitive and has real-world importance. Query-language preference can be of great importance in today's tool use agent with online searching ability during inference time generation.

  3. The experimental setup is rigorous and well-controlled. The control holds the bilingual context fixed while varying only the query language. The paper also applies a counterfactual design that cleanly isolates the effect of the type of query language. The dataset construction is also intuitive and has high ecological validity (data from real Wikipedia data), which filters for facts that are later than the model knowledge cutoffs (combined with closed-book checks). The appendices offer extensive analyses for alternative confounders (e.g., perplexity priors, embedding similarity, and surface cues).

  4. The paper also offers complementary mitigation strategy and offer an thorough comparison. The authors propose and evaluate two distinct mitigation mechanisms: an inference-time mechanistic intervention (QPHA) and a training-time adaptation (QLoRA fine-tuning with custom prompts and contrastive objectives). The paper provides a thorough comparative analysis of the two approaches, clearly articulating the practical trade-offs.

Critical weaknesses
  1. The novelty can be overstated. The claim that no prior work has specifically studied query-language preference at generation time (Section 2, lines 151–153) appears too broad. As the paper itself acknowledges, Sharma et al. (2025) already investigate query-language preference during generation under cross-lingual knowledge conflicts. Although WALDO offers a different data construction pipeline, the underlying research question and phenomenon are not new.

  2. The related work comparison is insufficiently focused. The related-work subsections appear to be broad and do not consistently connect prior findings to the specific question of query-language preference. A more precise comparison should distinguish what is new in the problem formulation, experimental controls, findings, and interventions, etc.

  3. It is unclear whether QLoRA learns conflict resolution or a dataset-specific shortcut. In WALDO’s default configuration, the original answer is always supported by the non-English source-language document, while the conflicting answer is always injected into the English document. This can create a systematic correlation between document language and the desired answer. QLoRA may therefore reduce the reported preference gap by learning to favor the non-English document whenever evidence conflicts, including when the query is in English. Such a policy would improve the benchmark metrics without establishing that the model has learned to assess evidence reliability. More fundamentally, if the provided documents contain no reliable signal indicating which account is correct, what allows the model to identify the designated answer on unseen test instances? Although Appendix D.3 reports swapped-roles results, it is unclear whether the same adapters trained under the original configuration are evaluated directly on reversed instances, or whether the mitigation procedure is repeated with reversed training data. The paper can be strengthened by evaluating a single fixed adapter on a balanced mixture, in which the correct answer’s language varies independently of the query language. Then, we can see if the model can still appropriately acknowledge conflicts when the evidence is insufficient to resolve them.

  4. There is insufficient justification for treating query-language preference as uniformly undesirable. The paper assumes that query-language preference under conflicting evidence should be eliminated, but this normative assumption needs qualification. When sources appear equally authoritative and the model cannot independently verify their claims, query language may provide a useful signal of the user’s intended cultural or regional context. For questions involving local conventions, culturally situated concepts, or region-dependent interpretations, different answers across languages may reflect legitimate contextual differences rather than factual errors. Indiscriminately suppressing this preference could therefore reduce answer relevance and accuracy for culturally or geographically specific queries, potentially privileging a single supposedly universal account over valid local perspectives. The paper should distinguish conflicts over objectively verifiable facts from differences arising from cultural or regional context, and evaluate whether its mitigation preserves performance when language-sensitive source selection is appropriate.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer wBYk

We sincerely thank the reviewer for their insightful feedback and for recognizing the strength of our setup and the real-world impact of our paper. We address the concerns below.

W1+W2. Novelty claim that WALDO is the first to study query-language preference at generation time may be overstated, and the related work comparison is insufficiently focused.

A1+A2. Thank you for raising these points. Following the reviewer’s suggestion, we added Table 1, which situates our work relative to prior studies of query-language preference. As it shows, no prior work combines all three of the following: (1) a benchmark grounded in real-world sources, (2) a focus on cross-lingual knowledge disparities, and (3) mitigation approaches for query-language preference at generation time. We will revise the original wording to “no prior work mitigated query-language preference within the model itself at generation time” to clarify this scope and include the table in the revised manuscript.

Table 1. Comparison to prior works

PaperStage (Generation (G) / Retrieval (R))(1) Real(2) Disparity(3) Mitigation
Qi et al., 2025; Ki et al., 2026G✔✗✗
Sharma et al., 2025R+G✗✔✗
Li et al., 2025aR+G✔✔✗ (balancing)
Amiraz et al., 2025R✔✗✗ (query translation)
Wang et al., 2026; Jang et al., 2026R✔✗✗ (calibration)
Lee et al., 2026; Hong et al., 2026R✔✗✗ (balancing)
Park et al., 2026R+G✔✗✗ (calibration)
Huo et al., 2026R+G✔✗✗ (balancing)
Park & Lee, 2025; Ranaldi et al., 2026R+G✔✗✔ (document translation)
WALDO (Ours)G✔✔✔ (QPHA, QLoRA)

W3. QLoRA may learn dataset-specific shortcuts rather than judging for evidence reliability or conflict resolution; show whether the original adapter generalizes to (1) swapped roles and (2) balanced mixture.

A3. The original QLoRA adapters generalize to both swapped-role and balanced-mixture settings.

Thank you for this suggestion. Our swapped-role experiments (Appendix D.3) already use the same adapters trained on the original configuration. Following the reviewer’s suggestion, we additionally constructed a balanced mixture of the original and swapped-role datasets, so that the language of the document containing the original target fact varies independently of the query language. As Table 2 shows, the original adapter reduces query-language preference to near zero in both settings.

Table 2. QLoRA (Promote) under original, swapped-role, and balanced-mixture settings

Significance relative to their respective: ** p < .01, *** p < .001.

ModelLLaMA-3.1Qwen-3Aya-Expanse
Δ AgreeΔ ProbΔ AgreeΔ ProbΔ AgreeΔ Prob
Original2.3***4.3***4.3***7.7**0.3***4.9***
Swapped role-4.6***-0.3***0.4***0.0***-0.3***0.2***
Balanced mixture-1.2***-0.1***0.7***0.1***-0.3***-0.4***

Regarding the broader concern, ideally, models would surface such disparities to users. While QPHA and QLORA are not designed to teach conflict resolution or assess evidence reliability, we view them as an intermediate step toward that goal: a model is unlikely to weigh evidence fairly or surface disagreement while the query language still determines which evidence it relies on. By counteracting this dependence, our mitigations make the model’s evidence selection less dependent on query language. In turn, the model assigns more comparable probability to both candidate answers, so that both can be surfaced and presented as a conflict.

W4. Insufficient justification for treating query-language preference as uniformly undesirable. For questions involving culturally situated concepts, different answers across languages may reflect legitimate contextual differences.

A4. Thank you for raising this point! We agree that cultural specificity is important when interpreting query-language preference, and that in some settings, answers that differ by language can be legitimate. WALDO, however, is deliberately constructed so that query-language-dependent source selection is always inappropriate. We include only (1) instances with a knowledge gap between documents in two languages, and (2) instances in which we manually inject a factually incorrect claim in one document, rather than cases that allow culturally different interpretations. This design choice removes cultural interpretation as a potential confounder.

We would be happy to address any further questions or concerns you may have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

Dear Reviewers,

Thank you for your time and thoughtful feedback on our paper! We have uploaded a revised version to OpenReview that incorporates your comments and suggestions. All new revisions are highlighted in green:

  • Table 1: Added a comparison with prior work (Reviewer wBYk)
  • L455-456: Clarified the definition of knowledge conflict instances (Reviewer X3s3)
  • Section 6.2: Added a simple prompting baseline (Reviewer TZ7D)
  • Limitations, Appendix E: Described the human validation setup and results (Reviewer X3s3)
  • Conclusion & Discussion: Discussed the roles of QPHA and QLoRA (Reviewer wBYk, TZ7D)
  • Appendix B.5: Added conflict detection with an LLM-based classifier (Reviewer X3s3)
  • Appendix B.6: Added a conflict acknowledgment metric with results (Reviewer X3s3, TZ7D) and adjusted answer probability gap results (Reviewer X3s3)
  • Appendix D.3: Showed that our mitigation approaches generalize to balanced-mixture and knowledge-gap settings (Reviewer wBYk, X3s3, TZ7D)

We sincerely appreciate your reviews and would be glad to discuss further or answer any remaining questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.