- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates query-language preference in multilingual question answering when documents in different languages contain missing or conflicting knowledge. The authors introduce WALDO, a benchmark constructed from multilingual Wikipedia articles, and evaluate eight LLMs across five source languages. They find that query-language preference is relatively limited under knowledge gaps but becomes pronounced under knowledge conflicts, where models tend to follow the document matching the query language. The paper further explores two mitigation approaches, attention-head ablation (QPHA) and QLoRA fine-tuning, to reduce this preference.
Critical strengths
-
The paper studies an important and well-motivated problem. Understanding how query language influences information selection is relevant to the reliability and fairness of multilingual LLM-based systems.
-
The WALDO benchmark is a valuable contribution. It combines naturally occurring knowledge gaps from multilingual Wikipedia with controlled knowledge conflicts, providing a systematic testbed for studying query-language preference.
Critical weaknesses
-
The paper does not adequately define or evaluate the desired behavior under unresolved knowledge conflicts. When two documents provide conflicting information without sufficient evidence to determine which is reliable, a more appropriate response may be to explicitly report the disagreement rather than select either answer. Indeed, the Introduction advocates surfacing unresolved conflicts. However, the benchmark and its evaluation metrics do not assess whether models can recognize and report such disagreements. Moreover, both mitigation approaches encourage models to select the predefined original answer rather than explicitly acknowledge the conflict. Consequently, there is a fundamental mismatch between the paper's stated motivation and what its benchmark and mitigation experiments actually evaluate.
-
The mitigation experiments lack a simple and important prompting baseline. For example, a system prompt could explicitly instruct models to consider evidence from all languages equally and report conflicting information when the correct answer cannot be determined, rather than favoring the document matching the query language. It is unclear whether such a straightforward approach would already substantially mitigate the observed problem without requiring attention-head interventions or fine-tuning.
Questions for authors
What exactly is QLoRA intended to teach the model? Is the goal to enable the model to identify the factually correct answer from conflicting multilingual documents, regardless of the query language? If so, it is unclear how the proposed training objectives would enable this capability to generalize to unseen conflicts.
5 comments
Thank you for the detailed response and additional experiments.
I appreciate the clarification that QPHA and QLoRA primarily target query-language preference rather than conflict resolution. The new prompt baseline and generalization analyses are also useful and partially address my second concern and question.
However, my main concern remains. I understand that the experiments demonstrate query-language bias when models are asked to select an answer from conflicting multilingual evidence, and that the proposed methods aim to reduce this bias.
My concern is about the practical relevance of this experimental setting and the corresponding mitigation approaches. In realistic scenarios, when conflicting evidence cannot be reliably resolved, models could simply report the disagreement rather than being required to select one answer. The added prompt baseline still instructs models to "answer using the information you judge most reliable," rather than explicitly allowing them to report unresolved conflicts, as I originally suggested.
Therefore, I remain unconvinced that the current experimental setting and mitigation approaches adequately reflect how such knowledge conflicts should be handled in practice.
Response to Reviewer TZ7D
We thank the reviewer for carefully reading our response and for clarifying this concern!
Following the reviewer’s original suggestion, we ran an additional prompt baseline that explicitly allows models to report unresolved conflicts: "If the documents disagree on the answer and the information does not show which is correct, report this as an unresolved conflict and describe what each document says." We evaluated it on the same knowledge-conflict set, and measured conflict flagging on the same 100 randomly sampled instances per model using GPT-5.6 as an LLM judge.
We find two results. First, even when explicitly allowed to report the conflict, models rarely do so: Prompt (report) flags the conflict in only 3 of 300 responses (1.0%), compared with 2 of 300 (0.7%) for Vanilla. Second, as Table 3 shows, query-language preference persists under this prompt, remaining above QPHA and QLoRA on both metrics.
Together, these results show that models still favor query-language evidence and rarely surface conflicts even when explicitly instructed to. We therefore believe that reducing query-language preference is a prerequisite for reporting unresolved conflicts, as it allows models to weigh both candidate answers more comparably and, in turn, to report both accounts. We will include this baseline and discussion in the revised manuscript.
Table 3. Results for the new prompt baseline
Prompt (reliable): our earlier prompt baseline; Prompt (report): new baseline that instructs reporting unresolved conflicts.
Δ Agree, Δ Prob significance is relative to Vanilla: * p < .05, ** p < .01, *** p < .001.
| Approach | LLaMA-3.1 | Qwen-3 | Aya-Expanse | |||
|---|---|---|---|---|---|---|
| Metric | Δ Agree | Δ Prob | Δ Agree | Δ Prob | Δ Agree | Δ Prob |
| Vanilla | 63.0 | 29.9 | 59.0 | 12.1 | 58.8 | 33.5 |
| Prompt (reliable) | 58.7** | 22.3** | 59.6 | 14.3 | 56.0* | 30.8* |
| Prompt (report) | 56.5** | 23.6* | 57.9* | 11.9 | 58.1 | 31.0* |
| QPHA (Zero) | 56.3** | 19.4*** | 54.3* | 10.5 | 55.7* | 29.7* |
| QPHA (Mean) | 55.9** | 9.1*** | 55.4* | 10.1* | 55.0* | 28.7** |
| QLoRA (Promote) | 2.3*** | 4.3*** | 4.3*** | 7.7*** | 0.3*** | 4.9*** |
| QLoRA (Contrastive) | 1.5*** | 6.4*** | 1.4*** | 5.3*** | 0.9*** | 1.7*** |
We hope this addresses the reviewer’s concern, and we would be happy to answer any further questions!
Response to Reviewer TZ7D
We sincerely thank the reviewer for their insightful feedback and for finding the benchmark well-motivated and a valuable contribution. We respond to each point below.
A1. Thank you for raising this point. As suggested, we measured how often models explicitly flag the conflict between documents, using GPT-5.6 as an LLM judge. Across 100 sampled responses for each open-source model, only 2 of 300 (0.7%) flagged the conflicting accounts. Table 1 shows one such case. In general, models rarely surface conflicts and instead tend to follow the document that matches the query language, confirming the gap our work highlights: query-language preference is a key barrier to this desired behavior.
Table 1.Example of response that flags the conflictAQ1. Thank you for the question! Ideally, models would surface such disparities to users. While QPHA and QLORA are not designed to teach conflict resolution or assess evidence reliability, we view them as an intermediate step toward that goal: a model is unlikely to weigh evidence fairly or surface disagreement while the query language still determines which evidence it relies on. By counteracting this dependence, our mitigations make the model’s evidence selection less dependent on query language. In turn, the model assigns more comparable probability to both candidate answers, so that both can be surfaced and presented as a conflict. We see extending QLoRA toward explicit conflict surfacing as a natural direction for future work.
Regarding generalization, since QLoRA trains the preference rather than specific facts, its effect extends beyond the conflicts seen in training:
Table 2in our response to Reviewer wBYk).We will add these results and discussion to the revised paper.
A2. The prompt baseline reduces query-language preference for two of three models, but less than QPHA and QLoRA.
Thank you for raising this point. We added a prompt baseline with the system prompt "The language of the query does not necessarily indicate that either document is more reliable or complete. Evaluate both documents solely based on their content, and answer using the information you judge most reliable." As Table 2 shows, the prompt baseline significantly reduces both agreement and probability gap relative to vanilla for LLaMA-3.1 and Aya-Expanse, but not for Qwen-3. In all cases, it remains well above QPHA and QLoRA, suggesting that explicit instructions alone are insufficient to override query-language preference. We will include this baseline in the main paper of the revised manuscript.
Table 2.Comparison with the Prompt baseline (lower is better)Significance relative to Vanilla: * p < .05, ** p < .01, *** p < .001.
We would be happy to address any further questions or concerns you may have.