- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates cultural epistemic prejudice in large language models: the tendency of LLMs to make systematically biased assumptions about what users from different cultural backgrounds are likely to know. The authors introduce a controlled entity-disambiguation benchmark in which an LLM encounters two entities, one from the United States and one from another country, followed by an ambiguous statement such as "I don't know this university." The model must select the entity it believes the user is unfamiliar with.
Critical strengths
This work studies the assumptions that LLMs make about the knowledge of their interlocutors, with implications for personalization, education, information access, and culturally appropriate communication. The evaluation includes seven different LLMs, ten languages, and five knowledge domains. The comparison between Wikipedia visibility and entity frequencies in the Tulu3 post-training dataset is an interesting exploratory analysis.
Critical weaknesses
- The evaluation does not adequately establish epistemic prejudice. The paper interprets the selection of a non-US entity as evidence that the model assumes the user knows the US entity but not the non-US entity. However, this interpretation depends on an assumption that is not validated. Consider the example on page 4: "MIT and IIT Delhi are collaborating on a new research project. I don't know this university." The statement is genuinely ambiguous. There is no ground-truth referent. Selecting IIT Delhi could arise from multiple mechanisms:
- The model considers IIT Delhi less globally recognizable than MIT.
- The model defaults to explaining entities that are less frequent in its training data.
- The model follows a linguistic preference for the more recently mentioned entity.
- The model makes a contextual inference about the user's knowledge.
Only the last mechanism directly supports the intended construct of epistemic prejudice. The authors acknowledge the potential influence of entity frequency and positional effects, but these alternative explanations are not sufficiently separated from the measured outcome. Most importantly, the paper states that a non-US explanation rate above 50% indicates epistemic prejudice. However, equal selection probability is not necessarily the appropriate fairness or calibration baseline. If the two entities differ substantially in global familiarity, selecting the less recognizable entity may be a reasonable default under uncertainty.
- Linguistic and positional confounding remain substantial. The paper always introduces the US entity before the non-US entity in its primary templates. For example: "Boston and Istanbul ... I don't know this city." Because the non-US entity appears second, the evaluation risks confounding cultural familiarity with grammatical recency and other reference-resolution preferences. The authors recognize this and perform an entity-order robustness analysis in Appendix D. However, the appendix explicitly reports that reversing entity order causes substantial changes for cities, particularly under native-language prompting. This suggests that positional information meaningfully influences the outcome. This is an important issue because the paper's main evaluation interprets the same outcome as an epistemic attribution. The robustness analysis therefore does not fully resolve the concern; it instead reveals that the measurement is sensitive to a major nuisance variable.
Questions for authors
See weakness.
Thank you for the detailed and constructive feedback. We address the main concerns and questions below.
1. What the evaluation establishes
We intentionally use an ambiguous prompt with no assumed ground-truth referent. The entity pair stays fixed across conditions, and the user-context signal changes through language and, more directly, explicit location. We then test whether the model changes which entity it treats as unfamiliar. This context-conditioned comparison is the part of our design that directly captures the user-knowledge aspect of epistemic prejudice, rather than the raw non-US selection rate alone.
We agree that the other factors can affect this choice, but they play different roles. Global familiarity and training frequency can shape the model’s default familiarity judgment, and mention order can affect how the ambiguous reference is resolved; we examine these separately. The non-US entities were selected for their prominence within the corresponding local cultural contexts (Appendix A), rather than as arbitrary obscure alternatives. We also test pair-level frequency differences and find that larger US–non-US frequency gaps are associated with higher non-US selection across all six open models, consistent with exposure differences shaping the model’s default familiarity pattern (please also refer to our response to Nnoc27). Even when the user’s location is stated directly, the same overall familiarity pattern remains. The default choice therefore changes little even when relevant information about the user is made explicit.
We agree that 50% should not be interpreted as a fairness or calibration baseline. We use it only as an equal-selection reference and will revise the paper to make this distinction explicit.
2. Linguistic and positional confounding
We agree that mention order affects some settings, particularly cities under native-language prompting, and we will state this more clearly. The order-swap analysis shows that recency alone is insufficient to explain the overall pattern. Reversing the entity order changes the magnitude in some settings, but fails to produce a systematic reversal. If grammatical recency were the dominant mechanism, moving the non-US entity to the first position should reverse the preference much more consistently. Instead, the broader pattern remains across both orderings.