acceptodds
Under review as a conference paper at ICLR 2027

Known to Whom? Auditing Cultural Epistemic Prejudice in Large Language Models

Abstract

As large language models (LLMs) become increasingly central to how people search for and consume information, biases in their outputs risk becoming embedded in users’ knowledge-seeking practices. Recent work has documented biases in the knowledge LLMs represent and generate, but how they infer what users are likely to know remains largely unexplored. Here, we provide a large-scale analysis of one way these inferences can disadvantage users: epistemic prejudice, or LLMs’ tendency to systematically misjudge how familiar users are with entities from different cultural contexts. We develop a controlled disambiguation task in which LLMs have to choose which of two entities—one US and one non-US—a user is unfamiliar with, and test them with different cues about the user’s cultural background: English prompts, native-language (e.g., Hindi) prompts, and explicitly stated user location (e.g., India). Across seven LLMs, ten languages, and five domains, models show a strong tendency to predict that the user knows the US entity and is unfamiliar with the non-US entity. Native language and explicit location alter this tendency in some settings, but the preference strongly persists across many languages and domains. For example, for users asking in Hindi, LLMs predominantly assume that they are more familiar with US entities than with Indian entities, suggesting the presence of strong epistemic prejudice in those models. In further experiments, we find that epistemic prejudice cannot be easily reduced via debiasing instructions. Finally, we analyze LLM training data and find that, relative to US entities, non-US entities are far more underrepresented than real-world visibility would suggest. Together, these findings establish epistemic prejudice as a distinct and understudied form of bias in LLMs: not a bias in what models know, but in what models assume their users know.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Bias/Fairness, Interpretability, Evaluation, Multilingual NLP, Epistemic Bias

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_j1xoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates cultural epistemic prejudice in large language models: the tendency of LLMs to make systematically biased assumptions about what users from different cultural backgrounds are likely to know. The authors introduce a controlled entity-disambiguation benchmark in which an LLM encounters two entities, one from the United States and one from another country, followed by an ambiguous statement such as "I don't know this university." The model must select the entity it believes the user is unfamiliar with.

Critical strengths

This work studies the assumptions that LLMs make about the knowledge of their interlocutors, with implications for personalization, education, information access, and culturally appropriate communication. The evaluation includes seven different LLMs, ten languages, and five knowledge domains. The comparison between Wikipedia visibility and entity frequencies in the Tulu3 post-training dataset is an interesting exploratory analysis.

Critical weaknesses
  1. The evaluation does not adequately establish epistemic prejudice. The paper interprets the selection of a non-US entity as evidence that the model assumes the user knows the US entity but not the non-US entity. However, this interpretation depends on an assumption that is not validated. Consider the example on page 4: "MIT and IIT Delhi are collaborating on a new research project. I don't know this university." The statement is genuinely ambiguous. There is no ground-truth referent. Selecting IIT Delhi could arise from multiple mechanisms:
  • The model considers IIT Delhi less globally recognizable than MIT.
  • The model defaults to explaining entities that are less frequent in its training data.
  • The model follows a linguistic preference for the more recently mentioned entity.
  • The model makes a contextual inference about the user's knowledge.

Only the last mechanism directly supports the intended construct of epistemic prejudice. The authors acknowledge the potential influence of entity frequency and positional effects, but these alternative explanations are not sufficiently separated from the measured outcome. Most importantly, the paper states that a non-US explanation rate above 50% indicates epistemic prejudice. However, equal selection probability is not necessarily the appropriate fairness or calibration baseline. If the two entities differ substantially in global familiarity, selecting the less recognizable entity may be a reasonable default under uncertainty.

  1. Linguistic and positional confounding remain substantial. The paper always introduces the US entity before the non-US entity in its primary templates. For example: "Boston and Istanbul ... I don't know this city." Because the non-US entity appears second, the evaluation risks confounding cultural familiarity with grammatical recency and other reference-resolution preferences. The authors recognize this and perform an entity-order robustness analysis in Appendix D. However, the appendix explicitly reports that reversing entity order causes substantial changes for cities, particularly under native-language prompting. This suggests that positional information meaningfully influences the outcome. This is an important issue because the paper's main evaluation interprets the same outcome as an epistemic attribution. The robustness analysis therefore does not fully resolve the concern; it instead reveals that the measurement is sensitive to a major nuisance variable.
Questions for authors

See weakness.

1 comment

Thank you for the detailed and constructive feedback. We address the main concerns and questions below.

1. What the evaluation establishes

We intentionally use an ambiguous prompt with no assumed ground-truth referent. The entity pair stays fixed across conditions, and the user-context signal changes through language and, more directly, explicit location. We then test whether the model changes which entity it treats as unfamiliar. This context-conditioned comparison is the part of our design that directly captures the user-knowledge aspect of epistemic prejudice, rather than the raw non-US selection rate alone.

We agree that the other factors can affect this choice, but they play different roles. Global familiarity and training frequency can shape the model’s default familiarity judgment, and mention order can affect how the ambiguous reference is resolved; we examine these separately. The non-US entities were selected for their prominence within the corresponding local cultural contexts (Appendix A), rather than as arbitrary obscure alternatives. We also test pair-level frequency differences and find that larger US–non-US frequency gaps are associated with higher non-US selection across all six open models, consistent with exposure differences shaping the model’s default familiarity pattern (please also refer to our response to Nnoc27). Even when the user’s location is stated directly, the same overall familiarity pattern remains. The default choice therefore changes little even when relevant information about the user is made explicit.

We agree that 50% should not be interpreted as a fairness or calibration baseline. We use it only as an equal-selection reference and will revise the paper to make this distinction explicit.

2. Linguistic and positional confounding

We agree that mention order affects some settings, particularly cities under native-language prompting, and we will state this more clearly. The order-swap analysis shows that recency alone is insufficient to explain the overall pattern. Reversing the entity order changes the magnitude in some settings, but fails to produce a systematic reversal. If grammatical recency were the dominant mechanism, moving the non-US entity to the first position should reverse the preference much more consistently. Instead, the broader pattern remains across both orderings.

Reviewer_BeTcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which entity an LLM assumes a user does not know. Each prompt pairs a US and a non-US entity and asks the model to resolve an ambiguous reference. The study varies language and stated user location across several domains and models. Models often choose the non-US entity, even with local context. The authors interpret this pattern as cultural epistemic prejudice and explore training frequency as a possible source.

Critical strengths
  1. The paper asks a useful question about user modeling. What a model assumes users know is distinct from what the model itself knows, and it can affect the help users receive.
  2. The context changes reveal a repeatable behavioral pattern. Tests across languages and domains show that native-language and location cues do not have the same effect. The order-swap analysis is also valuable for interpreting this pattern.
Critical weaknesses

The task does not establish mistaken beliefs about user knowledge. There are no user-knowledge labels, and equal selection is not necessarily the right baseline for entities with different familiarity. Current non-US selection alone does not establish epistemic prejudice. The research lacks a real user profile modeling or proxy metric to provide proof to distinguish users' prior cultural knowledge distribution and to explain wrong attribution.

Questions for authors

What evidence distinguishes wrong user-knowledge attribution from ordinary ambiguity resolution or differences in entity familiarity? Is any user-level validation available?

2 comments

Thank you very much for your thoughtful feedback and for raising these concerns about how we interpret the task.

No user-knowledge labels

We would like to clarify that our task examines the assumptions models make about what users are likely to know; we do not assume that every individual attribution is incorrect. As noted in §4.2, neither language nor location determines what a particular user knows, and establishing this at the individual level would require user-knowledge annotations. We currently lack such validation and will note this more clearly as a limitation and direction for future work

Equal selection as a baseline

We agree that 50% should not be interpreted as a fairness or calibration baseline when entities differ in global familiarity. We use it as an equal-selection reference and will revise the wording accordingly. Our main question is whether the model's default choice changes once information about the user's cultural background is provided.

What evidence distinguishes wrong user-knowledge attribution from ordinary ambiguity resolution or differences in entity familiarity? Is any user-level validation available?

Our main evidence comes from holding the entity pair fixed and changing the user's linguistic or geographic context. This directly tests whether the model changes which entity it treats as unfamiliar when information about the user changes. Ambiguity resolution is the method we use to observe this choice; the context manipulation is what tests the user-knowledge aspect.

Entity familiarity can shape the model's default choice, which we examine in §6.2 through the training-data analysis. The non-US entities were also selected for their local prominence within the corresponding cultural contexts (Appendix A), rather than as arbitrary obscure alternatives. Please also refer to our response to Nnoc27, where Tulu3 identifies the intended country correctly in 97% of native-language prompts. This shows that, for Tulu3, the high non-US selection rate cannot be explained simply by failure to recognize the user's cultural context. The explicit-location condition strengthens this point further: the user's location is stated directly, yet non-US selection remains high.

We currently lack user-level validation. Our evaluation measures model assumptions across controlled user contexts; individual-level accuracy would require separate user-knowledge annotations, which we see as a valuable direction for future work.

Thanks for the author's response. I regard the weakness I proposed as still existing without solid user-level validation or convincing proxies. Therefore, I maintain my scores for the paper.

Reviewer_NnocOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether LLMs make US-centric assumptions about what users know.  Across 10 languages and 7 LLMs, this tendency does not fully disappear even when the prompt is written in the local language or states the user’s location. The authors suggest that skewed training data is one cause.

This paper asks, across many languages, whether LLMs treat users as people who already know things from the US. However, the paper only measures which of two names the model outputs, and this alone does not justify interpreting the choice as an assumption about the user.

Critical strengths

This paper raises the question of how LLMs account for what users already know. Mismatches such as explaining Delhi to a Hindi speaker directly affect the quality of explanations in search and education, so the question is worth to consider.

The authors evaluates 5,000 name pairs across 10 languages, 5 domains, and 7 LLMs. The design strengthens the cue about the user step by step and is easy to follow.

They shows that in a public fine-tuning dataset, US city names appear 6–103 times as often as non-US city names, whereas the gap in Wikipedia pageviews is only 2–16 times.

Critical weaknesses

This paper treats choosing US and non-US names equally as the unbiased baseline, but does not justify it. For English prompts, answering with the non-US name may be reasonable, since the US has the largest share of LLM users. It is therefore unclear what the measured bias is relative to.

They measures which name the model outputs when forced to choose one of two. The paper cannot rule out that the model simply picks the less widely known name or the one mentioned second.

Their method aggregates a single response per prompt and does not measure how confident the model is in its choice. The training-data analysis is also limited to counting city names and does not test whether less frequent names are in fact more likely to be chosen.

Questions for authors

(1) Did the authors write the local-language prompts in the native script or transliterated into Latin script? The main text and Figure 1 disagree.

(2) If the model is asked directly, with the same prompt, where the user is from, does it answer correctly?

(3) This paper does not cite two closely related works: Jin et al., “Implicit Personalization in Language Models” (Findings of EMNLP 2024), and Naous et al., “Having Beer after Prayer?” (ACL 2024). How does this paper differ from them?

1 comment

Thank you for the careful reading. Below, we respond to each of your comments and questions in turn.

50% as an unbiased baseline

We agree that 50% should not be interpreted as an unbiased or normative fairness baseline. When the user's background is unknown, explaining the less globally familiar non-US entity may indeed be a reasonable default. Our main question is what happens once information about the user's cultural context is provided. We therefore use 50% only as an equal-selection reference point and will revise this in the paper to make this explicit.

Could model simply pick the less familiar or second-mentioned entity?

The ambiguity is intentional (§4.1), as the task probes which entity the model treats as unfamiliar to the user. Greater global visibility of a US entity is not equivalent to greater familiarity for users from the corresponding non-US context. The non-US entities were therefore selected for their local prominence within the respective cultural contexts (Appendix A), rather than as arbitrary or obscure alternatives. We also find that the same entity pairs can lead to different selections depending on the user's linguistic context (Figure 2), indicating that global familiarity alone is insufficient to explain the pattern.

We examine mention order separately in Appendix D (Figure 7). Models select the same entity in 66–94% of cases regardless of order, and the non-US entity remains preferred in both orderings. While order affects some settings, particularly cities, the broader pattern therefore cannot be explained simply by a preference for the second-mentioned entity.

Single response per prompt and frequency analysis

Our main evaluation reports aggregate selection rates rather than calibrated confidence for individual decisions (§5). To test whether a single sampled response is representative of the model's choice, we additionally evaluate response stability for Tulu3 by sampling ten responses per prompt at temperature 0.7 on 100 English city prompts. Among the 93 prompts with valid repeated samples, 96.8% produce the same choice across all valid samples and 98.9% show at least 80% agreement. The non-US entity is the majority choice for 92 of the 93 prompts, indicating that the observed selections are highly stable under repeated sampling.

To address the second point, we additionally test whether pair-level frequency differences are related to model choices. Using city frequencies from the Tulu3 SFT mixture, we compute the log-frequency gap between the US and non-US city and test whether it predicts selection of the non-US city as unfamiliar. We find a positive association across all six open models. From the smallest to the largest frequency-gap quartile, non-US selection increases from 67.4% to 97.2% for Command-R, 74.4% to 95.8% for Mistral, and 74.8% to 92.1% for Tulu3. Larger frequency gaps are therefore associated with a greater likelihood of selecting the non-US entity as unfamiliar, consistent with our discussion in §6.2 that uneven exposure may shape models' default familiarity assumptions. Our broader question is how these defaults change when information about the user is provided.

Native script or transliteration?

The native-language prompts use the corresponding native scripts (e.g., Devanagari for Hindi). The Hindi example in Figure 1 was transliterated into Latin script only for illustration. We will revise Figure 1 to make this distinction explicit.

Can the model infer where the user is from?

Recognizing the user's cultural context and using it to infer likely familiarity are separate steps. Our explicit-location condition isolates the second step by stating the user's location directly in the prompt. Even then, non-US selection remains at 85.9%, indicating that the geographic context is only weakly reflected in the familiarity attribution. We additionally test this directly for Tulu3. On a balanced sample of 100 native-language prompts across the ten language conditions, Tulu3 identifies the intended country correctly in 97% of cases. It is correct on all prompts in nine of the ten contexts and reaches 70% for Spanish/Mexico. This shows that, for Tulu3, the high non-US selection rate cannot simply be explained by a failure to recognize the cultural context conveyed by the prompt.

Related work cite

We thank the reviewer for pointing out these works and will add them to the revised paper. Jin et al. (2024) study implicit personalization, and Naous et al. (2024) examine Western cultural bias in model outputs. Our work instead asks what cultural knowledge models treat users as familiar with, and whether this changes with linguistic or geographic context.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.