acceptodds
Under review as a conference paper at ICLR 2027

CULTURAL ENTITY–PRACTICE COUPLING ACROSS CONTEXTS AND MODELS

Abstract

Cultural knowledge involves relations between entities and practices under particular social and situational conditions. We introduce Cultural Entity–Practice Coupling, an evidence-grounded framework for characterizing language models’ cultural relational competence through three connected dimensions: bidirectional relational retrieval, representational readability, and behavioral modifiability. Using 2,184 cultural records associated with six countries, we construct paired retrieval tasks with culturally grounded distractors and answer-position controls. Most evaluated models retrieve the supported relations in both directions, with variation across cultural contexts and presentation conditions. Contextconditioned entity and practice states support partial additive reconstruction of answer-anchor representations, and cross-template correspondence readout reveals transferable pairing information. Cue-conditioned residual interventions improve aggregate accuracy by 3.20 to 7.76 percentage points across three models and increase mean gold-answer margins in the four-choice analysis. Comparisons with matched controls characterize model- and direction-dependent intervention selectivity. Together, these findings establish complementary evidence for the retrieval, readability, and behavioral modifiability of cultural relations. The framework provides a structured way to characterize cultural relation knowledge beyond aggregate QA accuracy.

Primary area:
interpretability and explainable AI
Keywords:
Cultural Knowledge, Entity–Practice Coupling, Large Language Models, Cultural Representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gHXgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Cultural Entity–Practice Coupling, a framework for evaluating whether language models encode cultural knowledge as relations between entities and practices rather than as isolated facts. Each record links a cultural entity, such as a festival, to a documented practice under specific social conditions, supported by a source passage. Using 2,184 records from six countries, the authors test bidirectional retrieval through multiple-choice questions with same-country distractors, fit linear readouts that reconstruct answer-position hidden states from entity and practice representations, and train a cue-conditioned controller that injects residual perturbations to improve accuracy. They report that most models answer correctly in both directions, that the readouts generally rank true pairs above swapped ones, and that the intervention raises accuracy by 3 to 8 points on three models. The evidence-grounded dataset construction is careful, but the central claim is harder to accept. All relations come from highly public sources likely seen in pretraining, so strong accuracy and readable pairings are expected even from memorization alone. The readout extracts all states from the same forward pass, and the intervention is outperformed by a cue-independent control. Including less documented or debatable cultural relations would make the hidden-state analysis far more informative about how cultural entities are organized in representation space.

Critical strengths

The dataset is the strongest part of this work. Each of its 2,184 records is linked to a verbatim passage from an authoritative source and checked by a panel of three models, with human corrections where needed. The distractors are also carefully chosen: they are actual practices documented in the same country and category, making them plausible alternatives rather than arbitrary wrong answers. These choices make the task more challenging and give me greater confidence in the labels than in many existing cultural benchmarks. I see the dataset as the most reusable contribution, and it could support a solid benchmark paper with a narrower scope. Still, relying on well-documented public sources limits how much the benchmark can tell us about cultural competence beyond recalling encyclopedic facts. This strength also does not resolve my concerns about the soundness of the RQ2 and RQ3 analyses.

Critical weaknesses
  1. The data cannot separate relational knowledge from memorization. All relations come from highly public sources (government cultural sites, museums, UNESCO, encyclopedias) that are very likely in pretraining data. The paper’s stated goal is to measure relational competence beyond surface recognition, but with likely-memorized data, high accuracy, readable pairings, and steerability are all expected even under pure recall. There is no contamination check, lexical-overlap baseline, or evaluation on less-documented relations.

  2. RQ2 and RQ3 are not supported by the paper’s own controls. In RQ2, all states come from the same forward pass, so the readout mainly captures within-prompt dependence, and no counterfactual-pair control shows it tracks cultural validity. In RQ3, the controller is trained on gold-answer gradients, so accuracy gains are expected for any MCQ task, and a cue-independent global vector outperforms the cue-conditioned intervention in all six settings (Table 8), contradicting the central claim. Since RQ2 and RQ3 are what distinguish the paper from a standard benchmark, the contribution reduces to an MCQ evaluation on well-documented facts.

Questions for authors

My questions for the authors are:

1.All relations come from highly public sources likely seen in pretraining. Could you evaluate on relations unlikely to be memorized, such as less-documented community practices, post-cutoff sources, or fictional entity–practice pairs introduced in context? Does bidirectional accuracy hold, does a directional asymmetry appear, and does the RQ2 readout still separate valid from swapped pairs? If results hold, it would substantially address my main concern.

2.The paper reports only aggregate breakdowns by country, position, and direction. Did you analyze the cases where the relation is not retrieved? Are failures concentrated in particular categories, less-documented items, or one direction? Do RQ2 readout errors and RQ3 gains concentrate on the same items?

  1. In the balanced 2×2 test, accuracy drops from 85–91% (four options) to 70–80% (matched swaps). Does this indicate that a substantial part of four-option performance comes from cues other than the relation itself? Could you report a simple lexical baseline (e.g., BM25 or embedding similarity between stem and options) to show how much of the task is solvable by surface matching?

  2. In Table 8, a cue-independent global-gradient vector outperforms the cue-conditioned intervention in all six settings. What evidence remains that the cue conditioning, rather than a generic task direction, drives the RQ3 effect? Since the controller is trained on gold-answer margin gradients, have you applied the same procedure to a non-cultural MCQ task, and can the intervention steer the model toward a specific alternative relation rather than only toward the gold answer?

Clarifications:

1.Gemma-4-12B scores at chance and shows an inverted RQ2 pattern. Have you verified its label tokenization and chat template? Table 9’s values imply per-country denominators of roughly 34 to 103, not 364. What population is this table computed on? In the E→P direction, which practice span feeds the RQ2 readout? If it is the gold option’s tokens, how do you rule out leakage of the answer location?

Reviewer_KuQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Cultural Entity–Practice Coupling as a three-dimensional framework (bidirectional relational retrieval, representational readability, and behavioral modifiability) for characterizing cultural relational knowledge in LLMs. Seven LLMs are evaluated in a curated dataset of 2,184 cultural records from six countries using closed-book QA, hidden-state representation analysis, and activation interventions. The findings suggest that models can bidirectionally retrieve many documented cultural associations. They also show that entity and practice representations partially predict answer representations and that learned interventions can improve performance under certain experimental conditions.

Critical strengths
  1. The bidirectional evaluation protocol is potentially useful for identifying asymmetries in retrieving cultural associations.
  2. Connecting behavioral retrieval, representational analysis, and controlled interventions is potentially interesting. It can help investigate discrepancies between information that is recoverable from model representations and information that models successfully retrieve.
  3. The experiment setup seems plausible and rigorous.
Critical weaknesses
  1. Unclear scientific contribution and motivation: The paper does not convincingly establish the scientific advancement offered by the proposed framework beyond combining existing methods for relational retrieval, representation probing, and activation steering. Its practical relevance to cross-cultural settings is also insufficiently motivated. The paper is also dense and difficult to read. The explanations should be considerably improved.
  2. Insufficient evidence of cultural specificity: It remains unclear whether the findings reveal distinctive properties of cultural knowledge or general characteristics of factual associations in LLMs. Comparisons with non-cultural relational tasks would help establish the novelty and significance of the findings.
  3. Insufficient dataset validation details: The authors describe LLM-based validation supplemented by human spot-checks but do not provide sufficient details about the number of validated samples, annotators, inter-annotator agreement, human–LLM agreement, or disagreement resolution. These details are essential for establishing the reliability and cultural validity of the dataset.
  4. The main results are largely aggregated across countries, obscuring potential cultural biases and differences in retrieval performance. Given the paper's central claims, a more systematic country-level analysis is necessary.
  5. Entity, practice, and answer representations are extracted from the same contextualized forward pass, making it difficult to isolate genuine relational information from shared contextual information. Stronger baselines, including entity-only and practice-only readouts, are needed.
  6. The paper states that different variants of the same record are kept within a single data split, but it is unclear whether semantically related entities, practices, or supporting passages appear across training and evaluation sets. Entity-disjoint or source-disjoint experiments would strengthen the validity of the findings.
  7. Several technical terms and experimental procedures are insufficiently explained, making the manuscript difficult to follow. For example, the meaning of "specified composer" (line 168), the controls introduced around line 209, and variables in the bidirectional joint accuracy equation (lines 214–215) require clearer definitions. The abbreviation "LSE" in Equation 3 should be introduced.
  8. Gemma-4-12B performs near chance on the primary QA task, substantially worse than the other evaluated models, without adequate investigation. The authors should examine potential model-specific prompting or scoring issues and justify why intervention experiments cover only three of the seven models.
Questions for authors
  1. It would be great if the authors can explain what new scientific understanding of cultural knowledge does the proposed framework provides beyond existing approaches to relational retrieval, representation probing, and activation steering.
  2. How was the cultural validity of the 2,184 records established? Specifically, how many samples were manually validated, what were the qualifications and cultural backgrounds of the annotators, what were the human–human and human–LLM agreement rates, and how were disagreements or culturally ambiguous associations resolved?
  3. How do the authors explain Gemma's low performance? Does changing and finetuning the prompt change the results?
Reviewer_e5idOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Cultural Entity–Practice Coupling, a framework for studying how language models retrieve, represent, and modify contextualized cultural relationships. Using an evidence-grounded dataset of 2,184 cultural records from six countries, with paired multiple-choice questions and culturally plausible distractors, it examines three things:

Bidirectional retrieval: whether models can map entities to practices and practices back to entities. Representational readability: whether hidden states can reconstruct answer representations and tell valid entity–practice pairs from mismatched ones. Behavioral modifiability: whether learned, cue-conditioned residual-stream interventions can improve relational predictions.

Across seven models, retrieval is generally strong in both directions, though it varies by model, cultural context, and answer position. Hidden states support partial additive reconstruction and tell valid pairs from mismatched ones better than chance. On three models, interventions raise accuracy by 3.20–7.76 percentage points, but comparisons with matched controls show that how selective they are varies. The framework offers a unified way to probe cultural relational knowledge beyond standard QA accuracy.

Critical strengths

Structured, evidence-grounded evaluation of cultural relationships. Rather than testing isolated facts, the paper models how cultural entities, practices, and social contexts relate to each other. Source-supported answers, plausible distractors, bidirectional questions, and answer-position controls make the evaluation of contextual compatibility systematic, and the balanced 2×2 experiments control for how often each component appears. This matters because standard cultural QA accuracy may not show whether models consistently recognize these relationships.

Behavior, representations, and interventions studied together. The paper analyzes cultural relational knowledge at three levels instead of relying on accuracy alone. Seven models, held-out readouts, cross-template analysis, and multiple intervention controls add depth. A key insight is that information recoverable from representations can differ from task performance, as Gemma shows. The source-shuffled, pair-swapped, opposite-direction, and global-gradient controls also show how selective activation steering is and where it falls short.

Critical weaknesses

W1. The representation analysis does not separate relational knowledge from input context. In RQ2, entity, practice, and answer-anchor representations all come from the same forward pass over the full question, so each is conditioned on the question and answer options rather than encoded independently. Successful reconstruction and pair ranking may therefore reflect information already present in the shared input, or general representational properties, rather than an internal representation of the cultural relationship. The authors acknowledge this, and the cross-template and balanced-pair controls help, but they do not settle the interpretation. Option-blind or independently encoded controls would be more convincing. This matters because representational readability is one of the paper’s three core contributions, yet the results mainly show that contextual information can be recovered, not that a distinct relational representation exists.

W2. The interventions show little evidence of culture-specific steering beyond generic task-level gains. The 3.20–7.76 point gains in RQ3 are hard to attribute. In Table 8, the global-gradient control improves the gold-answer margin more than the directional intervention in every model–template comparison, and the source-shuffled control matches or beats it for Qwen and Llama. The authors acknowledge a large shared task-level signal, but this suggests much of the gain may come from general multiple-choice optimization rather than a cultural or relation-conditioned mechanism. The behavioral improvements still stand, but the mechanistic claim is weaker. Comparisons with generic relational reasoning tasks and a clear estimate of the extra benefit from the cultural cue would strengthen it.

Questions for authors

Q1. Isolating relational signal (RQ2). All states come from the same full-QA pass, so how do you rule out that reconstruction and pair ranking rely on the shared question and options rather than the entity–practice link? Would an option-blind readout, or entity and practice encoded in separate passes, keep the effect? Does a readout trained on mismatched pairs in the same template reach similar accuracy?

Q2. Specificity of interventions (RQ3). The global-gradient control beats the directional intervention on gold-answer margin everywhere in Table 8, and source-shuffled steering matches or beats it for Qwen and Llama. Which comparison best supports a relation-specific effect? Could you report the directional gain minus the strongest control per condition, and test whether the learned directions transfer to non-cultural relational MCQs? If they transfer, that would point to generic steering.

Q3. Annotation validity. Given the heavy reliance on model-assisted extraction and verification, what share of records did humans verify, with what cultural expertise, and what were the error and disagreement rates? How often were distractors judged as also correct, and does accuracy differ between human-verified and model-only subsets?

Q4. Coupling across the three dimensions. Do retrieval, readability, and modifiability correlate per relation? For example, are relations with stronger readouts retrieved more reliably, or helped more by interventions? Without such links, the “unified framework” reads as three parallel studies. A per-relation correlation or a predictive analysis, such as whether readout strength predicts intervention gain, would support the unifying claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.