- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Cultural Entity–Practice Coupling, a framework for evaluating whether language models encode cultural knowledge as relations between entities and practices rather than as isolated facts. Each record links a cultural entity, such as a festival, to a documented practice under specific social conditions, supported by a source passage. Using 2,184 records from six countries, the authors test bidirectional retrieval through multiple-choice questions with same-country distractors, fit linear readouts that reconstruct answer-position hidden states from entity and practice representations, and train a cue-conditioned controller that injects residual perturbations to improve accuracy. They report that most models answer correctly in both directions, that the readouts generally rank true pairs above swapped ones, and that the intervention raises accuracy by 3 to 8 points on three models. The evidence-grounded dataset construction is careful, but the central claim is harder to accept. All relations come from highly public sources likely seen in pretraining, so strong accuracy and readable pairings are expected even from memorization alone. The readout extracts all states from the same forward pass, and the intervention is outperformed by a cue-independent control. Including less documented or debatable cultural relations would make the hidden-state analysis far more informative about how cultural entities are organized in representation space.
Critical strengths
The dataset is the strongest part of this work. Each of its 2,184 records is linked to a verbatim passage from an authoritative source and checked by a panel of three models, with human corrections where needed. The distractors are also carefully chosen: they are actual practices documented in the same country and category, making them plausible alternatives rather than arbitrary wrong answers. These choices make the task more challenging and give me greater confidence in the labels than in many existing cultural benchmarks. I see the dataset as the most reusable contribution, and it could support a solid benchmark paper with a narrower scope. Still, relying on well-documented public sources limits how much the benchmark can tell us about cultural competence beyond recalling encyclopedic facts. This strength also does not resolve my concerns about the soundness of the RQ2 and RQ3 analyses.
Critical weaknesses
-
The data cannot separate relational knowledge from memorization. All relations come from highly public sources (government cultural sites, museums, UNESCO, encyclopedias) that are very likely in pretraining data. The paper’s stated goal is to measure relational competence beyond surface recognition, but with likely-memorized data, high accuracy, readable pairings, and steerability are all expected even under pure recall. There is no contamination check, lexical-overlap baseline, or evaluation on less-documented relations.
-
RQ2 and RQ3 are not supported by the paper’s own controls. In RQ2, all states come from the same forward pass, so the readout mainly captures within-prompt dependence, and no counterfactual-pair control shows it tracks cultural validity. In RQ3, the controller is trained on gold-answer gradients, so accuracy gains are expected for any MCQ task, and a cue-independent global vector outperforms the cue-conditioned intervention in all six settings (Table 8), contradicting the central claim. Since RQ2 and RQ3 are what distinguish the paper from a standard benchmark, the contribution reduces to an MCQ evaluation on well-documented facts.
Questions for authors
My questions for the authors are:
1.All relations come from highly public sources likely seen in pretraining. Could you evaluate on relations unlikely to be memorized, such as less-documented community practices, post-cutoff sources, or fictional entity–practice pairs introduced in context? Does bidirectional accuracy hold, does a directional asymmetry appear, and does the RQ2 readout still separate valid from swapped pairs? If results hold, it would substantially address my main concern.
2.The paper reports only aggregate breakdowns by country, position, and direction. Did you analyze the cases where the relation is not retrieved? Are failures concentrated in particular categories, less-documented items, or one direction? Do RQ2 readout errors and RQ3 gains concentrate on the same items?
-
In the balanced 2×2 test, accuracy drops from 85–91% (four options) to 70–80% (matched swaps). Does this indicate that a substantial part of four-option performance comes from cues other than the relation itself? Could you report a simple lexical baseline (e.g., BM25 or embedding similarity between stem and options) to show how much of the task is solvable by surface matching?
-
In Table 8, a cue-independent global-gradient vector outperforms the cue-conditioned intervention in all six settings. What evidence remains that the cue conditioning, rather than a generic task direction, drives the RQ3 effect? Since the controller is trained on gold-answer margin gradients, have you applied the same procedure to a non-cultural MCQ task, and can the intervention steer the model toward a specific alternative relation rather than only toward the gold answer?
Clarifications:
1.Gemma-4-12B scores at chance and shows an inverted RQ2 pattern. Have you verified its label tokenization and chat template? Table 9’s values imply per-country denominators of roughly 34 to 103, not 364. What population is this table computed on? In the E→P direction, which practice span feeds the RQ2 readout? If it is the gold option’s tokens, how do you rule out leakage of the answer location?