- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FACADE, a benchmark designed to evaluate whether LLMs can distinguish behavioral compliance with a cultural norm from alignment with the underlying value. It constructs 9,416 stories from 2,354 norms across 75 countries and regions, organizes them into four behavior–value categories, and evaluates 20 LLMs under different prompting conditions. The paper reports that models struggle particularly with surface compliance, where behavior appears to follow a norm but does not reflect its underlying value.
Critical strengths
-
The Behavior–Value matrix is a useful design choice because it creates matched contrasts rather than relying exclusively on isolated cultural appropriateness questions. The four-quadrant setup helps distinguish failures involving surface-level compliance from failures involving explicit non-compliance. This makes the benchmark more diagnostically informative than a single aggregate accuracy score.
-
The benchmark covers 2,354 norms and evaluates 20 LLMs. The authors also conduct human quality audits, report a human performance baseline, test a second story generator, and examine prompt sensitivity. In particular, the persistence of the Surface Compliance gap across models and story generators makes the main empirical observation more convincing than a result based on a single model family or generation pipeline.
Critical weaknesses
-
The gold labels are determined by the benchmark's operationalization of an extracted underlying value: F/P are positive and S/N are negative. However, the evaluated models see only the country or region and the story, not the source norm or the extracted value. Consequently, the benchmark measures whether models infer and apply the authors' intended value interpretation from the narrative, which is not necessarily equivalent to understanding culturally appropriate behavior in the corresponding population. The authors appropriately acknowledge that their sources do not establish population-wide cultural consensus, but this limitation still constrains the interpretation of the main findings as evidence of cultural misalignment. The human baseline demonstrates that the intended labels are recoverable, but does not by itself establish that these labels represent culturally representative judgments.
-
Although the authors instruct the generator to avoid explicit evaluative language, Surface Compliance stories necessarily include additional conduct that contradicts the intended value. Such details may provide generic cues of selfishness, disrespect, or interpersonal harm. A model could therefore reject some Surface Compliance stories by recognizing broadly negative behavior without recovering the particular value attached to the source norm. The human baseline and cross-generator evaluation establish that the phenomenon is not trivially absent, but do not fully separate value-sensitive reasoning from sensitivity to general negative cues. More targeted controls would help clarify the mechanism behind the reported error pattern.
Questions for authors
-
How should readers interpret the benchmark's gold labels when the underlying values are extracted from source descriptions and verified by annotators who are not necessarily members of the relevant cultural communities? Do the authors have any population-matched or culture-specific validation that can establish whether the labels reflect judgments made by people from the corresponding contexts, rather than only fidelity to the selected source and the authors' operationalization?
-
Can the authors evaluate a controlled subset in which generic negative sentiment or interpersonal harm is held approximately constant while the relationship between the focal behavior and the norm's underlying value varies? Such an analysis would help determine whether the Surface Compliance gap specifically reflects sensitivity to form–value conflicts rather than general detection of undesirable behavior.
-
How were the five human participants recruited, and what cultural or regional familiarity did they have with the evaluated norms? Could the authors report participant-level variation or clarify the extent to which the human baseline should be interpreted as a general human benchmark rather than a population-representative cultural reference?
-
Beyond introducing the benchmark, what generalizable technical insight does FACADE provide about LLMs, and how might that insight inform model training or evaluation beyond this specific task?
4 comments
Part2: The response to question 2.
Q2:
Can the authors evaluate a controlled subset in which generic negative sentiment or interpersonal harm is held approximately constant while the relationship between the focal behavior and the norm's underlying value varies? Such an analysis would help determine whether the Surface Compliance gap specifically reflects sensitivity to form–value conflicts rather than general detection of undesirable behavior.
A2:
Thank you for pointing out that generic negative cues may influence models’ judgments of Surface Compliance. We constructed a controlled 2 × 2 subset from 1,000 candidate norms. For each norm, we started with the original F and S action descriptions and inserted the same neutral conduct or its matched negative counterpart at corresponding positions. This additional conduct was unrelated to the target value, and the two versions differed only in negativity. We then made local edits within a shared narrative framework to produce four stories. All four versions retained the same Expected Behavior; F preserved the target value, whereas S violated it. This design allows us to compare F and S under the same cue condition and examine the influence of negative cues while holding Value Alignment constant.
Following independent verification, 962 sets passed verification for all four stories and were evaluated using four models. Within each cue condition, F and S received similar ratings of target-irrelevant negativity: 1.1 versus 1.2 in the neutral condition and 3.6 versus 3.6 in the negative condition. The 90% confidence intervals for the paired rating differences in both conditions fell within the prespecified equivalence bounds of ±0.5 rating points.
Without the additional negative cues, F and S accuracies were 98.3% and 76.5%, respectively, a gap of 21.8 percentage points. With negative cues, the corresponding accuracies were 96.8% and 78.1%, respectively, a gap of 18.7 percentage points. Adding negative cues reduced the Yes rates for F and S by 1.5 and 1.6 percentage points, respectively, indicating a small shift toward No in both quadrants. Models’ distinction between preservation and violation of the target value remained largely unchanged, and S remained harder to judge than F under both conditions.
We have expanded the Matched target-irrelevant negativity control paragraph in §4.3.5 of the main text to describe the experimental design and main findings. The complete construction procedure, independent verification, equivalence checks, and evaluation results across four models are provided in Appendix L.2 and Table 20. We further discuss the scope and limitations of this experiment in Appendix O. All corresponding revisions are highlighted in blue.
Part3: The response to question 3.
Q3:
How were the five human participants recruited, and what cultural or regional familiarity did they have with the evaluated norms? Could the authors report participant-level variation or clarify the extent to which the human baseline should be interpreted as a general human benchmark rather than a population-representative cultural reference?
A3:
Thank you for highlighting the need to clarify the recruitment, cultural or regional familiarity, participant-level variation, and interpretation of our human baseline.
First, we recruited five participants from different cultural backgrounds through campus announcements and selected them through a screening test. Before formal evaluation, the participants received standardized training from experts in translation and interpreting at our institution. Appendix D of the revised manuscript now reports this recruitment and training information.
Second, Appendix D provides the participants’ anonymized cultural/regional backgrounds. The participants were familiar with their own cultural and regional contexts, broadening the regional coverage of the study.
Third, we have added an anonymized table in Appendix D reporting each participant’s F/P/S/N accuracy and CIR, together with the mean and between-participant standard deviation. All five participants independently evaluated the same 1,000 norms comprising 4,000 stories. CIR requires all four judgments for a norm to be correct and is therefore calculated separately for each participant.
Fourth, the human baseline should be interpreted as a study-specific human performance reference under the benchmark’s evaluation conditions, not as a population-representative cultural reference. It provides evidence that these participants can recover the intended form–value distinction from the narratives and supports comparison with models on the same items.
Part4: The response to question 4.
Q4:
Beyond introducing the benchmark, what generalizable technical insight does FACADE provide about LLMs, and how might that insight inform model training or evaluation beyond this specific task?
A4:
FACADE provides a generalizable technical insight: LLMs can correctly judge clear-cut cases yet accept behavior that follows a rule’s outward form while violating its underlying purpose. This matters when models apply policies or evaluate behavior, where reliable decisions require recognizing both surface compliance and justified exceptions.
For model training, F and P provide positive examples, while S and N provide negative examples. P and S serve as hard positives and hard negatives, respectively, for supervised learning or preference training. As a preliminary prompt-time application, using one fixed complete four-shot F/P/S/N demonstration set improved CIR by 3.0–7.0 percentage points and S accuracy by 4.0–6.0 points across four models (§4.4; Appendix N).
For evaluation, improvements on S should be assessed alongside performance on F and P to determine whether apparent gains result from indiscriminate rejection. Evaluating complete contrast sets also tests whether a model makes consistent judgments across cases grounded in the same underlying norm.
For benchmark construction, a broader principle is to cross potential shortcut cues with task correctness and examine all four combinations. For example, a reading-comprehension benchmark could cross high/low lexical overlap with correct/incorrect answers. Comparing different levels of lexical overlap while holding answer correctness fixed tests sensitivity to surface cues; comparing correct and incorrect answers while holding lexical overlap fixed tests whether models recognize the substantive distinction. Performance across these groups can reveal weaknesses hidden by aggregate accuracy and inform targeted training.
Part1: The response to question 1.
Q1:
How should readers interpret the benchmark's gold labels when the underlying values are extracted from source descriptions and verified by annotators who are not necessarily members of the relevant cultural communities? Do the authors have any population-matched or culture-specific validation that can establish whether the labels reflect judgments made by people from the corresponding contexts, rather than only fidelity to the selected source and the authors' operationalization?
A1:
Thank you for highlighting the distinction between fidelity to the selected source and cultural representativeness. FACADE’s gold labels should be interpreted as judgments grounded in the selected source norms: F/P indicate that the behavior depicted in the story preserves the Underlying Value extracted from the source norm, whereas S/N indicate that it violates that value.
Before formal annotation, annotators received standardized training from experts in the School of Foreign Languages and Literature. The training instructed annotators to consider the specific cultural contexts described in the source materials when verifying whether the extracted Underlying Value was supported by the source and whether the behavioral evidence in each story supported the corresponding label (Appendix D).
The revised main evaluation explicitly provides models with the same Underlying Value and asks whether the protagonist’s behavior preserves it. The results therefore reflect models’ ability to apply a given value to a specific context. The existing human verification supports the labels’ consistency with the source norms and story evidence.