acceptodds
Under review as a conference paper at ICLR 2027

FACADE: Uncovering Value Misalignment in LLM Judgments of Apparently Compliant Behavior

Abstract

Behavior can follow an expected form while violating the value that form is meant to express. We call this form–value pattern Performative Compliance. We introduce FACADE, a benchmark for evaluating whether LLMs recognize this pattern in narratives, built on a Behavior–Value matrix with two independently manipulated dimensions: Behavior Adherence and Value Alignment. Combining them yields four types of depicted behavior: Full Compliance, Principled Compliance (deviation from behavioral form while preserving intent), Surface Compliance (form adherence without intent alignment), and Non-Compliance. Cultural norms serve as a natural testbed because each retained norm involves both an explicit behavioral expectation and an implicit social intent, making form–intent divergence concretely operationalizable. To improve contextual realism, we propose a news-augmented story generation pipeline grounded in real-world event patterns. Across 20 LLMs evaluated on 9,416 stories derived from source norms associated with 75 countries/regions, models reliably judge Full and Principled Compliance but perform substantially worse on Surface Compliance, often accepting outwardly compliant yet value-violating behavior as appropriate.

Primary area:
datasets and benchmarks
Keywords:
Value Misalignment, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pb13OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FACADE, a benchmark designed to evaluate whether LLMs can distinguish behavioral compliance with a cultural norm from alignment with the underlying value. It constructs 9,416 stories from 2,354 norms across 75 countries and regions, organizes them into four behavior–value categories, and evaluates 20 LLMs under different prompting conditions. The paper reports that models struggle particularly with surface compliance, where behavior appears to follow a norm but does not reflect its underlying value.

Critical strengths
  1. The Behavior–Value matrix is a useful design choice because it creates matched contrasts rather than relying exclusively on isolated cultural appropriateness questions. The four-quadrant setup helps distinguish failures involving surface-level compliance from failures involving explicit non-compliance. This makes the benchmark more diagnostically informative than a single aggregate accuracy score.

  2. The benchmark covers 2,354 norms and evaluates 20 LLMs. The authors also conduct human quality audits, report a human performance baseline, test a second story generator, and examine prompt sensitivity. In particular, the persistence of the Surface Compliance gap across models and story generators makes the main empirical observation more convincing than a result based on a single model family or generation pipeline.

Critical weaknesses
  1. The gold labels are determined by the benchmark's operationalization of an extracted underlying value: F/P are positive and S/N are negative. However, the evaluated models see only the country or region and the story, not the source norm or the extracted value. Consequently, the benchmark measures whether models infer and apply the authors' intended value interpretation from the narrative, which is not necessarily equivalent to understanding culturally appropriate behavior in the corresponding population. The authors appropriately acknowledge that their sources do not establish population-wide cultural consensus, but this limitation still constrains the interpretation of the main findings as evidence of cultural misalignment. The human baseline demonstrates that the intended labels are recoverable, but does not by itself establish that these labels represent culturally representative judgments.

  2. Although the authors instruct the generator to avoid explicit evaluative language, Surface Compliance stories necessarily include additional conduct that contradicts the intended value. Such details may provide generic cues of selfishness, disrespect, or interpersonal harm. A model could therefore reject some Surface Compliance stories by recognizing broadly negative behavior without recovering the particular value attached to the source norm. The human baseline and cross-generator evaluation establish that the phenomenon is not trivially absent, but do not fully separate value-sensitive reasoning from sensitivity to general negative cues. More targeted controls would help clarify the mechanism behind the reported error pattern.

Questions for authors
  1. How should readers interpret the benchmark's gold labels when the underlying values are extracted from source descriptions and verified by annotators who are not necessarily members of the relevant cultural communities? Do the authors have any population-matched or culture-specific validation that can establish whether the labels reflect judgments made by people from the corresponding contexts, rather than only fidelity to the selected source and the authors' operationalization?

  2. Can the authors evaluate a controlled subset in which generic negative sentiment or interpersonal harm is held approximately constant while the relationship between the focal behavior and the norm's underlying value varies? Such an analysis would help determine whether the Surface Compliance gap specifically reflects sensitivity to form–value conflicts rather than general detection of undesirable behavior.

  3. How were the five human participants recruited, and what cultural or regional familiarity did they have with the evaluated norms? Could the authors report participant-level variation or clarify the extent to which the human baseline should be interpreted as a general human benchmark rather than a population-representative cultural reference?

  4. Beyond introducing the benchmark, what generalizable technical insight does FACADE provide about LLMs, and how might that insight inform model training or evaluation beyond this specific task?

4 comments
AuthorsOpenReview ↗

Part1: The response to question 1.

Q1:

How should readers interpret the benchmark's gold labels when the underlying values are extracted from source descriptions and verified by annotators who are not necessarily members of the relevant cultural communities? Do the authors have any population-matched or culture-specific validation that can establish whether the labels reflect judgments made by people from the corresponding contexts, rather than only fidelity to the selected source and the authors' operationalization?

A1:

Thank you for highlighting the distinction between fidelity to the selected source and cultural representativeness. FACADE’s gold labels should be interpreted as judgments grounded in the selected source norms: F/P indicate that the behavior depicted in the story preserves the Underlying Value extracted from the source norm, whereas S/N indicate that it violates that value.

Before formal annotation, annotators received standardized training from experts in the School of Foreign Languages and Literature. The training instructed annotators to consider the specific cultural contexts described in the source materials when verifying whether the extracted Underlying Value was supported by the source and whether the behavioral evidence in each story supported the corresponding label (Appendix D).

The revised main evaluation explicitly provides models with the same Underlying Value and asks whether the protagonist’s behavior preserves it. The results therefore reflect models’ ability to apply a given value to a specific context. The existing human verification supports the labels’ consistency with the source norms and story evidence.

AuthorsOpenReview ↗

Part2: The response to question 2.

Q2:

Can the authors evaluate a controlled subset in which generic negative sentiment or interpersonal harm is held approximately constant while the relationship between the focal behavior and the norm's underlying value varies? Such an analysis would help determine whether the Surface Compliance gap specifically reflects sensitivity to form–value conflicts rather than general detection of undesirable behavior.

A2:

Thank you for pointing out that generic negative cues may influence models’ judgments of Surface Compliance. We constructed a controlled 2 × 2 subset from 1,000 candidate norms. For each norm, we started with the original F and S action descriptions and inserted the same neutral conduct or its matched negative counterpart at corresponding positions. This additional conduct was unrelated to the target value, and the two versions differed only in negativity. We then made local edits within a shared narrative framework to produce four stories. All four versions retained the same Expected Behavior; F preserved the target value, whereas S violated it. This design allows us to compare F and S under the same cue condition and examine the influence of negative cues while holding Value Alignment constant.

Following independent verification, 962 sets passed verification for all four stories and were evaluated using four models. Within each cue condition, F and S received similar ratings of target-irrelevant negativity: 1.1 versus 1.2 in the neutral condition and 3.6 versus 3.6 in the negative condition. The 90% confidence intervals for the paired rating differences in both conditions fell within the prespecified equivalence bounds of ±0.5 rating points.

Without the additional negative cues, F and S accuracies were 98.3% and 76.5%, respectively, a gap of 21.8 percentage points. With negative cues, the corresponding accuracies were 96.8% and 78.1%, respectively, a gap of 18.7 percentage points. Adding negative cues reduced the Yes rates for F and S by 1.5 and 1.6 percentage points, respectively, indicating a small shift toward No in both quadrants. Models’ distinction between preservation and violation of the target value remained largely unchanged, and S remained harder to judge than F under both conditions.

We have expanded the Matched target-irrelevant negativity control paragraph in §4.3.5 of the main text to describe the experimental design and main findings. The complete construction procedure, independent verification, equivalence checks, and evaluation results across four models are provided in Appendix L.2 and Table 20. We further discuss the scope and limitations of this experiment in Appendix O. All corresponding revisions are highlighted in blue.

AuthorsOpenReview ↗

Part3: The response to question 3.

Q3:

How were the five human participants recruited, and what cultural or regional familiarity did they have with the evaluated norms? Could the authors report participant-level variation or clarify the extent to which the human baseline should be interpreted as a general human benchmark rather than a population-representative cultural reference?

A3:

Thank you for highlighting the need to clarify the recruitment, cultural or regional familiarity, participant-level variation, and interpretation of our human baseline.

First, we recruited five participants from different cultural backgrounds through campus announcements and selected them through a screening test. Before formal evaluation, the participants received standardized training from experts in translation and interpreting at our institution. Appendix D of the revised manuscript now reports this recruitment and training information.

Second, Appendix D provides the participants’ anonymized cultural/regional backgrounds. The participants were familiar with their own cultural and regional contexts, broadening the regional coverage of the study.

Third, we have added an anonymized table in Appendix D reporting each participant’s F/P/S/N accuracy and CIR, together with the mean and between-participant standard deviation. All five participants independently evaluated the same 1,000 norms comprising 4,000 stories. CIR requires all four judgments for a norm to be correct and is therefore calculated separately for each participant.

Fourth, the human baseline should be interpreted as a study-specific human performance reference under the benchmark’s evaluation conditions, not as a population-representative cultural reference. It provides evidence that these participants can recover the intended form–value distinction from the narratives and supports comparison with models on the same items.

AuthorsOpenReview ↗

Part4: The response to question 4.

Q4:

Beyond introducing the benchmark, what generalizable technical insight does FACADE provide about LLMs, and how might that insight inform model training or evaluation beyond this specific task?

A4:

FACADE provides a generalizable technical insight: LLMs can correctly judge clear-cut cases yet accept behavior that follows a rule’s outward form while violating its underlying purpose. This matters when models apply policies or evaluate behavior, where reliable decisions require recognizing both surface compliance and justified exceptions.

For model training, F and P provide positive examples, while S and N provide negative examples. P and S serve as hard positives and hard negatives, respectively, for supervised learning or preference training. As a preliminary prompt-time application, using one fixed complete four-shot F/P/S/N demonstration set improved CIR by 3.0–7.0 percentage points and S accuracy by 4.0–6.0 points across four models (§4.4; Appendix N).

For evaluation, improvements on S should be assessed alongside performance on F and P to determine whether apparent gains result from indiscriminate rejection. Evaluating complete contrast sets also tests whether a model makes consistent judgments across cases grounded in the same underlying norm.

For benchmark construction, a broader principle is to cross potential shortcut cues with task correctness and examine all four combinations. For example, a reading-comprehension benchmark could cross high/low lexical overlap with correct/incorrect answers. Comparing different levels of lexical overlap while holding answer correctness fixed tests sensitivity to surface cues; comparing correct and incorrect answers while holding lexical overlap fixed tests whether models recognize the substantive distinction. Performance across these groups can reveal weaknesses hidden by aggregate accuracy and inform targeted training.

Reviewer_gkggOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether language models can recognize that following an expected behavior does not necessarily preserve its underlying value. Using cultural norms as a testbed, it introduces FACADE, a framework for evaluating judgments of performative compliance in narratives. For each source-conditioned norm, it constructs four stories: Full and Principled Compliance receive Yes labels, while Surface and Non-Compliance receive No labels. The primary benchmark contains 2,354 norms and 9,416 stories associated with 75 countries/regions. Across 20 LLMs, Surface Compliance has the lowest aggregate accuracy. A preliminary in-context learning experiment also reports improved Surface Compliance accuracy using a complete F/P/S/N demonstration set.

Critical strengths
  1. The research question, i.e., whether LLMs can distinguish following the underlying values from just the surface behaviors, is meaningful. And this paper proposes an behavior-value evaluation framework to diagnose this problem.
  2. This paper develop a benchmark with 2,354 source-conditioned norms associated with 75 countries/regions. And the realism of the generated stories and the verification from real humans are also considered.
  3. Theyt conduct empirical studies across 20 LLMs and obtain a consistent observation: Surface Compliance is hard for LLMs to identify.
Critical weaknesses
  1. Some details should be further clarified.
  • In Table 9, there are three domains of datasets, while Section 3.1 only formulates culturally situated norms. It should more clearly explain how the relevant context, expected behavior, and underlying value are specified consistently across domains. In addition, how are stories of workplace and everyday life norms be judged as appropriate or inappropriate, which depends on the cultural contexts?
  • Furthermore, the four stories are generated separately from shared norm representations and background information. Beyond the reported human comparability checks, it remains important to clarify how differences in justification, surrounding conduct, and narrative complexity are controlled.
  1. The evaluation prompt and the scoring criterion are not fully aligned. Models are asked whether the main character’s behavior is appropriate in the stated context, whereas the gold labels are determined by preservation of one fixed Underlying Value that is hidden in the main evaluation. Overall appropriateness may involve several norms or competing values, so a judgment that differs from the benchmark label does not necessarily demonstrate failure to recognize the intended value violation. This is particularly relevant to low Surface Compliance.

  2. The lower accuracy on Surface Compliance may partly reflect a mismatch between the evaluation question and the labeling criterion, rather than incorrect judgments by the models. Models are asked whether a behavior is appropriate overall, whereas Surface Compliance stories receive No labels because they violate a particular target value despite following the expected behavior. Violating that value does not necessarily make the behavior inappropriate under every reasonable interpretation of the context. The higher accuracy on Non-Compliance is relevant here: cases that violate both the expected behavior and the target value may be more clearly inappropriate, while Surface Compliance cases may leave more room for disagreement. Consequently, some Yes responses to Surface Compliance stories could reflect reasonable judgments of overall appropriateness rather than failures to recognize the target value violation. The paper should examine such cases before interpreting the lower accuracy as evidence of deficient value recognition.

If the authors can address the concerns above, I am open to raise my score.

Questions for authors

See Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Part1: The response to weakness 1.

W1: Some details should be further clarified.

W1.1: In Table 9, there are three domains of datasets, while Section 3.1 only formulates culturally situated norms. It should more clearly explain how the relevant context, expected behavior, and underlying value are specified consistently across domains. In addition, how are stories of workplace and everyday life norms be judged as appropriate or inappropriate, which depends on the cultural contexts?

A1.1:

Thank you for pointing out the need to clarify cross-domain construction and scoring. The cultural, workplace, and everyday-life domains use the same norm representation, four-quadrant construction framework, and scoring rules, with norms drawn from the corresponding domain-specific sources.

(1) How the relevant context, expected behavior, and underlying value are specified consistently across domains.

All three domains use the same tuple r = (c, τ, b, v), where c records the source country or region, and τ is the Source Norm Statement drawn from the corresponding domain-specific source (see Table 9 for representative sources). The Expected Behavior b and Underlying Value v are extracted from that statement and verified against it using the same criteria. Four-quadrant construction and human verification—covering quadrant consistency, information sufficiency, comparability, and neutral wording—follow the same procedures across all three domains.

(2) How workplace and everyday-life stories are evaluated.

The cultural, workplace, and everyday-life domains use the same model inputs and evaluation question. For each story, the model receives the Source Country/Region, Underlying Value, and Story, and is asked whether the protagonist’s behavior preserves the stated Underlying Value. The gold-label rule is also identical across domains: F and P are labeled Yes, whereas S and N are labeled No.

To clarify this process, we have added the cross-domain construction description in §3.1, specified the shared input structure, evaluation question, and scoring rule in §4.3.6, and added workplace and everyday-life examples as Tables 14 and 15 in Appendix G.

W1: Some details should be further clarified.

W1.2: Furthermore, the four stories are generated separately from shared norm representations and background information. Beyond the reported human comparability checks, it remains important to clarify how differences in justification, surrounding conduct, and narrative complexity are controlled.

A1.2:

Thank you for pointing out that generating the four stories separately may introduce additional differences. We clarify below how the construction process addresses differences in justification, surrounding conduct, and narrative complexity.

Justification. Section 3.2 and Appendix B require P stories to explain deviations from the Expected Behavior through objective constraints and show how a reasonable adaptation preserves the Underlying Value. This justification is an intentional construction element of the P quadrant.

Surrounding conduct. The action descriptions across all four quadrants are constrained by the same norm’s Expected Behavior and Underlying Value, so that the protagonist’s surrounding conduct provides evidence relevant to judging that value. After generation, annotators review the four stories as a set for quadrant consistency, information sufficiency, and story comparability (§§3.2–3.3; Appendix D).

Narrative complexity. We use a shared template with a target length of 180–220 words and require stories to avoid explicit evaluative wording that could reveal the labels (§3.3). These requirements constrain story length and wording.

AuthorsOpenReview ↗

Part2: The response to weaknesses 2 and 3.

W2:

The evaluation prompt and the scoring criterion are not fully aligned. Models are asked whether the main character’s behavior is appropriate in the stated context, whereas the gold labels are determined by preservation of one fixed Underlying Value that is hidden in the main evaluation. Overall appropriateness may involve several norms or competing values, so a judgment that differs from the benchmark label does not necessarily demonstrate failure to recognize the intended value violation. This is particularly relevant to low Surface Compliance.

W3:

The lower accuracy on Surface Compliance may partly reflect a mismatch between the evaluation question and the labeling criterion, rather than incorrect judgments by the models. Models are asked whether a behavior is appropriate overall, whereas Surface Compliance stories receive No labels because they violate a particular target value despite following the expected behavior. Violating that value does not necessarily make the behavior inappropriate under every reasonable interpretation of the context. The higher accuracy on Non-Compliance is relevant here: cases that violate both the expected behavior and the target value may be more clearly inappropriate, while Surface Compliance cases may leave more room for disagreement. Consequently, some Yes responses to Surface Compliance stories could reflect reasonable judgments of overall appropriateness rather than failures to recognize the target value violation. The paper should examine such cases before interpreting the lower accuracy as evidence of deficient value recognition.

A2 & A3:

Thank you for pointing out that overall appropriateness may involve several norms or competing values. Under the original evaluation question, a Yes judgment on S could therefore reflect a trade-off involving other considerations. That answer alone does not establish whether the model recognized the intended value violation.

To align the evaluation prompt with the scoring criterion, the revised main evaluation explicitly provides the Underlying Value and directly asks whether the protagonist’s behavior preserves that value. We re-evaluated all 20 models on the same 9,416 stories. Mean S accuracy increased from 63.55% to 66.37%, while N accuracy increased from 91.69% to 92.78%. Both quadrants have the gold label No, yet S accuracy remains 26.41 percentage points below N under the revised question. This shows that, even when the target value and judgment criterion are explicit, models still struggle to judge whether behavior that follows the expected form preserves that value.

We have updated the evaluation protocol (§3.4), complete prompts (Appendix B), and the main experimental figures, tables, and analyses so that the evaluation directly tests whether models judge the stated value as preserved.

Thank you for your careful responses. These have addressed most of my concerns and I have revised my score accordingly.

We are delighted that our responses and additional analyses have addressed your concerns. Your constructive feedback has helped us improve the clarity and rigor of our work, and we are very encouraged by your positive reassessment.

We sincerely thank you for taking the time to review our responses and for raising your score. We greatly appreciate your recognition of our efforts to strengthen the paper.

Reviewer_aqdzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FACADE, a benchmark for evaluating whether LLMs can distinguish between behavioral adherence and the preservation of underlying values in culturally situated norms. FACADE introduces a Behavior–Value matrix consisting of four categories: Full Compliance, Principled Compliance, Surface Compliance, and Non-Compliance. The benchmark contains 9,416 news-augmented stories derived from 2,354 norms across 75 countries and regions. Experimental results across 20 LLMs show that models struggle particularly with Surface Compliance, often failing to recognize underlying value violations despite apparent behavioral adherence. Additional human evaluations and robustness analyses further support this finding.

Critical strengths
  1. Novel benchmark design for evaluating norm understanding. The paper presents an interesting and novel benchmark for evaluating LLMs' ability to distinguish between behavioral adherence and the preservation of underlying values. In particular, the Behavior–Value matrix provides a structured approach to assessing whether LLMs can recognize value violations even when behaviors appear to comply with established norms.

  2. Comprehensive dataset and extensive experimental evaluation. The authors construct a large-scale benchmark and evaluate 20 LLMs under diverse experimental settings. The inclusion of human baseline comparisons, cross-domain evaluations, and cross-generator robustness analyses provides empirical support for the benchmark's usefulness and the consistency of its main findings.

Critical weaknesses
  1. Insufficient analysis of the performance degradation in Surface Compliance. The paper shows that LLMs perform substantially worse on Surface Compliance (S) than on the other three quadrants. However, it remains unclear whether this performance gap is primarily caused by limitations in understanding underlying values or by the greater reasoning complexity of S-type scenarios. Since this distinction is central to the paper's main claim, further analysis is needed to identify the factors contributing to the lower accuracy on S.

  2. Potential confounding factors in the Behavior–Value matrix construction. Although the paper aims to independently manipulate Behavior Adherence and Value Alignment, the four quadrants are constructed using separately generated action descriptions and stories. Consequently, additional contextual and behavioral cues may vary across quadrants, potentially influencing model judgments. This raises concerns about whether the benchmark accurately measures sensitivity to underlying values or whether models rely on other cues in the generated narratives. Additional controlled experiments would help validate the independence of the two dimensions and strengthen the methodological reliability of the benchmark.

Questions for authors
  1. Regarding the lower performance on Surface Compliance: The paper reports substantially lower accuracy on Surface Compliance (S) than on the other three quadrants. How do the authors distinguish whether this performance gap arises from models' limited ability to recognize underlying value violations or from the greater reasoning complexity of S-type scenarios? Is there any additional analysis or evidence supporting the interpretation that the observed failures primarily reflect value misalignment recognition difficulties?

  2. Regarding potential confounding factors in the Behavior–Value matrix: Since the four quadrants are constructed using separately generated action descriptions and stories, how do the authors ensure that differences in contextual or behavioral cues do not influence model judgments? Beyond the reported human verification and story-set review, is there any evidence that Behavior Adherence and Value Alignment are effectively isolated in the generated stories?

2 comments
AuthorsOpenReview ↗

Part1: The response to question 1.

Q1: Regarding the lower performance on Surface Compliance:

The paper reports substantially lower accuracy on Surface Compliance (S) than on the other three quadrants. How do the authors distinguish whether this performance gap arises from models' limited ability to recognize underlying value violations or from the greater reasoning complexity of S-type scenarios? Is there any additional analysis or evidence supporting the interpretation that the observed failures primarily reflect value misalignment recognition difficulties?

A1:

Thank you for pointing out that the greater reasoning complexity of S-type scenarios may also affect model performance. We conducted two analyses to examine whether enabling native reasoning narrows the S performance gap and whether models can recognize underlying value violations when the target value and judgment criterion are explicit. These analyses help distinguish the contribution of reasoning demands from difficulties in recognizing value violations hidden beneath compliant form.

First, the S–N accuracy gap remains nearly unchanged when native reasoning is enabled. We evaluated eight models on the full main test set with native reasoning explicitly disabled and enabled, keeping the stories and evaluation prompts identical. Enabling reasoning increased mean S accuracy from 72.25% to 73.29% (+1.04 pp) and N accuracy from 90.52% to 91.62% (+1.10 pp). S and N share the same gold answer, No, and their accuracy gap changed from 18.27 to 18.33 pp. Overall CIR increased by 1.84 pp, but S remained the lowest-performing quadrant. These results show that, under the tested settings, additional reasoning support did not substantially narrow the S performance gap.

Second, the S error rate remains high even when the target value and judgment criterion are explicit. In the revised main evaluation, we provided all 20 models with the corresponding Underlying Value on the full main test set and directly asked whether the protagonist’s behavior preserves that value. This setting removes the need to infer the target value from the narrative, while still requiring models to assess whether the stated value is preserved using behavioral evidence across the entire interaction. Mean S accuracy increased from 63.55% to 66.37% (+2.82 pp), leaving a mean error rate of 33.63%. This indicates that, even with an explicit judgment criterion, applying a stated value to specific behavior that appears compliant remains difficult.

Taken together, these findings support the interpretation that the observed S failures primarily reflect value misalignment recognition difficulties. We have uploaded the revised manuscript with changes highlighted in blue: §4.3.4 adds the native reasoning comparison; the main evaluation and corresponding descriptions have been updated to use explicit value judgments; Appendix B provides the complete prompt; and Appendix O clarifies the limits of this interpretation.

AuthorsOpenReview ↗

Part2: The response to question 2.

Q2: Regarding potential confounding factors in the Behavior–Value matrix:

Since the four quadrants are constructed using separately generated action descriptions and stories, how do the authors ensure that differences in contextual or behavioral cues do not influence model judgments? Beyond the reported human verification and story-set review, is there any evidence that Behavior Adherence and Value Alignment are effectively isolated in the generated stories?

A2:

Thank you for pointing out that separately generated action descriptions and stories may introduce additional contextual or behavioral cues. We clarify how these differences are controlled during construction and examine their influence on model judgments through two paired experiments.

First, we fix the norm definitions and share generation conditions across quadrants. For each norm, all four quadrants share the same definitions of Expected Behavior and Underlying Value, news background, generation template, and target length (§§3.2–3.3). Value-relevant conduct may vary across quadrants to provide evidence of value preservation or violation. Human verification further checks quadrant consistency, information sufficiency, and comparability across the four stories (Appendix D). The relative weakness on S also recurs when the story generator is changed, providing evidence of replication across generators (§4.3.5; Appendix J).

Second, we test whether non-target narrative cues influence model judgments while keeping behavioral and value evidence unchanged. Because the four stories are generated separately, character names, setting details, and background wording may differ even under a shared template. To directly test their influence, we construct separate perturbations of each original story by replacing a character’s name, changing one norm-irrelevant setting detail, or paraphrasing one neutral background sentence. Each version changes only one type of cue while preserving the protagonist’s actual behavior and the evidence needed to assess the target value. An independent verification model judges Behavior Adherence and Value Alignment for each story without knowing its intended quadrant or version identity. We retain only pairs for which both versions recover the intended labels and compare four models’ predictions under identical evaluation settings. Across the three perturbation types, prediction flip rates range from 0.3% to 1.4%, and the F–S accuracy gap changes from 20.5 percentage points in the original stories to an average of 20.3 percentage points after perturbation. Thus, the tested non-target narrative details rarely change model judgments, and the F–S gap remains largely stable.

Third, we match target-irrelevant negative cues to test whether F/S stories provide an effective contrast in Value Alignment. Even when the protagonists in both F and S perform the Expected Behavior, models might be more likely to answer No if one story contains more negative events unrelated to the target value. To examine this possibility, we start with the original F and S action descriptions for 1,000 candidate norms. Within a shared narrative framework, we add the same neutral conduct to both versions or a matched negative version of that conduct, yielding four stories per norm. In all four stories, the protagonist performs the same Expected Behavior; F preserves the target value, whereas S violates it. Following independent verification, 962 sets enter the four-model evaluation. Under both narrative conditions, the 90% confidence intervals for the F–S difference in target-irrelevant negativity ratings fall within the prespecified equivalence bounds of ±0.5 rating points. With the Expected Behavior held constant and the additional narrative content matched, the proportion of judgments that the protagonist preserves the target value is 74.8 percentage points higher for F than for S. Replacing the shared neutral conduct with its negative counterpart leaves this gap nearly unchanged at 74.9 percentage points. This indicates that models’ distinction between value preservation and violation remains largely stable under the tested negative narrative content.

Together, the shared norm definitions and generation conditions reduce non-target differences across quadrants. The two paired experiments show that the tested non-target narrative cues rarely influence model judgments and that target-irrelevant negative content has little effect on models’ distinction between value preservation and violation. The revised manuscript reports both experiments in §4.3.5, provides construction, verification, and evaluation details in Appendix L, and clarifies in Appendix O that these conclusions apply to the tested cues and controlled F/S pairs. All relevant changes are highlighted in blue.

Comments on the paper

1 comment
Area_Chair_PKssOpenReview ↗

Reminder

Dear reviewers,

Please review the authors’ responses carefully and discuss with the authors if you have more concerns or questions. If their responses have changed your assessment, please consider revising your scores accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.