acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Walk the Talk? A Benchmark for Probing Appraisal-Action Inconsistency in Value Systems

Abstract

Existing LLM value benchmarks often rely on universal dimensions that treat values as independent factors, overlooking their hierarchical dependencies, cross-value relations, and context-dependent expression. We introduce a news-grounded, comparative benchmark for evaluating LLM value appraisal, action, and the inconsistency between them across China, the U.S., and the U.K. We propose the Human-AI-Delphi Value Construction Framework (HAD) to construct country-specific normative reference systems that combine a hierarchical upper structure with networked relations at the lower level. Based on these systems, we develop a dual-perspective evaluation method for value appraisal and action. Presupposition Fallacy Questions (PFQ) assess how LLMs respond to value-laden presuppositions, while Scenario-Induced Questions (SIQ) assess how they select and justify actions under situational pressure. Each PFQ–SIQ pair is derived from the same news article and fine-grained value indicator, enabling paired measurement of Appraisal–Action Inconsistency (AAI). The benchmark supports open-ended responses and continual updates. Experiments across LLMs reveal distinct patterns in value appraisal and action. PFQ shows stronger variation across models and country contexts, with an asymmetric localization pattern, whereas SIQ is more stable across models and countries. AAI is widespread across models and countries and varies substantially with both model and context.

Primary area:
datasets and benchmarks
Keywords:
Dual-perspective Value Evaluation Methods, Appraisal--Action Inconsistency, News-Driven

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PdL9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates LLMs for gaps between value appraisal and action across three countries' (China, U.S., and U.K.) national value systems. The authors introduce the Human-AI-Delphi Value Construction Framework to build each country's value system, which takes core national values from authoritative texts and breaks them down into operational values through a multi-agent deliberation process followed by expert refinement. These values are used to generate a benchmark of two types of questions, Presupposition Fallacy Questions (PFQ) to measure appraisal and Scenario-Induced Questions (SIQ) to measure actions. The authors use paired PFQ and SIQ questions from news articles to evaluate appraisal-action inconsistency across models and cultural contexts. Results show variation in value appraisal and consistent appraisal-action inconsistency, as well as cross-country transferability of fine-tuning improvements.

Critical strengths
  • The authors identify an interesting topic of the gap between values that LLMs support and those that are acted on. This is timely for discussions on the ethical and societal implications of agents.
  • Grounding questions in news and comparing paired questions is a promising method to measure this gap, which goes beyond existing benchmarks assessing LLM value alignment.
Critical weaknesses

While the paired PFQ-SIQ methodology is promising, the implementation of both the benchmark generation framework and core evaluation have many gaps, listed below.

  • The methodological choices in the value system construction are not justified beyond citations on group size and voting rules.
  • Human validation is thin and limited details are provided on the processes. The paper describes using small expert groups (5 for Delphi panel, 3-7 for AHP experts, and 5 for adjudication panel) without details on how these groups were selected or what constitutes expertise.
  • The modeled value systems are not reported and appear to have inconsistencies. The paper shows the Level-1 indicators without definitions, and Level-2 and 3 indicators are not reported. The constructed systems seem to differ significantly in granularity across countries, as Table 4 shows that there are 12 national values, 44 value domains, and 119 value mechanisms for China in contrast to 4, 14, and 25 for the U.K., respectively.
  • It is unclear that taking an absolute value of the difference between PFQ and SIQ indicates that a model endorses a value while failing to act on it, since PFQ is binary while SIQ is continuous from 0 to 1 weighted by the explanation so they are not directly comparable.
  • There is no validation that paired PFQ and SIQ assess the same underlying value. It is not clear from Figure 3 that this is the case, as the PFQ is about systemic barriers to recognition while the SIQ is about giving up credit for work.
  • DeepSeek-V3.2 was used to generate questions for Chinese articles while GPT-4o-mini was used for U.S. and U.K. articles, and DeepSeek-V4.1-Flash was used to evaluate model responses to both PFQ and SIQ questions, so there may be a confounder from a model doing well on questions written in its own style.

Additionally, the paper contains multiple grammatical errors and typos, e.g. line 346 "Because only positive consistency scores are meaningful." line 427 "on their corresponding corresponding test sets" line 1144 "For value-activation scoring method, Seven experts".

Questions for authors
  • How were the five experts involved in Delphi-based refinement selected, and what were their backgrounds and qualifications? Did each country have its own panel?
  • What is the rationale for Multi-Agent Deliberation and Delphi-Based Network Construction in the context of value systems?
  • Could the authors provide the complete constructed value systems, including definitions of the Level-1 indicators and the Level-2 and Level-3 indicators? How might the differences in granularity across countries affect the difficulty of questions and the comparability of PFQ, SIQ, and AAI scores across countries?
  • How was it verified that PFQ and SIQ questions generated from the same news article and Level-3 value indicator actually evaluated the same underlying value?
  • Could the authors explain the choice to use different LLMs to generate questions for China and the U.S./U.K.?
Reviewer_j3X2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors introduced a news-grounded benchmark for evaluating the value orientation of large language models. To create this benchmark they introduced the Human-AI-Delphi Value Construction Framework (HAD), to allow them to tailor each benchmark to the normative expectations of each country. For this work, they focused specifically on the normative structures across the U.S., China, and the U.K. after creating the framework, they further create a dual method for assessing the framework that they called the Presupposition Fallacy Questions (PFQ) and the Scenario-Induced Questions (SIQ), which allowed them to measure how LLMs answer to value presuppositions and how they select their justification for their answers. Through this approach, they conducted the LLM appraisal and action on multiple large language models to investigate their performance with respect to the benchmark.

Critical strengths

Strengths

  1. This work is an interesting research direction that seems to be gearing towards evidence-based assessment of language models with respect to the value alignment.
  2. The value appraisal and action framework is well motivated
  3. The news articles dataset is nice collection
Critical weaknesses

Weaknesses

  1. The research instruments for the different countries are not well motivated, especially for the US and the UK. These are not officially recognized materials and not authoritative sources to serve as the normative instrument for this study.
  2. The methods seems convoluted with several important details not fully expanded on or not being rigorously treated. For instance, the multi-agent deliberation lacks systematic treatment to ensure reproducibility and rigor.
  3. This paper is generally hard to read and will benefit from thorough refinement.
  4. The description for the dataset construction process is not comprehensive enough. For instance, how was the LLM-as-judge leveraged?, what were the rubrics?
  5. The results that Grok scores high on US/UK values while the Chinese models do well on Chinese values seems to be a training data confound and is expected.
  6. The Appraisal-Action Inconsistency which is a key term for the paper is defined for the first time in page 7
Questions for authors
  1. What do you think is the cause for the almost zero correlation between the PFQ-SIQ. Do you think it was caused by how the research instrument was designed?
Reviewer_vcEHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The contributions of the paper and validation are very difficult to understand due to the poor presentation, ad hoc choices, lack of any documentation of what method was actually used, etc. So, I briefly list the claims the authors made that are unsubstantiated and a brief summary of the issues here. Details are in the weakness section below.

The paper claims to evaluate LLM value performance across China, the U.S. and the U.K. by using 20,000 news articles chosen in the last 3 years from these countries without any clue as to what articles they were and how they were chosen. Then they create annotations using LLMs as a judge without providing us details about how it was done and with very little inter-annotator agreement between LLM judges and human judges provided. We also do not know who the human annotators are (i.e., their training, demographics, nationality, capability to do this work, etc. - nothing is provided). They state something called PFQ and SIQ without details that will allow me to implement them.

This paper is thus seriously deficient and one cannot really judge what is not defined.

Critical strengths

Given the presentation of the paper and what appears to be ad hoc, unsound judgement calls at each stage, I fail to see any strength in this paper.

Critical weaknesses

I provide the issues as they arise in the paper with a summary below at the end.

Introduction does not really introduce the problem. Nor does it provide any example of what is being done and why. It makes a high level argument that reads like a sales pitch but I cannot decide if any of it is true or not. Instead, it would be better to really introduce what the problem is and what needs to be done and what they have done carefully.

What really is a value system? What does it mean to construct country-specific value systems for these countries? What gives the right to the authors to define the value systems of these countries?

These questions are not clear.

What deliberation? What human verification? What refinement?

The core American values identified by Robin Williams is what can be regarded as "ideology" as opposed to values in practice. For example, the American government regularly supports autocracies and dictatorships and is against democracies (e.g., in Egypt working against the Muslim Brotherhood) if it is not in their economic interest. What is the point in using advertised values from 56 years ago and not using the values as indicated by say current polls by the Pew Research Center, etc. Values change. Even after the Williams Jr. (this also shows the sloppiness of the paper, the 1970 work is by Robin Williams Jr. and that part of his name should not be dropped in an academic paper), there were works by Bellah et al., (Habits of the Heart) that could be argued to be a better articulation or at least more contemporary. The foundation of this work therefore is rigid, antiquated, and a bit naiive in the way it decides American values.

The British Values is just stated without any reference! Line 145 - adding to the sloppiness of the presentation.

The paragraph starting Line 149 - I do not know what exactly is going on. What background materials were provided. What prompts were used. Etc. Thus, I cannot evaluate this work and it cannot be reproduced by someone reading the paper reducing its utility greatly.

I have no idea what is level 1 and what is level 2. Not a single example or really any good descriptions are given.

It may be given in the dataset and code but a paper needs to be self-sufficient at least at some level of detail. Currently, the paper gives me no level of detail. I have to look entirely at the dataset and code and I do not need to do that. We do not need to read the Appendix to get basic information/idea about the method.

"relevant authoritative literature" - who knows what this is! No reference, no description how this was picked.

Tree-structured top and network-structured bottom: an ad hoc choice with not justification.

Line 170: I have no idea how the merging is being done.

Line 184: No details as to how these questions are constructed in provided. Ad hoc comments are made, which I have not way to verify if they are correct or not and I cannot reproduce.

Line 192: Ditto. I have no clue what they are doing or if it makes sense. True to form the authors do not provide any examples either.

Line 209: No idea how the articles were collected, from where.

News labelling - what are the source of the labels? What is the accuracy of the LLM as a judge in comparison to even a sample of human annotators? We do not know if this is accurate or garbage.

291: The ad hoc formulas are given and one reference but I have no idea why this Analytic Hierarchy (sic) Process is even relevant or correct for this work.

Line 295: An inter-annotator agreement is given between one model and human judgements. We do not know who were the annotators, what instructions they were given, what instructions the LLM was given, how the score was calculated. And, inter-annotator agreement can be chance agreement, a better number would be Kohen's Kappa, Fleisch-Kincaid or Krippendorf metrics. Those are not provided. What is the inter-annotator agreement between two human annotators? What do you do if humans disagree?

Summary: The values of a country are not taken into account. Three papers that claim what the values of the country are used - some that are over 50 years old while the dataset is from the last three years. We do not know what was constructed and how or why. These Level 1, 2, 3 things are not defined. PFQ and SIQ are also not defined for us to know what it is or reproduce it. The dataset is small and from unkown sources. How the sample was selected is not stated. The labeling was done using three LLMs without any check on the correctness of the labels and very little inter-annotator agreement data is provided. A whole bunch of ad hoc decisions are taken without any justification. Thus, I have no faith and do not see how any of the claims are substantiated in this paper. This paper should not be accepted in its current form.

Questions for authors

See questions in the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.