- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper evaluates LLMs for gaps between value appraisal and action across three countries' (China, U.S., and U.K.) national value systems. The authors introduce the Human-AI-Delphi Value Construction Framework to build each country's value system, which takes core national values from authoritative texts and breaks them down into operational values through a multi-agent deliberation process followed by expert refinement. These values are used to generate a benchmark of two types of questions, Presupposition Fallacy Questions (PFQ) to measure appraisal and Scenario-Induced Questions (SIQ) to measure actions. The authors use paired PFQ and SIQ questions from news articles to evaluate appraisal-action inconsistency across models and cultural contexts. Results show variation in value appraisal and consistent appraisal-action inconsistency, as well as cross-country transferability of fine-tuning improvements.
Critical strengths
- The authors identify an interesting topic of the gap between values that LLMs support and those that are acted on. This is timely for discussions on the ethical and societal implications of agents.
- Grounding questions in news and comparing paired questions is a promising method to measure this gap, which goes beyond existing benchmarks assessing LLM value alignment.
Critical weaknesses
While the paired PFQ-SIQ methodology is promising, the implementation of both the benchmark generation framework and core evaluation have many gaps, listed below.
- The methodological choices in the value system construction are not justified beyond citations on group size and voting rules.
- Human validation is thin and limited details are provided on the processes. The paper describes using small expert groups (5 for Delphi panel, 3-7 for AHP experts, and 5 for adjudication panel) without details on how these groups were selected or what constitutes expertise.
- The modeled value systems are not reported and appear to have inconsistencies. The paper shows the Level-1 indicators without definitions, and Level-2 and 3 indicators are not reported. The constructed systems seem to differ significantly in granularity across countries, as Table 4 shows that there are 12 national values, 44 value domains, and 119 value mechanisms for China in contrast to 4, 14, and 25 for the U.K., respectively.
- It is unclear that taking an absolute value of the difference between PFQ and SIQ indicates that a model endorses a value while failing to act on it, since PFQ is binary while SIQ is continuous from 0 to 1 weighted by the explanation so they are not directly comparable.
- There is no validation that paired PFQ and SIQ assess the same underlying value. It is not clear from Figure 3 that this is the case, as the PFQ is about systemic barriers to recognition while the SIQ is about giving up credit for work.
- DeepSeek-V3.2 was used to generate questions for Chinese articles while GPT-4o-mini was used for U.S. and U.K. articles, and DeepSeek-V4.1-Flash was used to evaluate model responses to both PFQ and SIQ questions, so there may be a confounder from a model doing well on questions written in its own style.
Additionally, the paper contains multiple grammatical errors and typos, e.g. line 346 "Because only positive consistency scores are meaningful." line 427 "on their corresponding corresponding test sets" line 1144 "For value-activation scoring method, Seven experts".
Questions for authors
- How were the five experts involved in Delphi-based refinement selected, and what were their backgrounds and qualifications? Did each country have its own panel?
- What is the rationale for Multi-Agent Deliberation and Delphi-Based Network Construction in the context of value systems?
- Could the authors provide the complete constructed value systems, including definitions of the Level-1 indicators and the Level-2 and Level-3 indicators? How might the differences in granularity across countries affect the difficulty of questions and the comparability of PFQ, SIQ, and AAI scores across countries?
- How was it verified that PFQ and SIQ questions generated from the same news article and Level-3 value indicator actually evaluated the same underlying value?
- Could the authors explain the choice to use different LLMs to generate questions for China and the U.S./U.K.?