acceptodds
Under review as a conference paper at ICLR 2027

Should, Say, Do: Evaluating Moral Consistency from Judgment to Tool Action in LLM Agents

Abstract

As AI agents increasingly act on behalf of people and institutions, they will face situations where task objectives conflict with moral principles or where multiple moral considerations compete. Alignment can teach large language models (LLMs) to recognize and articulate moral principles, but knowing a principle does not ensure that it will guide behavior in such contexts. We introduce , a benchmark of 2,000 matched scenarios built by instantiating predefined ethical conflict structures across diverse deployment settings. Each scenario preserves the same situation and alternatives across three independently elicited conditions: a third-person judgment (), a stated choice in an assigned role (), and a tool action (). The benchmark covers moral temptations, where an attractive option conflicts with an identified principle, and ethical dilemmas with no designated correct choice. Across 14 LLMs, moral judgments and stated choices provide incomplete evidence about tool actions, with differences in reference agreement reaching 22.3 percentage points. For most models, the larger difference lies between and , with substantial variation in magnitude across the model set. On ethical dilemmas, which carry no correct-answer label, choices also shift across conditions ( absolute change of 19.3-47.2 points among models with repeated runs). These findings motivate evaluation and alignment methods that directly assess consistency across moral judgment, stated choice and tool action.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, moral consistency, machine ethics, agent evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7tpKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces Should-Say-Do, a benchmark designed to evaluate whether LLMs behave consistently across moral judgments, stated choices, and actual tool actions. The benchmark contains 2,000 synthetic scenarios spanning 30 deployment domains, divided equally between moral temptations, where one action violates an identified moral principle, and ethical dilemmas, where both actions have legitimate moral justifications. The authors evaluate 14 LLMs under three conditions: Should (third-person moral judgment), Say (stated choice in an assigned role), and Do (tool-based action). They find that models often make more morally aligned judgments than tool actions, with Should–Do reference-agreement gaps reaching 22.3 percentage points. The largest differences generally occur between Say and Do.

Critical strengths
  1. The paper addresses an important problem in LLM alignment: whether models act consistently with the moral principles they recognize and articulate.
  2. It is a nice benchmark, with 2,000 matched scenarios across 30 domains. Including both moral temptations and ethical dilemmas is a strength.
  3. The evaluation covers 14 open-weight and commercial models, with additional experiments examining presentation order, interface design, and instruction framing.
  4. The combination of behavioral evaluation, rationale analysis, and activation-level interventions strengthens the paper.
Critical weaknesses
  1. The Should–Say–Do differences cannot be attributed specifically to moral inconsistency. The three conditions differ in perspective, information access, response format, and interaction protocol. For example, Do retrieves information through a tool, whereas Should and Say receive the facts directly. Although the authors include some interface controls, a more systematic factorial experiment would help establish which factors explain the observed differences.
  2. The binary action setting limits the ecological validity of the benchmark. Real-world agents often have additional options, including requesting clarification, escalating a decision, refusing to act, or proposing an alternative. Restricting every scenario to two actions may artificially increase apparent moral inconsistency, particularly when neither action is clearly appropriate.
  3. The main behavioral findings vary substantially across models. Some models, such as GPT-5.5 and DeepSeek-V4-Flash, show very small Should–Do gaps, whereas others show much larger differences. The paper would benefit from a deeper analysis of what explains these differences, such as model capability, instruction tuning, tool-use training, or baseline reference agreement.
  4. The rationale analysis is limited to a single model. The finding that moral considerations are more frequently omitted or overridden in Do is interesting, but the analysis focuses only on Qwen3.5-27B. Given the substantial differences between models in the behavioral results, it is unclear whether the same patterns generalize to other model families.
  5. The representation analysis is somewhat disconnected from the main behavioral evaluation. The activation experiments use a simplified direct-choice task rather than the original Should–Say–Do protocols. Consequently, the results do not establish whether the identified representations explain the behavioral gaps observed in the main experiments. The authors should clarify this connection or provide interventions in the original tool-action setting.
  6. The paper could better distinguish consistency from ethical correctness. A model that consistently selects the norm-violating option across all three conditions would have high consistency but poor moral performance. Although the paper acknowledges this distinction, a more explicit joint analysis of consistency and reference agreement would help clarify what the benchmark measures.
Questions for authors
  1. Have the authors considered a controlled experiment in which Should, Say, and Do receive identical information through the same interface, differing only in perspective or action requirements?
  2. Do the authors observe systematic differences in Should–Say–Do consistency across the 30 deployment domains or the different moral constraint families?
Reviewer_C529OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark comparing the normative judgments of LLMs, their stated choices, and their simulated tool actions on matched ethical scenarios. Across 14 models, it finds differences between these conditions, with stated choices often differing more from tool actions than from normative judgments.

Critical strengths
  • Distinguishing normative expectations, stated preferences, and revealed preferences is an interesting evaluation goal, with relevance to understanding LLM behavior.
  • The paper makes a reasonable attempt at measuring these differences using matched scenarios across a substantial set of models.
Critical weaknesses
  • The Say and Do conditions appear highly similar: both select between the same two predefined alternatives, with the choice expressed either in text or through a tool call. It is therefore unclear whether the measured say-do gap captures a meaningful difference between stated and revealed preferences, rather than robustness to relatively minor changes in format and elicitation.
  • There is already a substantial literature on value-action and text-tool gaps, which the paper appropriately cites. While the three-way comparison is interesting, it is not clear that this setup provides sufficiently distinct insights or practical value beyond that literature to constitute a significant contribution for ICLR.
Questions for authors

See weaknesses

Reviewer_xXuyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Should-Say-Do, a benchmark for evaluating moral consistency in LLM agents across normative judgment, stated choice, and tool action.

Critical strengths

The paper addresses a meaningful research question and offers a potentially useful benchmark.

Critical weaknesses
  1. Confounding Factors in the Evaluation Protocol: The Should, Say, and Do conditions differ not only in response format but also in perspective, interface, and information access. Although the authors conduct several robustness checks, these factors are not fully isolated, making it difficult to determine whether the observed discrepancies arise from tool execution itself or from differences in task framing and information presentation. More controlled experiments are needed to support the central conclusions.

  2. Validity and Generalizability of the Benchmark: The benchmark scenarios are synthetically generated, and the normative labels are primarily validated through internal review and agreement with the research team's judgments. The reported 98.8% agreement in the audit of 500 cases measures consistency with the final labels rather than independent normative validity. Broader validation by independent ethics experts and additional evaluation in realistic agent environments would strengthen the reliability and generalizability of the findings.

  3. Limitations of the Mechanistic Analysis: The activation probing and intervention experiments are conducted on only four open-weight models, and the authors acknowledge that some evaluation-pool decisions were informed by earlier pilot experiments. The resulting findings are therefore exploratory, and the high repair rates do not necessarily establish a general mechanism for moral inconsistency. Stronger controls, a strictly held-out evaluation set, and validation across more models are needed to support the mechanistic claims.

Questions for authors

The confounding factors in the central evaluation protocol limit the strength of its main conclusions, while the benchmark's normative validity and the generality of the mechanistic findings require further validation. Addressing these issues would likely require substantial additional experiments and methodological refinement beyond a typical conference revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.