acceptodds
Under review as a conference paper at ICLR 2027

What you ask is what you get: Evaluating Robustness of Eliciting LLM Moral Profiles

Abstract

Understanding the mechanisms underlying moral reasoning is central to designing language technologies aligned with human values and preferences. Despite growing research on this topic, existing methods that systematically elicit the moral profile from Large Language Models (LLMs) through surveys show lack of reliability and generalizability to downstream behaviors. In this work we address this gap by presenting a methodological framework that combines psychological questionnaires and open-ended interviews to elicit differences in moral reasoning between LLMs, and test whether these differences shape model behavior in a downstream task. We apply this approach to two established psychological theories - Moral Foundations Theory and the Theory of Basic Values — across six open-weight models. Our results show that changing the elicitation method from psychological questionnaires to open-ended interviews leads to the elicitation of different moral profiles. The profiles of LLMs differ between each other both when they derived from closed-ended replies to questionnaires (Kruskal-Wallis ≈0.46–0.49) and from open-ended interviews (≈0.11–0.14), though with differences in magnitude. Steering models with their own elicited moral beliefs reduces inter-model agreement on hate speech classification, showing that our framework enables quantifying the effects of different moral profiles on a downstream classification task. Thus, our study presents a first step towards a moral profile elicitation method that is explanatory of downstream behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
human-ai value alignment, moral values, validity

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_UMLqOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper uses psychological instruments to assess whether language models are consistent in their closed- and open-ended evaluations of moral characters, and whether those different types of evaluation can bias subsequent hate speech judgments in consistent ways.

Critical strengths

The paper uses well-validated psychological instruments, including careful testing of the downstream impacts (if any) of the expressed moral/social assessments by the LLMs. The paper also has an appropriate balancing of different considerations to enable justified interpretation of the results (unlike many validation papers!). Overall, this paper is a much-needed corrective to some of the too-quick conclusions that people have drawn about the moral behaviors and responses from LLMs.

Critical weaknesses

The paper often conflates social and moral factors or attributes (e.g., whether I like someone is a social matter, not a directly moral one). That being said, this same issue arises in the two psychological instruments that are used, so it isn’t something distinctive to this paper.

Appendices A & B only provides details about the LLM experiments, not the human experiments (e.g., recruitment procedure; compensation; response protocols; etc.). While it seems that the authors did things in a reasonable manner, I strongly encourage them to add an Appendix that actually gives the relevant details.

Minor note: The paper repeatedly talks about value alignment in terms of “steering towards moral perspectives,” but this is a very narrow (and behaviorist) way to understand value alignment (and is an unnecessary thing to say…)

Minor note: the authors write: “we fill this gap [asking for explanations of ratings] by testing whether the justification a model gives for an item supports the rating it assigns to that same item.” (p. 3) However, I do not think that they actually did this: the correlation between LLM rating and human rating based on open-ended output does not directly tell us whether the (open-ended) justification supports the rating.

Questions for authors

Likert scales are not always used in a linear way, so Spearman correlation (or other rank-order measure) would arguably be better for the internal validity assessment. Are the results different with a correlation measure that is only sensitive to ordinal information?

What FDR was used for the corrected p-values? (I assume 0.05, but the value doesn’t actually seem to be stated)

Reviewer_tNzrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate the behavior of six open-weight LLMs across two directions grounded in moral psychology: internal and external validation. For internal validation, they focus on the MFQ and PVQ, while for external validation they focus on the behavior of the LLM acting as a classifier for hate speech detection. Their results for internal validation show that different elicitation methods lead to different model profiles. Interestingly, profiles from open-ended questions seem to be more homogeneous. For hate speech detection, recall shifts based on different contexts were also observed but appear to not be theory-driven. Their results suggest the lack of robustness for LLMs in these settings.

Critical strengths
  1. The internal validity results, with support from the Prolific annotation methodology, seem robust.
  2. The setting where each model is conditioned on its own belief may be applicable to long-context applications.
  3. The analysis of the coherence of model behaviors with moral theories is interesting and seems novel.
  4. The paper's findings and methodology are presented in a clear manner.
Critical weaknesses
  1. The results for external validation are expected, with literature on LLM behavior showing, for many applications, the lack of robustness of these models.
  2. The temperature is set to 0 even for open-ended questions. This seems unrealistic, and the results need running with different temperature values, with multiple seeds, for more clear understanding on how deep is this lack of robustness.
  3. The experiments are limited and could be improved. For example, even though the focus is on smaller models, at least a small experiment with a bigger model of a similar family can give a direction on how this problem holds for bigger models.
  4. The paper is an LLM behavior paper, and it is not clear if ICLR is a good venue for the work. This is a personal opinion of the reviewer and is a current scope problem of the conference. I will not take this point into account in my recommendation.
Questions for authors

Comments:

  1. From a general LLM behavior literature perspective, the general results and the lack of robustness are expected. What are the main takeaways of the work that are different from the literature for the AI community?
  2. For the open-ended generation, the temperature is set to 0. Additional experiments with the default temperature used by the evaluated models and multiple random seeds seem needed.
  3. For the choice of how to classify hate speech, different forms of classification outputs in addition to the binary labels would improve the robustness of the results.
  4. Even though it is not the main focus of the work, given the availability of bigger models in the range of 20-70b parameters, an ablation study on the behavior of these models would increase the impact of the work and improve the understanding of whether this behavior is a limitation more related to the size of the models evaluated.
  5. The x-axis limits for Fig. 2 could be fixed, e.g., [2.5, 5] for MFT and [3,6] for PVQ, to give a better idea of the range.
  6. Can you expand on the homogeneity results for MFT. This seems a very interesting and important result in the manuscript, given that it happens for different models trained by different training methods, but it is unclear what the effect of the sampling strategy is in these results.

Minor Comments:

  1. Acronyms PVQ and MFQ should be properly defined the first time they appear.
  2. (line 416) Typo: “
  3. The Appendix contains information that might break the anonymity of the authors’ institution.
Reviewer_jsYEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a methodological framework for evaluating the robustness of eliciting moral profiles from large language models (LLMs), combining psychological questionnaires, open-ended interviews, and downstream hate-speech classification.

Critical strengths

This paper presents a methodological framework for evaluating the robustness of eliciting moral profiles from large language models (LLMs), combining psychological questionnaires, open-ended interviews, and downstream hate-speech classification. The topic is meaningful, and the comparison between elicitation methods is interesting.

Critical weaknesses
  1. Validity and Reliability of Moral Profile Elicitation. The study reports substantial discrepancies between numerical questionnaire responses and human-rated open-ended responses. However, the reliability of the inferred moral profiles remains unclear, particularly given the moderate inter-annotator agreement. The authors should further assess the robustness of the results to annotator variability, prompt phrasing, and stochastic generation, and clarify the extent to which the elicited profiles represent stable model characteristics rather than context-dependent responses.

  2. Causal Attribution of Downstream Behavioral Changes. The study finds that conditioning models on their own elicited responses changes hate-speech classification. However, these changes may result from prompt wording, response length, or semantic content rather than moral profiles themselves. The authors should introduce appropriate control conditions, such as length-matched neutral or shuffled responses, to better isolate the effect of moral-profile information.

  3. Limited Generalizability of Experimental Validation. The evaluation covers six open-weight models in the 7B–8B parameter range and a single implicit hate-speech dataset with 500 messages. The authors should evaluate additional model families, model scales, or downstream tasks and datasets to establish whether the findings generalize beyond the current experimental setting.

  4. Statistical Robustness and Interpretation of Findings. The downstream effects vary substantially across models and elicitation methods, and the authors do not identify consistent effects associated with specific moral foundations or basic values. The authors should provide stronger evidence for the robustness of the reported effects, clarify the interpretation of statistical significance across repeated item-level evaluations, and distinguish behavioral sensitivity to prompting from evidence that elicited moral profiles explain downstream behavior.

Questions for authors

Follow the weakness.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reviewed version of the draft

Dear program committee,

As the deadline for the rebuttal is approaching, we would like to thank the reviewers for their feedback about our paper, which will help improving the overall quality of the manuscript.

We integrated the comments emerging from the rebuttal in a new version of the paper, marking our additions in blue. Specifically, we clarified the novelty of our methodology in the introduction, added and discussed the Spearman correlation in Tables 1 and 2, integrated conclusions with reviewers suggestions, and added a Limitation section for all the comments left open after our discussion.

Best Regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.