- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper uses psychological instruments to assess whether language models are consistent in their closed- and open-ended evaluations of moral characters, and whether those different types of evaluation can bias subsequent hate speech judgments in consistent ways.
Critical strengths
The paper uses well-validated psychological instruments, including careful testing of the downstream impacts (if any) of the expressed moral/social assessments by the LLMs. The paper also has an appropriate balancing of different considerations to enable justified interpretation of the results (unlike many validation papers!). Overall, this paper is a much-needed corrective to some of the too-quick conclusions that people have drawn about the moral behaviors and responses from LLMs.
Critical weaknesses
The paper often conflates social and moral factors or attributes (e.g., whether I like someone is a social matter, not a directly moral one). That being said, this same issue arises in the two psychological instruments that are used, so it isn’t something distinctive to this paper.
Appendices A & B only provides details about the LLM experiments, not the human experiments (e.g., recruitment procedure; compensation; response protocols; etc.). While it seems that the authors did things in a reasonable manner, I strongly encourage them to add an Appendix that actually gives the relevant details.
Minor note: The paper repeatedly talks about value alignment in terms of “steering towards moral perspectives,” but this is a very narrow (and behaviorist) way to understand value alignment (and is an unnecessary thing to say…)
Minor note: the authors write: “we fill this gap [asking for explanations of ratings] by testing whether the justification a model gives for an item supports the rating it assigns to that same item.” (p. 3) However, I do not think that they actually did this: the correlation between LLM rating and human rating based on open-ended output does not directly tell us whether the (open-ended) justification supports the rating.
Questions for authors
Likert scales are not always used in a linear way, so Spearman correlation (or other rank-order measure) would arguably be better for the internal validity assessment. Are the results different with a correlation measure that is only sensitive to ordinal information?
What FDR was used for the corrected p-values? (I assume 0.05, but the value doesn’t actually seem to be stated)
Reviewed version of the draft
Dear program committee,
As the deadline for the rebuttal is approaching, we would like to thank the reviewers for their feedback about our paper, which will help improving the overall quality of the manuscript.
We integrated the comments emerging from the rebuttal in a new version of the paper, marking our additions in blue. Specifically, we clarified the novelty of our methodology in the introduction, added and discussed the Spearman correlation in Tables 1 and 2, integrated conclusions with reviewers suggestions, and added a Limitation section for all the comments left open after our discussion.
Best Regards,
The authors