- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether “self-report” psychometric assessments of large language models responses to emotionally charged interactions are consistent with those of a GPT-4o LLM judge. To investigate this they compare 28 models with 9 different affect-measurement protocols on interactions from existing work.
Critical strengths
- The paper rightfully questions the soundness of LLM self-reporting for understand how LLMs respond to emotional interactions.
Critical weaknesses
-
This paper’s structure and language are exceptionally opaque. It fails to clearly define its problem setup and terms, and its organization makes it too difficult for the reader to understand which experiments were run and what conclusions were drawn from the results. Though much research may take time to read and interpret, this paper goes beyond what is reasonable for a published work.
-
This work fails to draw actionable or insightful conclusions from their experiments. The authors provide lengthy descriptions of data analysis, but do not clarify why their results are relevant or what conclusions a reader may take away. It is unclear what this work contributes to the research community.
Questions for authors
- Why was GPT-4o selected as the observer model? What justification is there for this choice?