- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents a human study on anthropomorphic language when describing AI and people's perception.
Critical strengths
- anthropomorphic portryal of AI is widespread and worth consideration
Critical weaknesses
- Is this the right venue?
Questions for authors
-
My chief concern is that perhaps an ML conference is not the right venue? Something like CSCW/Facct/Nature Human Behavior might be more fitting?
-
Is recommender systems the right A/B testing against AI? I feel like the anthro description of recommender systems in Table 1 is unconvincing: it is just kind of hard to imagine it as a human in the first place, so it might be a weak baseline?
-
Would participants have strong prior registered opinion about the questions surveyed?
-
The results and analysis is too short (for the main paper), only one illustrative figure 2 without statistics, demographic breakdowns, anything deeper than the headline result.
3 comments
I would like to thank the authors for their clarifications. I still have reservations about venue fit as well as whether the depth of insights meet the standard. The AC could eventually weigh in on this.
Thank you for the response. We appreciate your continuous engagement and input on our work.
Response to Reviewer CD9s
Thank you for your feedback! Below we respond to the points raised:
Scope of paper for ICLR
Anthropomorphism has been discussed widely in the ML community and the broader field, and our work provides empirical grounding on the question in order to inform how AI systems are designed, evaluated, and discussed, which we believe is of direct interest to the ICLR community. We chose to submit this paper here given that the CFP listed “societal considerations including fairness, safety, privacy”, and explicitly encouraged authors to “take the opportunity to be ambitious” highlighting “integration with … humanistic fields outside the historical domain of ‘core ML’”.
Choice of recommender systems
We chose recommender systems as a type of AI system that is perceived differently from LLMs to examine whether the effects of anthropomorphism extend beyond LLMs, so we do not intend for recommender systems to be an equally anthropomorphized baseline. We examine whether anthropomorphic framing affects either technology and recognize that the magnitude of any effect may differ across technologies, although our results show that the effects are negligible in both cases in our setting.
Participant priors on survey questions
We control for any previous exposure or prior beliefs on the topics by measuring the shift in each participant’s pre and post responses, rather than only comparing post responses across treatments. From our raw pre-survey data (included in Appendix J), we also see that pre-survey responses were similar across conditions and showed relatively neutral stances. Questions on a 1-4 scale had pre-survey responses ranging from 1.8-3.2, questions on a 1-3 scale had responses from 1.7-2.3, and questions on a 0-1 scale had responses from 0.1-0.4, so we did not observe any ceiling effects.
Analysis details
We included main findings across four experiments in pages 7-9 of the paper, specifically the effects of the Doomsday packet, effects of anthropomorphic vs non-anthropomorphic packets, differences in effects across technologies, and changes in writing style, and will aim to include more details of statistical analysis in the next version. Additional figures and raw results are in Appendices J-P due to space constraints, and demographic breakdowns are in Appendix B.