acceptodds
Desk rejected at ICLR 2027

Privacy of the Mind: How the Law Should Respond to Emotional Processing Through AI Chatbots

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

People increasingly use AI chatbots not only to retrieve information or complete tasks, but to process emotions, reflect on personal experiences, and think through intimate problems. These interactions externalize cognitive and emotional processes that have traditionally occurred within the mind or through socially protected relationships such as diaries, friendships, and therapy. Yet existing approaches to AI privacy largely treat these interactions as exchanges of sensitive data, overlooking the role of the AI system in the cognitive-emotional process itself. We examine whether this phenomenon reflects a salient form of privacy in the age of AI that ought to be recognized and legally protected. We propose privacy of the mind as a distinct legal concept grounded in historical U.S. legal traditions protecting freedom of thought, cognitive liberty, and mental health integrity. We examine how AI-mediated conversation externalizes inner voice and outsources inner work, and identify design features that can invite further emotional disclosure, foster dependency, and create an appearance of confidentiality. These features make the interaction bidirectional: users expose increasingly intimate aspects of their inner lives to AI systems, while those systems simultaneously participate in shaping the cognitive and emotional processes through which users understand themselves. We then situate privacy of the mind within a century-long legal evolution protecting the boundaries of human thought—from First Amendment protections of cognition, to Fifth Amendment protections against self-incrimination, and to protections arising in mental health law. We argue that privacy of the mind in the AI era cannot be secured through data governance alone; it requires privacy to shape interaction design, system behavior, and the legal frameworks governing AI-mediated emotional processing.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy Law, AI-mediated Emotional Processing, Design, AI Regulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NpoQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces dangers regarding AI-human motion interaction.

Critical strengths

The topic is important.

Critical weaknesses

I cannot find core contributions. The paper lacks clear logics. The paper covers three aspects: interaction, data governance, and law enforcement. These are too broad for a good research paper.

Questions for authors

What is the research question that this paper handles?

Reviewer_TX92OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses the extent to which AI is an extension of self, how that gives service providers access to the thought and emotion processes of users, and proposes a legal framework how such access should be understood. Access is understood both in terms of access to information about those processes, but importantly also how service providers can influence thought and emotional processes.

Critical strengths
  • The questions addressed are exciting, timely and likely of critical importance for the well-being of humans.
  • The proposed legal framework is clearly defined and potentially a very useful link between technological research and development and legal scholars.
Critical weaknesses
  • It remains for the ACs to verify whether this is within the scope of the event. I favour accepting it, but I do understand that some might object, as it goes quite far into psychology and law, without much technical content.
  • I believe this paper would give a useful background information to this work Leschanowsky, et al (2024). Evaluating privacy, security, and trust perceptions in conversational AI: A systematic review. Computers in Human Behavior, 159, 108344.
Questions for authors

What is the intended audience for this paper? Is it for legal scholars or for the experts in technology?

Reviewer_hUv1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the privacy implications of using AI chatbots for emotional processing and personal reflection. Its central argument is that protecting conversational records alone is insufficient, since the chatbot may also shape the user's thoughts and emotions. The authors propose Privacy of the Mind (PoTM), covering both exposure of intimate information and harmful interference with the cognitive-emotional process. They relate this proposal to existing privacy and legal frameworks and discuss interventions at the design, data-governance, and legal levels. The contribution is conceptual and normative, supported by prior studies and illustrative cases rather than new experiments.

Critical strengths

The distinction between protecting data and protecting the interaction is valuable. Sections 4.2-4.4 explain why a chatbot is not merely a passive recipient of personal information: its responses may influence how a user interprets a difficult experience. Table 1 develops this distinction by separating design choices, data controls, and legal responsibilities. This is the paper's strongest contribution, as retention policies or encryption alone would not address all of these concerns.

Critical weaknesses
  1. The contribution of PoTM beyond existing frameworks remains insufficiently established. As discussed in Sections 4.7-4.8, intellectual privacy already concerns the formation of ideas, while cognitive liberty addresses both access to mental information and manipulation of thought. The paper needs to clarify what additional protection or analytical value PoTM provides. Its scope is also difficult to apply: a conversation about a breakup may combine emotional processing, companionship, and assistance with writing a message. It is unclear which aspects would fall under PoTM and what additional obligations would follow for a provider. Table 1 identifies reasonable interventions but does not resolve these questions. A worked example showing where PoTM leads to a different duty, remedy, or design decision would help establish the contribution. The overlap with earlier work concerns novelty; it does not establish plagiarism.

  2. Several evidentiary claims require correction or stronger support. a) Lines 139-141 describe the Anthropic result as "increasing vulnerability discoverable after every 300 interactions." The cited report instead states that severe user vulnerability occurred in roughly one in 300 interactions. This is a prevalence estimate, not evidence of increasing vulnerability within an individual after repeated interactions. In addition, the 1.9% figure in lines 131-135 matches Chatterji et al., How People Use ChatGPT (2025), rather than the cited Phang et al. study, and refers to messages rather than chats. These differences affect the interpretation of the evidence. The roughly 10% estimate from publicly shared sessions also requires a discussion of sampling limitations before comparison with a platform sample. b) Section 3.2 characterizes six mechanisms as deliberate design choices without sufficient direct evidence. In particular, lines 195-201 describe a system responding to suicidal thoughts as engineered to validate those thoughts and continue the conversation indefinitely. This claim requires product-specific support or narrower wording. Observed model failures, hypothesized commercial incentives, and documented design choices should be distinguished. c) The supporting materials provide limited detail for independent assessment. There is no appendix documenting source selection or applying the framework to a concrete case, and I found no code repository or released dataset. For this legal contribution, reproducibility primarily concerns whether readers can retrace the evidence and reasoning, rather than execute code. Some cited studies rely on provider-controlled data, and the paper provides no independent empirical validation of PoTM. There is also no ablation or structured comparison isolating the contribution of each proposed component. A conventional algorithmic ablation is not necessary here, but a comparative case analysis would strengthen the argument.

  3. The topic may relevant to ICLR's interest in privacy and the societal implications of AI, and the focus on the interaction itself is valuable. However, the contribution beyond existing frameworks is not yet sufficiently demonstrated, and the evidentiary claims require more careful treatment. These concerns, rather than the absence of a new algorithm or benchmark, motivate my recommendation. A clearer demonstration of what PoTM changes, together with corrected and better-supported claims, could change my assessment.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.