acceptodds
Under review as a conference paper at ICLR 2027

Beyond Small Talk: Modeling Disclosure Trajectories in Human–AI Conversation

Abstract

Conversational agents are increasingly used for social and emotional support, where disclosure unfolds over time. Yet evaluations typically examine isolated conversational states, and in analyses of naturally occurring conversations, disclosure history and current disclosure are confounded, making their effects difficult to isolate. We present a framework grounded in Social Penetration Theory that constructs predefined disclosure trajectories and evaluates how preceding disclosure history shapes model behavior under matched current disclosure. Across multiple models and relational behavior benchmarks, the same current disclosure elicits different responses depending on the preceding trajectory. In the trajectories tested, these differences track accumulated exposure to deeper disclosure rather than any single moment of vulnerability or the order of disclosures, and most persist for several turns after disclosure returns to shallow levels. Models thus carry conversational history forward. For several behaviors, they diverge most consistently from human supporters in the Emotional Support Conversation corpus (ESConv) not when users open up but when they step back, as disclosure recedes or changes direction. These results suggest that relational behavior is better understood as a property of an interaction history than of any single response. Our framework provides a systematic way to expose history-dependent relational behavior that evaluations of isolated conversational states may miss.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM evaluation, Multi-turn evaluation, self-disclosure, synthetic conversation generation, counterfactual evaluation, sycophancy, social impact, explainability, transparency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UycmOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The current work attempts to model what is known as a disclosure trajectory in the context of multi-turn interactions between a user and an LLMs. In particular, their framework leverages doctrines from Social Penetration Theory in order to provide a rigorous investigation of how previous disclosures impact model behavior in current disclosures.

The contributions of the work are thus twofold - first they provide a benchmark-agnostic framework which is able to birth multi-turn conversations that also include disclosure trajectories. Second, they demonstrate the utility of this framework via controlled experiments that demonstrate the impact of such trajectories downstream.

Critical strengths
  1. the background from Social Penetration theory is well provided and necessary, as it is a level of detail and importance to the work that the average audience member for this work may not be as familiar with.
  2. Good diverse choice of models for evaluation purposes.
  3. the investigation into how prior disclosure history shapes model behavior in the future (RQ1) is thoroughly developed and the experimental methodology therein is sound with respect to the conclusions that were able to be drawn.
Critical weaknesses
  1. the use of a single model for persona construction as well as conversation generation is a bit worrisome, especially as the authors do not readily provide evidence of quality control and assurance that the subsequent generations are not only viable but also are of diverse and high quality. Despite mentioning a generate-evaluate-correct loop, it would be good if the authors provided more insight and detail into the specific ways in which this evaluation occurs.
Questions for authors

See above.

3 comments

How do we ensure the quality and diversity of generated conversations?

We thank the reviewer for raising these concerns. We agree that the quality of the generated conversations is important since our experiments depend on producing disclosures at specific depth levels. We have also revised the manuscript to better explain our generation and evaluation procedures, following the reviewer's comment that parts of the paper were difficult to follow.

We summarize our validation results, generation acceptance rates, and analyses excluding fallback conversations in the table below.

EvaluationResultsReference
Disclosure depth validationLi et al. (2026a) validated their method using 320 human-annotated conversations, with F1F_1 scores of 0.78–0.87 and Cohen's κ\kappa of 0.68–0.75.Li et al. (2026a)
Evaluator comparisonDepth matching rates of 93.2% using Phi-4 and 88.4% using GPT-4.1. Both showed similar difficulties in generating D3 disclosures.Appendix A.2
Generation acceptance ratesD1: 99.6%, D2: 99.7%, D3: 76.3%, D4: 99.9%.Appendix A.9
RQ1 without fallback conversations130/1,456 significant comparisons, compared with 196/1,456 in the full analysis. Of the original 196 significant comparisons, 121 remained significant.Appendix A.18
RQ2b without fallback conversationsThe persistence patterns remained similar after excluding conversation chains containing fallback turns.Appendix A.18

We use Qwen3.6-35B-A3B to construct the personas and generate the conversations, but use Phi-4 to evaluate whether the generated disclosures match their intended depths. We explain the generation and correction procedure in Appendix A.8. For each disclosure turn, Qwen is given the target depth, the relevant persona information, and the six most recent dialogue turns. Each generated utterance is then checked for its disclosure depth, conversational style, and persona leakage before we accept it.

For disclosure depth, we follow Li et al. (2026a), who evaluate informational, cognitive, and emotional disclosure. Since Li et al. used GPT-4.1 while we used Phi-4, we also repeated the generation experiment with GPT-4.1 to examine whether changing the evaluator would affect the results. As shown in the table above, both evaluators achieved relatively high depth matching rates, although generating D3 disclosures remained more difficult (see Appendix A.2).

We also check for style violations, such as formulaic expressions, restricted punctuation patterns, and utterances longer than two sentences. For persona leakage, we use keyword matching to check whether the utterance explicitly reveals the assigned EMS or other internal generation information. When a candidate fails any of these checks, we provide feedback on what needs to be corrected and generate another candidate. We allow up to 12 attempts for each disclosure turn.

As reported in the table above, the acceptance rates were above 99% for D1, D2, and D4, but were lower for D3. When no candidate passed all three checks after 12 attempts, we selected a fallback candidate by prioritizing disclosure depth matching, followed by the absence of persona leakage and the number of style violations. If multiple candidates still satisfied these criteria equally, we selected the most recent attempt. These fallback turns were recorded separately (see Appendix A.8). We also repeated the analyses excluding conversation chains containing fallback turns (see Appendix A.18). Although fewer comparisons remained significant in RQ1, excluding these chains also reduced the sample size. We therefore cannot determine whether this decrease was due to fallback removal or reduced statistical power. However, we found that the persistence patterns in RQ2b remained similar under this restriction.

Regarding diversity, we did not simply ask Qwen to generate different personas. We selected six conversational domains based on recurring themes in studies of human–AI interactions by Skjuve et al. (2022) and Ta-Johnson et al. (2022): Career, Friendship, Family, Romantic Relationships, Future/Aspirations, and Social Belonging. We also selected seven Early Maladaptive Schemas (EMS) from Young et al. (2003), focusing on interpersonal vulnerabilities relevant to self-disclosure. The choice of EMS was further informed by Ang et al. (2025), who identified EMS-related characteristics in naturally occurring disclosures.

Combining the six topics and seven EMS gave us 42 persona configurations. This allows us to vary not only the topic of discussion of the personas, but also their underlying concerns when discussing the same topic. For each configuration and disclosure trajectory, we further generated three independent conversation chains, giving us 126 conversations per trajectory for each evaluated model. We also sampled and reviewed the generated conversations to check whether the discussions remained relevant to their assigned conversational domains. In Appendix A.10, we included examples to illustrate how the generated disclosures reflect the assigned topics and underlying EMS across different disclosure depths. The selection rationale is provided in Appendices A.4 and A.6.

To ensure consistency in the generation setup, we also kept Qwen as the generator throughout our experiments. While this does not establish whether the same findings would hold with other generators, the framework itself is not restricted to Qwen. Future evaluations could repeat these experiments using other generators and incorporate human assessments of conversational naturalness and diversity.

Dear Reviewer Uycm,

We apologize for following up again. As the discussion period is coming to an end, we would like to ask if you might have time to review our rebuttal. We have addressed the critical weaknesses raised and acknowledge your ratings on presentation and informativeness. We have thus revised our manuscript to improve on reading flow as well as clarity on the validation of our methods and additional qualitative investigations in the results section.

We would appreciate it if you could let us know whether our responses have sufficiently addressed your concerns, or if there is anything else you would like us to clarify. If you find our responses satisfactory, we would also be grateful if you could consider revising your score. Thank you very much for your time and consideration!

Best, Authors

Reviewer_FxLpOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper studies the role of longer term historical context in human-AI conversations and conversational trajectories in shaping disclosure of a current utterance. The use of a simulation set up to create disclosure trajectories in different categories that can then be matched to real Human-AI conversations from the ESConv dataset is a neat way of generating a benchmark without the need to record new data. The premise of the paper is good. The topic is very important to the ongoing discourse on the potential harms of AI-chatbots.

Critical strengths
  1. The paper is clearly written and the procedure is straightforward.
  2. the experimental framework used to categorize and analysis human-AI dialogues with respect to disclosure looks thorough and well thought out.
  3. related work is extensive
Critical weaknesses
  1. Social Penetration Theory application is insufficiently argued given more modern discourse on this topic: this paper is largely more on the applied side. It is important and solid incremental work. Given that the authors showcase how well they know the literature, I do not understand why they would stick to a theory as old as Social Penetration Theory. By the author's own observations (line 49: "naturalistic disclosure histories are heterogeneous: depth, duration, topic, temporal ordering, and exposure to vulnerable disclosure often vary together"). It seems therefore rather old fashioned. Here are some modern critiques:
  • The assumption of more depth being associated with high intimacy does not hold. This study showed that relationship context matters with respect to depth vs relational intimacy: Effects of self- and partner’s online disclosure on relationship intimacy and satisfaction: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0212186
  • Shallow communication can create feelings of connection in the context of humour use. I see this as an example of how ignoring breadth (topic/context/function) can oversimplify. This study is only looking at social network sites but seems relevant when considered against other humour use literature. The function of self-disclosure on social network sites: Not only intimate, but also positive and entertaining self-disclosures increase the feeling of connection: https://www.sciencedirect.com/science/article/pii/S0747563214006943?via%3Dihub.
    See also literature on humour use as social bonding: e.g. Fraley, B. and Aron, A. (2004), The effect of a shared humorous experience on closeness in initial encounters. Personal Relationships, 11: 61-78. https://doi.org/10.1111/j.1475-6811.2004.00071.x
  • Assumptions of uniformity needs further discussion. Subjectivity of the disclosure depth is not discussed but seems relevant to be at least validated. Again, this is a social media reference but the subjectivity argument holds, especially given the claim of a "benchmark agnostic framework" (line 065). Hichang Cho, Bart Knijnenburg, Alfred Kobsa, and Yao Li. 2018. Collective Privacy Management in Social Media: A Cross-Cultural Validation. ACM Trans. Comput.-Hum. Interact. 25, 3, Article 17 (June 2018), 33 pages. https://doi.org/10.1145/3193120
  • functions of disclosure aside from intimacy can exist e.g. for self-confirmation, acceptance of inner self: Skok, K. (2025). Intimacy in social media: A theoretical analysis of extimacy. Culture & Psychology, 31(1), 243-258. https://doi.org/10.1177/1354067X241312246.
  • lack of accounting for reciprocity and negotiation of disclosure and intimacy . Choi, M., & Toma, C. L. (2022). An Experiment on the Effects of Self-disclosure on Perceived Partner Responsiveness and Intimacy in Zero-Acquaintance Relationships. Communication Studies, 73(3), 297–313. https://doi.org/10.1080/10510974.2022.2084429
  1. Operationalization of SPT applies only depth and not the breath component The current operationalization focuses on depth only while ignoring breadth. I would have expected to see a more nuanced analysis of how breadth (topic) and depth interplay. Even so, given that there are so many variants on SPT that bring in other factors, I wonder why modelling disclosure trajectories alone independent of others like amount/frequency/duration of disclosures, valence of disclosures, conscious intent, accuracy/honesty etc would see quite relevant for downstream tasks. see eg. Self-Presentation in Online Personals: The Role of Anticipated Future Interaction, Self-Disclosure, and Perceived Success in Internet Dating: https://journals.sagepub.com/doi/10.1177/0093650205285368

  2. **No human validation of the identified ESConv generate trajectories nor EMS trajectories ** : Perhaps I missed it but I could not find a clear explanation of how (if any) sanity checks were done to ensure that the ESM matched trajectories or the generated set reflected disclosures as agreed upon by human coders.

Questions for authors
  • Why just depth? there is so much that could have been done to understand various relationships between depth and breath. Given the nuances involved, this seems too simplifying an assumption.
  • Why SPT? There are many critiques of SPT from modern communication scholars (see weaknesses mentioned above)
  • How does the proposed framework help applications related to understanding social and emotional support when all contextual information that could have been extracted in a more nuanced and complex framework using more modern theory has been simplified so much?
  • My understanding of the current construction is that the chatbot was not also rated with respect to disclosure given that intimacy and disclosure are typically social and reciprocal effects that are negotiated between interactants. However, not accounting for LLMs that are e.g. explicitly designed to elicit disclosure seems to be a missed opportunity here.
  • how accurate is Lit et al. 2026a and Phi-4? Could this have affects the results of RQ2 where the order of disclosures was important?Couldn't there also be subjectivity/uncertainty there that needs to be accounted for in subsequent analyses?
  • Is this more likely an NLP paper? My rating is a weak accept. I find it hard to calibrate what looks like a paper for NLP conferences at ICLR. This feels to me like a very sold paper for a conference like EMNLP. I am happy to be enlightened on this point by the AC, other reviewers or the authors.
7 comments

We would like to thank the reviewer for their thoughtful and constructive feedback and for recognizing the value of our work. We summarize and address their concerns in six sections below. We have also revised the manuscript to clarify our theoretical motivation, methodological choices, and scope of contribution based on the reviewer's feedback.

1. Why SPT, given the critiques from modern communication research?

We agree that Social Penetration Theory (SPT) was originally developed to explain human interpersonal relationships and does not fully capture the complex nature of human–AI interactions. However, research on Computers as Social Actors (CASA) has demonstrated that people apply interpersonal social rules, including reciprocity, when interacting with computers (Nass & Moon, 2000). Related work has also demonstrated that reciprocal interaction can influence people's willingness to disclose personal information to computers (Moon, 2000). These findings thus motivated us to examine self-disclosure in human–AI conversations, although we also recognize that human–AI interactions may involve distinct social dynamics (Gambino et al., 2020).

We also acknowledge the modern critiques highlighted by the reviewer. Prior research has shown that disclosure depth does not necessarily imply greater relational intimacy, as this relationship depends on communication context (Lee et al., 2019). Similarly, shallow communication and humor can foster connection (Utz, 2015; Fraley & Aron, 2004), while communicative intent and reciprocity also influence interpersonal relationships. These findings challenge the assumption that deeper disclosure necessarily leads to greater intimacy, but they do not invalidate disclosure depth as a dimension of self-disclosure. We therefore draw on SPT specifically for its established distinction between disclosure depth and breadth, rather than adopting it as a comprehensive account of relationship development. Our study examines how preceding disclosure histories influence LLM relational behavior, not whether deeper disclosure produces greater relational intimacy. We have clarified this distinction in our revised Introduction section.

2. Why focus on depth rather than breadth and other dimensions of disclosure?

Our aim is not to capture every aspect of self-disclosure, but to understand how what users have disclosed earlier in a conversation affects the way LLMs respond later. We focus on disclosure depth because it captures how personal or vulnerable the information shared by the user is. This is particularly relevant to the relational behaviors we examine in this study, such as empathy, validation, and framing, as we want to understand how these model behaviors change in response to earlier user disclosures.

We also agree with the reviewer that depth is not the only factor that may influence model behavior. Other aspects mentioned, such as disclosure breadth, valence, communicative intent, and the wider conversational context, could also play a role. Our framework does not assume otherwise. In fact, our additional analyses in Appendix A.15 show that conversational domains and vulnerability profiles can influence behavioral trajectory shapes, although these effects are model dependent. While the analyses in Appendix A.15 do not independently manipulate disclosure breadth, they further highlight the importance of conversational context.

3. How do we account for the subjectivity and reliability of disclosure depth measurements?

We agree with the reviewer that disclosure depth can be subjective and influenced by individual and cultural context. As highlighted by Cho et al. (2018), social constructs may not be interpreted consistently across cultural settings. We therefore do not assume that our disclosure levels represent a universal interpretation of how personal or intimate a disclosure is.

To operationalize disclosure depth, we draw on the taxonomy of Li et al. (2026a), which distinguishes informational, cognitive, and emotional disclosure. Their coding framework was evaluated against human annotations. We also tested the robustness of our implementation to the choice of disclosure evaluator and examined whether excluding generated conversations containing fallback turns affected our findings. We summarize these results below.

EvaluationResultsReference
Human validation of disclosure codingEvaluated against 320 human-annotated conversations, with F1F_1 scores of 0.78–0.87 and Cohen's κ\kappa of 0.68–0.75.Li et al. (2026a)
Disclosure evaluator comparisonTarget-depth matching rates of 93.2% using Phi-4 and 88.4% using GPT-4.1. Both showed similar difficulties at D3.Appendix A.2
RQ1 excluding fallback conversations130/1,456 significant comparisons, compared with 196/1,456 in the full analysis. Of the original 196 significant comparisons, 121 remained significant.Appendix A.18

While Li et al. (2026a) used GPT-4.1 for disclosure annotation, our study replaces it with Phi-4. We therefore repeated the generation experiment using GPT-4.1 as the disclosure evaluator. As shown in the table above, both configurations achieved relatively high depth matching rates, although D3 disclosures remained more difficult to generate. While this comparison does not directly establish agreement between the evaluators, it suggests that the difficulty in generating D3 disclosures is not specific to Phi-4.

We also repeated our analyses after excluding conversation chains containing fallback turns. Although fewer comparisons remained significant in RQ1, excluding these chains also reduced the sample size. We therefore cannot determine whether the decrease was due to fallback removal or reduced statistical power. We further conducted five sensitivity analyses of the human trajectory matching procedure (see Appendix A.11.7).

We recognize that these analyses do not provide direct human validation of our aggregated four disclosure levels or show whether they are interpreted consistently across cultures. However, they help us understand how much our results depend on the disclosure generation and trajectory matching procedures.

We also want to clarify what we mean by "benchmark agnostic." Our framework allows the same disclosure trajectories to be evaluated using different behavioral benchmarks. We do not mean that disclosure depth is interpreted the same way across individuals or cultures. We have clarified this in our revised contribution statement.

4. How does the framework help evaluate social and emotional support despite simplifying conversational context?

Our findings demonstrate why examining disclosure history is relevant to evaluating social and emotional support. Importantly in our study, disclosure depth is used to structure our experimental conditions, while the models continue to receive the natural language conversations, including the contextual information expressed within them.

Across our RQs, we found that preceding disclosure history can affect model responses, that some of these effects persist after disclosure returns to shallower levels, and that models differ from human supporters in how their relational behaviors change. We summarize the relevant findings below.

AnalysisResultsReference
RQ1: Disclosure history sensitivity196 of 1,456 comparisons (13.5%) showed significant differences after FDR correction, despite holding the current user utterance constant.Appendix A.16
RQ1: FramingFraming showed significant differences in 30 of 56 comparisons (53.6%).Tables 30 and 31
RQ2b: Persistence of earlier disclosureOf the 67 initially significant model and behavioral metric combinations, 52 remained detectable after four shallow turns, with a median of 61% of the initial difference remaining.Appendix A.17.2
RQ3: Framing trajectory shapesUnder Ascending disclosure, 3 of 8 models showed significant trajectory shape differences from human supporters, compared with 8 of 8 models under Descending disclosure.Section 4.3, Appendix A.20

For RQ1, we found that preceding disclosure history could influence whether a model agrees with or questions the user's interpretation of a situation, even when the current message remains the same. For example, Qwen3-32B accepted the user's explanation about focusing on work and climbing under an Ascending history, but interpreted the same explanation as a way of avoiding underlying emotions under a Flat-D4 history (see Section 4.1). We believe this has relevance to emotional support because the model's interpretation of the user's current concerns may be affected by what they had disclosed earlier, which may not be apparent when evaluating individual responses.

In RQ2b, we have also found that the effects of earlier vulnerable disclosures could persist even after the conversation returned to shallower levels. This raises an emotional support question of whether users would actually appreciate models continuing to take their earlier concerns into account or perceive it as unnecessary emotional reassurance (e.g. Social Sycophancy). Although our study cannot answer this question, it allows us to examine how long these differences remain in subsequent responses.

For RQ3, we observed differences in how models and human supporters changed their relational behaviors, particularly when disclosure became shallower or changed direction. In our study, we do not take on the assumption that human supporters represent an ideal standard of emotional support. Rather, these comparisons provide a reference for understanding how models respond differently as user disclosure changes. This also helps us identify when and how model behaviors differ from human supporters, which may be useful for evaluating their responses in conversations involving emotional support.

Our findings thus suggest that evaluating individual responses may not be sufficient to understand how models behave throughout a conversation. However, we also recognize that these behavioral differences do not necessarily mean that models provide better or worse social and emotional support. We have not yet examined whether users notice these differences or how they affect their experience of the conversation. Drawing on prior research by Ho et al. (2018) and Sundar (2020), future studies could examine whether responses generated under different preceding disclosure histories affect how supported users feel and whether they would be willing to disclose more.

Reviewer_eXnCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the depth of users’ prior self-disclosure shapes how language models respond to the same current disclosure. Across four models and 26 behavioral measures, 119 of 832 comparisons between disclosure histories (14.3%) showed statistically significant differences after correction for multiple testing. Behavioral differences were more common when both the depth and duration of prior disclosure varied. When users returned to shallow disclosure, 30 of the 38 initially detectable differences remained detectable four turns later, although they generally diminished in magnitude. For several behaviors, models differed more consistently from human supporters in how they adapted when disclosure became shallower or changed direction than when it deepened.

Critical strengths

A strength of the paper is its use of self-disclosure trajectories to examine how prior disclosures shape a model’s response while keeping the user’s current utterance fixed. This design reflects an important feature of human conversation: the same remark can be understood differently depending on what was shared earlier. It therefore offers a promising way to model this aspect of human dialogue.

Critical weaknesses

Human comparison: The study reports several differences between model and human responses. Given these differences, it is unclear how well the proposed approach captures the way humans respond to changes in self-disclosure.

Sensitivity to model changes: The findings are necessarily based on the model versions tested in this study. It remains unclear how much the results would change if these models were updated or replaced.

Diversity of models: Although the human comparison includes eight models, the main controlled experiments cover only four. Testing a wider range of models would provide stronger evidence that the findings generalize.

Although the authors compare disclosure trajectories within the same persona, it remains unclear whether the persona itself has a larger influence on model responses than the disclosure trajectory does.

Questions for authors

The authors should explain more clearly what the observed differences between chatbots and human supporters imply.

7 comments

We thank the reviewer for raising this question. We will first respond to the question posed by the reviewer and then discuss some of the critical weaknesses mentioned in the next three points.

1. What do the observed differences between models and human supporters imply?

Our intention in RQ3 is not to show that models reproduce human conversational behavior, but to use human supporters as a reference for understanding how models change their relational behaviors as user disclosure evolves.

Prior research on interpersonal communication has highlighted the importance of responsiveness to a person's disclosure, rather than simply how much support is expressed (Laurenceau et al., 1998; Arican-Dinc & Gable, 2023). This is relevant to our findings because we have observed differences not only in the overall levels of relational behaviors, but also in how these behaviors changed across different disclosure trajectories. For example, Sycophancy/Agreement trajectory shapes differed from the human reference for 2 of 8 models under Ascending disclosure, compared with 7 of 8 under Descending disclosure. A similar pattern was also observable for the Framing property, where these differences increased from 3 of 8 to 8 of 8 models (see Table 2). These results imply that differences between models and humans may be more apparent when users become less vulnerable, rather than only when they disclose more.

We would like to further highlight differences in particularly relevant behaviors such as sycophancy and companionship reinforcement. For example, Cheng et al. (2026) demonstrated that sycophantic AI responses can influence users' judgments and also reduce their willingness to repair interpersonal conflicts. In our study, all eight models showed higher overall levels of Sycophancy/Agreement and Retention/Engagement than the human reference across all three trajectories. Although we did not measure the consequences of these behaviors for users, our findings are still suggestive that it may be important to evaluate not only whether models exhibit such behaviors, but also how they change as users become more or less vulnerable.

To better explain these differences in practice, we have added qualitative examples in Section 4.3 of our revised manuscript. In the example, we illustrate how a human supporter adjusted their emotional acknowledgement according to the seeker’s immediate concerns, whereas we see a model providing more emotional reassurance after earlier vulnerable disclosures. Despite these examples not being content matched, they still help illustrate the differences in Validation observed in our quantitative results (see Table 2).

We further discuss the implications of these differences in Section 5. Particularly, we do not know whether users would perceive the model's responses as attentive to their earlier concerns or as unnecessary emotional reassurance. For future studies, we can draw on Sundar's (2020) HAII-TIME framework to examine whether these differences affect users' perceptions of the chatbot and their experience of being understood or supported.

We acknowledge and recognize that ESConv serves as a human reference rather than an ideal standard of supportive behavior. Moreover, the human and generated conversations are matched on disclosure trajectories rather than identical conversational content. We therefore do not interpret these differences as evidence that models provide worse emotional support or cause harm. Instead, they highlight differences in behavioral adaptation that motivates further investigation, particularly whether users notice these differences and how they affect their experience of being understood or supported.

2. How sensitive are the findings to changes in model versions?

We agree with the reviewer that updating or replacing a model may change the magnitude of the observed differences or which behaviors are most sensitive to preceding disclosure history. However, we also found several consistent patterns across the different models evaluated in our study. We summarize these findings in the table below:

AnalysisResultsReference
RQ1: Disclosure history sensitivityAll seven models showed significant differences depending on preceding disclosure history. The proportion of significant comparisons ranged from 5.3% (Phi-4) to 25.5% (Qwen3-32B).Appendix A.16
RQ2b: Persistence of earlier disclosureAll seven models showed decreasing effects after disclosure returned to shallower levels. Of the 67 initially significant model and behavioral metric combinations, 52 remained detectable after four shallow turns. The proportion of the initial difference remaining ranged from 38% to 78% across models.Appendix A.17.2, Table 34
RQ3: Comparison with human supportersAll eight models showed higher overall Empathy, Validation, Sycophancy/Agreement, and Retention/Engagement than the human reference across all three disclosure trajectories.Appendix A.19
RQ3: GPT generations (overall Sycophancy/Agreement)Under Descending disclosure, GPT-4.1 showed a difference of +1.039 from the human reference, compared with +0.420 for GPT-5.6 Sol. Both were significant (p<.001p < .001).Appendix A.20, Table 38
RQ3: GPT generations (trajectory shape)Under Descending disclosure, GPT-4.1 showed a significant difference in trajectory shape from the human reference (p<.001p < .001), whereas GPT-5.6 Sol did not (p=.826p = .826).Appendix A.20, Table 38

From our evaluation, we can also examine the results for GPT-4.1 and GPT-5.6 Sol in RQ3. Despite being from different GPT generations, both showed higher overall Sycophancy/Agreement than the human reference across all three trajectories. However, as shown in the table, we also found differences in how their behaviors changed over the conversation. For instance, under Descending disclosure, GPT-4.1 showed a significant difference in trajectory shape from the human reference, whereas GPT-5.6 Sol did not. This shows that while both GPT generations exhibited higher overall Sycophancy/Agreement than humans, their differences from the human trajectory were not necessarily the same.

These results show that some behavioral patterns were consistent across the models we evaluated, although the magnitude of the differences and how these behaviors changed over time varied between models. While we cannot establish whether these findings would remain unchanged with future model updates, our framework allows us to repeat the same evaluations to examine which behaviors remain consistent and which change across model versions.

3. Why do the main controlled experiments evaluate only four models?

We thank the reviewer for this suggestion, which led us to extend our controlled RQ1 and RQ2 experiments to three additional models (Gemini 3.8-Flash, DeepSeek-V3.2, and GPT-4.1). The main findings hold for these models from different developers, providing stronger evidence that they generalize. We used the same persona configurations, trajectory conditions, three behavioral benchmarks, and analysis code as in our original submission. Re-running this code on the original four models reproduced every reported number. As the magnitude of these effects already varied across the original four models, we asked whether the same patterns, rather than the same magnitudes, appear in the new models. The results are summarized below.

AnalysisOriginal 4 modelsAdditional 3 modelsAll 7 models
RQ1: Disclosure history sensitivity119/832 (14.3%)77/624 (12.3%)196/1,456 (13.5%)
RQ2a: Joint depth and exposure85/416 (20.4%)44/312 (14.1%)129/728 (17.7%)
RQ2a: Mean depth with peak properties fixed10/1,248 (0.8%)7/936 (0.7%)17/2,184 (0.8%)
RQ2b: Significant effects remaining after four shallow turns30/38 (79%)22/29 (76%)52/67 (78%)
RQ2b: Decay slope per turn, β₁ (95% CI)−0.0102 [−0.0121, −0.0083]−0.0096 [−0.0119, −0.0073]−0.0100 [−0.0114, −0.0085]
Counts are FDR-significant comparisons out of comparisons tested. RQ2a rows use the extreme pairwise contrasts reported in Section 4.2.1.

For RQ1, the three additional models also showed significant differences in relational behavior depending on preceding disclosure history. The proportion of significant comparisons ranged from 10.1% to 13.5%, which was within the range observed for the original four models (5.3% to 25.5%). Across all seven models, 196 of 1,456 comparisons (13.5%) were significant after FDR correction, including 14.8% of comparisons at the shallowest current disclosure depth (D1). GPT-4.1 and GPT-5.6 Sol, two generations from the same developer, showed similar sensitivity (13.5% and 11.5%), which also bears on the reviewer's question about model updates (see our response to Point 2).

For RQ2a, the additional models showed the same pattern as the original four. Changing disclosure depth and exposure duration together (comparing the greatest-exposure trajectory with Flat-D1) produced significant differences in 14.1% of comparisons for the additional models, and 17.7% across all seven models. However, when we varied mean disclosure depth while keeping the peak depth and its position fixed, only 0.7% of comparisons were significant (0.8% across all seven models). Across all seven models, mean depth also had a larger standardized coefficient than peak depth in 66% of cells and than peak position in 74%, and varying peak depth or peak position alone rarely produced significant differences (at most 1.2% and 2.6% of comparisons, respectively). Hence, our findings continue to suggest that accumulated exposure to deeper disclosures may matter more than changes in mean disclosure depth when the peak properties are held constant.

For RQ2b, we found that the effects of earlier vulnerable disclosures continued to persist after the conversation returned to shallower levels, although they gradually decreased. Of the 29 initially significant model and behavioral metric combinations for the additional models, 22 remained detectable after four shallow turns. All three models also showed negative decay slopes, with confidence intervals excluding zero (β1\beta_1 ranging from −0.0114-0.0114 to −0.0066-0.0066), similar to what we observed for the original four models.

We recognize that evaluating seven models does not mean that our findings will necessarily hold across all language models, and the magnitude of individual effects varied across models, as it did among the original four. We have added these results, including per-model breakdowns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for the reviews and helpful suggestions. We have revised our paper following the feedback provided, particularly to improve the clarity of our methodology, findings, and the scope of our work. We have also conducted additional experiments and robustness analyses to address some of the concerns raised by the reviewers.

We have uploaded a revised manuscript, and the main changes are summarized below.

  1. Clarified our evaluation framework and its theoretical motivation (Section 1). We have defined our framework as TraDE (Trajectory-based Disclosure Evaluation) and clarified its purpose. We have also revised our discussion of Social Penetration Theory to better explain why we focus on disclosure depth. Particularly, we clarify that we do not assume deeper disclosure necessarily implies greater relational intimacy, as this also depends on the communication context.

  2. Improved the clarity of our methodology (Section 3). We have revised the descriptions of our persona construction, conversation generation, human trajectory matching, and statistical analyses. We have also included examples of generated utterances from D0 to D4 in Table 1 to better illustrate how we distinguish between disclosure depths.

  3. Provided further validation of our generated conversations (Section 3, Appendices A.2, A.8–A.10, and A.18). We have included additional details on our generation and correction procedure, reported the acceptance rates at each disclosure depth, and compared Phi-4 against GPT-4.1 as our disclosure depth evaluator. We also repeated our analyses after excluding conversation chains containing fallback turns to examine whether these affected our findings.

  4. Examined the effects of conversational domains and EMS (Section 4, Appendix A.15). Following concerns about the influence of our persona configurations, we examined whether conversational domains and Early Maladaptive Schemas (EMS) affected model behavior. We found that these effects varied across models and did not show a consistent pattern. We have included these analyses to better explain how these factors may influence our results.

  5. Extended RQ1 and RQ2 to three additional models (Sections 4.1–4.2, Appendices A.16–A.18). To address concerns about generalizability, we expanded our controlled experiments from four to seven models by including Gemini 3.8-Flash, DeepSeek-V3.2, and GPT-4.1. We observed similar findings across the additional models. For RQ1, 196 of 1,456 comparisons (13.5%) were significant after FDR correction. For RQ2a, joint changes in disclosure depth and exposure produced more significant differences than varying mean depth while keeping peak properties fixed (17.7% compared with 0.8%). For RQ2b, 52 of 67 (78%) initially significant effects remained detectable after four shallow turns. We have included the additional results and their respective model breakdowns in the appendix.

  6. Added robustness analyses for our human trajectory matching (Section 3.4, Appendix A.11.7). We conducted five robustness checks to examine selection bias, sample size uncertainty, unequal sample sizes, DTW acceptance thresholds, and matching quality on our human comparisons. We found that our results remained broadly similar under different matching conditions, and have included the analyses in our appendix.

  7. Included qualitative examples for RQ1, RQ2b, and RQ3 (Sections 4.1–4.3). We have added examples of actual model responses to better explain our quantitative findings. Particularly, these examples illustrate how preceding disclosure history can change a model's interpretation of the same current message, how earlier vulnerable disclosures can continue to affect later responses, and how model responses differ from human supporters as disclosure changes.

  8. Revised our Discussion to better explain the implications and limitations of our work (Section 5). We have clarified how TraDE can be used to examine history-dependent model behavior, while acknowledging that our experiments do not explain the internal mechanisms behind these differences or whether users would perceive them positively. We have also included a short reflection on the role of chatbot reciprocity, particularly how a chatbot's own responses may influence subsequent user disclosure, and the importance of examining privacy awareness in future studies.

We thank the Area Chair and Reviewers once again for their feedback and suggestions. We hope these revisions have helped address the concerns raised and improve the overall clarity of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.