acceptodds
Under review as a conference paper at ICLR 2027

Right Destination, Wrong Journey: Synthetic Counseling Dialogues Miss Human Interactional Dynamics

Abstract

LLM-generated counseling dialogues are increasingly used for training and evaluating mental-health systems, yet existing evaluations focus largely on response quality and final outcomes rather than how support unfolds over time. We conduct a trajectory-centered evaluation of process fidelity across nine human, reconstructed, and synthetic counseling datasets. Our results show that synthetic dialogues often reach plausible endpoints but differ from human support in recovery dynamics and turn-by-turn responsiveness. While generation design can shift global properties such as recovery timing and trajectory structure, contingent patient–counselor responsiveness remains consistently lower in fidelity across models and generation paradigms. Our experiments on controlled synthetic datasets show that targeted generation interventions can narrow the gap with real data on several trajectory-level measures, but do not fully recover interactional fidelity, with turn-level responsiveness remaining limited. We release three controlled synthetic datasets comprising 2,050 dialogues, together with annotations and the evaluation framework.

Primary area:
datasets and benchmarks
Keywords:
Mental Health, NLP application in Healthcare, Synthetic Dialogue Evaluation, Emotional Trajectories, Emotion States, User Simulation, Conversational Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dxw5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a trajectory-centered framework for evaluating process fidelity in counseling dialogues. The framework assesses turn-level emotional and distress dynamics, counselor strategies and their associations with subsequent patient states, recovery timing and structure, and trajectory diversity. Applying this framework to nine human, reconstructed, and synthetic counseling datasets, the authors find that synthetic dialogues often reach plausible emotional endpoints while following recovery trajectories that differ from those in human support conversations. They also identify persistent gaps in contingent counselor–patient responsiveness across the models and generation paradigms examined.

Critical strengths

The paper addresses an important and worthwhile research question, and its focus on emotional trajectories in evaluating synthetic counseling dialogues is well motivated. The difference in trajectory-level control and turn-by-turn responsiveness between synthetic and real data can inform the design of data synthesis methods.

The four evaluation dimensions provide complementary perspectives, and the experiments include meaningful intermediate controls, where the evaluations are conducted on multi-type datasets, including real, semi-real, and fully synthetic interactions.

Critical weaknesses

The main findings rely on automated emotion labels and a predefined mapping from emotion categories to distress scores, while the Chinese datasets additionally undergo translation into English. However, the dataset-specific agreement with human annotations, labeling accuracy, or replication using alternative annotation models are not reported. Consequently, the observed differences may reflect linguistic style, translation effects, or annotation bias, whose contributions remain unclear.

Questions for authors
  1. How does the framework distinguish reduced distress from changes in emotional expression, such as gratitude or increased willingness to disclose? What annotation evidence supports interpreting the selected low-distress states as recovery across the counseling settings studied?

  2. Is process fidelity intended to be defined relative to peer support or to the goals and modality of each counseling setting? What will clinically appropriate trajectories in motivational interviewing or multi-session psychotherapy be classified when they differ from ESConv?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Dxw5 (1/2)

We thank the reviewer for the thoughtful comments and their recognition of the importance of our research question, the complementary perspectives provided by our four evaluation dimensions, and the meaningful controls across real, semi-real, and fully synthetic interactions. We address each concern below.

Weakness 1: Emotion annotation reliability and translation effects

(1) Translation of Chinese datasets into English.

We wanted to cover datasets across different languages while keeping the emotion-labeling pipeline consistent. We considered annotating each dataset in its original language or translating the Chinese datasets into English. We chose translation so that all datasets could be evaluated in the same language, using the same emotion taxonomy, annotation model, and scoring procedure. This provides a common basis for comparing emotional trajectories across datasets.

We acknowledge that translation may affect emotional nuance, and systematic human validation would be needed to quantify its effects on emotion labels and derived distress scores. We will make this limitation explicit in the paper and identify a systematic comparison of native-language and translation-based annotation as future work.

(2) Emotion-labeling performance across datasets.

Emotion recognition is a well-studied area in NLP. Prior work reports promising zero-shot performance of LLMs across emotion-labeling tasks, including conversational and multilingual settings [1–3]. These studies support the use of LLMs for emotion annotation, while also showing that performance varies with the dataset and linguistic context, particularly when emotional expressions are implicit [4].

To provide evidence specific to our annotation pipeline, we conducted the additional cross-model comparison below.

(3) Additional experiments using alternative annotation models.

We compared GPT-5.6 Luna’s predictions with those of Claude Opus 4.8 and Gemini 3.6 Flash on 705 client turns from 50 ESConv conversations. Across the three model pairs, we observed substantial agreement on fine-grained emotion labels (Cohen’s κ = 0.62–0.68), strong agreement on emotion polarity (79.0–87.5%; κ = 0.67–0.79), and high correlations in derived distress scores (r = 0.81–0.85). These results have been added to Section 4.1 and Appendix G, highlighted in blue in the revised draft.

Table R1. Cross-model emotion annotation agreement on 705 client turns from 50 ESConv conversations.

Model pairExact (%)κPolarity (%)Polarity κDistress rExact / synonym (%)
Luna–Opus 4.869.90.6887.50.790.8580.9
Luna–Gemini 3.6 Flash67.20.6481.80.720.8475.9
Opus 4.8–Gemini 3.6 Flash65.20.6279.00.670.8174.0

Luna’s agreement with both models is comparable to their agreement with each other. The agreement extends beyond exact emotion labels to polarity and derived distress scores, which are directly relevant to our trajectory analysis. In this comparison, Luna achieves this consistency at approximately one-twentieth the inference cost of Opus 4.8 and one-third that of Gemini 3.6 Flash, making a uniform annotation pipeline cost-efficient at the scale of our study. Although dataset-specific human validation is still a limitation of our study which we will clarify in the limitation section, these results above support cross-model annotation consistency.

Our evaluation focuses on comparisons under a consistent annotation procedure. Applying the same emotion-labeling model and scoring procedure across datasets provides a standardized basis for comparing emotional trajectories and interactional dynamics. Systematic annotation biases may still affect the estimated differences, but the framework is model-agnostic and can be applied with alternative annotation models to further assess robustness.

AuthorsOpenReview ↗

Response to Reviewer Dxw5 (2/2)

Question 1: Distinguishing distress reduction from changes in emotional expression

We use distress reduction as an indicator of emotional improvement. The original ESConv study similarly uses the decrease in self-reported emotional intensity to measure improvement [5]. This provides a precedent for distress-focused evaluation, although our label-derived distress scores are a proxy for expressed emotion.

In our analysis, “recovery” refers to reduced expressed distress within a conversation. We agree that emotion labels alone cannot fully distinguish different reasons that led to shifts toward positive emotion, and will add this to our limitation.

To further assess the interpretation of the patterns identified by our quantitative analysis, we will conduct a qualitative human evaluation of representative counseling trajectories, examining whether observed changes reflect emotional relief or alternative processes, such as gratitude and increased disclosure. Although it possibly will take longer than the rebuttal period, we will also incorporate illustrative case studies in the final version to contextualize our finding once the qualitative study is complete.

Question 2: Defining process fidelity across counseling settings

We’s like to clarify that process fidelity should be interpreted relative to the goals and modality of each counseling setting. Our goal is to provide a transferable method for comparing trajectories, rather than establish a single ground truth for all settings.

For example, when evaluating a synthetic motivational interviewing dataset, the framework can compare the generated conversations with authentic motivational interviewing data, including private reference data where available. The same approach can be adapted to multi-session psychotherapy, using reference data and interpretations appropriate to that setting.

The framework also supports relative comparisons between datasets or groups of datasets, as demonstrated in our paper. These comparisons characterize their trajectory distributions and interactional patterns; comparisons with authentic data from the same setting provide a stronger basis for assessing synthetic process fidelity. We chose ESConv as a common reference because it is one of the few publicly available authentic human-support datasets with complete conversations and annotations.

We will clarify this scope in the paper. The contribution we hope to share with the community is a transferable evaluation method, together with insights into the process-level distributions and characteristics of current datasets when grouped as synthetic, semi-synthetic, and authentic.


We thank the reviewer again for recognizing our methods and findings, and for the insightful questions and concerns raised. The additional experiments results and clarifications are included in the uploaded revised draft, highlighted in blue, and we will incorporate the qualitative study described above. We hope our revisions address your main concerns, and welcome any remaining questions or concerns.

References

[1] Niu et al. Rethinking Emotion Annotations in the Era of Large Language Models. IEEE Transactions on Affective Computing, 2025.

[2] Feng et al. Affect Recognition in Conversations Using Large Language Models. SIGDIAL, 2024.

[3] Safdarian et al. UT-NLP at SemEval-2025 Task 11: Evaluating Zero-Shot Capability of GPT-4o mini on Emotion Recognition via Role-Play and Contrastive Judging. SemEval, 2025.

[4] Sosea and Caragea. Hard Emotion Test Evaluation Sets for Language Models. Findings of NAACL, 2025.

[5] Liu et al. Towards Emotional Support Dialog Systems. ACL-IJCNLP, 2021.

AuthorsOpenReview ↗

Follow up on Rebuttal Discussion

Dear Reviewer,

Thank you again for your detailed review. As the discussion period is coming to an end, we wanted to briefly follow up to see whether our rebuttal and additional experiments have adequately addressed your concerns. We would be happy to provide any further clarification or answer any remaining questions.

Thank you for your time and consideration!

Reviewer_Me4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing evaluations of mental health corpora predominantly focus on "response quality" and "final outcomes," while in-depth investigations into how support unfolds over time remain limited. Centering on the concept of process fidelity, the authors conduct an evaluation across nine human, reconstructed (semi-real), and synthetic counseling datasets. The core contribution lies in proposing metrics across four dimensions—turn-level emotional change, recovery structure, patient–counselor responsiveness, and trajectory diversity—to quantify process fidelity, followed by extensive experiments that reveal the persistent limitations of interactional fidelity across the studied synthetic corpora.

Critical strengths

To uncover and quantify the dynamic disparities between authentic and synthetic counseling dialogues, the authors propose a targeted suite of metrics centered on process fidelity. Specifically, these include stage-adjusted conditional mutual information (stage-adjusted CMI) to eliminate spurious dependencies introduced by shared conversation schedules, recovery onset and last-mile share to characterize the pacing and progression of recovery, as well as trajectory archetype clustering. By mathematically decoupling surface-level conversational fluency from authentic emotional trajectories, these metrics provide substantial methodological value for both the generation and evaluation of counseling dialogue corpora.

Critical weaknesses
  1. The setup of several key experiments is insufficient to support the generality of their conclusions. (1) The prompt-tuning experiments in Table 5 are conducted on a single base model, GPT-4o. However, the synthetic-generation ablation reported in the same section shows that "model choice has the largest effect" within the zero-shot corpus. The conclusions drawn from Table 5 should therefore be validated across multiple diverse base models. (2) The coverage of generation architectures is narrow. The three controlled conditions built by the authors span only two paradigms: single-pass (Zero-shot) and dual-agent generation; the published synthetic corpora under comparison do not go beyond this range either. Given that multi-agent collaborative generation frameworks(e.g., pipelines with planner/critic roles) have become increasingly common since last year [1-3], no corresponding dataset is included to validate the conclusions. (3) Some conclusions are stated more uniformly than the per-dataset results warrant. Table 4 shows that the published synthetic corpus SimPsyDial performs comparably to real data on per-strategy downstream distress reduction, and in Table 3, the largest gaps of Stable% (56.9%/43.7% vs. 0.8%) and SCR (0.064/0.121 vs. 0.386) come mainly from the authors' own unconstrained baseline setups (Baseline DA and Dual Agent). Whether a uniform conclusion across all synthetic corpora can be drawn thus requires further justification; the authors should clearly identify which process-level gaps hold consistently across all synthetic datasets.

  2. The paper lacks necessary significance testing, which weakens the credibility of its conclusions. No p-values or formal tests of cross-corpus differences are reported anywhere; uncertainty is quantified only through confidence intervals and permutation-based bias correction. For example, the differences between real and synthetic data in Figure 3 and Table 3 are presented purely descriptively, and the Tier-1 improvements in Table 5 rely on small samples of 100+100 conversations. Without testing, genuine effects cannot be distinguished from noise.

[1] MultiAgentESC: A LLM-based Multi-Agent Collaboration Framework for Emotional Support Conversation.(EMNLP 2025)

[2] StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation. (Findings of ACL 2026)

[3] PsyChain: A Collaborative Chain-of-Agents Framework for Generating Personalized and Professional Counseling Dialogues. (Findings of ACL 2026)

Questions for authors
  1. It appears that recent work has already combined process-level supervision signals with reinforcement learning to build stronger psychological counseling corpora / AI counseling models[1-3]. Do the conclusions of this paper still apply?
  2. The average number of turns varies considerably across datasets (minimum 25.9, maximum 86.6), yet all trajectories are interpolated onto a fixed set of 20 progress points. Have the authors considered how changing the number of interpolation points (e.g., 40/50) would affect the trajectory shapes and the related metrics?

[1] Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoning(ACL 2026) [2] MIThinker: A Plug-and-Play Policy-Optimized Thinker For Motivational Interviewing Counseling(Findings of ACL 2026) [3] Flexible Thinking for Multimodal Emotional Support Conversation via RL(EMNLP Findings 2025)

6 comments
AuthorsOpenReview ↗

Response to Reviewer Me4J (1/4)

We thank the reviewer for the thoughtful and constructive comments on our paper. We appreciate their recognition of the substantial methodological value of our work for the generation and evaluation of counseling dialogue corpora. We address each concern below.


Weakness 1: Generalizability of conclusions

(1) Prompt-tuning experiments are conducted on a single base model (GPT-4o).

We thank the reviewer for raising this concern. We would like to clarify that Table 5 was intended as an exploratory experiment examining whether process-level fidelity can be improved through explicit behavioral instructions alone.

To address this concern, we extended our experiments to three additional frontier models: GPT-6.1 Sol (September 2026), Claude Sonnet 5.5 (September 2026), and Gemini 3.1 Pro (February 2026). Given the high API costs of generating multi-turn conversations, we generated 100 conversations per prompt-tuning condition. We acknowledge that this sample size limits statistical power, and larger-scale evaluations would be valuable in future work.

The results are summarized below, with detailed analyses reported in Appendix E.7–E.9.

GPT-6.1 Sol

MeasureMetricControlTunedESConvResult
Tier 1: Varies with protocol
RecoveryEnd positive (%)70.071.067.3✗
Recovery onset0.590.430.56✗
Last-mile share0.810.630.50✓
Post-onset persistence (%)16.819.753.4✗
Archetype coverageJSD to ESConv0.5340.4970.00✗
Dominant archetype (%)72.060.030.0✓†
Tier 2: Persists across prompt interventions
Strategy mixStrategy evenness (J)0.730.720.89✗
Top-2 strategy share (%)66.665.634.6✗
Strategy-channel couplingStage-adjusted CMI (bits)0.0100.0090.039✗
Emotion-channel couplingStage-adjusted CMI (bits)0.0010.0020.019✗
Turn-level reliefStrategy-explained Δd (%)1.71.11.9✗
Mean Δd per counselor turn-0.009-0.009-0.041✗

Claude Sonnet 5.5

MeasureMetricControlTunedESConvResult
Tier 1: Varies with protocol
RecoveryEnd positive (%)64.071.067.3✗
Recovery onset0.420.440.56✗
Last-mile share0.640.510.50✓
Post-onset persistence (%)19.627.753.4✓†
Archetype coverageJSD to ESConv0.3040.1970.00✓
Dominant archetype (%)56.051.030.0✓†
Tier 2: Persists across prompt interventions
Strategy mixStrategy evenness (J)0.700.730.89✓†
Top-2 strategy share (%)75.370.034.6✗
Strategy-channel couplingStage-adjusted CMI (bits)0.0090.0120.039✗
Emotion-channel couplingStage-adjusted CMI (bits)0.0050.0020.019✗
Turn-level reliefStrategy-explained Δd (%)0.80.81.9✗
Mean Δd per counselor turn-0.012-0.012-0.041✗

Gemini 3.1 Pro (Appendix E.9)

MeasureMetricControlTunedESConvResult
Tier 1: Varies with protocol
RecoveryEnd positive (%)44.051.067.3✓
Recovery onset0.520.490.56✗
Last-mile share0.620.520.50✓
Post-onset persistence (%)24.027.153.4✗
Archetype coverageJSD to ESConv0.3480.2650.00✓†
Dominant archetype (%)51.050.030.0✗
Tier 2: Persists across prompt interventions
Strategy mixStrategy evenness (J)0.650.710.89✓†
Top-2 strategy share (%)71.968.534.6✗
Strategy-channel couplingStage-adjusted CMI (bits)0.0110.0080.039✗
Emotion-channel couplingStage-adjusted CMI (bits)0.0090.0080.019✗
Turn-level reliefStrategy-explained Δd (%)1.00.41.9✗
Mean Δd per counselor turn-0.009-0.009-0.041✗

Summary of additional experiments

  • Recovery-related metrics are more responsive to prompt tuning. All three models achieve at least a 30% reduction in the gap to ESConv for last-mile recovery share. Claude Sonnet 5.5 also improves archetype-distribution similarity, while Gemini 3.1 Pro improves positive-ending rate.
  • Improvements in interactional metrics remain limited. None of the three models achieves a 15% gap reduction in strategy-channel coupling, emotion-channel coupling, or turn-level relief.

These additional experiments suggest that the distinction between recovery outcomes and interactional dynamics observed in our original GPT-4o experiments is not specific to a single base model. Even with frontier models, we observe a consistent difference between recovery-related metrics and interactional dynamics.

AuthorsOpenReview ↗

Response to Reviewer Me4J (2/4)

Weakness 1: Generalizability of conclusions

(2) Lack of multi-agent collaborative generation frameworks and datasets.

We thank the reviewer for highlighting the importance of evaluating more recent multi-agent collaborative generation frameworks. Our original experiments followed the mainstream datasets used in prior work and primarily cover single-pass and dual-agent generation, and we agree that extending the evaluation to more sophisticated architectures would strengthen the generalizability of our findings. We also considered this during the experimental design, but finding both real and high-quality synthetic datasets is difficult. We reached out to some authors but did not receive a reply.

We closely investigate the three representative frameworks cited by the reviewer. However, MultiAgentESC does not release public dataset and focuses on multi-agent collaboration for counselor response generation, rather than generating complete patient–counselor dialogues. Since our evaluation requires full conversations to measure recovery trajectories and interactional dynamics, its released resources were not directly suitable for our analysis. StoryMI only releases a dataset of 20 conversations generated by 5 models, which is not enough for our analysis. For PsyChain, the authors released a test set containing 1,047 synthetic counseling dialogues, while the full generated corpus does not appear to be publicly accessible. Nonetheless, these are all good frameworks, and we have cited them in our related work on in our revised draft on lines 137-144.

To directly address the reviewer's concern, we extend our evaluation to the 1,047 publicly available PsyChain dialogues with an average of about 37 turns. PsyChain employs a collaborative chain-of-agents architecture with specialized agents responsible for client simulation, counseling, process monitoring, and supervision, substantially broadening the generation architectures covered by our study.

Our findings about PsyChain remain consistent with our central conclusions and further validates our finding in an expanded task setting. PsyChain exhibits a higher positive-ending rate than ESConv but lower sentiment volatility. More importantly, despite its collaborative multi-agent design, PsyChain shows weaker stage-adjusted strategy coupling and emotion coupling, as well as lower inter-conversation trajectory diversity.

These findings extend our conclusions to a more sophisticated multi-agent generation paradigm: producing conversations with favorable emotional endpoints does not necessarily reproduce the interactional dependencies or diversity of recovery pathways observed in real counseling. We have incorporated PsyChain into Tables 3–4, Figure 3, and the corresponding significance analyses in Appendix F.

(3) Some conclusions are stated more uniformly than the per-dataset results warrant.

We thank the reviewer for this observation. In our paper, we acknowledge that some process-level properties vary with the generation paradigm and are dataset-dependent.

Regarding which process-level gaps hold consistently across all synthetic datasets:

  • Distress trajectories: All eight synthetic corpora differ significantly from ESConv in both overall trajectory distance and mean-centered trajectory shape (Holm-adjusted p < 0.001; Appendix F.1).
  • Sentiment dynamics: All eight differ significantly in sentiment change rate (SCR) and positive-ending rate (p < 0.001; Appendix F.5), although the direction varies by dataset.
  • Interactional coupling: All eight exhibit significantly weaker emotion-channel coupling than ESConv (p < 0.01; Appendix F.3).
  • Trajectory diversity: All eight exhibit significantly lower inter-conversation trajectory diversity than ESConv (p < 0.001; Appendix F.4).

Other findings are more dependent on the generation paradigm. For example, strategy-channel coupling is significantly weaker in six of eight synthetic corpora, while CACTUS and PsychEval do not differ significantly from ESConv. Recovery timing, persistence, strategy distributions, and individual strategy effects also vary across datasets.

We have clarified these distinctions in the revised paper Section 5.1 (line 356-362).


References:

[1] MultiAgentESC: A LLM-based Multi-Agent Collaboration Framework for Emotional Support Conversation. (EMNLP 2025)

[2] StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation.(Findings of ACL 2026)

[3] PsyChain: A Collaborative Chain-of-Agents Framework for Generating Personalized and Professional Counseling Dialogues. (Findings of ACL 2026)

*As described, we have included a comparison and cited in the paper.

AuthorsOpenReview ↗

Response to Reviewer Me4J (3/4)

Weakness 2: Lack of significance testing

We have added statistical significance tests with Holm correction for all four analyses in Figure 3(a–d) and Table 3, with detailed results reported in Appendix F.1–F.5. We use conversation-level permutation tests for continuous metrics, Fisher's exact tests for binary proportions, and permutation tests for conditional mutual information and trajectory diversity. All reported p-values are Holm-adjusted for multiple comparisons. Detailed per-corpus results are provided in Appendix F.

Summary of Statistical Significance Tests (also updated in rebuttal revision Table 3, Appendix F 1-5).

All tests compare each corpus against ESConv, with Holm correction for multiple comparisons.

AnalysisMain findingsStatistical significanceAppendix
Distress trajectories (Fig. 3a)All 10 corpora differ from ESConv in both overall distress trajectories (RMS distance) and mean-centered trajectory shape. Only AnnoMI shows no significant difference in mean distress level.RMS: 10/10, p < 0.001; shape-only RMS: 10/10, p < 0.001; mean level: 9/10, p < 0.001.F.1
Strategy downstream effects (Fig. 3b)Affirmation and Reassurance differs in 9/10 corpora (except PsychEval). Providing Suggestions differs in 6/10. PsyChain differs in Question, Affirmation, Suggestions, and Others.Affirmation: 9/10, p < 0.05; other significant tests vary by strategy (p < 0.05 or p < 0.01). Holm correction across 73 eligible tests.F.2
Strategy–emotion coupling (Fig. 3c)Strategy coupling differs in 8/10 corpora (except CACTUS and PsychEval). Emotion coupling differs in 8/10 (except AnnoMI and PsyDial). All 8 synthetic corpora have weaker emotion coupling, while 6/8 have weaker strategy coupling.Strategy: 8/10, p < 0.05; emotion: 8/10, p < 0.01.F.3
Inter-conversation diversity (Fig. 3d)All 10 corpora have lower mean pairwise trajectory diversity than ESConv. PsyChain has the second-lowest diversity (0.6591 vs. 1.0762 for ESConv), after Baseline DA (0.5831).10/10, p < 0.001.F.4
Patient-level sentiment dynamics (Table 3)All 8 synthetic corpora differ in sentiment change rate (SCR) and positive-ending rate. Across all 10 corpora, SCR differs in 9/10 (except AnnoMI), while positive-ending rate differs in 10/10.Synthetic: 8/8 for both metrics, p < 0.001; all corpora: SCR 9/10, positive-ending rate 10/10, p < 0.001.F.5

Overall, these results demonstrate that our main conclusions are statistically robust across multiple process-level measures and comparison datasets, with significant differences persisting after Holm correction for multiple comparisons.

Reviewer_7uCQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a trajectory-centered framework for evaluating the process fidelity of synthetic counseling dialogues. Their findings suggest that synthetic dialogues can reproduce plausible emotional endpoints and some global recovery patterns, yet often fail to capture the interactions observed in human support conversations. Controlled prompt interventions improve several trajectory-level properties but have limited effects on interactional responsiveness. The paper highlights the need to evaluate synthetic counseling data not only by dialogue quality and final outcomes, but also by the interaction processes through which those outcomes emerge.

Critical strengths
  1. The paper addresses an important and underexplored issue in synthetic counseling dialogue generation: matching the final emotional outcome does not necessarily imply fidelity to the interactional process. Evaluating how emotional states evolve and how counselor behavior relates to subsequent patient states provides a valuable perspective beyond conventional response-quality and endpoint-based evaluations.
  2. The proposed framework examines several complementary dimensions. This multidimensional analysis helps identify different types of process-level discrepancies that may be overlooked by one single metric.
  3. Useful diagnostic interventions. The targeted prompt experiments investigate whether the observed gaps can be reduced through generation instructions. Holding the model and generation architecture fixed provides an informative setting for examining how different trajectory properties respond to prompting.
Critical weaknesses
  1. Emotiion labels lacking validation. The authors replace the emotion labeling LLM from Llama to GPT-5.6-Luna, while lacking the examination of the performance of this LLM judgement. It's not evident that there is a significant improvement of labeling accuracy or human consensus.
  2. This paper use emotion labels and distress scores to characterize the behaviour of the counselor. However, it's not such confident that a synthetic counselor responds appropriately to the specific content of a patient's preceding message. In particular, conditional dependence between counselor behavior and subsequent patient emotion is not equivalent to genuine contextual adaptation or a causal therapeutic effect.
Questions for authors
  1. The authors mentioned to translate Chineses dataset to English in the emotion labeling process, why do you translate the corpus into English rather than translating the taxonomy into Chinese? Since some expressions may vary in these two languages and this translation process may lead to misinterpretation in different languages.
  2. How do the authors validate that the proposed metrics genuinely capture interactional responsiveness rather than merely statistical dependence between counselor behavior and subsequent patient emotion? Have the authors considered expert judgments of whether each counselor response appropriately addresses the patient's preceding message?
  3. The authors are encouraged to provide more details about the performance of GPT-5.6-Luna in emotion labeling process, since emotion labels are the basis of further investigations of this paper.
  4. Given the overlap with EMPATH and RealCBT, could the authors explicitly identify the novel methodological components of this work and quantify the additional insights they provide beyond existing trajectory-based evaluations?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 7uCQ (1/3)

We thank the reviewer for recognizing the importance of investigating interactional process fidelity beyond final emotional outcomes, as well as the value of our multidimensional evaluation framework in identifying complementary process-level discrepancies. We also appreciate the positive assessment of our controlled prompt-tuning experiments as an informative approach to understanding how different aspects of process fidelity respond to generation instructions. We address each concern as below.


Weakness 1 and Question 3: Emotion-label validation and choice of GPT-5.6 Luna

We follow EMPATH’s emotion-labeling methodology. Its emotion inventory was developed through qualitative analysis of authentic CTL conversations with social work researchers, and its labeling pipeline was evaluated against human annotations [1]. EMPATH used a local Llama model because CTL data are subject to a Data Use Agreement and cannot be processed through external APIs. Since our datasets are publicly available, we chose API-based annotation with GPT-5.6 Luna for scalability and efficiency across approximately 600K utterances.

To assess annotation consistency, we compared Luna with Claude Opus 4.8 and Gemini 3.6 Flash on 705 client turns from 50 ESConv conversations. This cross-model comparison covers fine-grained emotion labels, polarity, and derived distress scores. Across the three model pairs, fine-grained emotion agreement is substantial (Cohen’s κ = 0.62–0.68), polarity agreement is stronger (79.0–87.5%; κ = 0.67–0.79), and derived distress scores are highly correlated (r = 0.81–0.85).

Table R1. Cross-model emotion annotation agreement on 705 client turns from 50 ESConv conversations.

Model pairExact (%)κPolarity (%)Polarity κDistress rExact / synonym (%)
Luna–Opus 4.869.90.6887.50.790.8580.9
Luna–Gemini 3.6 Flash67.20.6481.80.720.8475.9
Opus 4.8–Gemini 3.6 Flash65.20.6279.00.670.8174.0

Luna’s agreement with both models is comparable to their agreement with each other, at approximately one-twentieth the inference cost of Opus 4.8 and one-third that of Gemini 3.6 Flash in this comparison. These results have been added to Section 4.1 and Appendix G, highlighted in blue in the rebuttal revision.

In addition, our model-agnostic evaluation framework focuses more on comparison rather than dependent on a particular annotation model. By applying the same emotion-labeling model and scoring procedure consistently across all datasets, we ensure a standardized basis for comparing emotional trajectories and interactional dynamics. Although systematic annotation biases may still affect the estimated differences, the proposed framework is model-agnostic and can be applied with alternative emotion annotation models to further assess robustness.

AuthorsOpenReview ↗

Response to Reviewer 7uCQ (2/3)

Weakness 2 and Question 2: Interpreting coupling and validating contextual responsiveness

We agree that statistical dependence alone does not establish contextual appropriateness or causal therapeutic effects. We address and clarify these related concerns together.

First, our framework is designed to characterize observable interactional dynamics rather than establish causal therapeutic effects. Specifically, our stage-adjusted CMI analysis measures how much information counselor behavior provides about the patient’s next emotional state beyond their current state. The stage-adjusted permutation baseline shuffles counselor labels within groups sharing the same current patient state and conversation stage. Comparing observed coupling against this baseline helps distinguish interactional associations from correlations arising from shared conversation progression.

Second, establishing causal therapeutic effects would require counterfactual evidence that is generally unavailable in existing counseling corpora. This would require comparing patient outcomes under alternative counselor responses while holding relevant patient conditions fixed. Our observational datasets do not support this comparison, but our methods may be applied if such datasets are available. We will clarify this scope and avoid causal interpretations of downstream distress changes.

Third, we agree that further assessment that provides a complementary validation would be helpful, and we plan to conduct a qualitative human evaluation to further assess the interpretation of the patterns identified by our quantitative analysis. We will examine representative counseling trajectories from real and synthetic datasets to provide qualitative analysis that would provide more insights. As the study may take longer than the rebuttal period, we plan to include the analysis and illustrative case studies in the final version once it is complete. As part of this analysis, we plan to assess counselor responses along three dimensions: Context-specific understanding (whether responses recognize the patient’s expressed concerns and circumstances); Strategy appropriateness (whether the supportive approach is suitable given the preceding context); Response specificity (whether responses address patient-specific information rather than rely on generic supportive language).

AuthorsOpenReview ↗

Response to Reviewer 7uCQ (3/3)

Question 1: Why translate Chinese conversations rather than the emotion taxonomy?

We wanted to cover datasets across different languages while keeping the emotion-labeling pipeline consistent. We considered annotating each dataset in its original language or translating the Chinese datasets into English. We chose translation so that all datasets could be evaluated in the same language, using the same emotion taxonomy, annotation model, and scoring procedure. This provides a common basis for comparing emotional trajectories across datasets.

We acknowledge that translation may affect emotional nuance, and systematic human validation would be needed to quantify its effects on emotion labels and derived distress scores. We will make this limitation explicit in the paper and identify a systematic comparison of native-language and translation-based annotation as future work.

Question 4: Contributions beyond EMPATH and RealCBT

We reuse EMPATH’s emotion-labeling foundation and some trajectory representations, and do not claim these components as new. EMPATH primarily studies one authentic crisis-support setting, with a secondary comparison to two synthetic conditions [1]. RealCBT compares authentic CBT with CACTUS using VAD/UED emotional-arc statistics [2].

Our contribution extends this work into a multidimensional, controlled evaluation of synthetic process fidelity across nine datasets in the original study, covering peer support, motivational interviewing, long-term mental-health support, CBT, and multi-session therapy. We add measures of recovery onset, last-mile change, and post-recovery persistence; stage-adjusted counselor–client coupling; cross-corpus trajectory coverage; and controlled generation interventions.

Moreover, compared to [1, 2], our study goes beyond detection and asks what kind of improvements can be made based on the detections of discrepancies. We systematically investigate which dimensions of process fidelity can be improved through generation design and which remain resistant to targeted interventions and observe a consistent asymmetry as reported in Section 5.4. This design allows us to examine which differences vary across datasets and generation methods, and whether targeted generation changes reduce them. This distinction between prompt-steerable trajectory properties and persistent interactional gaps is a central empirical contribution of our work. The findings also motivate using process-level measures to inform trajectory-aware learning objectives, including reinforcement learning rewards.


We thank the reviewer again for the thoughtful comments. The revisions are included in the revised rebuttal draft, highlighted in blue. We hope these responses address your concerns and welcome any further questions.

References

[1] EMPATH: Tracing Multi-Level Emotion Dynamics in Crisis Counseling Dialogues. 2026.

[2] Feel the Difference? A Comparative Analysis of Emotional Arcs in Real and LLM-Generated CBT Sessions. Findings of EMNLP, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback. As the discussion period draws to a close, we would greatly appreciate your feedback on whether our revisions address your concerns. We posted our responses and revised manuscript on November 12, and we would like to briefly highlight the additional experiments addressing your main concerns:

  • Annotation consistency (7uCQ, Dxw5): On 705 client turns from 50 ESConv conversations, GPT-5.6 Luna shows substantial agreement with Claude Opus 4.8 and Gemini 3.6 Flash on emotion labels (κ=0.62\kappa=0.62–0.680.68) and polarity (κ=0.67\kappa=0.67–0.790.79), with highly correlated distress scores (r=0.81r=0.81–0.850.85), at substantially lower inference cost.
  • Additional base models (Me4J): We repeated the prompt-tuning experiments with GPT-6.1 Sol, Claude Sonnet 5.5, and Gemini 3.1 Pro. None achieves a 15% gap reduction on strategy coupling, emotion coupling, or turn-level relief, indicating that the interactional gaps persist beyond GPT-4o under the tested interventions.
  • Multi-agent generation (Me4J): We added PsyChain (1,047 dialogues), which also exhibits weaker coupling and lower trajectory diversity than ESConv, providing additional evidence from collaborative multi-agent generation.
  • Significance testing (Me4J): Holm-corrected tests show that all synthetic corpora differ significantly from ESConv in distress trajectories, emotion-channel coupling, and trajectory diversity. We also clarify which findings are dataset-dependent.
  • Interpolation sensitivity (Me4J): Results remain stable across K=20K=20–5050 (r≥0.966r\geq0.966, τ≥0.96\tau\geq0.96), supporting robustness to trajectory resolution.

We also clarified the non-causal scope of our claims, the definition of recovery, and the limitations concerning translation and human validation (7uCQ, Dxw5).

We hope these additions address the main concerns raised and are happy to asnwer any further questions or suggestions. As the discussion period draws to a close, we would greatly appreciate your feedback.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.