- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a trajectory-centered framework for evaluating process fidelity in counseling dialogues. The framework assesses turn-level emotional and distress dynamics, counselor strategies and their associations with subsequent patient states, recovery timing and structure, and trajectory diversity. Applying this framework to nine human, reconstructed, and synthetic counseling datasets, the authors find that synthetic dialogues often reach plausible emotional endpoints while following recovery trajectories that differ from those in human support conversations. They also identify persistent gaps in contingent counselor–patient responsiveness across the models and generation paradigms examined.
Critical strengths
The paper addresses an important and worthwhile research question, and its focus on emotional trajectories in evaluating synthetic counseling dialogues is well motivated. The difference in trajectory-level control and turn-by-turn responsiveness between synthetic and real data can inform the design of data synthesis methods.
The four evaluation dimensions provide complementary perspectives, and the experiments include meaningful intermediate controls, where the evaluations are conducted on multi-type datasets, including real, semi-real, and fully synthetic interactions.
Critical weaknesses
The main findings rely on automated emotion labels and a predefined mapping from emotion categories to distress scores, while the Chinese datasets additionally undergo translation into English. However, the dataset-specific agreement with human annotations, labeling accuracy, or replication using alternative annotation models are not reported. Consequently, the observed differences may reflect linguistic style, translation effects, or annotation bias, whose contributions remain unclear.
Questions for authors
-
How does the framework distinguish reduced distress from changes in emotional expression, such as gratitude or increased willingness to disclose? What annotation evidence supports interpreting the selected low-distress states as recovery across the counseling settings studied?
-
Is process fidelity intended to be defined relative to peer support or to the goals and modality of each counseling setting? What will clinically appropriate trajectories in motivational interviewing or multi-session psychotherapy be classified when they differ from ESConv?
5 comments
Response to Reviewer Dxw5 (2/2)
Question 1: Distinguishing distress reduction from changes in emotional expression
We use distress reduction as an indicator of emotional improvement. The original ESConv study similarly uses the decrease in self-reported emotional intensity to measure improvement [5]. This provides a precedent for distress-focused evaluation, although our label-derived distress scores are a proxy for expressed emotion.
In our analysis, “recovery” refers to reduced expressed distress within a conversation. We agree that emotion labels alone cannot fully distinguish different reasons that led to shifts toward positive emotion, and will add this to our limitation.
To further assess the interpretation of the patterns identified by our quantitative analysis, we will conduct a qualitative human evaluation of representative counseling trajectories, examining whether observed changes reflect emotional relief or alternative processes, such as gratitude and increased disclosure. Although it possibly will take longer than the rebuttal period, we will also incorporate illustrative case studies in the final version to contextualize our finding once the qualitative study is complete.
Question 2: Defining process fidelity across counseling settings
We’s like to clarify that process fidelity should be interpreted relative to the goals and modality of each counseling setting. Our goal is to provide a transferable method for comparing trajectories, rather than establish a single ground truth for all settings.
For example, when evaluating a synthetic motivational interviewing dataset, the framework can compare the generated conversations with authentic motivational interviewing data, including private reference data where available. The same approach can be adapted to multi-session psychotherapy, using reference data and interpretations appropriate to that setting.
The framework also supports relative comparisons between datasets or groups of datasets, as demonstrated in our paper. These comparisons characterize their trajectory distributions and interactional patterns; comparisons with authentic data from the same setting provide a stronger basis for assessing synthetic process fidelity. We chose ESConv as a common reference because it is one of the few publicly available authentic human-support datasets with complete conversations and annotations.
We will clarify this scope in the paper. The contribution we hope to share with the community is a transferable evaluation method, together with insights into the process-level distributions and characteristics of current datasets when grouped as synthetic, semi-synthetic, and authentic.
We thank the reviewer again for recognizing our methods and findings, and for the insightful questions and concerns raised. The additional experiments results and clarifications are included in the uploaded revised draft, highlighted in blue, and we will incorporate the qualitative study described above. We hope our revisions address your main concerns, and welcome any remaining questions or concerns.
References
[1] Niu et al. Rethinking Emotion Annotations in the Era of Large Language Models. IEEE Transactions on Affective Computing, 2025.
[2] Feng et al. Affect Recognition in Conversations Using Large Language Models. SIGDIAL, 2024.
[3] Safdarian et al. UT-NLP at SemEval-2025 Task 11: Evaluating Zero-Shot Capability of GPT-4o mini on Emotion Recognition via Role-Play and Contrastive Judging. SemEval, 2025.
[4] Sosea and Caragea. Hard Emotion Test Evaluation Sets for Language Models. Findings of NAACL, 2025.
[5] Liu et al. Towards Emotional Support Dialog Systems. ACL-IJCNLP, 2021.
Follow up on Rebuttal Discussion
Dear Reviewer,
Thank you again for your detailed review. As the discussion period is coming to an end, we wanted to briefly follow up to see whether our rebuttal and additional experiments have adequately addressed your concerns. We would be happy to provide any further clarification or answer any remaining questions.
Thank you for your time and consideration!
Response to Reviewer Dxw5 (1/2)
We thank the reviewer for the thoughtful comments and their recognition of the importance of our research question, the complementary perspectives provided by our four evaluation dimensions, and the meaningful controls across real, semi-real, and fully synthetic interactions. We address each concern below.
Weakness 1: Emotion annotation reliability and translation effects
(1) Translation of Chinese datasets into English.
We wanted to cover datasets across different languages while keeping the emotion-labeling pipeline consistent. We considered annotating each dataset in its original language or translating the Chinese datasets into English. We chose translation so that all datasets could be evaluated in the same language, using the same emotion taxonomy, annotation model, and scoring procedure. This provides a common basis for comparing emotional trajectories across datasets.
We acknowledge that translation may affect emotional nuance, and systematic human validation would be needed to quantify its effects on emotion labels and derived distress scores. We will make this limitation explicit in the paper and identify a systematic comparison of native-language and translation-based annotation as future work.
(2) Emotion-labeling performance across datasets.
Emotion recognition is a well-studied area in NLP. Prior work reports promising zero-shot performance of LLMs across emotion-labeling tasks, including conversational and multilingual settings [1–3]. These studies support the use of LLMs for emotion annotation, while also showing that performance varies with the dataset and linguistic context, particularly when emotional expressions are implicit [4].
To provide evidence specific to our annotation pipeline, we conducted the additional cross-model comparison below.
(3) Additional experiments using alternative annotation models.
We compared GPT-5.6 Luna’s predictions with those of Claude Opus 4.8 and Gemini 3.6 Flash on 705 client turns from 50 ESConv conversations. Across the three model pairs, we observed substantial agreement on fine-grained emotion labels (Cohen’s κ = 0.62–0.68), strong agreement on emotion polarity (79.0–87.5%; κ = 0.67–0.79), and high correlations in derived distress scores (r = 0.81–0.85). These results have been added to Section 4.1 and Appendix G, highlighted in blue in the revised draft.
Table R1. Cross-model emotion annotation agreement on 705 client turns from 50 ESConv conversations.
Luna’s agreement with both models is comparable to their agreement with each other. The agreement extends beyond exact emotion labels to polarity and derived distress scores, which are directly relevant to our trajectory analysis. In this comparison, Luna achieves this consistency at approximately one-twentieth the inference cost of Opus 4.8 and one-third that of Gemini 3.6 Flash, making a uniform annotation pipeline cost-efficient at the scale of our study. Although dataset-specific human validation is still a limitation of our study which we will clarify in the limitation section, these results above support cross-model annotation consistency.
Our evaluation focuses on comparisons under a consistent annotation procedure. Applying the same emotion-labeling model and scoring procedure across datasets provides a standardized basis for comparing emotional trajectories and interactional dynamics. Systematic annotation biases may still affect the estimated differences, but the framework is model-agnostic and can be applied with alternative annotation models to further assess robustness.