- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PSYCHO-R1, a counseling agent for long-horizon psychological counseling. The main idea is to train the counselor in an interactive environment, where the client has a private psychological profile and provides information during the conversation. Overall, I think the problem is important and the paper is interesting.
Critical strengths
1:The problem is important. Current counseling LLMs usually focus on empathy or response quality, while this paper considers how to actively control the whole counseling process. I think this direction is meaningful.
2: The experimental results are strong. PSYCHO-R1 achieves 4.71 overall score and performs better than both general-purpose models and specialized counseling models.
Critical weaknesses
-
The evaluation relies too much on LLM simulation and LLM judge. The training interaction is based on an LLM client simulator and an LLM evaluator. During evaluation, the counseling dialogues are also generated in an interactive simulated environment, and GPT-5.5 is used to score 33 evaluation items. However, psychological counseling is a very human-centered task. Good performance with an LLM client does not mean good performance with real users. The paper does not provide evaluation from real counselors, psychologists, or human users. Therefore, it is difficult to know whether the very high scores, such as 4.99 or 4.96 out of 5, really represent better counseling quality.
-
There may be some evaluation bias because GPT-5.5 is used in both data generation and evaluation. The SFT data is generated by GPT-5.5, while GPT-5.5 is also used as the final evaluator.
This may introduce some model-specific preference. For example, PSYCHO-R1 is first trained to imitate GPT-5.5-generated counseling dialogues. Then GPT-5.5 is used to judge whether its counseling behavior is good. It is possible that the evaluator prefers some response style similar to its own generated data.
Questions for authors
Please see above.
2 comments
Follow-up on Our Rebuttal
Thank you again for your constructive feedback. If your schedule permits, we would appreciate hearing your thoughts on our rebuttal before the discussion period closes. We would be happy to clarify any points or address any remaining concerns.
Official Comment by Authors
Thank you for your thoughtful review. We sincerely appreciate your recognition of the importance and potential impact of our work. We have carefully addressed the comments and incorporated the corresponding revisions into the updated PDF, with key changes highlighted in blue for clarity. Below, we provide point-by-point responses.
W1: No evaluation from real counselors, psychologists, or human users.
To address your concern, we conduct complementary human evaluations (Section 3.3, Figure 4, and Appendix F) from two perspectives: 1) User Experience Evaluation, 2) Human Expert Validation.
1) User Experience as Clients. To assess user experience, we recruit three psychology students. Each student role-plays clients based on the same 20 assigned profiles and engages in separate conversations with PSYCHO-R1 and three baseline models. Then they conduct blind pairwise comparisons across four user-centered evaluation dimensions: Acceptance, Insight, Empowerment, and Helpfulness.
The win rates for each model (excluding ties) are shown below:
Conclusion: PSYCHO-R1 ranked first across all four dimensions. Consistent with the automated evaluation, these results suggest that PSYCHO-R1 provides a better perceived user experience in role-play counseling.
2) Human Expert Validation. We recruit two licensed psychologists to independently evaluates 50 counseling dialogues per model. They assess counseling competence on a five-point Likert scale across four dimensions: Strategicness, Informativeness, Progression, and Professionalism. Results are shown in Table 5.
Table 5: Counseling competence scores assigned by GPT-5.5, Grok-4.6, and two licensed psychologists.
Conclusion: PSYCHO-R1 ranks first across all four dimensions. Compared with Claude-Sonnet-5, the strongest baseline, it achieves higher scores in Informativeness (4.74 vs. 3.93) and Progression (4.97 vs. 4.43). These results provide expert evidence that PSYCHO-R1 better elicits key information and guides conversations toward new understanding and appropriate next steps.
W2: Preference Bias in GPT-5.5 Judgments.
To assess the reliability of GPT-5.5 as an evaluator, we conduct a Cross-Evaluator Validation Study (Appendix F.3), in which both GPT-5.5 and an independent evaluator, Grok-4.6, assess the samples used in the Human Expert Validation along the same evaluation dimensions. Results are shown in Table 5.
We calculate Spearman’s rank correlations across the 16 model–dimension score pairs, with results shown below:
Conclusion: GPT-5.5’s scores exhibit strong positive correlations with both Grok-4.6’s scores and human expert ratings. And PSYCHO-R1 always ranks first across all four dimensions under all three evaluator groups. These findings indicate that our reported improvements are consistently supported by both human experts and a third-party automated evaluator, supporting the reliability of GPT-5.5 as a judge for comparative counseling assessment.