acceptodds
Under review as a conference paper at ICLR 2027

PSYCHO-R1: Advancing Proactive Guidance for Long-Horizon Psychological Counseling

Abstract

LLM-based counselors can generate empathetic and professionally grounded responses, yet they struggle to proactively guide therapeutic progress across multi-turn interactions—a capability essential to effective counseling. This limitation stems in part from insufficient modeling of dynamic decision-making under information asymmetry, which requires state inference of clients to guide proactive inquiry and intervention. It is further compounded by trajectory-level advantages during optimization, which provide coarse supervision and fail to distinguish the contributions of individual counselor turns to subsequent conversational progress. To this end, we introduce PSYCHO-R1, a proactive counselor agent trained via a GRPO variant in an agentic interactive framework with three components: 1) an information-asymmetric multi-agent environment with a client simulator conditioned on a private profile and an evaluator providing turn-level rewards; 2) hierarchical decision-making linking state inference to proactive guidance; and 3) critic-free credit assignment using local lookahead returns to compute turn-level group-relative advantages. Experimental results demonstrate that PSYCHO-R1 achieves the best performance across all 11 dimensions spanning counselor-level, client-level, and therapeutic alliance metrics, outperforming both specialized mental-health models and substantially larger general-purpose LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Counselor Agent, Proactive Counseling, Agentic Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8xsNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PSYCHO-R1, a counseling agent for long-horizon psychological counseling. The main idea is to train the counselor in an interactive environment, where the client has a private psychological profile and provides information during the conversation. Overall, I think the problem is important and the paper is interesting.

Critical strengths

1:The problem is important. Current counseling LLMs usually focus on empathy or response quality, while this paper considers how to actively control the whole counseling process. I think this direction is meaningful.

2: The experimental results are strong. PSYCHO-R1 achieves 4.71 overall score and performs better than both general-purpose models and specialized counseling models.

Critical weaknesses
  1. The evaluation relies too much on LLM simulation and LLM judge. The training interaction is based on an LLM client simulator and an LLM evaluator. During evaluation, the counseling dialogues are also generated in an interactive simulated environment, and GPT-5.5 is used to score 33 evaluation items. However, psychological counseling is a very human-centered task. Good performance with an LLM client does not mean good performance with real users. The paper does not provide evaluation from real counselors, psychologists, or human users. Therefore, it is difficult to know whether the very high scores, such as 4.99 or 4.96 out of 5, really represent better counseling quality.

  2. There may be some evaluation bias because GPT-5.5 is used in both data generation and evaluation. The SFT data is generated by GPT-5.5, while GPT-5.5 is also used as the final evaluator.
    This may introduce some model-specific preference. For example, PSYCHO-R1 is first trained to imitate GPT-5.5-generated counseling dialogues. Then GPT-5.5 is used to judge whether its counseling behavior is good. It is possible that the evaluator prefers some response style similar to its own generated data.

Questions for authors

Please see above.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your thoughtful review. We sincerely appreciate your recognition of the importance and potential impact of our work. We have carefully addressed the comments and incorporated the corresponding revisions into the updated PDF, with key changes highlighted in blue for clarity. Below, we provide point-by-point responses.

  • W1: No evaluation from real counselors, psychologists, or human users.

To address your concern, we conduct complementary human evaluations (Section 3.3, Figure 4, and Appendix F) from two perspectives: 1) User Experience Evaluation, 2) Human Expert Validation.

1) User Experience as Clients. To assess user experience, we recruit three psychology students. Each student role-plays clients based on the same 20 assigned profiles and engages in separate conversations with PSYCHO-R1 and three baseline models. Then they conduct blind pairwise comparisons across four user-centered evaluation dimensions: Acceptance, Insight, Empowerment, and Helpfulness.

The win rates for each model (excluding ties) are shown below:

DimensionPSYCHO-R1Claude-Sonnet-5RLVERPsyDTLLM
Acceptance88.31%81.82%10.00%29.31%
Insight91.77%72.50%18.64%23.73%
Empowerment86.49%78.77%25.00%17.86%
Helpfulness92.62%75.64%17.96%20.93%

Conclusion: PSYCHO-R1 ranked first across all four dimensions. Consistent with the automated evaluation, these results suggest that PSYCHO-R1 provides a better perceived user experience in role-play counseling.

2) Human Expert Validation. We recruit two licensed psychologists to independently evaluates 50 counseling dialogues per model. They assess counseling competence on a five-point Likert scale across four dimensions: Strategicness, Informativeness, Progression, and Professionalism. Results are shown in Table 5.

Table 5: Counseling competence scores assigned by GPT-5.5, Grok-4.6, and two licensed psychologists.

JudgeModelStrategicnessInformativenessProgressionProfessionalism
GPT-5.5Claude-Sonnet-54.343.824.684.08
PsyDTLLM2.902.863.362.90
RLVER3.462.963.743.26
PSYCHO-R14.644.284.784.34
Grok-4.6Claude-Sonnet-54.003.424.223.94
PsyDTLLM3.323.103.463.38
RLVER3.202.943.522.88
PSYCHO-R14.224.084.924.28
Human ExpertsClaude-Sonnet-54.253.934.433.91
PsyDTLLM3.773.854.093.69
RLVER3.773.833.933.71
PSYCHO-R14.794.744.974.56

Conclusion: PSYCHO-R1 ranks first across all four dimensions. Compared with Claude-Sonnet-5, the strongest baseline, it achieves higher scores in Informativeness (4.74 vs. 3.93) and Progression (4.97 vs. 4.43). These results provide expert evidence that PSYCHO-R1 better elicits key information and guides conversations toward new understanding and appropriate next steps.

  • W2: Preference Bias in GPT-5.5 Judgments.

To assess the reliability of GPT-5.5 as an evaluator, we conduct a Cross-Evaluator Validation Study (Appendix F.3), in which both GPT-5.5 and an independent evaluator, Grok-4.6, assess the samples used in the Human Expert Validation along the same evaluation dimensions. Results are shown in Table 5.

We calculate Spearman’s rank correlations across the 16 model–dimension score pairs, with results shown below:

Evaluator ComparisonSpearman’s ρp-value
GPT-5.5 vs. Grok-4.60.89313.2014 × 10⁻⁶
GPT-5.5 vs. Human Experts0.85622.3116 × 10⁻⁵

Conclusion: GPT-5.5’s scores exhibit strong positive correlations with both Grok-4.6’s scores and human expert ratings. And PSYCHO-R1 always ranks first across all four dimensions under all three evaluator groups. These findings indicate that our reported improvements are consistently supported by both human experts and a third-party automated evaluator, supporting the reliability of GPT-5.5 as a judge for comparative counseling assessment.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your constructive feedback. If your schedule permits, we would appreciate hearing your thoughts on our rebuttal before the discussion period closes. We would be happy to clarify any points or address any remaining concerns.

Reviewer_gHT5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

To address the challenge that psychological counseling models struggle to proactively guide therapeutic progress in multi-turn interactions, this paper proposes PSYCHO-R1, a proactive counselor agent trained via an improved GRPO method within an agentic interactive framework. The core contributions include: (1) proposing a fine-grained optimization method that refines credit assignment to each individual counselor turn using local lookahead returns and advantage backfilling; and (2) equipping the counselor model with state-aware decision-making and generation capabilities, which select counseling strategies and generate responses based on inferred client states. Extensive experiments demonstrate that PSYCHO-R1 substantially outperforms both general-purpose LLMs and specialized psychological counseling models across three key dimensions: counselor-level, client-level, and therapeutic alliance. Ablation studies further confirm the effectiveness of the proposed reinforcement learning method.

Critical strengths
  1. The authors design an improved RL strategy based on GRPO tailored to the characteristics of psychological counseling scenarios. Compared with the original GRPO, which provides only trajectory-level holistic feedback, the proposed method refines the optimization signal at the level of each counselor turn through local lookahead rewards, group-wise normalization, and advantage backfilling, thereby achieving turn-level credit assignment. This design is reasonably original and well suited to optimizing the capabilities of multi-turn dialogue models.

  2. On the experimental side, the authors employ multi-dimensional evaluation metrics and extensive experiments to validate the effectiveness of PSYCHO-R1 in interactive counseling (Table 1), as well as its improvement on the psychology QA benchmarks and its preservation of general capabilities (Table 2). The ablation study in Table 4 further confirms that, compared with SFT alone, the proposed RL strategy indeed improves the feedback from the client’s perspective.

  3. The evaluation framework employed by the authors, which is composed of 33 items, 11 dimensions, and 3 aspects, is comprehensive and well-formulated. In particular, dimensions like Strategy & Timing (ST) and Session Coherence & Progression (SCP) correspond precisely to the core capabilities targeted by the proposed method, allowing the improvements brought by the method to be measured directly and accurately.

  4. The paper is well-written, fluent, and easy to follow. The appendix supplements the main text with necessary details, including the evaluation rubrics, prompt templates, and training configurations.

Critical weaknesses
  1. Lack of human evaluation as a reference. The main interactive-counseling results are all obtained through prompt-driven evaluation by GPT-5.5, with no human expert ratings reported, nor any alignment analysis between human judgments and LLM scores. For research on psychological counseling models, the absence of such validation would undermines the credibility of the evaluation results.

  2. The evaluation set consists of only 150 cases seeded from PsyQA, which is limited in both source diversity and scale. It would be preferable to provide results on additional interactive-counseling benchmark to better demonstrate the performance of the proposed method.

Questions for authors
  1. In Section 2.2, the counseling action space is divided into four strategies: Fact Inquiry (FQ), Cognitive Inquiry (CQ), Cognitive Intervention (CI), and Behavior Intervention (BI). Is this division supported by established psychological theories (like CCD) or authoritative references, or is it an empirical design?

  2. In interactive dialogue simulations, the behavioral of the client simulator naturally affect the final evaluation results. Although detailed prompt constraints are applied to its behavior, have the authors considered instantiating the client simulator with different LLMs? Currently, it appears that only GPT-OSS-20B was used. If a different model were used instead, how would the evaluation results in Table 1 be affected?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We greatly appreciate the reviewer for your encouraging remarks and constructive comments. We have carefully addressed the comments and incorporated the corresponding revisions into the updated PDF, with key changes highlighted in blue for clarity. Below, we provide point-by-point responses.

  • W1: Lack of Human Evaluation and Alignment Analysis.

To address this concern, we conduct complementary human evaluations (Section 3.3, Figure 4, and Appendix F) from three perspectives : 1) User Experience Evaluation, 2) Human Expert Validation, 3) Cross-Evaluator Validation Study.

1) User Experience as Clients. To assess user experience, we recruit three psychology students. Each student role-plays clients based on the same 20 assigned profiles and engages in separate conversations with PSYCHO-R1 and three baseline models. Then they conduct blind pairwise comparisons across four user-centered evaluation dimensions: Acceptance, Insight, Empowerment, and Helpfulness.

The win rates for each model (excluding ties) are shown below:

DimensionPSYCHO-R1Claude-Sonnet-5RLVERPsyDTLLM
Acceptance88.31%81.82%10.00%29.31%
Insight91.77%72.50%18.64%23.73%
Empowerment86.49%78.77%25.00%17.86%
Helpfulness92.62%75.64%17.96%20.93%

Conclusion: PSYCHO-R1 ranked first across all four dimensions. Consistent with the automated evaluation, these results suggest that PSYCHO-R1 provides a better perceived user experience in role-play counseling.

2) Human Expert Validation. We recruit two licensed psychologists to independently evaluates 50 counseling dialogues per model. They assess counseling competence on a five-point Likert scale across four dimensions: Strategicness, Informativeness, Progression, and Professionalism. Results are shown in Table 5.

Table 5: Counseling competence scores assigned by GPT-5.5, Grok-4.6, and two licensed psychologists.

JudgeModelStrategicnessInformativenessProgressionProfessionalism
GPT-5.5Claude-Sonnet-54.343.824.684.08
PsyDTLLM2.902.863.362.90
RLVER3.462.963.743.26
PSYCHO-R14.644.284.784.34
Grok-4.6Claude-Sonnet-54.003.424.223.94
PsyDTLLM3.323.103.463.38
RLVER3.202.943.522.88
PSYCHO-R14.224.084.924.28
Human ExpertsClaude-Sonnet-54.253.934.433.91
PsyDTLLM3.773.854.093.69
RLVER3.773.833.933.71
PSYCHO-R14.794.744.974.56

Conclusion: PSYCHO-R1 ranks first across all four dimensions. Compared with Claude-Sonnet-5, the strongest baseline, it achieves higher scores in Informativeness (4.74 vs. 3.93) and Progression (4.97 vs. 4.43). These results provide expert evidence that PSYCHO-R1 better elicits key information and guides conversations toward new understanding and appropriate next steps.

3) Cross-Evaluator Validation Study. To assess the reliability of GPT-5.5 as a judge, we have GPT-5.5 and an independent judge, Grok-4.6, evaluate the samples used in 2) Human Expert Validation along the same evaluation dimensions (Appendix F.3). Results are shown in Table 5.

We calculate Spearman’s rank correlations across the 16 model–dimension score pairs, with results shown below:

Evaluator ComparisonSpearman’s ρp-value
GPT-5.5 vs. Grok-4.60.89313.2014 × 10⁻⁶
GPT-5.5 vs. Human Experts0.85622.3116 × 10⁻⁵

GPT-5.5’s scores exhibit strong positive correlations with both Grok-4.6’s scores and human expert ratings. And PSYCHO-R1 always ranks first across all four dimensions under all three evaluator groups. These findings indicate that our reported improvements are consistently supported by both human experts and a third-party automated evaluator, supporting the reliability of GPT-5.5 as a judge for comparative counseling assessment.

AuthorsOpenReview ↗

Official Comment by Authors

  • W2: Evaluations on Additional Interactive-counseling Benchmark.

To assess generalization of PSYCHO-R1, we conduct supplementary experiments on CARE-Bench[1] (Appendix H). We use Qwen-3.8-Max as the client simulator and GPT-5.6-sol as the evaluator. Results are in shown Table 7.

Table 7: Results on CARE-Bench.

ModelWAI GoalWAI TaskWAI BondWAI Avg.BLRI Cogn.BLRI Affect.BLRI Differ.BLRI Inner.BLRI Avg.CCS-R ProbingCCS-R Environ.CCS-R Reflect.CCS-R ChangeCCS-R Avg.
Claude-Sonnet-52.903.653.943.502.572.682.292.592.594.194.474.143.764.11
Deepseek-v4-Flash3.343.883.913.712.572.642.392.692.604.104.444.114.164.18
PsyDTLLM1.531.952.962.151.941.791.522.121.883.843.662.971.722.91
PsyLLM1.101.241.831.39-1.33-0.91-0.66-1.01-1.081.992.091.631.121.64
Psyche-R11.161.312.001.49-0.53-0.63-0.25-0.35-0.512.332.251.611.191.74
RLVER1.441.712.641.931.261.161.320.931.182.713.012.531.722.43
Psycho-R1 (Ours)3.693.873.943.832.562.732.282.632.604.424.624.374.304.40

Conclusion: PSYCHO-R1 achieves average scores of 3.83, 2.60, and 4.40, ranking first or tied for first on all three scales and outperforming all evaluated open-source counseling models. Compared with PsyDTLLM, the strongest open-source specialized counseling baseline, its average scores are higher by 1.68, 0.72, and 1.49 points, respectively. More analysis in Appendix H.

[1] Bichen Wang, Yixin Sun, Junzhe Wang, Hao Yang, Xing Fu, Yanyan Zhao, Si Wei, Shijin Wang, and Bing Qin. Care-bench: A benchmark of diverse client simulations guided by expert principles for evaluating llms in psychological counseling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 39378–39386, 2026a.

  • Q1: Four-strategy division theory-based or empirically designed?

To our knowledge, no existing literature explicitly classify counselors’ actions into these four strategies. In fact, our strategy design is inspired by the Helping Skills Model[4] and cognitive behavioral therapy (CBT)[5].

Helping Skills Model organizes the counseling process around exploration, insight, and action.

CBT emphasizes individuals’ interpretations of situations and the interactions among cognitions, emotions, and behaviors, while using case conceptualization tools, such as the Cognitive Conceptualization Diagram (CCD), to guide cognitive and behavioral interventions.

Inspired by these theoretical foundations, we first anchor proactive counseling strategies around two complementary functions: inquiry and intervention. We then further differentiate these strategies along more specific dimensions: facts versus cognition for inquiry, and cognition versus behavior for intervention.

[4] Clara E Hill. Helping skills: Facilitating exploration, insight, and action. American Psychological Association, 1999.

[5] Judith S Beck. Cognitive behavior therapy: Basics and beyond. Guilford Publications, 2020.

AuthorsOpenReview ↗

Official Comment by Authors

  • Q2: Different model as Client Simulator.

To examine this sensitivity, we conduct additional evaluations using Gemini-3.6-Flash and Deepseek-v4-Flash as alternative client simulators (Appendix G). The results are shown in Table 6.

Table 6: Results under different client simulators.

ClientCounselorEUAIGESTSCPIQPAvg.GoalTaskBondAvg.SUIEPBWEPBAvg.Overall Avg.
Gemini-3.6-FlashClaude-Sonnet-54.944.864.594.804.484.733.724.054.354.044.693.684.534.304.40
Deepseek-v4-Flash4.994.884.824.844.914.894.434.554.404.464.854.244.744.614.68
PsyDTLLM3.963.662.872.893.393.352.642.813.583.013.952.403.623.323.22
RLVER3.543.452.472.923.173.112.833.153.543.174.242.813.693.583.22
Psycho-R1 (Ours)5.004.954.834.924.914.924.574.484.404.494.964.064.414.484.68
Deepseek-V4-FlashClaude-Sonnet-54.704.294.044.284.084.283.003.233.833.353.642.923.803.453.79
Gemini-3.5-Flash4.994.784.644.764.644.764.074.224.454.244.683.724.334.244.48
PsyDTLLM4.103.963.293.143.643.632.743.073.603.134.062.433.473.323.39
RLVER3.503.342.502.853.233.082.732.793.563.033.832.333.393.183.08
Psycho-R1 (Ours)4.994.974.874.914.944.944.394.324.364.354.933.904.144.334.61

Conclusion: PSYCHO-R1 achieves best or joint-best overall scores with Gemini-3.6-Flash (4.68) and Deepseek-v4-Flash (4.61). It also ranks first or tied for first on all five counselor-level dimensions under both simulators. These results support its robustness across client simulators and suggest that our framework narrows the gap between compact open-source and powerful API models in interactive counseling.

The additioal experiments address my concern. I will remain my rating.

Reviewer_N3uUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper looks at LLM-based psychological counseling and asks how to train a counselor that guides a multi-turn session, deciding when to ask questions and when to intervene, rather than just giving empathetic replies. The authors fine-tune Qwen3-8B on GPT-generated dialogues and then train it with RL against an LLM-simulated client whose profile is hidden from the counselor. At each turn, the model writes a CBT-style summary of the client’s state, picks one of four strategies (two types of inquiry, two types of intervention), and then responds. An LLM evaluator scores every turn, and each turn’s reward is the sum of the scores over the next three turns (if I got it correctly), instead of one reward for the whole dialogue. On 150 simulated clients, scored by GPT on several dimensions, the model beats specialized counseling models and larger general LLMs such as Sonnet 5.

Critical strengths
  • The paper frames counseling as a sequence of decisions under missing information, which is closer to how counseling actually works. The client’s profile is hidden, so the counselor has to keep an explicit picture of the client’s state (the CCD) and decide at each turn whether to ask more or to intervene. Most earlier work instead trains models to imitate single responses given a fixed dialogue history. The learned behavior also makes sense, the model starts with fact-gathering questions, moves to questions about thoughts and beliefs, and ends with cognitive and behavioral interventions. I like this framing.

  • The reward per turn is simple and could be reused elsewhere. Each turn is rewarded with the sum of rewards over the next few turns, rather than sharing a single reward for the whole dialogue. I think that nothing about this is specific to counseling, so it could be reused in other multi-turn RL setups.

Critical weaknesses
  • The evaluation is done entirely by LLMs and is very close to what the model was trained on. The clients are simulated, every score comes from GPT 5.5, and no humans or clinicians were involved. The client-level and alliance scores are GPT 5.5’s guesses from the transcript, not reports from clients. Several evaluation dimensions closely match the training reward (strategy appropriateness, information gain), and both are built around the same four-strategy scheme the model is trained to follow. GPT 5.5 also generated the SFT data, so the judge may favor the model’s style. The paper also doesn’t say which model plays the client at test time, or maybe I missed it. No confidence intervals or seeds are reported.

  • The credit-assignment experiments don’t isolate what they claim to show. The paper doesn’t say what reward the vanilla GRPO baseline uses. If it isn’t the same per-turn rewards summed over the dialogue, then the comparison mixes up reward design with credit assignment.

  • There is no comparison with existing turn-level methods the paper cites (MICA, Turn-PPO, GiGPO).

Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the valuable feedback and positive assessment of our framework design. We have carefully addressed the comments and incorporated the corresponding revisions into the updated PDF, with key changes highlighted in blue for clarity. Below, we provide point-by-point responses.

  • W1-1: Evaluation lacks participation from real users.

To assess user experience, we recruit three psychology students. Each student role-plays clients based on the same 20 assigned profiles and engages in separate conversations with PSYCHO-R1 and three baseline models. Then they conduct blind pairwise comparisons across four user-centered evaluation dimensions: Acceptance, Insight, Empowerment, and Helpfulness. More details of human evaluation are provided in Section 3.3, Figure 4, and Appendix F.

The win rates for each model (excluding ties) are shown below:

DimensionPSYCHO-R1Claude-Sonnet-5RLVERPsyDTLLM
Acceptance88.31%81.82%10.00%29.31%
Insight91.77%72.50%18.64%23.73%
Empowerment86.49%78.77%25.00%17.86%
Helpfulness92.62%75.64%17.96%20.93%

Conclusion: PSYCHO-R1 ranked first across all four dimensions. Consistent with the automated evaluation, these results suggest that PSYCHO-R1 provides a better perceived user experience in role-play counseling.

  • W1-2: Evaluation does not involve clinical psychologists.

We recruit two licensed psychologists to independently evaluates 50 counseling dialogues per model. They assess counseling competence on a five-point Likert scale across four dimensions: Strategicness, Informativeness, Progression, and Professionalism. Results are shown in Table 5. More details of human evaluation are provided in Section 3.3, Figure 4, and Appendix F.

Table 5: Counseling competence scores assigned by GPT-5.5, Grok-4.6, and two licensed psychologists.

JudgeModelStrategicnessInformativenessProgressionProfessionalism
GPT-5.5Claude-Sonnet-54.343.824.684.08
PsyDTLLM2.902.863.362.90
RLVER3.462.963.743.26
PSYCHO-R14.644.284.784.34
Grok-4.6Claude-Sonnet-54.003.424.223.94
PsyDTLLM3.323.103.463.38
RLVER3.202.943.522.88
PSYCHO-R14.224.084.924.28
Human ExpertsClaude-Sonnet-54.253.934.433.91
PsyDTLLM3.773.854.093.69
RLVER3.773.833.933.71
PSYCHO-R14.794.744.974.56

Conclusion: PSYCHO-R1 ranks first across all four dimensions. Compared with Claude-Sonnet-5, the strongest baseline, it achieves higher scores in Informativeness (4.74 vs. 3.93) and Progression (4.97 vs. 4.43). These results provide expert evidence that PSYCHO-R1 better elicits key information and guides conversations toward new understanding and appropriate next steps.

  • W1-3: Preference Bias in GPT-5.5 Judgments.

To assess the reliability of GPT-5.5 as a judge, we conduct a Cross-Evaluator Validation Study (Appendix F.3), in which both GPT-5.5 and an independent evaluator, Grok-4.6, assess the samples used in W1-2 along the same evaluation dimensions. Results are shown in Table 5.

We calculate Spearman’s rank correlations across the 16 model–dimension score pairs, with results shown below:

Evaluator ComparisonSpearman’s ρp-value
GPT-5.5 vs. Grok-4.60.89313.2014 × 10⁻⁶
GPT-5.5 vs. Human Experts0.85622.3116 × 10⁻⁵

Crucially, GPT-5.5’s scores exhibit strong positive correlations with both Grok-4.6’s scores and human expert ratings. And PSYCHO-R1 always ranks first across all four dimensions under all three evaluator groups. These findings indicate that our reported improvements are consistently supported by both human experts and a third-party automated evaluator, supporting the reliability of GPT-5.5 as a judge for comparative counseling assessment.

AuthorsOpenReview ↗

Official Comment by Authors

  • W1-4: Evaluation is very close to what the model was trained on.

To assess model generalization, we conduct supplementary experiments on CARE-Bench[1] (Appendix H). We use Qwen-3.8-Max as the client simulator and GPT-5.6-sol as the evaluator. Results are in shown Table 7.

Table 7: Results on CARE-Bench.

ModelWAI GoalWAI TaskWAI BondWAI Avg.BLRI Cogn.BLRI Affect.BLRI Differ.BLRI Inner.BLRI Avg.CCS-R ProbingCCS-R Environ.CCS-R Reflect.CCS-R ChangeCCS-R Avg.
Claude-Sonnet-52.903.653.943.502.572.682.292.592.594.194.474.143.764.11
Deepseek-v4-Flash3.343.883.913.712.572.642.392.692.604.104.444.114.164.18
PsyDTLLM1.531.952.962.151.941.791.522.121.883.843.662.971.722.91
PsyLLM1.101.241.831.39-1.33-0.91-0.66-1.01-1.081.992.091.631.121.64
Psyche-R11.161.312.001.49-0.53-0.63-0.25-0.35-0.512.332.251.611.191.74
RLVER1.441.712.641.931.261.161.320.931.182.713.012.531.722.43
Psycho-R1 (Ours)3.693.873.943.832.562.732.282.632.604.424.624.374.304.40

Conclusion: PSYCHO-R1 achieves average scores of 3.83, 2.60, and 4.40, ranking first or tied for first on all three scales and outperforming all evaluated open-source counseling models. Compared with PsyDTLLM, the strongest open-source specialized counseling baseline, its average scores are higher by 1.68, 0.72, and 1.49 points, respectively. More analysis in Appendix H.

[1] Bichen Wang, Yixin Sun, Junzhe Wang, Hao Yang, Xing Fu, Yanyan Zhao, Si Wei, Shijin Wang, and Bing Qin. Care-bench: A benchmark of diverse client simulations guided by expert principles for evaluating llms in psychological counseling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 39378–39386, 2026a.

  • W1-5: Unspecified Client Model at Test Time.

At test time, client simulator is powered by GPT-OSS-20B. We have clarified this to Section 3.1. Thank you for pointing this out.

AuthorsOpenReview ↗

Official Comment by Authors

  • W1-6: No Confidence Intervals or Seeds Are Reported.

For confidence intervals, to quantify uncertainty in our main evaluation results, we compute 95% confidence intervals using a case-level bootstrap (Appendix I). Results are shown in Table 8.

For seeds, we apologize that we did not explicitly set random seeds during evaluation.

Table 8: 95% Case-Level Bootstrap Confidence Intervals for the Main Results

ModelEUAIGESTSCPIQPGoalTaskBondSUIEPBWEPBOverall Avg.
Claude-Sonnet-5[4.87, 4.93][4.73, 4.82][4.39, 4.50][4.58, 4.68][4.61, 4.71][3.80, 3.94][4.29, 4.41][4.33, 4.43][4.71, 4.86][3.85, 3.97][4.39, 4.55][4.41, 4.49]
Gemini-3.5-Flash[4.56, 4.68][4.29, 4.43][4.05, 4.20][4.07, 4.22][4.57, 4.69][3.71, 3.85][4.14, 4.27][4.07, 4.18][4.37, 4.54][3.73, 3.85][4.15, 4.28][4.16, 4.26]
Qwen-3.7-Plus[4.72, 4.81][4.42, 4.52][4.21, 4.35][4.45, 4.55][4.59, 4.69][3.88, 4.02][4.36, 4.47][4.36, 4.43][4.64, 4.79][3.95, 4.04][4.35, 4.48][4.36, 4.44]
Deepseek-v4-Flash[4.65, 4.73][4.46, 4.57][4.18, 4.31][4.39, 4.51][4.66, 4.76][3.83, 3.97][4.36, 4.47][4.16, 4.28][4.64, 4.78][3.92, 4.04][4.36, 4.50][4.33, 4.41]
MeChat[2.08, 2.22][1.88, 2.08][1.44, 1.56][1.38, 1.51][1.84, 1.99][1.92, 2.14][2.68, 2.95][2.74, 2.94][3.60, 3.79][2.43, 2.67][3.59, 3.79][2.28, 2.42]
CPsyCounX[2.52, 2.70][2.01, 2.25][1.50, 1.66][1.70, 1.87][2.43, 2.66][1.71, 1.97][2.06, 2.35][2.55, 2.74][2.76, 3.10][1.91, 2.19][2.64, 2.97][2.16, 2.35]
PsyDTLLM[3.71, 3.85][3.58, 3.76][3.13, 3.33][3.46, 3.65][3.60, 3.77][3.42, 3.62][3.73, 3.92][3.62, 3.73][4.03, 4.19][3.42, 3.62][4.05, 4.21][3.61, 3.75]
PsyLLM[3.60, 3.74][3.16, 3.32][2.45, 2.64][2.67, 2.88][3.39, 3.57][2.94, 3.16][3.39, 3.61][3.35, 3.50][3.74, 3.91][3.08, 3.33][3.68, 3.88][3.21, 3.37]
Psyche-R1[2.53, 2.69][2.03, 2.24][1.53, 1.71][1.74, 1.93][2.56, 2.76][2.00, 2.30][2.35, 2.67][2.63, 2.85][3.20, 3.46][2.19, 2.51][2.89, 3.18][2.31, 2.52]
RLVER[3.74, 3.84][3.53, 3.70][2.98, 3.17][3.11, 3.30][3.73, 3.90][3.56, 3.80][3.96, 4.19][3.74, 3.86][4.18, 4.35][3.56, 3.80][4.18, 4.38][3.66, 3.81]
Psycho-R1 (Ours)[4.99, 5.00][4.94, 4.98][4.65, 4.75][4.72, 4.79][4.93, 4.98][4.58, 4.68][4.63, 4.72][4.48, 4.55][4.96, 5.00][3.99, 4.04][4.49, 4.62][4.69, 4.73]
  • W2: Unclear Reward Definition for the Vanilla GRPO Baseline.

We clarify that PSYCHO-R1 and the outcome-supervised vanilla GRPO baseline use the same turn-level reward function defined in Section 2.3. The turn-level reward is the weighted sum of the four dimension scores.

Then the turn-level rewards rtr_t are summed over the entire trajectory τ\tau to obtain a single outcome reward:

Rτ=∑t=1Trt,R_{\tau} = \sum_{t=1}^{T} r_t,

where rtr_t denotes the reward at turn tt and TT is the number of turns.

Vanilla GRPO uses this trajectory-level outcome reward for supervision.

So the comparison is fair, we have clarified this in Section 3.5. Thank you for pointing this out.

  • W3: Lack of Comparison with Existing Turn-Level Methods (MICA, Turn-PPO, and GiGPO).

We thank the reviewer for raising this point. Below, we explain the challenges in directly applying or reproducing these methods in our scenario.

GiGPO[2] is primarily evaluated on agent tasks with clearly defined goals, including embodied interaction, online shopping, and retrieval-augmented question answering. Its step-level credit assignment relies on within-group comparisons of actions taken at shared or recurring states across sampled trajectories. However, psychological counseling involves continuous interaction between clients and counselors. Dialogue trajectories therefore diverge after the first utterance, making state matching difficult.

Turn-PPO[3] does not open-source.

MICA does not release the trained model. Moreover, we found that the authors withdrew the arXiv preprint on October 21, 2026, during the ICLR review period. We will therefore consider removing this reference in a subsequent revision.

[2] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. In Advances in Neural Information Processing Systems, volume 38, pp. 46375–46408, 2025.

[3] Junbo Li, Peng Zhou, Rui Meng, Meet P Vadera, Lihong Li, and Yang Li. Turn-ppo: Turn-level advantage estimation with ppo for improved multi-turn rl in agentic llms. In Findings of the Association for Computational Linguistics: EACL 2026, pp. 6227–6243, 2026a.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your constructive feedback. If your schedule permits, we would appreciate hearing your thoughts on our rebuttal before the discussion period closes. We would be happy to clarify any points or address any remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their thorough review and constructive feedback. This summary explains how we addressed each of the major concerns raised during review. We have incorporated all revisions into the updated manuscript and highlighted the key changes in blue for ease of reference.

Strengths Highlighted by the Reviewers

  • Reviewer 8xsN found our paper interesting and considered the problem we address important. The reviewer also described our focus on proactively guiding the entire counseling process as a meaningful research direction.

  • Reviewer N3uU appreciated our overall framework, in which the model learns when and how to use four counseling strategies in an interactive environment where the client’s profile is hidden from the counselor. The reviewer noted that this framing more closely reflects how counseling works in practice and found the learned progression from fact and cognitive inquiry to cognitive and behavioral interventions sensible.

  • Reviewer gHT5 described our turn-level credit assignment method as reasonably original and well suited to improving multi-turn dialogue models. The reviewer also found our automated evaluation framework comprehensive and well-formulated.

Reviewer Concerns and Our Responses

  1. Lack of human evaluation (Section 3.3, Figure 4, and Appendix F). We add two types of human evaluation, one using blind pairwise comparisons of client role-play conversations and the other using independent ratings by two licensed psychologists. PSYCHO-R1 ranks first on all four dimensions in each evaluation.

  2. Reliability of GPT-5.5 as an evaluator (Appendix F.3). Across 16 model–dimension pairs, GPT-5.5’s ratings show strong positive rank correlations with those of Grok-4.6 (ρ = 0.8931, p = 3.2014 × 10⁻⁶) and the licensed psychologists (ρ = 0.8562, p = 2.3116 × 10⁻⁵).

  3. External validation (Appendices G and H). PSYCHO-R1 achieves the highest mean score on each of the three CARE-Bench scales. In the cross-simulator evaluation, it scores 4.68 (joint first) and 4.61 (first) overall with Gemini-3.6-Flash and DeepSeek-V4-Flash as client simulators, respectively.

  4. Vanilla GRPO reward and baseline comparisons (Section 3.5 and rebuttal discussion). We clarify that PSYCHO-R1 and vanilla GRPO use the same turn-level reward function and differ in how they assign credit. In our response to Reviewer N3uU (W3), we also explain why direct comparisons with MICA, Turn-PPO, and GiGPO are impractical in our setting.

  5. Confidence intervals and evaluation setup (Section 3.1 and Appendix I). We identify GPT-OSS-20B as the client simulator used in the main evaluation and report 95% case-level bootstrap confidence intervals in Table 8.

Overall, we believe that PSYCHO-R1, a proactive counseling agent optimized through turn-level credit assignment, represents a meaningful contribution to research on interactive psychological counseling. Our empirical results provide strong support for this conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.