- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work tackles multi-session counseling by building both a simulated client and a counselor. The client updates its state after every session using the session dialogue, its state before that session, and its persona. The counselor updates its state using the session dialogue and its state before the session; it doesn't get access to the client's states or persona. They use this setup, along with 150 client personas, to create a dialogue dataset consisting of 4 sessions per client. They experiment with finetuning different models on this created dataset and compare their performance with the same models finetuned on other datasets. The authors also introduce a metric to measure cross-session coherence.
Critical strengths
-
The approach looks interesting and simulates what would happen in real multi-session encounters. The client's state is affected by the conversation and their character, while the counselor can only learn through the conversation and can only infer the client's evolving state from it.
-
The proposed MCAF metric is an important addition as a measure of cross-session coherence. To my knowledge, no other metric exists for this purpose.
-
The released multi-session datasets might be used to build better synthetic clients/counselors.
Critical weaknesses
-
The MCAF metric, which is an important contribution of this work, is not human-validated. I did notice that this is acknowledged in the limitations section, but the authors human-validated the within-session metrics, which is great, so it raises the question of why they skipped validating the reliability of their proposed cross-session metric.
-
The ablation suggests that other approaches for state tracking perform similarly to the more complex pipeline proposed by the authors, which means that the added complexity might be unnecessary and using summaries of previous sessions might suffice.
Questions for authors
for the ablation study, did the authors compute statistical significance? If so, are these differences (for the memory-enabled configurations) statistically significant?
3 comments
I hope our clarification was helpful. If anything remains unclear, we would be glad to provide a brief additional clarification before the discussion period closes.
I thank the authors for their response.
W1: Thanks for the clarification.
W2: The response here reinforces my point: there really isn't a noticeable difference between the proposed approach and simpler approaches; the only significant differences are in words/utterance and number of turns. One cannot argue that it's better (or worse) to have a bigger value for either of these two metrics; what matters is the effectiveness of the session, which is measured by the other metrics.
On statistical significance. I disagree: If I understand correctly from this work, there are 150 personas; one can still measure statistical significance with that number.
Overall, I appreciate the work and maintain my score.
Response to Reviewer KNe8
We thank the reviewer for the thoughtful comments.
W1 — Human validation of MCAF. We supplemented a human evaluation of the cross-session metric. Eleven evaluators performed a double-blind evaluation on 30 randomly sampled groups: 10 PsyMem, 10 native human PsychEval [1], and 10 negative examples. Negatives were built by concatenating Session 1 and Session 4 of the same case, or sessions from unrelated cases; their surface topics may appear coherent, but they contain identity and memory inconsistencies.
We used boundary anchoring: PsychEval as the high-quality reference, concatenated negatives as the low-quality reference, with PsyMem between them for calibration. Agreement between human ratings and MCAF at both anchors supports MCAF's validity. Evaluators used a 1–5 scale (1 = lacking coherence, 5 = coherent) across four MCAF-aligned dimensions: memory consistency, state evolution, alliance continuity, and overall coherence.
Mean over 10 dialogues/condition; Q1–Q3 are the core dimensions underlying MCAF.
Results: PsyMem 4.5/5, PsychEval 5/5, negatives 1/5; MCAF also assigns lower scores to negatives. Human and automatic MCAF ratings show consistent trends at both anchors, supporting MCAF's reliability. Representative excerpts (PsyMem / PsychEval / negative) will be included in the revision.
W2 — Is the added complexity unnecessary? MCAF is slightly higher for PsyMem (4.96 vs. 4.95), and scores are close across configurations; MCAF primarily measures memory coherence and may not fully capture differences in generation behavior. We therefore also analyze interaction length and task alignment.
Full in absolute values; parentheses = difference vs. Full; ±0 = no change.
The difference in interaction length is clearer: PsyMem averages 61 turns vs. 33 (Summary) and 34 (Shared Memory), with 105/87/85 words per utterance. On WAIOS Task, PsyMem matches Summary (5.92 vs. 5.92) and is 0.17 above Shared Memory (5.75). This is consistent with the role-asymmetric design: client-private state and counselor dialogue-derived memory are updated separately, so the two roles need to keep exchanging new information in later sessions. When summaries or memory are shared, interaction length drops substantially (28 and 27 fewer turns), suggesting that information asymmetry is one factor sustaining long-horizon interaction, rather than redundant complexity.
Q1 — Statistical significance in the ablation. Given the number of variants and limited per-variant sample size, we primarily report effect sizes rather than making per-variant significance claims. In the larger-scale downstream fine-tuning experiments we did conduct a systematic statistical analysis, including multi-judge ratings and correlation analysis (e.g., GPT-4o vs. DeepSeek-V4-Flash/Qwen3.6), to assess stability. From the table above, the role-asymmetric memory module maintains strong coherence while mainly affecting interaction behavior and task alignment: the clearest effect is interaction length (61 vs. 33/34 turns, i.e., 28/27 fewer; 105 vs. 87/85 words per utterance), with WAIOS Task matched to Summary (5.92) and +0.17 over Shared Memory (5.75).
References:
[1] Pan et al. (2026). PsychEval. Findings of ACL 2026, pp. 22192–22212.